S1330
Interdisciplinary - Education in radiation oncology
ESTRO 206
Proffered Paper 1841 Human-in-the-loop GPT-4o exam items for radiation oncology education: blinded psychometric equivalence to expert multiple choice questions Philipp Linde 1 , Florian Fichter 2 , Markus Dietlein 3 , Ferdinand Sudbrock 3 , Kambiz Afshar 4 , Hendrik Dapper 1 , Emmanouil Fokas 1 , Anna-Lena Hillebrecht 5 , Tobias Raupach 6 , Matthias Carl Laupichler 6 1 Department of Radiation Oncology, Cyberknife and Radiation Therapy, Faculty of Medicine and University Hospital of Cologne, University of Cologne, Cologne, Germany. 2 Institute for Diagnostic and Interventional Radiology, Faculty of Medicine and University Hospital of Cologne, University of Cologne, Cologne, Germany. 3 Department of Nuclear Medicine, Faculty of Medicine and University Hospital of Cologne, University of Cologne, Cologne, Germany. 4 Institute for General Practice and Palliative Care, Hannover Medical School, Hannover, Germany. 5 Department of Prosthetic Dentistry, Centre for Dental Medicine, Medical Centre, University of Freiburg, Freiburg, Germany. 6 Institute of Medical Education, University Hospital Bonn, Bonn, Germany Purpose/Objective: Large language models (LLMs) could accelerate assessment in imaging-centric training, but psychometric equivalence (PE) and acceptability remain uncertain. To determine whether human-in- the-loop GPT-4o–generated multiple-choice questions (MCQs) achieve PE to expert-written items for radiation oncology education in medical students and physicians, and whether examinees can detect item origin. Material/Methods: In a preregistered, single-center, blinded, within- subject study, medical students (n=82) and physicians (n=46) completed a 48-item mock exam comprising 24 human-authored and 24 topic-paired GPT-4o MCQs (origin masked), covering radiation oncology, radiology, and nuclear medicine themes. Primary endpoints were item difficulty and discrimination. Secondary endpoints were per-item source identification (human vs LLM) and blinded expert ratings (appropriateness, didactic quality) with inter- rater reliability. LLM items were produced with a version-pinned model and expert review within a human-in-the-loop workflow. Results: Across 128 participants, item difficulty was comparable (human 0.65±0.22 vs LLM 0.67±0.20; t(45.52)= − 0.446, p=.658). In students, medians were 0.58 vs 0.59 (z= − 0.413, p=.680); in physicians, 0.76±0.17 vs 0.78±0.18 (t(45.94)= − 0.399, p=.692). Discrimination was similar overall (0.27±0.12 vs
explored participants’ perceptions and experiences of the VR education. Results: Participants (median age 65 years; range 31–87) included 23 (56%) Stage 0–I, 14 (34%) Stage II, and 4 (10%) Stage III.Mean STAI-6 scores decreased significantly from baseline (41.2 ± 15.6) to end-of- treatment (25.4 ± 8.9; p < 0.001). Wilcoxon and Mann- Whitney U tests confirmed significant reductions across all timepoints, with the steepest decline between T1 and T2, indicating early reassurance and improved emotional readiness. Variability in scores narrowed over time, suggesting greater emotional stability. Changes in self-reported understanding, confidence, and preparedness were not statistically significant, though qualitative data indicated strong perceived benefit.Five qualitative themes emerged:Relief through understanding - “It showed the exact process I’d be going through, removing fear of the unknown.”Seeing is believing - “As a visual learner, it was much easier to understand what I’m expected to do.”Empowerment and calm - “Prepared me for the real experience, I know what to expect.”Clarity and reassurance - “Explained in layman’s terms, gentle and reassuring.”Minor technical issues - Headset comfort and visual focus.Patients consistently described the VR as “just like being there,” reporting reduced fear and increased emotional readiness. Conclusion: Among the 41 participants, immersive VR education significantly reduced anxiety and was perceived as clear, reassuring, and empowering. Although confidence and preparedness improvements were not statistically significant, qualitative findings indicate meaningful perceived value, warranting further evaluation of VR interventions. VR education transforms how radiation therapy is explained replacing fear with familiarity and empowering patients through immersive, patient-centred communication. References: Mayr NA, Borm KJ, Kalet AM, et al. Reducing Cardiac Radiation Dose From Breast Cancer Radiation Therapy With Breath Hold Training and Cognitive Behavioral Therapy. Top Magn Reson Imaging. 2020;29(3):135- 148. doi:10.1097/RMR.0000000000000241Dower K, Halkett GKB, Dhillon H, Naehrig D, O'Connor M. Eliciting the views of left breast cancer patients' receiving deep inspiration breath hold radiation therapy to inform the design of multimedia education and improve patient-centred care for prospective patients. J Med Radiat Sci. 2024;71(3):384-395. doi:10.1002/jmrs.790 Keywords: virtual reality, deep inspiration breath hold
Made with FlippingBook - Share PDF online