Quality Action Assurance: Multimodal Verification of Examiner Claims in VR OSCEs
Authors: Harry Rogers, Sally Shiels, Ashley Tomlinson, James Thomas, James Aylward, Nathan Gauge, Helen Higham, Alison Noble
Organizations: Department of Engineering Science, University of Oxford, Oxford, UK · Nuffield Department of Clinical Neurosciences, University of Oxford, Oxford, UK · Department for Continuing Education, University of Oxford, Oxford, UK
Objective Structured Clinical Examinations (OSCEs) are the gold standard for assessing clinical competence, yet scoring remains vulnerable to examiner subjectivity, fatigue, and cognitive bias. Standard examiner validation via inter-rater statistics lacks explanatory power regarding the source of errors, as it neither analyzes examiner reasoning nor verifies examiner claims against actual events. Thus, we introduce Quality Action Assurance (QAA), a multimodal framework that verifies examiner claims in Virtual Reality (VR) pediatric OSCEs by comparing actions claimed by examiners against a reference record of events constructed from video, VR logs, and actor annotations. QAA combines a constrained temporal action alignment model, which performs action localization and actor source attribution, with a large language model that extracts examiner claims and checks them against the record. Across a 5-fold cross-validation, QAA achieves 99.2% ± 0.7% Actor F1 and 93.4% ± 1.9% W@16 for temporal alignment. Overall, QAA detects examiner errors with 69.9% precision and 76.7% recall; in retrospective evaluation, correcting the detected errors raises the share of factually correct transcripts from 39.2% to 79.2%, supporting fairer OSCE quality assessment.