Toward Scalable Co-located Practical Learning: Assisting with Computer Vision and Multimodal Analytics
Authors: Xinyu Li, Linxuan Zhao, Yueqiao Jin, Yuchen Liu, Jin Zhou, Roberto Martinez-Maldonado, Dragan Gasevic, Lixiang Yan
Organizations: Centre for Learning Analytics at Monash, Monash University, Melbourne, Australia · Department of Civil and Environmental Engineering, Monash University, Melbourne, Australia · Faculty of Education and School of Computing & Data Science, The University of Hong Kong, Hong Kong · School of Education, Tsinghua University, Beijing, China
Co-located practical learning leaves evidence in visible actions around patients, task resources and room zones, but these traces are often recovered through live observation or retrospective video review. Fixed wide-angle video could reduce sensing burden, yet a debriefing pipeline must do more than detect behaviours: it must maintain detection after small camera-position shifts, relate the detector-derived behaviour trace to instructor-labelled outcomes and preserve room-zone context. This study evaluates a fixed-camera pipeline in repeated nursing simulation. Using a harmonised six-code taxonomy, we tested YOLO26 target-only training and two-stage source-to-target adaptation across two same-room side-view data sources. We then converted detections from 51 instructor-labelled sessions into one-second behaviour and behaviour-zone traces for rate, ordered-network, transition-network and sequence analyses. Two-stage adaptation improved mean mAP50 from 0.815 to 0.848 for the 2021 target view and from 0.690 to 0.855 for the smaller 2022 target view; with a balanced target quota of N=22, the 2022 model reached 0.850 mAP50. In the detector-derived behaviour trace analyses, higher phone use characterised low task-performance sessions. Zone labels changed the interpretation of patient interaction: primary patient-care-zone interaction was stronger in higher-performance sessions, while secondary-zone interaction was stronger in lower-performance sessions. Ordered and transition network models showed that ordered room-zone relations contributed beyond behaviour frequency, with the strongest task-performance classifier using zoned and co-presence features. The resulting trace is most appropriate for searchable simulation debriefing, where instructors inspect detected moments rather than receive automated assessment scores.
Figures & tables
Figure 1: Masked wide-angle frame from the nursing simulation room. The overlay shows how detected learner actions were linked to patient-care and secondary room zones for debriefing-oriented trace construction.
Item
Assessment statement
T1
The students demonstrated a structured approach to patient assessment and management.
T2
The students recognised and responded to early signs of patient deterioration.
T3
The students appropriately activated medical emergency team calls.
T4
The students contributed to effective teamwork.
T5
The students communicated information effectively to the health care team.
T6
The students demonstrated an understanding of the roles of the multidisciplinary team.
Table 1: Instructor assessment items for task and collaboration performance.
Code
Behaviour
Interpretation
0
Information Handling
Learner interaction with notes, documentation sheets, laptops, or workstation materials
1
Medication/Equipment Interaction
Learner handling medication trays, trolleys, monitors, pumps, or other clinical resources
2
Other
Learner movement between zones, brief conversation, repositioning, or coordination not assigned to a focal task code
3
Patient Interaction
Learner interaction with the patient, patient bed, bedside monitor, or patient-facing care space
4
Sitting
Teacher-acted family-member distractor; treated as a detection control class and excluded from the learner trace
5
Using Phone
Learner phone use during the scenario
Table 2: Harmonised six-code behaviour taxonomy.
Figure 2: Fixed-camera computer-vision and MMLA pipeline. The pipeline converts repeated simulation video into searchable behaviour-zone traces for instructor-led debriefing.
Code
Behaviour
Instances
0
Information Handling
92
1
Medication/Equipment Interaction
51
2
Other
143
3
Patient Interaction
114
4
Sitting
47
5
Using Phone
16
Table 3: Corrected six-code holdout distribution.
Target view
Setting
Overall
Information
Medication/Equipment
Other
Patient
Sitting
Phone
2021 data
Target-only Full
0.815
0.784
0.692
0.726
0.781
0.927
0.980
2021 data
Two-stage N = 83
0.815
0.712
0.707
0.732
0.782
0.990
0.965
2021 data
Two-stage Full
0.848
0.793
0.778
0.777
0.822
0.993
0.927
2022 data
Target-only Full
0.690
0.609
0.518
0.671
0.634
0.915
0.790
2022 data
Two-stage N = 22
0.850
0.833
0.743
0.747
0.816
0.976
0.987
2022 data
Two-stage Full
0.855
0.847
0.755
0.795
0.784
0.967
0.980
Table 4: YOLO26 comparison on the corrected six-code holdout, averaged over seeds 1000 and 4000.
Outcome group
Sessions
Information
Medication/Equipment
Other
Patient
Phone
Task high
40
0.554
0.315
1.363
0.690
0.102
Task low
11
0.558
0.335
1.384
0.671
0.151
Collaboration high
30
0.549
0.332
1.389
0.710
0.099
Collaboration low
21
0.564
0.301
1.336
0.652
0.132
Table 5: Mean session rates by instructor-labelled group, using the basic learner trace.
Outcome group
Sessions
Primary patient
Secondary patient
Primary phone
Primary co-presence
Primary individual
Task high
40
0.558
0.132
0.099
0.483
0.312
Task low
11
0.435
0.236
0.151
0.398
0.362
Collaboration high
30
0.582
0.128
0.098
0.497
0.303
Collaboration low
21
0.459
0.193
0.128
0.419
0.351
Table 6: Behaviour-zone and co-presence rates by instructor-labelled group.
Outcome
Method
Feature set
AUC [95% CI]
Balanced accuracy
Strongest effect
q / FDR effects
Task
ENA
Basic learner
0.673 [0.448, 0.870]
0.635
phone--other , d = -1.16
.043 / 1
Task
ENA
Zoned learner
0.639 [0.425, 0.831]
0.552
P:Phone--S:Info , d = -1.24
.073 / 0
Task
ONA
Basic learner
0.700 [0.487, 0.881]
0.615
phone -> patient , d = -1.58
.007 / 1
Task
ONA
Zoned learner
0.775 [0.574, 0.948]
0.731
P:Phone -> P:Patient , d = -1.47
.039 / 1
Task
ONA
Zoned + co-presence
0.820 [0.624, 0.974]
0.801
P:Patient -> primary_ind , d = 1.49
.027 / 3
Task
TNA
Zoned + co-presence
0.709 [0.532, 0.863]
0.640
P:Info -> P:Phone , d = -1.77
.098 / 0
Table 7: Main classification rows for instructor-labelled outcomes. AUC CIs are session-bootstrap 95% intervals. The q column reports BH-corrected edge or transition p values within each method × feature-set × outcome family.
Figure 3: ONA difference plot for task performance using zoned learner features. Blue edges are stronger in the high group, orange edges are stronger in the low group, and chevrons indicate ordered direction from thin to thick.
Figure 4: ONA difference plot for collaboration performance using zoned learner features. Blue edges are stronger in the high group, orange edges are stronger in the low group, and chevrons indicate ordered direction from thin to thick.
Figure 5: TNA visualisation for collaboration performance using zoned learner features. The upper panel shows the global transition network after removing transitions below .10. The lower panel shows the high-low permutation contrast, with green transitions stronger in the high group, red transitions stronger in the low group and numeric labels reporting high-minus-low transition differences; this panel is descriptive because the zoned TNA transition effect did not survive FDR correction.
Assessing learner competency in clinical simulation requires expert observation that is time-intensive, difficult to scale, and subject to inter-rater variability. Vision-language models have emerged as a promising tool for understanding complex visual behavior. In this work, we investigate whether visual observations can provide educationally meaningful signals for competency assessment through a three-stage framework that (1) extracts action timelines from egocentric nursing simulation video using frozen visual encoders and few-shot learning, (2) derives sequence-level features and per-session recognition metrics, and (3) relates these to instructor-rated competency. Across 22 densely annotated sessions (3.8 hours, 493 actions), a frozen DINOv2 backbone with HMM Viterbi decoding achieves 57.4% MOF in leave-one-out 1-shot recognition. Surprisingly, we observe a negative trend between recognition accuracy and competency (rho = -0.524, p = 0.012 for mIoU), robust to six confound controls: more competent students produce diverse, harder-to-classify workflows, while simple sequence features show no such relationship. Per-item analysis identifies patient safety protocols and team communication as the expected behaviors most reflected in this pattern, and process model comparisons reveal that higher-competency students exhibit more protocol-consistent action transitions. These findings suggest that recognition accuracy may complement predicted action timelines as a pedagogically informative signal in automated competency assessment.
This paper studies how to adapt a computer vision object detector to an unknown environment under both a robot navigation time and annotation budget constraint. Our approach selects informative robot trajectories and image samples to retrain the detector, explicitly targeting its failure cases. Formally, the approach is an embodied variant of batch active learning, where at each round an agent has a limited navigation budget to collect candidate samples and a limited annotation budget for the most relevant images. We leverage spatial consistency to identify images with inconsistent labels, which are likely to provide the greatest improvement to the vision model. We evaluate the approach using different active learning objectives on large scenes from the AI2-THOR simulator and on a real-world setup using a Boston Dynamics Spot robot with the real-time object detector YOLOv5. Through comparison against several baselines, our experimental results show that spatial inconsistency helps guide the agent and select relevant images without external supervision, achieving the highest detection accuracy at the end of the adaptation process under the same budget. The open-source project can be found at https://mkabouri.github.io/embodied-active-learning-od
Inria Scool, the Universit´e de Lille and also with the Centre de Recherche en Informatique, Signal et Automatique de Lille (CRIStAL) · LAAS-CNRS, Universite de Toulouse, CNRS, Toulouse, France · Department of Electrical Engineering and Automation at Aalto University
On-screen learning behavior provides valuable insights into how students seek, use, and create information during learning. Analyzing on-screen behavioral engagement is essential for capturing students' cognitive and collaborative processes. The recent development of Vision Language Models (VLMs) offers new opportunities to automate the labor-intensive manual coding often required for multimodal video data analysis. In this study, we compared the performance of both leading closed-source VLMs (Claude-3.7-Sonnet, GPT-4.1) and open-source VLM (Qwen2.5-VL-72B) in single- and multi-agent settings for automated coding of screen recordings in collaborative learning contexts based on the ICAP framework. In particular, we proposed and compared two multi-agent frameworks: 1) a three-agent workflow multi-agent system (MAS) that segments screen videos by scene and detects on-screen behaviors using cursor-informed VLM prompting with evidence-based verification; 2) an autonomous-decision MAS inspired by ReAct that iteratively interleaves reasoning, tool-like operations (segmentation/ classification/ validation), and observation-driven self-correction to produce interpretable on-screen behavior labels. Experimental results demonstrated that the two proposed MAS frameworks achieved viable performance, outperforming the single VLMs in scene and action detection tasks. It is worth noting that the workflow-based agent achieved best on scene detection, and the autonomous-decision MAS achieved best on action detection. This study demonstrates the effectiveness of VLM-based Multi-agent System for video analysis and contributes a scalable framework for multimodal data analytics.
Likai Peng, Shihui Feng
Faculty of Education, The University of Hong Kong, Pokfulam, Hong Kong · Institute of Data Science, The University of Hong Kong, Pokfulam, Hong Kong