Toward Scalable Co-located Practical Learning: Assisting with Computer Vision and Multimodal Analytics
Authors: Xinyu Li, Linxuan Zhao, Yueqiao Jin, Yuchen Liu, Jin Zhou, Roberto Martinez-Maldonado, Dragan Gasevic, Lixiang Yan
Organizations: Centre for Learning Analytics at Monash, Monash University, Melbourne, Australia · Department of Civil and Environmental Engineering, Monash University, Melbourne, Australia · Faculty of Education and School of Computing & Data Science, The University of Hong Kong, Hong Kong · School of Education, Tsinghua University, Beijing, China
Co-located practical learning leaves evidence in visible actions around patients, task resources and room zones, but these traces are often recovered through live observation or retrospective video review. Fixed wide-angle video could reduce sensing burden, yet a debriefing pipeline must do more than detect behaviours: it must maintain detection after small camera-position shifts, relate the detector-derived behaviour trace to instructor-labelled outcomes and preserve room-zone context. This study evaluates a fixed-camera pipeline in repeated nursing simulation. Using a harmonised six-code taxonomy, we tested YOLO26 target-only training and two-stage source-to-target adaptation across two same-room side-view data sources. We then converted detections from 51 instructor-labelled sessions into one-second behaviour and behaviour-zone traces for rate, ordered-network, transition-network and sequence analyses. Two-stage adaptation improved mean mAP50 from 0.815 to 0.848 for the 2021 target view and from 0.690 to 0.855 for the smaller 2022 target view; with a balanced target quota of N=22, the 2022 model reached 0.850 mAP50. In the detector-derived behaviour trace analyses, higher phone use characterised low task-performance sessions. Zone labels changed the interpretation of patient interaction: primary patient-care-zone interaction was stronger in higher-performance sessions, while secondary-zone interaction was stronger in lower-performance sessions. Ordered and transition network models showed that ordered room-zone relations contributed beyond behaviour frequency, with the strongest task-performance classifier using zoned and co-presence features. The resulting trace is most appropriate for searchable simulation debriefing, where instructors inspect detected moments rather than receive automated assessment scores.
Figures & tables
Figure 1: Masked wide-angle frame from the nursing simulation room. The overlay shows how detected learner actions were linked to patient-care and secondary room zones for debriefing-oriented trace construction.
Item
Assessment statement
T1
The students demonstrated a structured approach to patient assessment and management.
T2
The students recognised and responded to early signs of patient deterioration.
T3
The students appropriately activated medical emergency team calls.
T4
The students contributed to effective teamwork.
T5
The students communicated information effectively to the health care team.
T6
The students demonstrated an understanding of the roles of the multidisciplinary team.
Table 1: Instructor assessment items for task and collaboration performance.
Code
Behaviour
Interpretation
0
Information Handling
Learner interaction with notes, documentation sheets, laptops, or workstation materials
1
Medication/Equipment Interaction
Learner handling medication trays, trolleys, monitors, pumps, or other clinical resources
2
Other
Learner movement between zones, brief conversation, repositioning, or coordination not assigned to a focal task code
3
Patient Interaction
Learner interaction with the patient, patient bed, bedside monitor, or patient-facing care space
4
Sitting
Teacher-acted family-member distractor; treated as a detection control class and excluded from the learner trace
5
Using Phone
Learner phone use during the scenario
Table 2: Harmonised six-code behaviour taxonomy.
Figure 2: Fixed-camera computer-vision and MMLA pipeline. The pipeline converts repeated simulation video into searchable behaviour-zone traces for instructor-led debriefing.
Code
Behaviour
Instances
0
Information Handling
92
1
Medication/Equipment Interaction
51
2
Other
143
3
Patient Interaction
114
4
Sitting
47
5
Using Phone
16
Table 3: Corrected six-code holdout distribution.
Target view
Setting
Overall
Information
Medication/Equipment
Other
Patient
Sitting
Phone
2021 data
Target-only Full
0.815
0.784
0.692
0.726
0.781
0.927
0.980
2021 data
Two-stage N = 83
0.815
0.712
0.707
0.732
0.782
0.990
0.965
2021 data
Two-stage Full
0.848
0.793
0.778
0.777
0.822
0.993
0.927
2022 data
Target-only Full
0.690
0.609
0.518
0.671
0.634
0.915
0.790
2022 data
Two-stage N = 22
0.850
0.833
0.743
0.747
0.816
0.976
0.987
2022 data
Two-stage Full
0.855
0.847
0.755
0.795
0.784
0.967
0.980
Table 4: YOLO26 comparison on the corrected six-code holdout, averaged over seeds 1000 and 4000.
Outcome group
Sessions
Information
Medication/Equipment
Other
Patient
Phone
Task high
40
0.554
0.315
1.363
0.690
0.102
Task low
11
0.558
0.335
1.384
0.671
0.151
Collaboration high
30
0.549
0.332
1.389
0.710
0.099
Collaboration low
21
0.564
0.301
1.336
0.652
0.132
Table 5: Mean session rates by instructor-labelled group, using the basic learner trace.
Outcome group
Sessions
Primary patient
Secondary patient
Primary phone
Primary co-presence
Primary individual
Task high
40
0.558
0.132
0.099
0.483
0.312
Task low
11
0.435
0.236
0.151
0.398
0.362
Collaboration high
30
0.582
0.128
0.098
0.497
0.303
Collaboration low
21
0.459
0.193
0.128
0.419
0.351
Table 6: Behaviour-zone and co-presence rates by instructor-labelled group.
Outcome
Method
Feature set
AUC [95% CI]
Balanced accuracy
Strongest effect
q / FDR effects
Task
ENA
Basic learner
0.673 [0.448, 0.870]
0.635
phone--other , d = -1.16
.043 / 1
Task
ENA
Zoned learner
0.639 [0.425, 0.831]
0.552
P:Phone--S:Info , d = -1.24
.073 / 0
Task
ONA
Basic learner
0.700 [0.487, 0.881]
0.615
phone -> patient , d = -1.58
.007 / 1
Task
ONA
Zoned learner
0.775 [0.574, 0.948]
0.731
P:Phone -> P:Patient , d = -1.47
.039 / 1
Task
ONA
Zoned + co-presence
0.820 [0.624, 0.974]
0.801
P:Patient -> primary_ind , d = 1.49
.027 / 3
Task
TNA
Zoned + co-presence
0.709 [0.532, 0.863]
0.640
P:Info -> P:Phone , d = -1.77
.098 / 0
Table 7: Main classification rows for instructor-labelled outcomes. AUC CIs are session-bootstrap 95% intervals. The q column reports BH-corrected edge or transition p values within each method × feature-set × outcome family.
Figure 3: ONA difference plot for task performance using zoned learner features. Blue edges are stronger in the high group, orange edges are stronger in the low group, and chevrons indicate ordered direction from thin to thick.
Figure 4: ONA difference plot for collaboration performance using zoned learner features. Blue edges are stronger in the high group, orange edges are stronger in the low group, and chevrons indicate ordered direction from thin to thick.
Figure 5: TNA visualisation for collaboration performance using zoned learner features. The upper panel shows the global transition network after removing transitions below .10. The lower panel shows the high-low permutation contrast, with green transitions stronger in the high group, red transitions stronger in the low group and numeric labels reporting high-minus-low transition differences; this panel is descriptive because the zoned TNA transition effect did not survive FDR correction.
Inria Scool, the Universit´e de Lille and also with the Centre de Recherche en Informatique, Signal et Automatique de Lille (CRIStAL) · LAAS-CNRS, Universite de Toulouse, CNRS, Toulouse, France · Department of Electrical Engineering and Automation at Aalto University