Online Mistake Detection in Instruction-Conditioned Procedural Video
Organizations: Faculty of computer and Information Science, University of Haifa
Abstract
An assistant monitoring a person performing a procedure is most useful when it can detect mistakes while there is still time to recover. This requires judging actions against task instructions as the video unfolds, identifying the attempted step, locating it in time, and assessing its execution. Existing online detectors do not use the instruction text, while instruction-conditioned methods operate offline. In this paper, we study online, instruction-conditioned mistake detection in egocentric videos. Using only observed video, task instructions, and prior predictions, our model identifies and localizes completed action attempts and classifies them as correct, modified, slip, corrections, or additions; omitted steps are determined after the video ends. We compare a modular pipeline, with separate step identification, temporal localization, and execution verification, to a joint model that predicts all outputs in a single step. To support supervised learning from mistakes, we repartition EgoPER to include erroneous videos in all splits. The approaches achieve similar step-segmentation [email protected] (93.6% and 92.1%), while the joint model achieves higher binary mistake-detection [email protected] (84.1% vs. 80.7%), with its largest gain on slips (56.4% vs. 37.5%). Component-level analysis suggests that error propagation between stages contributes to the modular approach's lower detection score. On test videos containing mistakes, both designs outperform released AEM and ESTANet checkpoints under our evaluation protocol.
Figures & tables
| Partition | Subset | Normal | Erroneous | Total |
| Official | Train + validation | 201 | 0 | 201 |
| Test | 15 | 173 | 188 | |
| Ours | Train + validation | 183 | 148 | 331 |
| Test | 33 | 25 | 58 |
| Method | EDA | [email protected] | O-Acc | O-IoU |
| Full test set (58 videos) | ||||
| Zero-shot | 88.7 | 8.9 | 100.0 | 8.32 |
| Modular (ours) | 97.7 | 80.7 | 99.6 | 79.1 |
| Joint (ours) | 97.8 | 84.1 | 97.8 | 95.4 |
| Erroneous subset (25 videos) | ||||
| Zero-shot | 74.8 | 9.1 | 100.0 | 16.31 |
| Modular | Joint | |||||||
| Test set | Mistake type | P | R | F1 | P | R | F1 | |
| All 58 | Correction | 5 | 40.0 | 80.0 | 53.3 | 30.0 | 60.0 | 40.0 |
| Modification | 33 | 92.9 | 78.8 | 85.2 | 87.5 | 84.8 | 86.2 | |
| Slip | 19 | 46.2 | 31.6 | 37.5 | 55.0 | 57.9 | 56.4 | |
| Addition | 16 | 62.5 | 93.8 | 75.0 | 65.0 | 81.3 | 72.2 | |
| Macro | – | 60.4 | 71.0 | 62.8 | 59.4 | 71.0 | 63.7 | |
| Full test set | Erroneous (25 videos) | |||
| Method | [email protected] | Pred. | [email protected] | Pred. |
| AEM [ 3 ] | – | – | 61.1 | 368 |
| ESTANet [ 9 ] | – | – | 46.2 | 387 |
| Modular (ours) | 92.1 | 696 | 91.5 | 283 |
| Joint (ours) | 93.6 | 724 | 94.6 | 284 |
| Test (58) | Erroneous (25) | ||||
| Component | Metric | All | Cond. | All | Cond. |
| Step Identifier | 2,679 | – | 965 | – | |
| Step macro F1 | 80.3 | – | 83.8 | – | |
| Accuracy | 88.9 | – | 90.1 | – | |
| Temporal Localizer | 871 | 656 | 333 | 266 | |
| Temporal [email protected] | 85.0 | 96.9 | 86.3 | 96.8 | |
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Block | Layers | LoRA targets |
| Linear attention | 48 | in_proj_qkv , in_proj_z , in_proj_a , in_proj_b , out_proj |
| Full attention | 16 | q_proj , k_proj , v_proj , o_proj |
| MLP | 64 | gate_proj , up_proj , down_proj |
| Model | Vision | Frames | Seq. | Gen. | Epoch |
| Step Identifier | adapted | 32 | 12,288 | 1 letter | 6 |
| Temporal Localizer | frozen | 54 | 20,480 | 64 | 4 |
| Execution Verifier | frozen | 40 | 20,480 | 128 | 4 |
| Joint | adapted | 54 | 20,480 | 192 | 6 |
| BERTScore F1 against | |||||
| Method | N | EM (%) | Own reference | Instruc- tion | Wrong reference |
| Modular (ours) | 41 | 75.6 | 0.899 | 0.730 | 0.631 |
| Joint (ours) | 59 | 86.4 | 0.944 | 0.685 | 0.643 |
| Method | Measurement | Mean | Median | P95 |
| Modular | Module A, one call | 1.75 | 1.49 | 3.06 |
| Module B, one call | 3.09 | 2.78 | 5.17 | |
| Module C, one call | 3.10 | 2.95 | 4.53 | |
| One accepted action | 7.63 | 7.18 | 11.81 | |
| Joint | One call (all) | 3.18 | 2.22 | 6.42 |
| Window with no action | 2.29 | 1.72 | 4.78 |
| Method | [email protected] | [email protected] | [email protected] | Frame acc. |
| AEM [ 3 ] | 53.9 | 50.1 | 34.0 | 74.0 |
| ESTANet [ 9 ] | 46.5 | 35.5 | 18.8 | 72.7 |
| Modular (ours) | 79.2 | 78.8 | 72.5 | 92.8 |
| Joint (ours) | 82.8 | 81.7 | 79.2 | 93.9 |
Egocentric Video Understanding
Egocentric video understanding studies how AI interprets first-person video to recognize actions, track interactions, reason about space and time, and assist users.
See all 224 papers