cs.CVNov 19, 2026

Online Mistake Detection in Instruction-Conditioned Procedural Video

Authors: Amit Feldman, Ilan Shimshoni, Rotem Dror

Organizations: Faculty of computer and Information Science, University of Haifa

Abstract

An assistant monitoring a person performing a procedure is most useful when it can detect mistakes while there is still time to recover. This requires judging actions against task instructions as the video unfolds, identifying the attempted step, locating it in time, and assessing its execution. Existing online detectors do not use the instruction text, while instruction-conditioned methods operate offline. In this paper, we study online, instruction-conditioned mistake detection in egocentric videos. Using only observed video, task instructions, and prior predictions, our model identifies and localizes completed action attempts and classifies them as correct, modified, slip, corrections, or additions; omitted steps are determined after the video ends. We compare a modular pipeline, with separate step identification, temporal localization, and execution verification, to a joint model that predicts all outputs in a single step. To support supervised learning from mistakes, we repartition EgoPER to include erroneous videos in all splits. The approaches achieve similar step-segmentation [email protected] (93.6% and 92.1%), while the joint model achieves higher binary mistake-detection [email protected] (84.1% vs. 80.7%), with its largest gain on slips (56.4% vs. 37.5%). Component-level analysis suggests that error propagation between stages contributes to the modular approach's lower detection score. On test videos containing mistakes, both designs outperform released AEM and ESTANet checkpoints under our evaluation protocol.

Figures & tables

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Post-Training VLMs for Video Mistake Detection

    Aug 28, 2026Federico Spurio, Olga Zatsarynna, Lars Doorenbos +3VLM Post-TrainingVision-Language Models

  2. MistExit: Learning to Exit for Early Mistake Detection in Procedural Videos

    Mar 15, 2026Sagnik Majumder, Anish Nethi, Ziad Al-Halah +1Temporal Video Understanding

  3. PROVIA: Procedure State Tracking for Online Mistake Detection in Egocentric Videos

    Sep 17, 2026Di Wen, Kailun Yang, Jimmy Weissert +7Egocentric Video UnderstandingState Tracking