MistExit: Learning to Exit for Early Mistake Detection in Procedural Videos

Sagnik Majumder; Anish Nethi; Ziad Al-Halah; Kristen Grauman

cs.CVMar 15, 2026

MistExit: Learning to Exit for Early Mistake Detection in Procedural Videos

Authors: Sagnik Majumder, Anish Nethi, Ziad Al-Halah, Kristen Grauman

Organizations: UT Austin · University of Utah

Abstract

We introduce the task of early mistake detection in video, where the goal is to determine whether a keystep in a procedural activity is performed correctly while observing as little of the streaming video as possible. To tackle this problem, we propose a method comprising a mistake detector and a reinforcement learning policy. At each timestep, the detector processes recently observed frames to estimate the keystep's correctness while anticipating future visual features, enabling reliable early mistake estimates. Meanwhile, the policy aggregates the detector outputs and visual observations over time and adaptively decides when to exit (i.e., stop processing incoming frames) while producing the final prediction. Using diverse real-world procedural video datasets, we demonstrate that our MistExit model achieves superior mistake detection accuracy while reducing the fraction of video observed compared to state-of-the-art models. Project: https://vision.cs.utexas.edu/projects/mist_exit.

Figures & tables

Figure 1: Our goal is to learn a policy that, given a streaming video of a keystep in a procedural activity, decides how much of the video to observe before exiting ( i.e . , stopping inference), such that a mistake detector conditioned on the recently observed frames can accurately determine whether the keystep is a mistake or not while minimizing the fraction of the video observed. E.g . , in a video like the one shown in row 2, a well-trained model may observe a drill bit (frame 3) and infer that the step will eventually result in a mistake—the glass breakage in later frames confirms this—and exit promptly, thereby using only about 12% of the video.

Figure 2: Our MistExit model for early mistake detection has two components: 1) a mistake detector $\mathcal{D}$ (top), and 2) an exit policy $\pi$ (bottom). At each timestep in a streaming keystep clip, $\mathcal{D}$ processes recent frames to predict the keystep’s mistake label and anticipate future features, improving the mistake detection quality. The policy $\pi$ takes the detector’s estimate and the latest frame, aggregates them over time, and decides when to exit. We train $\pi$ with a novel reward that encourages improving detection quality over time while promoting early and accurate exits.

Panel of Figure 3 — Figure 3: Early mistake detection results. Higher AP, lower OR is better.

Panel of Figure 4 — Figure 4: Left: Ablation on larger-scale CaptainCook4D (CC4D) [ 54 ] . Right: Early mistake detection results on larger-scale CC4D for different lengths ( $L$ ) of the anticipated feature sequence in our mistake detector (Sec. 4.1 ). Higher AP, lower OR is better for both plots.

Figure 5: Our model’s successful predictions on CaptainCook4D [ 54 ] (top 2 rows) and Assembly101 [ 61 ] (bottom 2 rows). Our model correctly detects mistakes by identifying cues such as incorrect technique—for example, the knife positioned to produce an abnormally thin slice in row 1—and signs of struggle by the actor, such as repeatedly moving the cabin back and forth in row 3. It can also correctly predict that a step will end in a successful execution by leveraging cues that indicate the correctness of the remaining portion of the step—for instance, a closed pepper container in row 2 may suggest that the actor will not add extra pepper to the eggs, while a correctly installed wheel in row 4 may indicate that the remaining wheels will also be installed correctly.

Figure 6: Qualitative examples illustrating our model’s frame-level attention scores at the exit point for CaptainCook4D [ 54 ] (top) and Assembly101 [ 61 ] (bottom). We observe that the model attends the most to frames that are strongly indicative of whether the step will ultimately be executed correctly or not. For example, in the CaptainCook4D case, the model focuses most on the frame where the knife makes a small cut on the bagel, suggesting that the subsequent slice will follow that cut and become too thin, leading to a mistake. In the Assembly101 example, the base aligns well with the chassis in frame 3, indicating that the step is being executed correctly and leading the model to assign the highest attention to that frame.

Model	AP %	OR %
FastForward [ 16 ]	56.2	5.0
Ours	56.8	3.7

Table 1: Early mistake detection results for our method and the state-of-the-art RL baseline FastForward [ 16 ] , averaged across 3 random seeds. Higher AP, lower OR is better

Temporal Video Understanding

Temporal video understanding studies how models track actions, events, and changes over time to answer questions, locate moments, and predict what happens next.

See all 358 papers

Post-Training VLMs for Video Mistake Detection
Aug 28, 2026Federico Spurio, Olga Zatsarynna, Lars Doorenbos +3VLM Post-Training Vision-Language Models
University of Bonn Bonn, Germany · Lamarr Institute for Machine Learning and Artificial Intelligence Bonn, Germany · Toyota Motor Europe Belgium
The Unreasonable Effectiveness of VLMs for Zero-shot Procedural Mistake Detection
Jun 19, 2026Serdar Ozsoy, Lars Doorenbos, Federico Spurio +2Temporal Action Segmentation Vision-Language Models
University of Bonn · Lamarr Institute for Machine Learning and Artificial Intelligence · Toyota Motor Europe
ESTANet: Efficient Online Error Detection in Procedural Videos via Prediction Inconsistency
Jun 24, 2026Shih-Po Lee, Reza Ghoddoosian, Faizan Siddiqui +2Online Anomaly Detection Temporal Video Understanding
Honda Research Institute, USA · Northeastern University

Aug 28, 2026J/K move · Enter open · S save

Post-Training VLMs for Video Mistake Detection

Federico Spurio, Olga Zatsarynna, Lars Doorenbos, Emad Bahrami+2

University of Bonn Bonn, Germany · Lamarr Institute for Machine Learning and Artificial Intelligence Bonn, Germany · Toyota Motor Europe Belgium

VLM Post-Training Vision-Language Models Video QA

Open paper AskPDF arXiv

Human mistakes are inevitable when following instructions, yet they can lead to severe consequences. As such, there has been an increased interest in developing methods for detecting mistakes in videos, with current methods mostly focusing on closed-set protocols. While successful in controlled settings, the closed-set assumption limits their wider applicability, as any changes to the task require collecting new data and re-training models. Instead, we argue that mistake detection methods should learn the general concept of a mistake, rather than overfitting to step-specific details. To reflect this, we introduce the Mistake Detection Video Question Answering (MD-VQA) protocol and accompanying benchmark. MD-VQA tests whether methods can discern if a step was executed correctly with respect to its description, for both seen and unseen actions. To address this important challenge, we propose the first video-language-model post-training technique for mistake detection. Our method uses a tailored reward function to encourage the model to identify discrepancies between an instruction and the corresponding video. Extensive evaluations demonstrate that this approach outperforms zero-shot, supervised fine-tuning, and post-training baselines. Notably, our method generalizes especially well to unseen procedures, for instance, with an improvement of up to 11.6% over the best-performing baseline on EP-VQA, paving the way toward general mistake detection. We release our code and benchmark at https://github.com/FedeSpu/mstk.

Abstract

Temporal Video Understanding

Explore similar work

Post-Training VLMs for Video Mistake Detection

The Unreasonable Effectiveness of VLMs for Zero-shot Procedural Mistake Detection

ESTANet: Efficient Online Error Detection in Procedural Videos via Prediction Inconsistency