cs.CVAug 28, 2026

Post-Training VLMs for Video Mistake Detection

Authors: Federico Spurio, Olga Zatsarynna, Lars Doorenbos, Emad Bahrami, Gianpiero Francesca, Juergen Gall

Organizations: University of Bonn Bonn, Germany · Lamarr Institute for Machine Learning and Artificial Intelligence Bonn, Germany · Toyota Motor Europe Belgium

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Human mistakes are inevitable when following instructions, yet they can lead to severe consequences. As such, there has been an increased interest in developing methods for detecting mistakes in videos, with current methods mostly focusing on closed-set protocols. While successful in controlled settings, the closed-set assumption limits their wider applicability, as any changes to the task require collecting new data and re-training models. Instead, we argue that mistake detection methods should learn the general concept of a mistake, rather than overfitting to step-specific details. To reflect this, we introduce the Mistake Detection Video Question Answering (MD-VQA) protocol and accompanying benchmark. MD-VQA tests whether methods can discern if a step was executed correctly with respect to its description, for both seen and unseen actions. To address this important challenge, we propose the first video-language-model post-training technique for mistake detection. Our method uses a tailored reward function to encourage the model to identify discrepancies between an instruction and the corresponding video. Extensive evaluations demonstrate that this approach outperforms zero-shot, supervised fine-tuning, and post-training baselines. Notably, our method generalizes especially well to unseen procedures, for instance, with an improvement of up to 11.6% over the best-performing baseline on EP-VQA, paving the way toward general mistake detection. We release our code and benchmark at https://github.com/FedeSpu/mstk.

Figures & tables

Figure 1: Mistake detection in seen and unseen domains. Our benchmark evaluates models not only on recipes present in the training data (left), but also tests their ability to generalize the abstract concept of “mistake” to novel tasks (right). While standard training strategies (Zero-Shot, SFT, and GRPO) fail to detect these subtle errors or struggle to generalize, our proposed post-training approach successfully grounds its reasoning to accurately identify mistakes across both scenarios. Examples from EP-VQA.
Figure 2: Example from CC-VQA Test Seen split. Question qq asks whether the cooking step “cut the english muffin into two even pieces with a knife” contains any mistakes in the video vv . The ground-truth answer αi\alpha_{i} is “YES” as the muffin is shredded by hand rather than cut with a knife.
\topline DatasetTrainTest SeenTest Unseen
 NormalErrorNormalErrorNormalError
EP-VQA35712706025453260
CC-VQA31861077484107459100
\bottomline      
Table 1: Number of samples for the EP-VQA and CC-VQA datasets.
Figure 3: Overview of our method. To improve the reasoning capabilities of VLMs for mistake detection, we post-train models with GRPO using a novel opposite reward function, which encourages different responses for video clip pairs demonstrating correct and incorrect executions of the same step. A: We pair an incorrectly performed step (bottom) with an “opposite” correct execution from a reference pool (top). The VLM processes both videos independently. B: The model earns an accuracy reward for correct answers, and an opposite reward if its answers for the two clips differ from one another. The opposite reward is only given if the accuracy reward is achieved. Through this training, the model’s ability to detect mistakes as subtle discrepancies between instructional step descriptions and their execution is improved.
\toplineCC-VQAEP-VQA
MethodSeenUnseenSeenUnseen
 RecPrecF1RecPrecF1RecPrecF1RecPrecF1
ZS0.00.00.00.00.00.00.00.00.00.00.00.0
ZS + R41.916.723.854.820.629.960.412.520.721.712.315.7
GPT-4o-mini + R---------50.020.429.0
SFT43.021.528.760.024.234.51.914.33.33.37.74.7
Table 2: Comparative evaluation on the CC-VQA and EP-VQA seen and unseen test splits. Our proposed post-training method learns a generalizable mistake representation that performs well for both seen and unseen videos.
Table 3: Recall for the seen and unseen test splits of CC-VQA. We show the number of samples for each mistake type in parentheses for seen (S) and unseen (U) splits. The number of training error samples is shown next to the name of the corresponding mistake category. Our method consistently scores among the best methods in mistake recall for both seen and unseen instructions.
More in this topic

VLM Post-Training

See all 11 papers

Explore similar work

CardsList
Jun 19, 2026

The Unreasonable Effectiveness of VLMs for Zero-shot Procedural Mistake Detection

Procedural mistake detection is important for quality control and user assistance across many disciplines. Recent work in this field has achieved significant gains by using the reasoning capabilities of Video-Language Models (VLMs) as components within multi-stage pipelines, which consist of separate modules for supervised temporal action segmentation, error detection, and explainability. Consequently, they remain dependent on tailored training datasets and require task-specific training, limiting their wider applicability. To remedy this, we introduce zero-shot procedural mistake detection and propose a unified Zero-shot Procedural Mistake detection (ZeProM) framework that jointly solves procedural mistake detection and temporal action segmentation with a single pre-trained VLM. By evaluating our framework on two canonical mistake detection benchmarks, EgoPER and CaptainCook4D, we find that ZeProM can perform these tasks successfully, while approaching, or even outperforming, the performance of fully supervised methods. For instance, we achieve a 4.4 point improvement in EDA and a 2.0 point improvement in [email protected] on average over all five EgoPER tasks compared to the strongest supervised methods. Overall, our results show the potential of unified methods for procedural mistake detection, and we hope this will steer the field away from highly complex pipelines and toward more generally applicable solutions.

Serdar Ozsoy, Lars Doorenbos, Federico Spurio +2

University of Bonn · Lamarr Institute for Machine Learning and Artificial Intelligence · Toyota Motor Europe

Temporal Action SegmentationVision-Language ModelsEgocentric Video Understanding
AskPDF
Mar 15, 2026

MistExit: Learning to Exit for Early Mistake Detection in Procedural Videos

We introduce the task of early mistake detection in video, where the goal is to determine whether a keystep in a procedural activity is performed correctly while observing as little of the streaming video as possible. To tackle this problem, we propose a method comprising a mistake detector and a reinforcement learning policy. At each timestep, the detector processes recently observed frames to estimate the keystep's correctness while anticipating future visual features, enabling reliable early mistake estimates. Meanwhile, the policy aggregates the detector outputs and visual observations over time and adaptively decides when to exit (i.e., stop processing incoming frames) while producing the final prediction. Using diverse real-world procedural video datasets, we demonstrate that our MistExit model achieves superior mistake detection accuracy while reducing the fraction of video observed compared to state-of-the-art models. Project: https://vision.cs.utexas.edu/projects/mist_exit.

Sagnik Majumder, Anish Nethi, Ziad Al-Halah +1

UT Austin · University of Utah

Temporal Video Understanding
AskPDF
Jun 1, 2026

Understanding-Enhanced Model Collaboration for Long-Tailed Egocentric Mistake Detection

In this report, we address the problem of determining whether a user performs an action incorrectly from egocentric video data. To this end, we propose an Understanding-Enhanced Model Collaboration Method (UE-MCM) that combines efficient coarse-grained video understanding with accurate fine-grained action reasoning. Specifically, UE-MCM contains a small model branch and a large model branch. The large model branch focuses on whether the fine-grained action itself is executed incorrectly, while the small model branch jointly takes the coarse-grained video and fine-grained segment as input to identify actions that may be locally correct but inconsistent with the overall workflow. The small model branch is built on a CLIP4CLIP video encoder initialized from a CLIP model enhanced by Diffusion Contrastive Reconstruction, and the large model branch uses the Qwen3-VL Embedding model to extract high-capacity representations from fine-grained action segments. The small-branch prediction and the large-branch prediction are then adaptively fused by a lightweight collaboration gate. To handle the long-tailed distribution of mistake instances, we optimize the classifiers with complementary objectives, including reweighted cross-entropy, AUC-oriented learning, and label-aware adjustment. The resulting system balances speed and accuracy, making it effective for detecting subtle, rare, and ambiguous mistakes in egocentric instructional videos.

Boyu Han, Qianqian Xu, Shilong Bao +3

State Key Laboratory of AI Safety, Institute of Computing Technology, CAS · School of Computer Science and Tech., University of Chinese Academy of Sciences · Beijing Academy of Artificial Intelligence +2

Egocentric Video UnderstandingTemporal Video Understanding
AskPDF