Post-Training VLMs for Video Mistake Detection

Federico Spurio; Olga Zatsarynna; Lars Doorenbos; Emad Bahrami; Gianpiero Francesca; Juergen Gall

cs.CVAug 28, 2026

Post-Training VLMs for Video Mistake Detection

Authors: Federico Spurio, Olga Zatsarynna, Lars Doorenbos, Emad Bahrami, Gianpiero Francesca, Juergen Gall

Organizations: University of Bonn Bonn, Germany · Lamarr Institute for Machine Learning and Artificial Intelligence Bonn, Germany · Toyota Motor Europe Belgium

Abstract

Human mistakes are inevitable when following instructions, yet they can lead to severe consequences. As such, there has been an increased interest in developing methods for detecting mistakes in videos, with current methods mostly focusing on closed-set protocols. While successful in controlled settings, the closed-set assumption limits their wider applicability, as any changes to the task require collecting new data and re-training models. Instead, we argue that mistake detection methods should learn the general concept of a mistake, rather than overfitting to step-specific details. To reflect this, we introduce the Mistake Detection Video Question Answering (MD-VQA) protocol and accompanying benchmark. MD-VQA tests whether methods can discern if a step was executed correctly with respect to its description, for both seen and unseen actions. To address this important challenge, we propose the first video-language-model post-training technique for mistake detection. Our method uses a tailored reward function to encourage the model to identify discrepancies between an instruction and the corresponding video. Extensive evaluations demonstrate that this approach outperforms zero-shot, supervised fine-tuning, and post-training baselines. Notably, our method generalizes especially well to unseen procedures, for instance, with an improvement of up to 11.6% over the best-performing baseline on EP-VQA, paving the way toward general mistake detection. We release our code and benchmark at https://github.com/FedeSpu/mstk.

Figures & tables

Figure 1: Mistake detection in seen and unseen domains. Our benchmark evaluates models not only on recipes present in the training data (left), but also tests their ability to generalize the abstract concept of “mistake” to novel tasks (right). While standard training strategies (Zero-Shot, SFT, and GRPO) fail to detect these subtle errors or struggle to generalize, our proposed post-training approach successfully grounds its reasoning to accurately identify mistakes across both scenarios. Examples from EP-VQA.

Figure 2: Example from CC-VQA Test Seen split. Question $q$ asks whether the cooking step “cut the english muffin into two even pieces with a knife” contains any mistakes in the video $v$ . The ground-truth answer $\alpha_{i}$ is “YES” as the muffin is shredded by hand rather than cut with a knife.

\topline Dataset	Train		Test Seen		Test Unseen
	Normal	Error	Normal	Error	Normal	Error
EP-VQA	3571	270	602	54	532	60
CC-VQA	3186	1077	484	107	459	100
\bottomline

Table 1: Number of samples for the EP-VQA and CC-VQA datasets.

Figure 3: Overview of our method. To improve the reasoning capabilities of VLMs for mistake detection, we post-train models with GRPO using a novel opposite reward function, which encourages different responses for video clip pairs demonstrating correct and incorrect executions of the same step. A: We pair an incorrectly performed step (bottom) with an “opposite” correct execution from a reference pool (top). The VLM processes both videos independently. B: The model earns an accuracy reward for correct answers, and an opposite reward if its answers for the two clips differ from one another. The opposite reward is only given if the accuracy reward is achieved. Through this training, the model’s ability to detect mistakes as subtle discrepancies between instructional step descriptions and their execution is improved.

\topline	CC-VQA						EP-VQA
Method	Seen			Unseen			Seen			Unseen
	Rec	Prec	F1	Rec	Prec	F1	Rec	Prec	F1	Rec	Prec	F1
ZS	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
ZS + R	41.9	16.7	23.8	54.8	20.6	29.9	60.4	12.5	20.7	21.7	12.3	15.7
GPT-4o-mini + R	-	-	-	-	-	-	-	-	-	50.0	20.4	29.0
SFT	43.0	21.5	28.7	60.0	24.2	34.5	1.9	14.3	3.3	3.3	7.7	4.7

Table 2: Comparative evaluation on the CC-VQA and EP-VQA seen and unseen test splits. Our proposed post-training method learns a generalizable mistake representation that performs well for both seen and unseen videos.

Panel of Table 3 — Table 3: Recall for the seen and unseen test splits of CC-VQA. We show the number of samples for each mistake type in parentheses for seen (S) and unseen (U) splits. The number of training error samples is shown next to the name of the corresponding mistake category. Our method consistently scores among the best methods in mistake recall for both seen and unseen instructions.

VLM Post-Training

See all 11 papers

The Unreasonable Effectiveness of VLMs for Zero-shot Procedural Mistake Detection
Jun 19, 2026Serdar Ozsoy, Lars Doorenbos, Federico Spurio +2Temporal Action Segmentation Vision-Language Models
University of Bonn · Lamarr Institute for Machine Learning and Artificial Intelligence · Toyota Motor Europe
MistExit: Learning to Exit for Early Mistake Detection in Procedural Videos
Mar 15, 2026Sagnik Majumder, Anish Nethi, Ziad Al-Halah +1Temporal Video Understanding
UT Austin · University of Utah
Understanding-Enhanced Model Collaboration for Long-Tailed Egocentric Mistake Detection
Jun 1, 2026Boyu Han, Qianqian Xu, Shilong Bao +3Egocentric Video Understanding Temporal Video Understanding
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS · School of Computer Science and Tech., University of Chinese Academy of Sciences · Beijing Academy of Artificial Intelligence +2

Jun 19, 2026J/K move · Enter open · S save

The Unreasonable Effectiveness of VLMs for Zero-shot Procedural Mistake Detection

Serdar Ozsoy, Lars Doorenbos, Federico Spurio, Gianpiero Francesca+1

University of Bonn · Lamarr Institute for Machine Learning and Artificial Intelligence · Toyota Motor Europe

Temporal Action Segmentation Vision-Language Models Egocentric Video Understanding

Open paper AskPDF arXiv

Procedural mistake detection is important for quality control and user assistance across many disciplines. Recent work in this field has achieved significant gains by using the reasoning capabilities of Video-Language Models (VLMs) as components within multi-stage pipelines, which consist of separate modules for supervised temporal action segmentation, error detection, and explainability. Consequently, they remain dependent on tailored training datasets and require task-specific training, limiting their wider applicability. To remedy this, we introduce zero-shot procedural mistake detection and propose a unified Zero-shot Procedural Mistake detection (ZeProM) framework that jointly solves procedural mistake detection and temporal action segmentation with a single pre-trained VLM. By evaluating our framework on two canonical mistake detection benchmarks, EgoPER and CaptainCook4D, we find that ZeProM can perform these tasks successfully, while approaching, or even outperforming, the performance of fully supervised methods. For instance, we achieve a 4.4 point improvement in EDA and a 2.0 point improvement in [email protected] on average over all five EgoPER tasks compared to the strongest supervised methods. Overall, our results show the potential of unified methods for procedural mistake detection, and we hope this will steer the field away from highly complex pipelines and toward more generally applicable solutions.

Abstract

VLM Post-Training

Explore similar work

The Unreasonable Effectiveness of VLMs for Zero-shot Procedural Mistake Detection

MistExit: Learning to Exit for Early Mistake Detection in Procedural Videos

Understanding-Enhanced Model Collaboration for Long-Tailed Egocentric Mistake Detection