Through a PRISM, Clearly: Step-Sequence Reasoning for Mistake Recognition in Procedural Videos

Olga Loginova; Oleksandr Bezrukov; Frank Keller

cs.CVOct 22, 2026

Through a PRISM, Clearly: Step-Sequence Reasoning for Mistake Recognition in Procedural Videos

Authors: Olga Loginova, Oleksandr Bezrukov, Frank Keller

Organizations: University of Trento, Trento, Italy · Gran Sasso Science Institute, L’Aquila, Italy · University of Edinburgh, Edinburgh, United Kingdom

Abstract

Procedural mistake recognition requires identifying deviations from valid multi-step activities. We study post-completion recognition from videos with benchmark-provided step boundaries. We introduce PRISM (Procedural mistake Recognition via Interpretable Semantic Modeling), which grounds video segments in language and maps their descriptions to procedural states using a lightweight hidden Markov model trained on correct executions. Reference alignment and semantic checks identify errors in step content and order. To support evaluation across heterogeneous benchmarks, we map dataset-specific annotations to four edit-based error types: insertion, deletion, substitution, and transposition. Across three benchmarks, we compare gold and VLM-generated descriptions in raw, semantic-role, and action-object forms. Decoder controls show consistent gains from the trained local decoder over nearest-candidate matching, with additional, dataset-dependent benefits from sequence decoding. Semantic role labeling improves Any-F1 in all six matched grounded comparisons, although the gains vary across datasets and do not translate into uniform improvements under separate configuration selection. These results establish language grounding as an effective interface for explicit procedural mistake reasoning and characterize the contributions of representation and decoding.

Figures & tables

Figure 1: PRISM representation of the valid Zoodles procedure and an erroneous execution from CaptainCook4D. Circles contain Step IDs, adjacent text gives step descriptions, and solid arrows trace the reference order. Red marks show a transposition of Steps 193, 194, and 198, a substitution at Step 193, and the deletion of Step 195.

Figure 2: The PRISM pipeline for post-completion mistake recognition. (1) For each step of each training video, we obtain a textual, visual, or multimodal encoding, ${\bm{x}}_{t}$ . (2) After PCA regularization, we obtain the representation of each step, ${\bm{z}}_{t}$ . (3) During training, the model $\theta_{\text{task}}$ learns from sequences of ${\bm{z}}_{t}$ for each task. Additionally, when we put all tasks together, we obtain a global HMM $\theta_{-1}$ . (4) At inference time, we obtain embeddings for each video segment in the same way as during training and compute the emissions $[\bm{E}]_{t,s}$ . (5) We then run Viterbi for sequence inference and keep the emission and transition scores for each step. (6) For each task, we also have a set of possible non-erroneous sequences. (7) We align the predicted sequence to the reference sequence using the selected sequence-alignment method. (8) Mismatches in the alignment indicate possible structural errors. Falling below the predefined thresholds for emission and transition scores indicates possible low-probability events. (9) During evaluation, we compare the predicted errors with the gold-standard error annotations of the dataset.

	EgoOops				EgoPER				CC4D
Family	Seq	Any-F1	Type-F1	AUC	Seq	Any-F1	Type-F1	AUC	Seq	Any-F1	Type-F1	AUC
Gold text	0.713	0.480	0.446	0.756	0.903	0.741	0.673	0.918	0.617	0.631	0.466	0.761
Vision	0.328	0.454	0.439	0.523	0.372	0.519	0.527	0.580	0.159	0.556	0.464	0.478
Gold + GPT-SRL	0.742	0.510	0.533	0.775	0.913	0.736	0.675	0.908	0.638	0.629	0.495	0.757
Qwen text	0.493	0.441	0.451	0.564	0.802	0.607	0.599	0.736	0.517	0.603	0.498	0.640
Qwen + GPT-SRL	0.499	0.409	0.407	0.623	0.816	0.601	0.594	0.742	0.485	0.592	0.505	0.648

Table 1: Family-level leaderboard. Each row reports all metrics from one complete configuration. Encoder and HMM settings may differ across rows. Seq denotes sequence accuracy, and CC4D denotes CaptainCook4D. Appendix E.1 lists the corresponding configurations.

	Gold				Qwen				Gemini
Dataset	Raw	GPT-SRL	$\bm{\Delta}$	Act.+Obj.	Raw	GPT-SRL	$\bm{\Delta}$	Act.+Obj.	Raw	GPT-SRL	$\bm{\Delta}$	Act.+Obj.
EgoOops	0.400	0.434	+0.034	0.318	0.321	0.356	+0.036	0.444	0.349	0.353	+0.003	0.414
EgoPER	0.681	0.725	+0.044	0.540	0.578	0.601	+0.024	0.550	0.602	0.631	+0.030	0.552
CaptainCook4D	0.629	0.629	+0.000	0.571	0.587	0.592	+0.005	0.566	0.595	0.596	+0.001	0.570

Table 2: Any-F1 by text source and representation. Each Raw–GPT-SRL pair uses identical downstream settings, and

\Delta

is GPT-SRL minus Raw. Act.+Obj. reports the best configuration within that representation family. Bold marks the highest score within each dataset–source group. Differences are computed before rounding.

		Nearest candidate		Greedy emission		Viterbi
Dataset	VLM	Raw	SRL	Raw	SRL	Raw	SRL
EgoOops	Qwen	0.337	0.335	0.339	0.351	0.321	0.356
EgoOops	Gemini	0.305	0.315	0.339	0.328	0.349	0.353
EgoPER	Qwen	0.547	0.524	0.578	0.601	0.578	0.601
EgoPER	Gemini	0.571	0.557	0.602	0.631	0.602	0.631
CaptainCook4D	Qwen	0.567	0.559	0.581	0.578	0.587	0.592

Table 3: Decoder controls under the matched Raw–SRL configurations. Nearest-candidate matching uses cosine similarity and no HMM. Greedy decoding uses learned emissions without start or transition probabilities. Viterbi uses the complete HMM. Bold marks the highest Any-F1 within each dataset–VLM–representation condition.

	EgoPER		CaptainCook4D
Method	EDA	AUC	Precision	EDA	AUC
AMNAR	0.644	0.685	0.668	0.723	0.602
AEM (pred. segments)	0.667	0.738	0.681	0.719	0.625
AEM (GT segments)	0.747	0.867	–	–	–
Ours (Qwen)	0.704	0.751	0.253	0.646	0.694
Ours (Qwen + GPT-SRL)	0.699	0.753	0.785	0.841	0.900

Table 4: System-level results under benchmark-specific protocols. PRISM uses benchmark-provided step boundaries. AEM results are shown for predicted and ground-truth (GT) segmentation. Baseline scores are taken from the original papers; all metrics in each PRISM row come from one configuration.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Native label / description	Unified label
CC4D	Order Error	Transposition
	Missing Step	Deletion
	Preparation Error	Substitution
	Measurement Error	Substitution
	Timing Error	Substitution
	Technique Error	Substitution

Appendix

Table 6: Mapping from benchmark-specific error labels to the unified edit-based taxonomy used in this work. For EgoOops, the rare others cases are resolved individually from the textual error descriptions. CC4D is CaptainCook4D.

Dataset	Type	ID	Step description	SRL representation
CC4D	modified	20	Microwave the plate, covered, on high for 5 minutes instead of 1.5 minutes	MICROWAVE(Agent: you, Object: plate(State: covered), Setting: high, Duration: minute(Quantity: 5), Manner: INSTEAD_OF(Duration: minute(Quantity: 1.5)))
	modified	28	Measure 1/8 teaspoon of salt using dirty measuring spoon and add it to the mug	ADD(Agent: you, Object: salt(Measurement: teaspoon(Quantity: 1/8)), Destination: mug, Instrument: dirty(measuring_spoon), Manner: after(MEASURE(Agent: you, Object: salt(Measurement: teaspoon(Quantity: 1/8)), Instrument: dirty(measuring_spoon))))
	canonical	54	Put all the Vegetables in a microwave-safe bowl	PUT(Agent: you, Object: vegetables(Quantity: all), Destination: bowl(Property: microwave_safe))
EgoOops	canonical	211	Put chopsticks through the cutout in the lid to widen the hole.	PUT(Agent: you, Object: chopsticks, Path: through(cutout(Location: in(lid))), Purpose: WIDEN(Agent: you, Object: hole))
	modified	10	Pour about 15mL of water into a cup, dip the tip of a red highlighter in the water and squeeze out a drop. Knock over an empty cup.	POUR(Agent: you, Object: water(Measurement: ml(Quantity: 15, Approx: about)), Destination: into(cup)); DIP(Agent: you, Object: tip(of(red_highlighter)), Destination: in(water(Location: in(cup)))); SQUEEZE_OUT(Agent: you, Object: drop); KNOCK_OVER(Agent: you, Object: empty_cup)
EgoPER	canonical	57	Put banana slices on tortilla	PUT(Agent: you, Object: banana_slices, Destination: on(tortilla))

Appendix

Table 7: Representative SRL examples from the three benchmarks. We include both canonical and modified step descriptions to show how the representation handles quantities, instruments, destinations, purposes, and multi-event modified steps. CC4D is CaptainCook4D.

Hyperparameter	Sym.	Tags	Meaning	Value effect	Default settings & rationale
Frames per step	$n$	t/i	Number of frames per step averaged into the image feature.	High: stronger denoising and object coverage, but more IO/latency and possible over-smoothing on very short steps; low: faster and crisper, but noisier and more sensitive to accidental frames.	3 : a good robustness–latency trade-off; using more frames may introduce irrelevant content in short steps but can help with long, stable steps.
PCA dimension	$d$	t/i	Output dimensionality after PCA (fit on train; applied at inference) before Gaussian emissions.	High: preserves fine semantics, but makes covariance estimation harder and can overfit; low: better conditioning and sample-efficiency, but too low drops discriminative cues.	128 : stable Ledoit–Wolf covariances with modest memory/compute; increase with ample per-class data.
Laplace smoothing	$\beta$	t	Add- $\beta$ smoothing for transition ${\bm{A}}$ and start prior $\bm{\pi}$ (counts+ $\beta$ , renormalize).	High: flatter ${\bm{A}},\bm{\pi}$ , safer on rare or omitted transitions, and less brittle; low: sharper priors that mirror train paths, but more risk of overfitting ordering quirks.	3.0 : applied before reference-based transition masking.
Banding (local moves)	band	t	For each valid reference sequence, let $i$ and $j$ be the positions of two states in that sequence. Mask $A_{ss^{\prime}}$ unless $\|i{-}j\|\leq 1$ in at least one valid reference, then renormalize each row; encodes locally ordered procedures.	On: restricts non-local transitions and produces more locally ordered paths; off: allows global jumps for branched workflows, but raises transposition false positives.	On with $\|i{-}j\|\leq 1$ : disabling recommended only if tasks truly need non-local transitions.
Jitter probability	$p_{\text{jit}}$	t	Probability to add Gaussian noise to a training vector before PCA.	High: more invariance to ambience and lighting, but too much blurs class boundaries; low: crisper clusters, but more brittle to domain shift.	0.15 : mild diversity without collapsing adjacent steps; co-tuned with $\sigma$ .

Appendix

Table 8: Core HMM hyperparameters for sequence modeling and training-time regularization. Tags indicate whether a hyperparameter is used during training ( t ) or inference ( i ).

Hyperparameter	Sym.	Tags	Meaning	Value effect	Default settings & rationale
Jitter std	$\sigma$	t	Noise scale in $\mathcal{N}(0,\sigma^{2}I)$ before PCA.	High: more robust, but can distort likelihoods and increase class overlap; low: minimal regularization.	0.3 in configurations that use jitter.
Per-modality normalization	l2/None	t/i	L2-normalize each modality before fusion and HMM; stabilizes scales and PCA.	l2: avoids dominance by high-variance channels and yields steadier thresholds; none: preserves raw magnitudes, but can skew fusion and Gaussian likelihoods.	l2 : matches CLIP best practice; lets $w$ modulate content, not scale.
Emission family	Gauss or Cosine	i	Observation model at inference: Gaussian log-likelihoods vs. cosine similarity (with temperature).	Gauss: calibrated log-likelihoods and good use of PCA plus Ledoit–Wolf; cosine: magnitude-invariant, but needs a temperature and may require transition rebalancing.	Gauss default; switch to Cosine for strongly angular features or when skipping PCA.
Cosine temperature	$T$	i	Scale for cosine emissions ( $E=T\!\cdot\!\cos$ ); controls emission/transition balance.	High: more emission-dominant, peak-following paths; low: more transition-dominant, smoother paths.	N/A for Gauss. For Cosine, tuned $T$ to match the dynamic range of log-transitions.
Emission scaling	$\alpha$	i	Global multiplier $\tilde{{\bm{E}}}=\alpha{\bm{E}}$ to retune emission vs. transition influence.	High: more emission-driven Viterbi, hence more sensitive to per-step scores; low: more transition-driven, smoother, and more prior-driven.	1.0 : adjustment needed only if domain shift skews score ranges.
Semantic substitution threshold	$\tau_{\mathrm{sem}}$	i	Cosine-similarity threshold between an observation and the mean embedding of its decoded state. A lower score produces a semantic Substitution when structural alignment retains the same Step ID.	High: detects subtler argument-level changes but may increase false positives; low: is more conservative but may miss modified executions.	0.85 in all reported experiments.

Appendix

Table 9: Core HMM hyperparameters for normalization and emission control. Tags indicate whether a hyperparameter is used during training ( t ) or inference ( i ).

Hyperparameter	Sym.	Tags	Meaning	Value effect	Default settings & rationale
Emission anomaly threshold	$\tau_{\mathrm{em}}$	i	Flag if the selected-state emission score is below $\tau_{\mathrm{em}}$ .	Higher thresholds flag more observations; lower thresholds are more conservative.	$-10$ for Gaussian emissions and $0$ for cosine emissions.
Transition anomaly threshold	$\tau_{\mathrm{tr}}$	i	Flag if $\log A_{S_{t-1},S_{t}}<\tau_{\mathrm{tr}}$ .	Higher thresholds flag more transitions; lower thresholds are more conservative.	$-23.5$ for CaptainCook4D and $-5$ for EgoPER and EgoOops.
Alignment mode	NW or difflib	i	Align prediction to a canonical variant to label Deletion, Insertion, Substitution, or Transposition.	NW: stable edit metric (use sub=2, gap=1 for reports). difflib: forgiving for narratives and qualitative inspection.	difflib is our default in the main experiments because it performs well across grounded settings; NW remains competitive and is preferred in some paper-facing or family-specific comparisons.
Global fallback HMM	task $=-1$	i	Task-agnostic model pooled over all tasks and available when task-specific parameters are missing.	Provides an implementation fallback when no task-specific model is available.	Implemented but not invoked in the reported experiments.
Vision weight (mm)	$w$	mm (t/i)	Fusion in joint space: $\bm{x}=w\,\bm{v}_{img}+(1-w)\,\bm{v}_{txt}$ (or scaled-concat in disjoint space).	High: vision-led. Low: text-led; better when phrasing disambiguates or clarifies visuals.	Task-dependent . Optimal $w{\approx}0.2$ – $0.4$ : for subtle manipulations; increase when distinctive visuals drive the step.
Use superstructure (mm)	on/off	mm (t/i)	Add structured phrases (objects/actions) as text before fusion.	On: boosts separability for visually similar steps; sensitive to noisy phrases. Off: simpler; may blur key discriminators.	Off for clean baselines; On in V + AO style variants with reliable extractors and prompt filtering.

Appendix

Table 10: Additional HMM hyperparameters for anomaly scoring, alignment, and multimodal variants. Tags indicate whether a hyperparameter is used during training ( t ), inference ( i ), or in a multimodal setting ( mm ); entries like mm (t/i) denote multimodal settings with separate training/inference use.

GPT-5.3 label	Step description	GPT-5.3	Qwen3.6-27B
Setting	Microwave the ramekin cup uncovered on high for 10 seconds	Setting: high	Manner: uncovered & on(high)
Surface	Brush 2 slices of baguette with olive oil on both sides	Surface: both_sides	Location: on(both_sides)
Position	Put three copper plates on the left column of the microplate	column(of(microplate), Position: left)	left_column( of(microplate))
Container	In the mug, whisk two eggs	eggs(Quantity: 2, Container: mug)	eggs(Quantity: 2), Location: in(mug)
Duration	Microwave the plate, uncovered, on high for 5 minutes	Duration: minute(Quantity: 5)	Temporal: for(minute(Quantity: 5))
Range	Continue to microwave for 15–30 more seconds	seconds(Range: 15-30)	seconds(Quantity: 15-30)

Appendix

Table 11: Examples of role-inventory differences between GPT-5.3 and Qwen3.6-27B. GPT-5.3 uses some low-frequency labels that Qwen3.6-27B often represents with broader roles, nested modifiers, or explicit event composition.

Difference	Step description	GPT-5.3	Qwen3.6-27B
Predicate choice	Use a butter knife to scoop nut butter from the jar; Use scissors to cut the bag in half	USE(Object: butter_knife, Purpose: SCOOP(Object: nut_butter, Origin: jar)) ; USE(Object: scissors, Purpose: CUT(Object: bag, Result: in_half))	SCOOP(Object: nut_butter, Instrument: butter_knife, Origin: jar) ; CUT(Object: bag, Instrument: scissors, Result: in(half))
Alternative action	Chop or grate the cucumber	CHOP_OR_GRATE(Object: cucumber)	CHOP(Object: cucumber) \| GRATE(Object: cucumber)
Event decomposition	Measure and add 2 tbsp water to the bowl	ADD(Object: water(Measurement: tablespoon(Quantity: 2)), Destination: bowl, Manner: MEASURED)	MEASURE(Object: water(Measurement: tablespoon(Quantity: 2))) & ADD(Object: water(...), Destination: to(bowl))
Predicate normalization	Slice mushrooms	SLICE(Object: mushrooms)	CUT(Object: mushrooms, Result: slices)
State change	Return to low heat	RETURN(Destination: to(low_heat))	SET(Object: heat, Result: low)
Error-event scope	Spill while cracking an egg in the bowl	SPILL(Temporal: WHILE(CRACK(Object: egg, Destination: in(bowl))))	CRACK(Object: egg, Destination: bowl, Manner: spill)

Appendix

Table 12: Differences between SRL representations generated by GPT-5.3 and Qwen3.6-27B. The two models use the same predicate–argument syntax but differ in predicate choice, role assignment, event decomposition, and scope representation. Examples are shortened where needed.

Component	Content
System instruction	You are a linguistic semantic analyzer. For each procedural step description, generate the semantic representation in roles.
Prompt guidance	Predicates are uppercase. Concrete manipulated entities are usually encoded under Object . Agent: you is used unless another agent is explicitly specified. Entity names are normalized as lowercase_with_underscores. Nested structures are allowed when needed, e.g., tip(of(red_highlighter)) or water(Location: in(cup)) . The prompt also gives typical role names such as Object , Coobject , Destination , Location , Instrument , Manner , Purpose , Temporal , Origin , and Result .
Selected few-shot examples	INSERT(Agent: you, Object: test_swab, Destination: into(nostril(of(her)))) ; ADD(Agent: you, Object: coffee_grounds, Origin: bowl, Destination: filter(Location: in(french_press))) ; ADD(Agent: you, Object: cut(onions), Coobject: egg(Location: in(mixing_bowl))) ; TOP(Agent: you, Object: cup, Coobject: with(salsa(Measurement: tablespoon(Quantity: 1)))) ; POUR(Agent: you, Object: egg(Quantity: 1), Destination: into(ramekin_cup)) .
Output schema	Strict JSON with one item per step, containing the fields id , step_description , and semantic_representation .
Prompt calibration and checks	Iterative manual inspection calibrates the role schema and nine-example prompt. Each generator then regenerates the complete dictionaries automatically. Scripted checks verify valid JSON, required fields, and balanced delimiters. No final role or argument structure is manually edited.
Covered vocabularies	Gold canonical descriptions, modified descriptions, and VLM-predicted descriptions not covered by the first two vocabularies. Each unique description is processed once. At grounded inference, the normalized VLM-generated description is the only SRL lookup key.

Appendix

Table 13: Core design of the prompt and schema used to generate SRL representations.

Dataset	Family	Encoder	$d$	Norm	Emission	Temp	Align
EgoOops	Vision	V-JEPA	1024	L2	Gaussian	–	NW
	Gold text	CLIP	128	none	cosine	6	NW
	Gold + GPT-SRL	CLIP	128	none	cosine	12	difflib
	Qwen text	Instructor-XL	64	L2	Gaussian	–	difflib
	Qwen + GPT-SRL	Instructor-XL	64	none	Gaussian	–	difflib
	Gemini text	all-MPNet	768	L2	Gaussian	–	difflib

Appendix

Table 14: Configurations used for the family-level leaderboard in Table 1 . CaptainCook4D and EgoPER configurations are selected by validation Any-F1. EgoOops configurations are transferred from the best mean validation result across these two datasets. EgoOops error-video labels do not enter selection.

Dataset	Encoder	Output $d$	PCA $d$	Norm	Emission	Temp.	Align
EgoOops	Instructor-XL	768	128	L2	cosine	8	NW
EgoPER	RoBERTa-SimCSE	1024	128	L2	Gaussian	–	NW
CaptainCook4D	RoBERTa-SimCSE	1024	256	L2	cosine	10	difflib

Appendix

Table 15: Fixed downstream configurations for the matched Raw–GPT-SRL comparisons and decoder controls. Output

d

is the encoder output dimension; PCA

d

is the dimension after projection.

Dataset	VLM	Representation	Encoder	PCA $d$	Norm	Emission	Align
EgoPER	Qwen	Raw	all-MiniLM	256	L2	Gaussian	difflib
	Qwen	GPT-SRL	RoBERTa-SimCSE	64	none	Gaussian	difflib
	Gemini	Raw	all-MiniLM	256	L2	Gaussian	difflib
	Gemini	GPT-SRL	RoBERTa-SimCSE	64	none	Gaussian	NW
CaptainCook4D	Qwen	Raw	RoBERTa-SimCSE	256	none	Gaussian	difflib
	Qwen	GPT-SRL	RoBERTa-SimCSE	256	L2	Gaussian	difflib

Appendix

Table 16: Configurations used for the PRISM rows in the system-level comparison in Table 4 . PCA

d

is the projected dimensionality.

Dataset	Qwen Exact	Gemini Exact	$\Delta$
EgoOops	0.425	0.496	+0.070
EgoPER	0.806	0.864	+0.057
CaptainCook4D	0.578	0.695	+0.117

Appendix

Table 17: Word-for-word grounding accuracy before Step-ID decoding.

\Delta

is Gemini minus Qwen; differences are computed before rounding.

Dataset	Grounding	Raw	GPT-SRL	Qwen-SRL
EgoOops	Qwen	0.321	0.356	0.341
EgoOops	Gemini	0.349	0.353	0.341
EgoPER	Qwen	0.578	0.601	0.584
EgoPER	Gemini	0.602	0.631	0.613
CaptainCook4D	Qwen	0.587	0.592	0.588
CaptainCook4D	Gemini	0.595	0.596	0.596

Appendix

Table 18: Matched Any-F1 with GPT-5.3 and Qwen3.6-27B SRLs. Every row uses the fixed downstream configuration reported in Table 15 .

Dataset	VLM	Exact	Same ID	Other ID	Ambiguous/open	Modified
EgoOops	Qwen	+0.021 (145)	+0.095 (21)	-0.006 (170)	-0.200 (5)	+0.034 (59)
EgoOops	Gemini	+0.000 (169)	+0.000 (19)	+0.020 (153)	–	+0.000 (59)
EgoPER	Qwen	+0.014 (1771)	-0.008 (121)	+0.000 (284)	+0.250 (20)	+0.054 (500)
EgoPER	Gemini	+0.006 (1897)	+0.000 (104)	-0.005 (182)	+0.077 (13)	+0.024 (500)
CaptainCook4D	Qwen	-0.004 (891)	-0.013 (80)	-0.013 (550)	+0.048 (21)	-0.006 (363)
CaptainCook4D	Gemini	+0.014 (1071)	+0.000 (130)	+0.013 (318)	-0.087 (23)	+0.025 (363)

Appendix

Table 19: Change in positionwise decoded-ID accuracy from Raw to GPT-SRL within each grounding stratum, before sequence alignment. Support appears in parentheses. Modified overlaps the first four strata. A dash marks an empty stratum.

		Qwen grounding			Gemini grounding
Dataset	Error type	Raw	GPT-SRL	Support	Raw	GPT-SRL	Support
EgoOops	Insertion	0.082	0.119	89	0.151	0.170	90
	Deletion	0.000	0.000	0	0.000	0.000	0
	Substitution	0.220	0.215	62	0.164	0.177	62
	Transposition	0.000	0.000	0	0.000	0.000	0
EgoPER	Insertion	0.203	0.189	199/195	0.185	0.208	209/207

Appendix

Table 20: Type-specific F1 and evaluation support for all matched grounded conditions in Table 2 . Support is

TP+FN

. Raw/GPT-SRL values are shown when support differs between conditions.

Dataset	VLM	Raw Any-F1 [95% CI]	GPT-SRL Any-F1 [95% CI]	$\bm{\Delta}$ [95% CI]
EgoOops	Qwen	0.321 [0.252, 0.380]	0.356 [0.295, 0.407]	+0.036 [+0.005, +0.076]
	Gemini	0.349 [0.287, 0.414]	0.353 [0.279, 0.426]	+0.003 [ $-$ 0.028, +0.035]
EgoPER	Qwen	0.578 [0.544, 0.611]	0.601 [0.565, 0.638]	+0.024 [+0.007, +0.041]
	Gemini	0.602 [0.569, 0.633]	0.631 [0.594, 0.668]	+0.030 [+0.011, +0.048]
CaptainCook4D	Qwen	0.587 [0.520, 0.643]	0.592 [0.525, 0.650]	+0.005 [ $-$ 0.007, +0.017]
	Gemini	0.595 [0.530, 0.649]	0.596 [0.533, 0.649]	+0.001 [ $-$ 0.013, +0.015]

Appendix

Table 21: Paired video-level bootstrap 95% confidence intervals for Any-F1 in the six matched grounded comparisons. We use 10,000 replicates and percentile intervals.

\Delta

is GPT-SRL minus Raw; each replicate uses identical sampled video IDs for both conditions. Differences are computed before rounding.

	EgoOops			EgoPER			CC4D
Weights $(w_{\mathrm{img}},w_{\mathrm{txt}})$	Seq	Any-F1	AUC	Seq	Any-F1	AUC	Seq	Any-F1	AUC
$(0.2,0.8)$	0.651	0.437	0.558	0.235	0.478	0.441	0.598	0.547	0.604
$(0.4,0.6)$	0.651	0.429	0.562	0.479	0.474	0.558	0.569	0.550	0.576
$(0.6,0.4)$	0.651	0.429	0.563	0.629	0.463	0.597	0.538	0.554	0.561
$(0.8,0.2)$	0.651	0.429	0.562	0.700	0.466	0.599	0.502	0.558	0.549

Appendix

Table 22: Multimodal fusion sweep over interpolation weights between visual and textual step embeddings. For each dataset, we fix one visual–textual configuration and vary only the fusion weights.

Dataset	Matched runs	$\Delta$ Seq	$\Delta$ Any-F1	$\Delta$ Type-F1	$\Delta$ AUC
EgoOops	160	-0.0003	-0.0003	+0.0016	-0.0002
EgoPER	240	+0.0007	-0.0014	-0.0005	+0.0005
CaptainCook4D	240	+0.0007	+0.0002	-0.0006	-0.0004

Appendix

Table 23: Matched multimodal comparison of GPT-SRL and Qwen-SRL. Deltas are computed as Qwen-SRL minus GPT-SRL. AUC denotes Step-AUC.

Dataset	Reference step	Observed / grounded variant	Semantic distinction
CC4D	Place the egg from the cup over the lettuce	Place the egg from the cup under the lettuce	Makes the relation contrast explicit: over vs. under .
	Invert the mug to release the cake onto a plate	Invert the mug to release the cake onto a bowl Invert the mug to release the cake onto an unclean plate	Isolates destination and destination-property changes while preserving the same event structure.
	Invert the mug to release the cake onto a plate	Use a spoon to invert the mug to release the cake onto a plate	Preserves the added instrumental subevent as nested structure.
	Measure and add 1.5 tbsp sugar to the mixing bowl	Qwen: Measure and add 1.5 tbsp brown sugar to the mixing bowl	Preserves the MEASURE & ADD structure and highlights the ingredient mismatch.
EgoOops	Put three zinc plates on the center column of the microplate using the pair of tweezers	Qwen: Put three copper plates on the left column of the microplate using the pair of tweezers	Makes both material and location drift explicit.

Appendix

Table 24: Examples of error-relevant distinctions that SRL can represent explicitly, including relations, destinations, properties, materials, and nested events. CC4D denotes CaptainCook4D.

Effect	Dataset	Reference step	Contrasting step	What action–object captures or loses
Preserves	EgoPER	Use knife to scoop jelly	Use spoon to scoop jelly	Preserves the tool distinction because tools remain in the object set.
	CC4D	Sprinkle 1 tablespoon of cheese on cup	Pour 1 tablespoon of cheese on cup	Preserves the predicate change.
Weakens	CC4D	Microwave just until cheese melts, about 10 seconds	Microwave for 30 seconds	Loses duration and completion-condition information.
	CC4D	Top cup with 1 tablespoon of salsa	Top cup with 1/2 tablespoon of salsa	Loses scalar quantity information.
	EgoPER	Slowly pour the rest of water in circular motion	–	Loses manner information.
	CC4D	Replace the top of the English muffin	Flip over the top muffin and replace it on top	Weakens multi-action structure.

Appendix

Table 25: Qualitative examples of what the action–object abstraction preserves and what it weakens. It remains informative for predicate and tool changes, but loses detail when the mistake depends on duration, scalar quantity, manner, or multi-action structure. CC4D denotes CaptainCook4D.

Type	Dataset	Example	Why it belongs to this type
Insertion	EgoOops	Extra grasp actions such as Grasp black light or Grasp a long blue thin cuboid block	Extra steps outside the successful reference sequence.
	EgoPER	Pour water from the second bowl to the first one	Corrective action inserted after an earlier mistake.
	EgoOops	Mix all the liquid and shine a black light on it	Composite action inserted outside the reference sequence.
Deletion	EgoPER	Put bananas annotated as Skipped this step	Reference step missing from the observed sequence.
Substitution	CC4D	Place the egg from the cup over the lettuce vs. ... under the lettuce	Same step slot, wrong relation.
	CC4D	Invert the mug to release the cake onto a plate vs. ... onto a bowl	Same step slot, wrong destination.

Appendix

Table 26: Qualitative examples of the unified edit-based taxonomy across benchmarks. The same four types remain interpretable despite different dataset-specific error labels and annotation styles. CC4D denotes CaptainCook4D.

			Nearest		Greedy		Viterbi
Dataset	VLM	Repr.	Seq	Any-F1	Seq	Any-F1	Seq	Any-F1
EgoOops	Qwen	Raw	0.501	0.337	0.501	0.339	0.575	0.321
	Qwen	GPT-SRL	0.499	0.335	0.501	0.351	0.587	0.356
	Gemini	Raw	0.560	0.305	0.572	0.339	0.619	0.349
	Gemini	GPT-SRL	0.560	0.315	0.569	0.328	0.613	0.353
EgoPER	Qwen	Raw	0.803	0.547	0.803	0.578	0.803	0.578

Appendix

Table 27: Decoder ablations under the matched configurations. Nearest-candidate matching uses no HMM. Greedy decoding retains the trained local decoder. Viterbi adds start and transition probabilities and global sequence search.

Video Reasoning

Video reasoning studies how AI systems use video evidence to infer events, actions, temporal relations, causes, and future states.

See all 211 papers

The Unreasonable Effectiveness of VLMs for Zero-shot Procedural Mistake Detection
Jun 19, 2026Serdar Ozsoy, Lars Doorenbos, Federico Spurio +2Temporal Action Segmentation Vision-Language Models
University of Bonn · Lamarr Institute for Machine Learning and Artificial Intelligence · Toyota Motor Europe
How to Correctly Make Mistakes: A Framework for Constructing and Benchmarking Mistake Aware Egocentric Procedural Videos
Apr 16, 2026Olga Loginova, Frank KellerBenchmark Construction Egocentric Video Understanding
University of Trento · Italy · University of Edinburgh +1
Post-Training VLMs for Video Mistake Detection
Aug 28, 2026Federico Spurio, Olga Zatsarynna, Lars Doorenbos +3VLM Post-Training Vision-Language Models
University of Bonn Bonn, Germany · Lamarr Institute for Machine Learning and Artificial Intelligence Bonn, Germany · Toyota Motor Europe Belgium

Jun 19, 2026J/K move · Enter open · S save

The Unreasonable Effectiveness of VLMs for Zero-shot Procedural Mistake Detection

Serdar Ozsoy, Lars Doorenbos, Federico Spurio, Gianpiero Francesca+1

University of Bonn · Lamarr Institute for Machine Learning and Artificial Intelligence · Toyota Motor Europe

Temporal Action Segmentation Vision-Language Models Egocentric Video Understanding

Open paper AskPDF arXiv

Procedural mistake detection is important for quality control and user assistance across many disciplines. Recent work in this field has achieved significant gains by using the reasoning capabilities of Video-Language Models (VLMs) as components within multi-stage pipelines, which consist of separate modules for supervised temporal action segmentation, error detection, and explainability. Consequently, they remain dependent on tailored training datasets and require task-specific training, limiting their wider applicability. To remedy this, we introduce zero-shot procedural mistake detection and propose a unified Zero-shot Procedural Mistake detection (ZeProM) framework that jointly solves procedural mistake detection and temporal action segmentation with a single pre-trained VLM. By evaluating our framework on two canonical mistake detection benchmarks, EgoPER and CaptainCook4D, we find that ZeProM can perform these tasks successfully, while approaching, or even outperforming, the performance of fully supervised methods. For instance, we achieve a 4.4 point improvement in EDA and a 2.0 point improvement in [email protected] on average over all five EgoPER tasks compared to the strongest supervised methods. Overall, our results show the potential of unified methods for procedural mistake detection, and we hope this will steer the field away from highly complex pipelines and toward more generally applicable solutions.

Abstract

Video Reasoning

Explore similar work

The Unreasonable Effectiveness of VLMs for Zero-shot Procedural Mistake Detection

How to Correctly Make Mistakes: A Framework for Constructing and Benchmarking Mistake Aware Egocentric Procedural Videos

Post-Training VLMs for Video Mistake Detection