What's the Catch? Evaluating Temporal Consistency in Vision-Language Models

Marek Hradil; Danae Sánchez Villegas

cs.CLAug 24, 2026

What's the Catch? Evaluating Temporal Consistency in Vision-Language Models

Authors: Marek Hradil, Danae Sánchez Villegas

Organizations: University of Copenhagen · The National Center for AI in Society (CAISA)

Abstract

Vision-language models (VLMs) achieve strong performance on video and image-sequence benchmarks, yet it remains unclear whether they capture temporal structure. To study this question, we formulate temporal grounding as an anomaly detection problem, providing a simple and controlled evaluation that directly tests sensitivity to temporal consistency. We introduce TimeCatch, where temporal anomalies are created by swapping consecutive frames and frame-level anomalies by replacing a frame with Gaussian noise. Models are evaluated on anomaly detection and localization tasks across four synthetic and real-world datasets, alongside a human study. Our evaluation reveals a substantial gap between frame-level and temporal anomaly detection. While VLMs consistently detect frame-level anomalies and often localize them accurately, under our main evaluation setting they generally perform near chance on temporal anomaly detection and show limited localization performance. Humans, in contrast, achieve near-ceiling performance on both tasks. Additional analyses across model scales, prompting strategies, sequence lengths, and visual similarity show that performance can improve under some conditions, while substantial gaps in temporal anomaly detection and localization remain. Together, these findings reveal a gap between frame-level and temporal anomaly detection. TimeCatch provides a controlled benchmark for evaluating temporal consistency in vision-language models.

Figures & tables

Figure 1: Overview of the proposed anomaly detection tasks. Given an image sequence, models detect or localize either temporal anomalies (swapped adjacent frames) or frame-level anomalies (corrupted frames). Under our main evaluation setting, VLMs perform well on frame-level tasks but generally remain close to chance on temporal anomaly tasks, whereas humans achieve high performance on both.

Figure 2: Example image sequences from the four benchmark datasets. The benchmark spans both synthetic (CLEVRER, CRAFT) and real-world (DriveLM, MTL-AQA) domains.

Figure 3: Sequence length distribution across the benchmark datasets. Sequences of 4 to 8 frames are used in the main experiments. Longer CRAFT sequences (9–16 frames) are reserved for evaluating the effect of sequence length on temporal anomaly detection.

Dataset	Domain	Sequences	Avg. Length
CLEVRER	Synthetic	4,997	7.5
CRAFT	Synthetic	858	6.9
DriveLM	Real-world	696	5.9
MTL-AQA	Real-world	338	4.5
CRAFT-Long	Synthetic	1,101	10.6

Table 1: Statistics of datasets included in TimeCatch.

	Temporal Detect				Temporal Localize				Frame Detect				Frame Localize
	CL	CR	DR	MT	CL	CR	DR	MT	CL	CR	DR	MT	CL	CR	DR	MT
Random	50.0	50.0	50.0	50.0	4.3	5.5	8.7	13.2	50.0	50.0	50.0	50.0	13.5	14.9	18.1	22.4
Qwen2.5-VL-7B	50.5	47.6	50.7	45.3	13.1	14.3	14.8	29.9	66.9	67.9	60.3	71.6	42.0	60.5	45.8	73.1
Qwen3-VL-8B	51.9	53.3	51.9	57.4	25.6	24.5	24.7	43.2	93.3	98.1	99.6	94.1	98.9	96.3	92.7	99.4
Gemma-4-E4B	49.2	48.0	53.9	56.2	15.0	14.7	22.3	29.9	77.2	80.5	82.0	76.6	33.6	46.5	61.9	88.2
InternVL3-8B	50.1	49.1	53.2	47.9	14.3	17.7	17.8	21.6	58.1	77.7	68.4	76.6	30.9	52.9	33.9	65.7

Table 2: Detection and localization accuracy (%) across datasets. Models reliably detect and localize frame-level anomalies but struggle on temporal anomaly tasks. Results with 95% Wilson score confidence intervals are reported in Table 5 Appendix D . CL: CLEVRER, CR: CRAFT, DR: DriveLM, MT: MTL-AQA.

Figure 4: Example from the temporal anomaly localization task on MTL-AQA. The highlighted frames indicate the swapped pair. Although the temporal inconsistency is readily identified by the human participant, all evaluated VLMs fail to localize the anomaly correctly.

Figure 5: Temporal anomaly detection and localization accuracy (%) on the human study subset. Humans consistently outperform all evaluated VLMs across datasets, highlighting a substantial gap between human and VLM performance on temporal reasoning. We also perform statistical significance tests for this experiment, reported in Table 7 Appendix D .

Figure 6: Temporal anomaly detection accuracy by LPIPS distance for Qwen3-VL-8B. Although larger perceptual differences between swapped frames slightly improve performance, accuracy remains close to chance across both synthetic and real-world datasets, indicating that visual similarity is not the primary limitation.

Figure 7: Temporal anomaly localization confusion matrix for Qwen3-VL-8B, averaged across all datasets. Rather than correctly localizing the anomaly, the model exhibits a strong bias toward predicting position 2, independent of the true anomaly location.

Figure 8: Effect of model scale on anomaly detection and localization accuracy for the Qwen3-VL family. Scaling improves frame-level anomaly performance and also yields gains in temporal anomaly detection, particularly from 4B to 32B, while temporal localization remains comparatively low. This suggests that model capacity contributes to temporal anomaly detection, but does not close the performance gap observed on temporal tasks.

Figure 9: Effect of prompting on temporal anomaly detection accuracy. Values show the change in accuracy (percentage points) relative to the base prompt (Prompt A) for Qwen3-VL-8B. Grey circles correspond to alternative prompt formulations, while orange diamonds and green squares denote removing scene descriptions and enabling reasoning, respectively.

Figure 10: Effect of sequence length on anomaly detection and localization accuracy across sequence lengths (4–16 frames). Frame-level detection remains high, whereas frame-level localization degrades as the number of frames increases. Temporal anomaly detection and localization show little dependence on sequence length. The final bucket ( $n=16$ ) contains only five samples and should be interpreted with caution (see Figure 3 ).

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Content	Video	Sequences	Avg. frames	Mean LPIPS
CRAFT	Abstract 2D objects, collisions	✓	$1{,}983\to 1{,}959$	$16.0\to 9.0$	$0.053\to 0.093$
CLEVRER	Abstract 3D objects, collisions	✓	$5{,}000\to 4{,}997$	$8.0\to 7.5$	$0.123\to 0.131$
DriveLM	Driving, egocentric	-	$696$	$5.9$	$0.428$
MTL-AQA	Olympic competitive diving	✓	$353\to 338$	$8.4\to 4.5$	$0.492\to 0.584$

Appendix

Table 3: Dataset statistics before and after the sampling and filtering pipeline. Arrows indicate the changes in the number of sequences, average sequence length, and mean LPIPS after preprocessing. The 1,959 filtered CRAFT sequences are partitioned into 858 main-evaluation sequences (4–8 frames) and 1,101 CRAFT-long sequences (9–16 frames).

Figure 11: LPIPS distributions of consecutive frame pairs before and after filtering. Filtering removes nearly identical frame pairs while preserving the overall distribution of perceptual differences. Synthetic datasets (CRAFT and CLEVRER) contain more visually similar consecutive frames than the real-world datasets due to their largely static backgrounds.

Figure 12: User interface for the temporal anomaly detection task. Participants viewed an image sequence and indicated whether the frames were presented in the correct temporal order. Scene descriptions were provided below the sequence to match the model evaluation setting.

Figure 13: Close-up of the image viewer. Viewer was beneficial for annotators to notice finer details, – it allows for quicker swapping between the images to emulate a more video-like experience.

	CL	CR	DR	MT
QW3-2B (0-shot)	49.6	50.5	50.0	44.1
QW3-2B (FT)	96.5	63.6	52.7	68.1
$\Delta$	+46.9	+13.1	+2.7	+24.0
QW3-8B (0-shot)	51.9	53.3	51.9	57.4

Appendix

Table 4: Temporal anomaly detection accuracy (%) of Qwen3-VL-2B (QW3-2B) before and after fine-tuning on the CLEVRER training split. Fine-tuning consistently improves performance on the held-out CLEVRER split and transfers to the unseen CRAFT, DriveLM, and MTL-AQA datasets. Zero-shot Qwen3-VL-8B (QW3-8B) is shown for reference.

Temporal Detect
	Random	Qwen2.5-VL-7B	Qwen3-VL-8B	Gemma-4-E4B	InternVL3-8B	InternVL3.5-8B
CLEVRER	50.0	50.5 [49.1, 51.9]	51.9 [50.5, 53.3]	49.2 [47.9, 50.6]	50.1 [48.7, 51.5]	49.6 [48.2, 51.0]
CRAFT	50.0	47.6 [44.2, 50.9]	53.3 [49.9, 56.6]	48.0 [44.7, 51.4]	49.1 [45.7, 52.4]	49.9 [46.5, 53.2]
DriveLM	50.0	50.7 [47.0, 54.4]	51.9 [48.2, 55.6]	53.9 [50.2, 57.6]	53.2 [49.4, 56.8]	49.9 [46.2, 53.6]
MTL-AQA	50.0	45.3 [40.0, 50.6]	57.4 [52.1, 62.6]	56.2 [50.9, 61.4]	47.9 [42.7, 53.2]	50.6 [45.3, 55.9]
Temporal Localize

Appendix

Table 5: Detection and localization accuracy (%) across datasets (same runs as Table 2 ), with 95% Wilson score confidence intervals. For temporal detect, most intervals include the 50% chance level.

	Temporal Detect				Temporal Localize				Frame Detect				Frame Localize
	CL	CR	DR	MT	CL	CR	DR	MT	CL	CR	DR	MT	CL	CR	DR	MT
QW2.5	-0.5	+4.4	-1.7	+7.7	+1.8	+3.1	+3.6	+4.7	+2.6	+2.2	+2.9	-4.7	+33.9	+19.3	+23.3	+13.0
QW3	+7.0	+2.3	-1.0	+17.2	+4.7	+2.1	-0.1	+0.6	+5.3	+0.6	-0.9	-4.1	+1.0	+2.6	+1.1	+0.0
GEM4	+0.7	+2.2	-5.0	-0.9	-0.2	+0.2	+2.3	-2.1	+7.5	+1.7	+5.6	+1.5	+2.5	-2.7	+5.7	+0.6
IVL3	+0.1	+2.6	-0.6	+8.9	+1.3	-2.9	+1.7	+2.4	+16.0	+8.7	+11.8	+0.9	+15.9	+12.7	+28.3	+20.1
IVL3.5	+0.5	+0.6	-2.0	+10.9	+2.5	+2.0	+10.9	+3.3	+9.8	+9.0	+11.9	+7.4	+39.6	+34.6	+55.0	+66.0

Appendix

Table 6: Change in accuracy (%) after removing scene descriptions. Positive values ( green ) indicate improved performance without scene descriptions, while negative values ( red ) indicate degradation. Removing scene descriptions has the largest effect on frame-level localization, whereas temporal anomaly performance changes only modestly.

Model	$n$	Human	Model	Gap	$p$	$n$	Human	Model	Gap	$p$
	Temporal Detect					Temporal Localize
Qwen2.5-VL-7B	167	83.8	51.5	+32.3	$1.1\times 10^{-8}$ ***	143	86.7	22.4	+64.3	$6.4\times 10^{-23}$ ***
Qwen3-VL-8B	167	83.8	58.1	+25.7	$6.1\times 10^{-6}$ ***	143	86.7	29.4	+57.3	$8.8\times 10^{-19}$ ***
Gemma-4-E4B	167	83.8	49.7	+34.1	$1.9\times 10^{-8}$ ***	143	86.7	26.6	+60.1	$6.6\times 10^{-23}$ ***
InternVL3-8B	167	83.8	53.3	+30.5	$9.1\times 10^{-9}$ ***	143	86.7	14.7	+72.0	$6.7\times 10^{-27}$ ***
InternVL3.5-8B	167	83.8	51.5	+32.3	$1.8\times 10^{-10}$ ***	143	86.7	25.2	+61.5	$8.8\times 10^{-22}$ ***

Appendix

Table 7: Human vs. model accuracy (%), McNemar exact test (Bonferroni corrected), averaged across all four datasets per model. Each model is compared to the human annotators on the identical samples it was evaluated on.

Figure 14: Temporal anomaly detection accuracy by LPIPS distance for all evaluated models. Accuracy remains close to chance across most perceptual similarity bins, indicating that larger visual differences between swapped frames do not consistently improve performance. The apparent increase in the final bin (

0.8

–

0.9

) is likely due to the small number of samples (

n=13

Figure 15: Temporal localization confusion matrices for all evaluated models, averaged across datasets and row-normalized. Two distinct error patterns emerge. Qwen3-VL and InternVL3.5 exhibit a strong bias toward predicting specific swap positions, whereas Gemma-4, Qwen2.5-VL, and InternVL3 produce more uniformly distributed predictions, consistent with near-random guessing.

Panel of Figure 16 — Figure 16: Qualitative examples illustrating success and failure modes of both humans and VLMs. Red borders mark the swapped frame pair. From top to bottom: (1) Both correct – swap present (CRAFT, detect): the pair at positions 3–4 is out of order, human and all five models correctly identify the swap. (2) Both correct – no swap (MTL-AQA, detect): the sequence is unmodified, human and all models correctly report no anomaly. (3) LLMs correct, human wrong (Drive-LM, detect): the sequence is unmodified, but the human incorrectly reports a swap, likely misled by rainy conditions and a camera turn between frames; all five models answer correctly. (4) Both wrong (MTL-AQA, localize): the swap at positions 3–4 (two nearly identical underwater frames) is missed by both the human annotator and all five models, who instead point to earlier positions in the sequence.

Explore similar work

Jun 15, 2026

Chronological Blindness: Benchmarking Temporal Reasoning in Vision-Language Models with CHRONOSIGHT

Human perception of visual scenes is inherently temporal. We instinctively recognise whether a fruit is ripening or rotting, whether construction is progressing or being demolished, and approximately how much time separates two photographs of the same subject. Whether large vision-language models (VLMs) share this competence remains an open and practically important question. We introduce CHRONOSIGHT, a rigorously controlled benchmark evaluating five dimensions of visual temporal reasoning: CHRONORANK (chronological ordering of image sequences), CHRONOLOCATE (ordinal stage localisation from a single image), CHRONODELTA (estimation of time elapsed between two images on a logarithmic scale), CHRONOREVERSE (detection of temporally reversed sequences), and CHRONOODD (identification of a temporal outlier within a set). The benchmark comprises 1{,}000 items across eight process families (biological growth, food transformation, physical weathering, construction, environmental change, human ageing, astronomical phenomena, and urban dynamics) spanning timescales from minutes to millennia. We evaluate eight open-source VLMs (500 M to 19 B parameters) under two prompting regimes and collect human performance baselines. Human performance averages 0.89 across tasks; the best open model (Qwen2.5-VL-7B) reaches 0.40 under direct prompting, a gap we term chronological blindness. Lightweight LoRA fine-tuning on 151 examples raises CHRONODELTA accuracy from near-zero to 0.43, transferring zero-shot to related tasks (CHRONOODD: 0.37; CHRONOREVERSE: 0.64)suggesting the bottleneck is partly instruction following rather than visual perception. Benchmark, code, and predictions will be released upon acceptance.

Parthaw Goswami, Jaynto Goswami Deep

Department of Computer Science, University of Missouri, Columbia, 65201, MO, USA · SAP, Prague, Czech Republic

VLM Evaluation Temporal Reasoning VLM Reasoning

Ask PDF

May 28, 2026

Tiny but Trusted: Efficient Vision-Language Reasoning for Time-Series Anomaly Detection

Recent advances in Vision-Language Models (VLMs) have achieved impressive performance across many tasks, yet prior studies report unsatisfactory performance when applying large language or multimodal models to finding abnormal patterns in sequential data. Public anomaly detection benchmarks typically provide interval annotations but not natural-language rationales, making it difficult to fine-tune VLMs to produce grounded, interpretable decisions. To address this gap, we construct VisAnomBench, a curated benchmark built from public time-series datasets and augmented with high-quality anomaly explanations selected from multiple large VLMs using fine-grained, task-specific rewards. Through fine-tuning on this benchmark, we develop VisAnomReasoner, a parameter-efficient VLM for time-series anomaly detection. Experimental results on VisAnomBench show that VisAnomReasoner achieves more accurate anomaly localization and consistently outperforms all baselines, with improvements of at least 21.23 and 23.87 percentage points in precision and F1, respectively. Additional experiments on the TSB-AD-U benchmark demonstrate strong cross-benchmark generalization, with VisAnomReasoner improving precision and F1 by 9.57 and 13.39 percentage points, respectively.

Xiaona Zhou, Muntasir Wahed, Tianjiao Yu +2

University of Illinois Urbana-Champaign · Sandia National Laboratories

Interpretable Anomaly Detection Anomaly Localization VLM Reasoning Time-Series Anomaly Detection

Ask PDF

May 20, 2026

TempGlitch: Evaluating Vision-Language Models for Temporal Glitch Detection in Gameplay Videos

Vision-language models (VLMs) are increasingly being explored for video game quality assurance, especially gameplay glitch detection. Most existing evaluations, however, treat glitches as static visual anomalies, asking models to detect failures from a single frame. We argue that this framing misses a key distinction: some glitches are spatial and visible in an isolated frame, whereas others are temporal and become evident only through changes across ordered frames. A preliminary study confirms this gap, showing that temporal glitches are substantially harder for VLMs to detect than spatial ones. To enable systematic evaluation of this underexplored setting, we introduce TempGlitch, a controlled gameplay video benchmark for temporal glitch detection. TempGlitch covers five temporal glitch types with balanced per-category samples, together with paired glitch-free videos that enable reliable binary evaluation. We evaluate 12 proprietary and open-weight VLMs across multiple frame-sampling settings. Our results show that current VLMs remain near chance on TempGlitch, often collapsing into either overly conservative behavior that misses most glitches or overly sensitive behavior that flags clean videos as glitchy. Moreover, denser frame sampling and larger model size do not reliably resolve these failures. TempGlitch provides a focused testbed for temporal reasoning, robust gameplay understanding, and automated glitch detection with VLMs. Code and data are available at the project website.

Yakun Yu, Ashley Wiens, Adrián Barahona-Ríos +4

University of Alberta · Sony Interactive Entertainment

VLM Evaluation Automated Software Testing Video Anomaly Detection Temporal Video Understanding

Ask PDF