Video Large Language Models (Video-LLMs) have achieved strong performance on many video understanding tasks, yet remain vulnerable to temporal hallucination in time-sensitive reasoning scenarios such as scene transition understanding and event ordering. Existing solutions either require costly retraining or introduce inefficient multi-pass inference and external supervision at test time. In this work, we show that temporal evidence is already encoded in Video-LLM hidden representations but is not fully exploited during standard inference. Motivated by this observation, we propose Single-Pass Orthogonal Decomposition (SPOD), a training-free framework that recovers suppressed temporal evidence through analytical hidden-state manipulation in a single forward pass. SPOD consists of two components: Frame Distinctiveness Analysis (FDA), which identifies salient frames from middle-layer hidden-state geometry, and Temporal Delta Subspace (TDS), which constructs orthogonal temporal and prior subspaces from weighted inter-frame hidden-state differences. By decomposing final hidden states into temporal, prior, and residual components and rescaling them independently, SPOD enhances dynamic information, suppresses static appearance bias, and preserves complementary information. Experiments on VidHalluc, EventHallusion, and VideoMMMU show that SPOD consistently outperforms strong inference-time baselines across temporal reasoning tasks and model architectures while retaining near-vanilla inference efficiency. These results suggest that improving temporal reasoning in Video-LLMs may depend not only on additional supervision, but also on recovering temporal evidence already present in their internal representations. Our code is available at https://github.com/VioAgnes/SPOD.
Figures & tables
Figure 1. Illustration of temporal hallucination in Video-LLMs and our proposed SPOD.
Figure 2. Frame-level temporal distinctiveness across video samples. Layer-aggregated scores ϕi separate high- and low-saliency frames, with a mean high-to-low ratio of 2.81× .
Figure 3. Comparison of frame semantic coherence and temporal-boundary diversity between FDA-selected and uniformly sampled random frames.
Figure 4. Cumulative singular-value energy and effective-rank statistics of FDA-weighted temporal-difference matrices used for temporal subspace construction.
Figure 5. SPOD pipeline. (1) FDA identifies salient frames from middle-layer hidden-state geometry. (2) TDS constructs temporal and prior subspaces from weighted inter-frame differences and text hidden states. (3) Final hidden states are decomposed into temporal, prior, and residual components and rescaled to recover temporal evidence in a single forward pass.
Model
BQA
MCQ
STH
TSH
Overall
VideoLLaMA2
Vanilla
75.77
83.35
56.55
58.17
68.46
TCD
76.77
83.65
55.19
56.67
68.07
DINO-HEAL
75.79
83.35
56.32
57.67
68.28
TA-AE
79.09
84.07
65.14
61.67
72.49
CGD
76.03
82.76
59.14
59.02
69.24
Table 1. Results on VidHalluc benchmark. Bold indicates the best result per metric within each model group.
Variant
BQA
MCQ
STH
TSH
Vanilla
81.47
81.24
67.60
84.00
Ours
84.73
88.45
68.64
85.67
Table 2. Results on Qwen2.5-VL-72B.
Model
Entire
Mix
Misleading
Overall
VideoLLaMA2
Vanilla
38.60
62.18
46.08
48.95
TCD
42.98
75.65
54.90
57.84
DINO-HEAL
38.60
62.69
46.08
48.96
TA-AE
43.86
70.47
57.70
57.34
CGD
42.11
63.21
47.06
50.79
Table 3. Results on EventHallusion benchmark.
Model
Perc.
Comp.
Adapt.
Overall
VideoLLaMA2
Vanilla
60.34
46.00
27.67
44.67
TCD
60.00
43.00
36.33
46.44
TA-AE
61.67
43.00
35.00
46.56
Ours
68.67
48.34
28.67
48.56
Qwen2.5-VL
Table 4. Results on VideoMMMU benchmark.
Category
Variant
STH
TSH
Vanilla
60.55
59.17
FDA Ablation
FDA only
62.90
61.50
FDA + Attn
63.50
61.90
Random frames
61.50
60.30
TDS Ablation
w/o FDA weights
62.50
61.10
TDS + FDA weights
63.80
62.20
Table 5. Ablation study and sensitivity analysis.
Figure 6. Qualitative example on EventHallusion with Qwen2.5-VL-7B, where bold green text denotes correct descriptions with accurate temporal details, and bold red text denotes hallucinated descriptions.
Figure 7. Joint comparison of inference throughput and BLEU scores across different methods.
Video Large Language Models (VideoLLMs) receive frames in sequential order and interpret how visual content evolves along the temporal axis, yet temporal reasoning remains a persistent weakness across architectures. Reversing the frame order of a video, a transformation that should invert temporal answers, often leaves the final prediction unchanged. We investigate where this failure originates by defining the temporal divergence vector τl, the layer-wise representational difference induced by reversing temporal order. Tracking its magnitude across layers reveals a consistent temporal divergence profile where the divergence peaks at intermediate layers and progressively diminishes toward the output. We confirm this peak is specific to temporal reasoning and functionally critical for predictions, establishing that VideoLLMs acquire temporal information at intermediate layers but fail to maintain it to the output. This progressive fading motivates our method, Temporal Activation Injection (TAI), which extracts τl at the peak of the profile for each input and reinjects it into subsequent layers following the measured decay. TAI requires no training and consistently improves temporal reasoning across three VideoLLMs and four benchmarks with negligible impact on non-temporal tasks. Code is available at https://github.com/Youngwoo-git/Before-It-Fades.
Youngwoo Shin, Yusung Ro, Minseo Kim +1
Korea Advanced Institute of Science and Technology (KAIST)
Video Large Language Models (Video-LLMs) remain prone to spatiotemporal hallucinations, often generating visually unsupported details or incorrect temporal relations. Existing mitigation methods typically treat hallucination as a uniform decoding failure, applying globally shared correction rules. We instead observe that decoder layers contribute differently to visual grounding and later linguistic composition, indicating that intervention must be layer-aware. Based on this insight, we propose STEAR, a layer-aware spatiotemporal evidence intervention framework. STEAR identifies high-risk decoding steps and selects token-conditioned visual evidence from grounding-sensitive middle layers. It uses this shared evidence for two coupled purposes: restoring missing local grounding in middle layers, and constructing temporally perturbed patch-level counterfactuals to falsify inconsistent reasoning during late-layer decoding. Consequently, STEAR mitigates both spatial and temporal hallucinations within an efficient single-encode inference framework. Experiments across representative Video-LLM backbones and challenging benchmarks demonstrate that STEAR consistently reduces hallucinations while improving faithfulness, temporal consistency, and robustness. Our results confirm that reliable video decoding relies on intervening on precise evidence at the right layer, rather than enforcing a global penalty. The code is provided in the Supplementary Material.
Linfeng Fan, Yuan Tian, Ziwei Li +1
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China · King Abdullah University of Science and Technology, Thuwal, Saudi Arabia
The Arrow-of-Time (AoT) task, determining whether a video plays forward or backward by recognizing temporal irreversibility, is one humans solve with near-perfect accuracy, yet frontier Video Large Language Models (Video-LLMs) perform only modestly above chance. This gap raises a key question: do visual backbones fail to encode temporal information, or does information bottleneck lie elsewhere in the Video-LLM architecture? We address this question by isolating the vision encoder from the Video-LLM and tracing temporal information across the encoder, projector, and LLM. We find that video-centric encoders with explicit temporal modeling encode strong temporal signals, whereas frame-centric encoders do not. However, when video-centric representations are passed through a standard Video-LLM architecture, performance often collapses, revealing a bottleneck of temporal information flow. We identify projector design as a key factor: Q-Former disrupts temporal information, while a time-preserved MLP projection substantially improves the LLM's access to such information. Our layer-wise analysis further shows temporal representation dynamics across encoder layers. Guided by these findings, we build a Video-LLM with temporal-aware video-centric encoder, time-preserved projector, and AoT supervision, surpassing human performance on AoTPPB with 98.1% accuracy, and improving broader temporal reasoning tasks by up to 6.0 points on VITATECS-Direction and 1.3 points on TVBench. Our results show that temporal reasoning in Video-LLMs requires both effective temporal encoding and reliable transfer of this information to the LLM.
Peitao Han, Fei Cheng, Lis K. Pereira +2
The University of Osaka · Center for Information and Neural Networks · National Institute of Information and Communications Technology +2