Chain-of-thought reasoning has substantially advanced video understanding in multimodal large language models (MLLMs). However, language-based reasoning traces do not provide a reliable mechanism for models to revisit relevant visual evidence, leading to visual forgetting and impairing their ability to reason over video content. Existing approaches maintain visual grounding during reasoning in two main ways: they either refer to evidence using textual coordinates, which specify symbolic locations rather than the visual content itself, or reinsert cropped regions or frames into the multimodal context, preserving visual details at the cost of substantial context expansion. To address these limitations, we introduce VideoLVR, which constructs latent visual evidence from local regions at relevant video moments, thereby providing visual support for intermediate reasoning steps. To supervise latent visual evidence generation, we introduce a visual token prediction objective that formulates visual token reconstruction as prediction over a vocabulary derived from the input video, thereby unifying visual evidence reconstruction and text generation within the same token prediction framework. This shared prediction framework enables MLLMs to reason over interleaved text tokens and latent visual states while integrating evidence from different moments. We further extend VideoLVR to reinforcement learning by treating sampled visual tokens as actions alongside text tokens and jointly optimizing the policy using final-answer rewards. By efficiently revisiting task-relevant visual information during reasoning, VideoLVR improves video understanding without substantially expanding the multimodal context. Experiments on multiple video benchmarks demonstrate its effectiveness, especially on tasks requiring fine-grained visual evidence.
Figures & tables
Figure 1: Overview of VideoLVR. (a) A teacher MLLM synthesizes evidence-grounded reasoning trajectories through rejection sampling and trajectory polishing. (b) Visual tokens from regions supporting the same reasoning step are grouped and ordered by timestamp to form latent targets. VideoLVR jointly trains text generation with next-token prediction and latent visual evidence generation with visual token prediction over the input-video visual vocabulary. (c) At inference with the supervised model, each decoder hidden state is projected into a latent visual state and fed back as the input for the next decoding step, enabling autoregressive reasoning over latent visual evidence without additional visual observations.
Method
Video-MMMU
TempComp.
MMVU
LV-R.
VideoMathQA
Avg
Percep.
Comp.
Adapt.
Avg
Qwen2.5-VL-7B-Instruct
60.00
44.00
24.00
42.67
61.97
53.40
71.20
22.62
50.37
Video-R1-7B
45.33
37.67
29.00
37.33
60.83
45.90
59.30
20.00
44.67
VideoRFT-7B
49.33
39.33
25.67
38.11
59.76
49.10
64.00
23.10
46.81
Open-o3-Video-7B
60.33
41.33
35.33
45.67
72.20
51.00
63.60
24.41
51.38
Mull-Tokens-7B
51.33
39.67
29.00
40.00
57.41
29.20
61.40
16.90
40.98
Table 1: Main results on five video reasoning benchmarks. VideoLVR-8B-RL denotes VideoLVR-8B after training with TVPO. Percep., Comp., and Adapt. denote the Perception, Comprehension, and Adaptation dimensions of Video-MMMU. TempComp. and LV-R. abbreviate TempCompass and LongVideo-Reason, respectively. Bold indicates the best reproduced result within each backbone family, while underlining denotes the second best. † denotes results reported under the original evaluation settings, which are excluded from ranking.
Loss
V-MMMU
TC
MMVU
LV-R.
VMQA
Avg
MSE
67.28
76.42
59.50
76.80
33.57
62.71
VTP
67.44
77.39
61.70
78.20
35.00
63.95
Table 2: Comparison of MSE and visual token prediction (VTP) for latent supervision. V-MMMU, TC, LV-R., and VMQA abbreviate Video-MMMU, TempCompass, LongVideo-Reason, and VideoMathQA, respectively.
Figure 2: Effect of the inference-time latent-step budget on five video benchmarks. Enlarged markers denote the best results.
Data
V-MMMU
TC
MMVU
LV-R.
VMQA
Avg
Standard RS
64.89
75.15
63.60
77.60
31.43
62.53
Ours
67.44
77.39
61.70
78.20
35.00
63.95
Table 3: Comparison of trajectories from standard rejection sampling and our evidence-grounded synthesis pipeline.
Setting
V-MMMU
TC
MMVU
LV-R.
VMQA
Avg
First-Region
66.89
76.76
60.70
76.90
34.52
63.15
Per-Region
66.56
77.05
60.10
77.30
34.52
63.11
Grouped
67.44
77.39
61.70
78.20
35.00
63.95
Table 4: Comparison of strategies for organizing localized visual regions into latent blocks.
Method
V-MMMU
TC
MMVU
LV-R.
VMQA
Avg
GRPO
67.11
77.33
59.40
77.70
35.00
63.31
TVPO
68.22
77.60
60.50
79.90
35.71
64.39
Table 5: Comparison of GRPO and TVPO, both initialized from the same supervised VideoLVR-8B model.
Figure 3: Comparison of training reward curves for GRPO and TVPO, shown as trailing 20-step moving averages.
τ
V-MMMU
TC
MMVU
LV-R.
VMQA
Avg
0.01
62.66
73.75
57.70
75.97
29.84
59.98
0.03
65.04
75.64
59.27
77.95
30.24
61.63
0.07
67.44
77.39
61.70
78.20
35.00
63.95
0.10
66.00
76.59
59.53
77.10
34.52
62.75
0.15
66.35
77.09
60.30
76.37
32.78
62.58
Table 6: Ablation of the visual token prediction temperature τ on the five main benchmarks.
Setting
V-MMMU
TC
MMVU
LV-R.
VMQA
Avg
Zero
65.11
76.81
61.00
76.30
34.52
62.75
Noise
65.00
76.86
60.70
76.10
31.67
62.07
Ours
67.44
77.39
61.70
78.20
35.00
63.95
Table 7: Effect of interventions on latent states during inference.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Statistic
Value
Training Trajectories
8,782
Avg. Groups
2.93
Avg. Regions
3.66
Multi-Region Groups
19.8%
Multi-Moment Groups
14.0%
Appendix
Table 8: Statistics of the supervised training set. Both averages are computed over all evidence-grounded reasoning trajectories; the last two rows report percentages over all evidence groups.
Figure 4: Localized visual regions in a representative training example. Red boxes mark the regions annotated in the evidence groups. Panels (a)–(b) form the first group, while panels (c)–(d) form the second.
Recent advancements in chain-of-thought (CoT) reasoning have shown promise in enhancing video understanding and reasoning capabilities of multimodal large language models (MLLMs). However, existing CoT-based MLLMs require labor-intensive CoT annotations and incur substantial training and inference overhead. While visual latent reasoning has emerged as a more efficient alternative, existing methods primarily focus on image tasks and heavily rely on additional supervision signals for visual latent generation (e.g., CoT traces, auxiliary images, or fine-grained annotations), limiting their scalability and transferability to video tasks. To bridge this gap, we introduce VideoLatent, a novel MLLM equipped with a latent injection module tailored for video understanding and reasoning. Specifically, VideoLatent learns to perform visual latent reasoning using a new latent self-forcing training paradigm, which comprises latent alignment and latent diversity objectives, and relies solely on standard video-question-answer triplets. Extensive experiments across 14 benchmarks demonstrate that our model consistently outperforms existing standard and latent MLLMs on general video understanding and complex video reasoning. Compared with Video-R1, our VideoLatent achieves superior computational efficiency, reducing training/inference overhead by ∼6×/∼68×. Moreover, experiments demonstrate that our method has strong generalizability to different MLLM backbones and different model scales.
Video reasoning requires Large Multimodal Models (LMMs) to remain grounded in dense evidence, yet existing systems largely adopt "read-once, generate-many" paradigm, in which visual grounding weakens during generation. This phenomenon has been widely observed and is known as Visual Anchoring Decay. To fill this gap, we introduce Latent Video Cache (Latent-VC), a recurrent latent visual cache inserted into the decoder to preserve compact visual memories throughout reasoning. The cache is trained with supervised contrastive cache alignment and vision-grounded GRPO with a latent grounding reward, while maintaining strict train-inference alignment through native decoder hidden states. Built on Qwen3.5-9B, Latent-VC consistently outperforms strong CoT and SFT+GRPO baselines across six video benchmarks, with especially clear gains on grounding-intensive and long-video tasks. In addition, it also achieves higher accuracy with substantially shorter responses, suggesting that latent visual caching improves video reasoning by preserving visual evidence rather than relying on longer textual chains.
Yongheng Zhang, Zhipeng Xu, Hao Wu +4
1Tencent Youtu Lab · 2Tsinghua University · University of Illinois at Chicago
Video reasoning has advanced with large multimodal models (LMMs), yet their inference is often a single pass that returns an answer without verifying whether the reasoning is evidence-aligned. We introduce Reinforce to Learn, Elect to Reason (RLER), a dual paradigm that decouples learning to produce evidence from obtaining a reliable answer. In RLER-Training, we optimize the policy with group-relative reinforcement learning (RL) and 3 novel task-driven rewards: Frame-sensitive reward grounds reasoning on explicit key frames, Think-transparency reward shapes readable and parsable reasoning traces, and Anti-repetition reward boosts information density. These signals teach the model to emit structured, machine-checkable evidence and potentiate reasoning capabilities. In RLER-Inference, we apply a train-free orchestrator that generates a small set of diverse candidates, parses their answers and cited frames, scores them by evidence consistency, confidence, transparency, and non-redundancy, and then performs a robust evidence-weighted election. This closes the loop between producing and using evidence, improving reliability and interpretability without enlarging the model. We comprehensively evaluate RLER against various open-source and RL-based LMMs on 8 representative benchmarks. RLER achieves state of the art across all benchmarks and delivers an average improvement of 6.3% over base models, while using on average 3.1 candidates per question, indicating a favorable balance between compute and quality. The results support a simple thesis: making evidence explicit during learning and electing by evidence during inference is a robust path to trustworthy video reasoning.
Songyuan Yang, Weijiang Yu, Jilin Ma +5
National University of Defense Technology · Sun Yat-sen University