Chain-of-thought reasoning has substantially advanced video understanding in multimodal large language models (MLLMs). However, language-based reasoning traces do not provide a reliable mechanism for models to revisit relevant visual evidence, leading to visual forgetting and impairing their ability to reason over video content. Existing approaches maintain visual grounding during reasoning in two main ways: they either refer to evidence using textual coordinates, which specify symbolic locations rather than the visual content itself, or reinsert cropped regions or frames into the multimodal context, preserving visual details at the cost of substantial context expansion. To address these limitations, we introduce VideoLVR, which constructs latent visual evidence from local regions at relevant video moments, thereby providing visual support for intermediate reasoning steps. To supervise latent visual evidence generation, we introduce a visual token prediction objective that formulates visual token reconstruction as prediction over a vocabulary derived from the input video, thereby unifying visual evidence reconstruction and text generation within the same token prediction framework. This shared prediction framework enables MLLMs to reason over interleaved text tokens and latent visual states while integrating evidence from different moments. We further extend VideoLVR to reinforcement learning by treating sampled visual tokens as actions alongside text tokens and jointly optimizing the policy using final-answer rewards. By efficiently revisiting task-relevant visual information during reasoning, VideoLVR improves video understanding without substantially expanding the multimodal context. Experiments on multiple video benchmarks demonstrate its effectiveness, especially on tasks requiring fine-grained visual evidence.
Figures & tables
Figure 1: Overview of VideoLVR. (a) A teacher MLLM synthesizes evidence-grounded reasoning trajectories through rejection sampling and trajectory polishing. (b) Visual tokens from regions supporting the same reasoning step are grouped and ordered by timestamp to form latent targets. VideoLVR jointly trains text generation with next-token prediction and latent visual evidence generation with visual token prediction over the input-video visual vocabulary. (c) At inference with the supervised model, each decoder hidden state is projected into a latent visual state and fed back as the input for the next decoding step, enabling autoregressive reasoning over latent visual evidence without additional visual observations.
Method
Video-MMMU
TempComp.
MMVU
LV-R.
VideoMathQA
Avg
Percep.
Comp.
Adapt.
Avg
Qwen2.5-VL-7B-Instruct
60.00
44.00
24.00
42.67
61.97
53.40
71.20
22.62
50.37
Video-R1-7B
45.33
37.67
29.00
37.33
60.83
45.90
59.30
20.00
44.67
VideoRFT-7B
49.33
39.33
25.67
38.11
59.76
49.10
64.00
23.10
46.81
Open-o3-Video-7B
60.33
41.33
35.33
45.67
72.20
51.00
63.60
24.41
51.38
Mull-Tokens-7B
51.33
39.67
29.00
40.00
57.41
29.20
61.40
16.90
40.98
Table 1: Main results on five video reasoning benchmarks. VideoLVR-8B-RL denotes VideoLVR-8B after training with TVPO. Percep., Comp., and Adapt. denote the Perception, Comprehension, and Adaptation dimensions of Video-MMMU. TempComp. and LV-R. abbreviate TempCompass and LongVideo-Reason, respectively. Bold indicates the best reproduced result within each backbone family, while underlining denotes the second best. † denotes results reported under the original evaluation settings, which are excluded from ranking.
Loss
V-MMMU
TC
MMVU
LV-R.
VMQA
Avg
MSE
67.28
76.42
59.50
76.80
33.57
62.71
VTP
67.44
77.39
61.70
78.20
35.00
63.95
Table 2: Comparison of MSE and visual token prediction (VTP) for latent supervision. V-MMMU, TC, LV-R., and VMQA abbreviate Video-MMMU, TempCompass, LongVideo-Reason, and VideoMathQA, respectively.
Figure 2: Effect of the inference-time latent-step budget on five video benchmarks. Enlarged markers denote the best results.
Data
V-MMMU
TC
MMVU
LV-R.
VMQA
Avg
Standard RS
64.89
75.15
63.60
77.60
31.43
62.53
Ours
67.44
77.39
61.70
78.20
35.00
63.95
Table 3: Comparison of trajectories from standard rejection sampling and our evidence-grounded synthesis pipeline.
Setting
V-MMMU
TC
MMVU
LV-R.
VMQA
Avg
First-Region
66.89
76.76
60.70
76.90
34.52
63.15
Per-Region
66.56
77.05
60.10
77.30
34.52
63.11
Grouped
67.44
77.39
61.70
78.20
35.00
63.95
Table 4: Comparison of strategies for organizing localized visual regions into latent blocks.
Method
V-MMMU
TC
MMVU
LV-R.
VMQA
Avg
GRPO
67.11
77.33
59.40
77.70
35.00
63.31
TVPO
68.22
77.60
60.50
79.90
35.71
64.39
Table 5: Comparison of GRPO and TVPO, both initialized from the same supervised VideoLVR-8B model.
Figure 3: Comparison of training reward curves for GRPO and TVPO, shown as trailing 20-step moving averages.
τ
V-MMMU
TC
MMVU
LV-R.
VMQA
Avg
0.01
62.66
73.75
57.70
75.97
29.84
59.98
0.03
65.04
75.64
59.27
77.95
30.24
61.63
0.07
67.44
77.39
61.70
78.20
35.00
63.95
0.10
66.00
76.59
59.53
77.10
34.52
62.75
0.15
66.35
77.09
60.30
76.37
32.78
62.58
Table 6: Ablation of the visual token prediction temperature τ on the five main benchmarks.
Setting
V-MMMU
TC
MMVU
LV-R.
VMQA
Avg
Zero
65.11
76.81
61.00
76.30
34.52
62.75
Noise
65.00
76.86
60.70
76.10
31.67
62.07
Ours
67.44
77.39
61.70
78.20
35.00
63.95
Table 7: Effect of interventions on latent states during inference.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Statistic
Value
Training Trajectories
8,782
Avg. Groups
2.93
Avg. Regions
3.66
Multi-Region Groups
19.8%
Multi-Moment Groups
14.0%
Appendix
Table 8: Statistics of the supervised training set. Both averages are computed over all evidence-grounded reasoning trajectories; the last two rows report percentages over all evidence groups.
Figure 4: Localized visual regions in a representative training example. Red boxes mark the regions annotated in the evidence groups. Panels (a)–(b) form the first group, while panels (c)–(d) form the second.