Multimodal large language models increasingly use visual chain-of-thought (Visual CoT) to reason about spatial, temporal, and embodied environments. By generating intermediate reasoning images, Visual CoT provides an intuitive mechanism for visual foresight but introduces substantial inference overhead, which is particularly problematic for proactive video reasoning. We ask whether models can learn to think visually during training while reasoning directly at inference. We introduce Internalized Visual Thinking (IVT), a post-training framework that jointly optimizes textual prediction and next-embedding prediction over unlabeled videos. Given a partially observed video, IVT predicts latent representations of future frames together with the target textual answer, encouraging the model to capture motion, object transitions, interactions, and latent intent. At inference, IVT generates the answer directly without synthesizing or re-encoding future frames. We conduct controlled studies across target representations, decoder designs, prediction horizons, data mixtures, training curricula, and predictive objectives. IVT improves over direct-answer fine-tuning on all six evaluation settings while retaining the same inference pathway. Compared with explicit Visual CoT, IVT achieves comparable or better performance and reduces average end-to-end latency by more than 5x. Together, our findings suggest that explicit pixel-space generation at inference time, as used in visual chain-of-thought, may not be necessary for effective proactive video reasoning. Predictive world modeling can be internalized during training to produce multimodal reasoners that are both more accurate and substantially more efficient.
Figures & tables
Figure 1: Three post-training paradigms for proactive video reasoning. Answer-Only SFT maps the observed input directly to the predicted textual state. Visual CoT first generates an explicit future visual state in pixel space and then conditions on it to produce the answer, so it pays for the image generation cost at inference. In contrast, our Internalized Visual Thinking (IVT) predicts the future visual state as a latent embedding target only during training ; at inference it generates the answer directly, without synthesizing an intermediate image, and therefore executes the same computation graph as Answer-Only SFT.
Latency ( ↓ )
Answer Quality ( ↑ )
Task
Method
Mean
P95
ROUGE-L
BERTScore
METEOR
CIDEr
Early Event Pred.
Answer-Only SFT
1.07
1.79
34.4
41.3
24.3
1.43
Visual CoT (generated frame)
6.56 (6.2 × )
8.72 (4.9 × )
36.4 ( + 5.9%)
43.2 ( + 4.5%)
29.8 ( + 22.4%)
1.69 ( + 17.9%)
Visual CoT (oracle frame)
—
—
41.9 ( + 21.6%)
48.3 ( + 17.0%)
34.2 ( + 40.6%)
2.05 ( + 43.4%)
Next Event Pred.
Answer-Only SFT
1.32
2.05
47.2
50.2
35.3
2.14
Visual CoT (generated frame)
6.55 (5.0 × )
7.12 (3.5 × )
46.8 ( − 0.8%)
49.9 ( − 0.7%)
36.9 ( + 4.3%)
2.13 ( − 0.6%)
Table 1: Accuracy–efficiency trade-off of Visual CoT. For each task, we report performance averaged across the validation subsets of Ego-Exo4D, Ego4D, and EPIC-KITCHENS-100. Conditioning on a generated future frame improves early-event prediction but provides limited benefit for next-event prediction, while substantially increasing mean inference latency. Replacing the generated frame with the ground-truth future frame improves both tasks, revealing a considerably higher upper bound when accurate future visual context is available. This gap suggests that the effectiveness of Visual CoT depends strongly on the fidelity of the generated future state, as intermediate images that fail to preserve plausible scene dynamics or task-relevant visual information may provide weak or misleading conditioning signals.
Figure 2: Comparison of different target representations. We report ROUGE-L, BERTScore, METEOR, and CIDEr, averaged across the three benchmarks for early-event and next-event prediction. The dashed line denotes the Answer-Only SFT baseline. Predicting a Flux-VAE latent consistently achieves the strongest performance across all metrics and both tasks, whereas semantic targets provide smaller and less consistent gains.
Figure 3: Effect of the understanding-to-prediction data-mixture ratio on Ego-Exo4D early-event prediction. We report performance throughout training for three sampling ratios. Understanding-heavy mixtures ( 3:1 and 5:1 ) improve more rapidly at early checkpoints but subsequently plateau or decline. In contrast, the balanced 1:1 mixture improves steadily and achieves the strongest result on all four metrics at 20k steps.
Figure 4: Comparison of predictive objectives on Ego-Exo4D early-event prediction. We compare rectified-flow matching and direct feature regression for VAE and DINOv2 future-state targets. Direct regression substantially outperforms flow matching for DINOv2 and is also competitive with flow matching for VAE across most checkpoints and metrics.
Method
ROUGE-L
BERTScore
METEOR
CIDEr
Answer-Only SFT
30.6
32.5
22.6
1.60
Two-Stage Training
28.6 ( − 6.5%)
30.7 ( − 5.6%)
21.3 ( − 5.5%)
1.54 ( − 3.5%)
Joint Training
34.5 ( + 12.6%)
36.5 ( + 12.4%)
27.4 ( + 21.4%)
2.02 ( + 26.3%)
Table 2: Joint optimization is more effective than sequential training. We report the results on Ego-Exo4D early-event prediction. Relative changes are computed against the Answer-Only SFT baseline. Two-stage training underperforms Answer-Only SFT on every metric, whereas joint training achieves the strongest result across all metrics. Best value in each column is shown in bold .
Figure 5: Interaction between decoder architecture and prediction horizon on Ego-Exo4D early-event prediction. We report the experimental results for each prediction horizon H , defined as the number of predicted future frames. The dashed line denotes the Answer-Only SFT baseline. Both architectures outperform it across all horizons. The shared Dense decoder performs best at short horizons, whereas the MoE architecture is comparatively less sensitive to horizon and reaches its strongest results around H=3 .
Early Event Pred. (top-1)
Next Event Pred. (top-5)
Method
ROUGE-L
BERTScore
METEOR
CIDEr
ROUGE-L
BERTScore
METEOR
CIDEr
External models fine-tuned on the same data
LLaVA-NeXT-Video-7B
24.3
5.5
17.4
1.11
44.2
42.3
33.7
2.40
VideoLLaMA3-7B
14.6
17.4
7.7
0.31
30.2
26.3
19.6
1.30
Qwen2.5-VL-7B
34.9
36.0
26.6
1.75
48.0
45.9
37.9
2.77
Qwen3-VL-8B
37.0
38.1
28.5
1.90
52.1
49.5
42.0
3.09
Table 3: Results on the full Ego-Exo4D validation set. We evaluate early-event prediction with top-1 decoding and next-event prediction with top-5 decoding using ROUGE-L, BERTScore, METEOR, and CIDEr. Answer-Only SFT uses the same matched backbone and training setup as IVT but removes future-state prediction, retaining only the textual prediction objective. External models are included as reference points, while bold and underlined values denote the best and second-best results within the controlled BAGEL-7B-based block.
Early Event Pred. (top-1)
Next Event Pred. (top-5)
Method
ROUGE-L
BERTScore
METEOR
CIDEr
ROUGE-L
BERTScore
METEOR
CIDEr
External models fine-tuned on the same data
LLaVA-NeXT-Video-7B
36.8
30.3
27.2
1.49
46.3
46.3
33.1
1.98
VideoLLaMA3-7B
15.6
19.9
9.8
0.44
24.3
22.2
13.3
0.63
Qwen2.5-VL-7B
35.8
38.6
26.0
1.39
49.2
47.9
35.6
2.12
Qwen3-VL-8B
39.5
41.9
29.1
1.59
52.9
51.8
39.7
2.37
Table 5: Results on the full EPIC-KITCHENS-100 validation set. We evaluate early-event prediction with top-1 decoding and next-event prediction with top-5 decoding using ROUGE-L, BERTScore, METEOR, and CIDEr. Answer-Only SFT uses the same matched backbone and training setup as IVT but removes future-state prediction, retaining only the textual prediction objective. External models are included as reference points, while bold and underlined values denote the best and second-best results within the controlled BAGEL-7B-based block.
Table 6: Training hyperparameters. Default configuration used for the target-representation ablations and main IVT experiments. Architecture and curriculum variants modify only the corresponding design choices described in the main paper.
Table 11
Answer-Only SFT
Visual CoT
IVT (Ours)
Task
Benchmark
Mean
P95
Mean
P95
Mean
P95
Early Event Prediction
Ego-Exo4D
1.42
2.45
7.08
10.54
1.52
2.34
Ego4D
0.83
1.27
6.38
7.28
0.81
1.12
EPIC-KITCHENS-100
0.95
1.64
6.24
8.34
0.87
1.30
Next Event Prediction
Ego-Exo4D
1.37
2.40
7.70
8.62
1.25
1.91
Ego4D
0.92
1.49
3.72
4.01
0.82
0.98
Appendix
Table 9: End-to-end inference latency. Per-sample latency is measured on a single B200 GPU with batch size 1 and bf16 precision over 200 validation examples per benchmark. Mean and P95 latency are reported in seconds. IVT retains an inference profile comparable to Answer-Only SFT, whereas Visual CoT incurs substantially higher latency due to intermediate future-frame generation and visual re-encoding.
Video reasoning aims to understand complex temporal events and causal relationships within videos. Recently, Chain-of-Thought (CoT) has been introduced to this field to enhance reasoning accuracy. However, existing CoT-based video reasoning methods primarily rely on text-only information for logical deduction, overlooking critical visual information during the inference process. Inspired by the human cognitive mechanism of reviewing visual segments during inference, we propose VTI-CoT, a Visual-Textual Interleaved CoT framework. VTI-CoT integrates textual reasoning steps with corresponding visual frames. Given the scarcity of visual-textual interleaved CoT in existing datasets, we develop an automated annotation pipeline to construct high-quality multimodal CoT data. Further, reasoning over long-form videos entails increasingly long CoT token sequences, which severely hinders training convergence and efficiency. To address this, we employ Optical Character Recognition (OCR)-based compression techniques to compress CoT supervision signals into a single canvas. Experimental results demonstrate that VTI-CoT achieves state-of-the-art performance among models of the same parameter scale while significantly improving training efficiency.
Shufan Zhang, Ziyue Lin, Bairun Wang +4
Beijing University of Posts and Telecommunications · University of Hong Kong · Beijing Shanwei Zhixing Technology Co., Ltd. +2
Video event prediction (VEP) requires models to infer unobserved future states from partial video evidence. Existing video MLLMs usually verbalize intermediate future reasoning in text space: once visual evidence is verbalized, fine-grained motion, geometry, and interaction cues can be lost, leading to plausible but visually ungrounded hallucinations. We introduce Future-L1, an interleaved latent visual reasoning framework that lets an MLLM alternate between language tokens and continuous latent visual spans during autoregressive decoding. To train this capability, we construct Future-L1-50K by selecting examples where future visual hints help prediction and align latent states to future-frame embeddings, then further optimize sampled latent trajectories with LA-DAPO, a latent-aware RL objective with outcome-contrastive and temporal-diversity rewards. Future-L1 achieves new state-of-the-art results on both benchmarks: on FutureBench, it improves Qwen3-VL-8B from 61.0 to 85.4 and exceeds the previous best Video-CoE by 10.4 points; on TwiFF-Bench, it improves the average score from 2.44 to 3.04. These results suggest that future-oriented video reasoning benefits from preserving intermediate visual semantics in latent space rather than translating every reasoning step into text.
Tianxiang Jiang, Linquan Wu, Sheng Xia +5
University of Science and Technology of China · Shanghai AI Laboratory · City University of Hong Kong +4
Recent advancements in chain-of-thought (CoT) reasoning have shown promise in enhancing video understanding and reasoning capabilities of multimodal large language models (MLLMs). However, existing CoT-based MLLMs require labor-intensive CoT annotations and incur substantial training and inference overhead. While visual latent reasoning has emerged as a more efficient alternative, existing methods primarily focus on image tasks and heavily rely on additional supervision signals for visual latent generation (e.g., CoT traces, auxiliary images, or fine-grained annotations), limiting their scalability and transferability to video tasks. To bridge this gap, we introduce VideoLatent, a novel MLLM equipped with a latent injection module tailored for video understanding and reasoning. Specifically, VideoLatent learns to perform visual latent reasoning using a new latent self-forcing training paradigm, which comprises latent alignment and latent diversity objectives, and relies solely on standard video-question-answer triplets. Extensive experiments across 14 benchmarks demonstrate that our model consistently outperforms existing standard and latent MLLMs on general video understanding and complex video reasoning. Compared with Video-R1, our VideoLatent achieves superior computational efficiency, reducing training/inference overhead by ∼6×/∼68×. Moreover, experiments demonstrate that our method has strong generalizability to different MLLM backbones and different model scales.