Joint Latent Reasoning and Observation for Agentic Video Understanding
Authors: Junzhang Liu, Hao Yuan, Qi Zhang, Yiming Wu, Long Chen, Zhecan Wang
Organizations: Hong Kong University of Science and Technology · Shanghai Artificial Intelligence Laboratory · Fudan University · The University of Hong Kong
Effective video agents must learn not only how to answer, but also where to look next. Transferring this capability to smaller models is challenging because each observation serves two purposes: supporting an answer and guiding the search for further evidence. Textual descriptions may capture what is currently visible while omitting cues needed for subsequent acquisition. We introduce Latent Agentic Reasoning for Video Analysis (LARVA), a framework that distills multi-round video agents through jointly learned latent reasoning and visual observations. Rather than communicating through captions, an observer passes compact continuous representations to a reasoner that selects the next acquisition action or produces the final answer. Crucially, action and answer supervision propagate through this interface, training the observer to communicate information useful for downstream decisions. Observation reconstruction and reasoning-distribution alignment provide auxiliary supervision from textual teacher trajectories. Across five video-understanding benchmarks, LARVA achieves strong performance with 4B and 7B backbones. On VideoHolmes, Video-MME, and LVBench, the 7B model outperforms a textual CoT student trained on the same teacher trajectories by 5.1, 2.5, and 3.7 percentage points, respectively. These results highlight the value of jointly learning how agents represent evidence and act on it when distilling video-understanding capabilities.
Figures & tables
Figure 1: When asked to compare the contestants’ attire, the CoT agent receives identical captions (“Gymnast in red, white, and blue”), confuses their identities, and skips ahead to the fourth contestant. LARVA samples around the transitions and correctly identifies no difference in attire.
Figure 2: Overview of LARVA. (a) During multi-round inference, the latent agent selects video acquisitions and integrates compact latent observations to guide subsequent tool calls and produce an answer. (b) Joint training combines observation reconstruction Lobs , CoT distribution alignment Lalign , and action supervision Lact . The alignment loss uses a frozen CoT agent to match text- and latent-conditioned reasoning distributions, supervising both the latent agent and observer.
Method
VideoHolmes ( Cheng et al., 2025 )
Video-MME ( Fu et al., 2025 )
LVBench ( Wang et al., 2025 )
LongVideoBench ( Wu et al., 2024 )
MLVU ( Zhou et al., 2024 )
Teacher models
DeepSeek-V4.1-Flash ( DeepSeek-AI et al., 2026 )
60.4
80.1
63.3
71.9
77.2
Qwen3.8-27B ( Qwen Team, 2026 )
50.3
72.5
56.8
67.4
75.2
Qwen2.5-VL-7B ( Bai et al., 2025b )
EVA ( Zhang et al., 2026 )
37.2
60.2
43.3
55.0
68.3
VideoMind ( Liu et al., 2026b )
41.0
58.2
40.8
56.3
64.4
Table 1: Main results. Accuracy (%) on five video-understanding benchmarks. Teacher models are listed for reference. CoT Policy and Direct Policy are controlled students distilled from the same teacher trajectories as LARVA.
Interface
Accuracy (%)
Reasoning
Observation
Joint
VideoHolmes ( Cheng et al., 2025 )
Video-MME ( Fu et al., 2025 )
LVBench ( Wang et al., 2025 )
Textual
Textual
✗
39.2
59.9
41.3
Latent
Textual
✗
42.7
61.9
43.6
Textual
Latent
✗
38.6
59.3
40.7
Textual
Latent
✓
43.5
60.8
42.0
Latent
Latent
✗
37.4
55.2
39.7
Table 2: Contributions of latent reasoning and observation. Accuracy (%) of students with textual or latent reasoning and observation interfaces, with (✓) or without (✗) joint training of the reasoner and observer, on Qwen2.5-VL-7B ( Bai et al., 2025b ) . All variants use the same training examples and tools.
Tokens
Method
Turns
Reasoning
Observation
Frames
Latency (s)
EVA ( Zhang et al., 2026 )
2.6
361.2
13984.4
71.4
11.8
VideoMind ( Liu et al., 2026b )
3.0
16.6
19253.2
511.5
21.4
LongVideoAgent † ( Liu et al., 2026a )
2.5
3196.9
168.8
21.7
30.8
CoT Policy
4.8
1752.5
1457.2
112.5
48.3
Direct Policy
4.4
0.0
1506.5
110.9
39.0
Table 3: Inference cost per question, averaged over VideoHolmes, Video-MME, and LVBench. Reasoning tokens are generated by the reasoner, and observation tokens are received by it. Latency is measured on NVIDIA H200 GPUs. All local models use 7B backbones; EVA and the students use Qwen2.5-VL-7B ( Bai et al., 2025b ) . † Relies on closed-source APIs, so latency includes API response time; excluded from Table 1 .
Trajectory
Answerer
VideoHolmes
Video-MME
LVBench
CoT
CoT
39.2
59.9
41.3
Latent
42.8
60.0
41.5
27B Visual
49.9
63.7
49.6
Latent
CoT
43.6
61.1
41.5
Latent
44.3
62.4
45.0
27B Visual
51.6
67.9
52.2
Table 4: Evidence quality via answerer swaps. Trajectories are generated by the CoT or latent student; at the terminal decision, the acquired evidence is passed to the listed answerer.
Model
Evidence-gain rate (%)
Uncertain (count)
Text reasoning + latent observation
11
7
Latent reasoning + textual observation
15
5
Full LARVA
35
8
Table 5: Human assessment of incremental evidence. Evidence-gain rate is adjudicated Yes divided by 100 questions. Uncertain cases remain in the denominator and are reported as counts. Both models receive the same 64-frame Overview and a 16-frame additional acquisition budget.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Left: Accuracy vs. latent observation token count. Right: Accuracy vs. latent reasoning token count. The performance increases with both hyperparameters and plateaus at around 8 tokens.
Similarity to teacher trajectory
Benchmark
Method
Accuracy
Tool Sequence
Temporal IoU
Optional Tools
VideoHolmes
Direct
34.6
54.6
36.2
26.1
CoT
39.2
57.3
38.2
29.5
LARVA
44.3
61.9
39.4
30.7
Video-MME
Direct
53.4
60.6
30.4
29.0
CoT
59.9
63.6
30.5
35.9
Appendix
Table 6: Similarity to teacher trajectories. Answer accuracy (%) and agreement between student and teacher acquisition trajectories on Qwen2.5-VL-7B. For each benchmark, the best result is in bold .
Video understanding develops methods for recognizing actions, tracking events, grounding evidence in time, and reasoning across visual and audio streams.
Institute of Computing Technology, Chinese Academy of Sciences, China · University of Chinese Academy of Sciences, China · College of Computer Science and Electronic Engineering, Hunan University, China +1