Multimodal large language models (MLLMs) can generate detailed video captions, yet they may overemphasize minor details while overlooking essential context and key events. We argue that not all visual evidence deserves equal attention, as the importance of visual information depends on the content of each video. Motivated by this observation, we propose Reviso, an agentic video captioning framework that learns what to observe and how to gather relevant visual evidence. From a low-frame-rate global preview, Reviso constructs a hierarchical Context--Focus--Points (CFP) observation plan that identifies the video context, prioritizes key aspects, and defines concrete observation targets and corresponding tools. Guided by this plan, Reviso performs goal-directed active perception by selectively revisiting temporal segments and combining tools for fine-grained inspection. It interprets the acquired evidence, updates its plan as needed, and integrates global and local information to generate the final caption. To learn this observation strategy, we develop a verification-guided pipeline that constructs high-quality training trajectories through factual verification and teacher-guided refinement. We further train Reviso using Group Relative Policy Optimization (GRPO), incorporating a key-fact reward for caption quality, a Plan Alignment Reward (PAR) for identifying important observation targets, and an Unproductive Tool-Call Penalty (UTCP) that adjusts observation-cost penalties based on caption quality. Experiments across four video captioning benchmarks show that Reviso outperforms the evaluated open-source baselines, while its generated captions better support downstream video question answering. These findings highlight the effectiveness and potential of priority-guided active observation for video captioning.
Figures & tables
Figure 1: Video captioning through active observation. Captioning priorities vary across video types, including action order in tutorials, characters’ expressions and actions in movie scenes, and product names and attributes in advertisements. Reviso identifies what matters in each video and selects tools to gather evidence for its caption.
Figure 2: Overview of Reviso. (a) A hierarchical Context–Focus–Points plan guides tool-assisted observation and caption generation. (b) Verified teacher trajectories provide supervised initialization. (c) GRPO rewards factual caption quality (KFR) and plan alignment (PAR), while penalizing unproductive tool use (UTCP).
Figure 3: Observation priorities across diverse video content. We illustrate representative visual aspects that may require prioritized attention in different video scenarios. Reviso identifies important information based on the content and context of each video, guiding targeted observation and evidence gathering.
Figure 4: Overview of the trajectory construction pipeline. There are six steps in the pipeline. We generate observation trajectories, verify them against ground-truth captions, and use a teacher model to refine unsuccessful attempts, retaining verified trajectories for supervised fine-tuning.
Model
DREAM-1K (%)
MCTS-VCB (%)
CaReBench (%)
VDC (Detailed)
Prec. ↑
Rec. ↑
F1 ↑
Prec. ↑
Rec. ↑
F1 ↑
Action F1 ↑
Object F1 ↑
TP score ↑
TP acc. ↑
Proprietary Models
GPT-6 Astra [ 21 ]
24.7
61.8
35.2
30.7
64.5
41.6
44.3
34.5
3.00
0.576
Gemini 3.8 Flash [ 13 ]
34.8
50.2
41.1
47.8
50.7
49.2
39.7
34.3
2.46
0.486
Open-Source Models
ARC-Qwen-Video-7B [ 11 ]
31.2
31.4
31.3
58.4
42.7
49.3
24.9
30.2
2.90
0.554
Table 1: Results on four video captioning benchmarks. Benchmark metrics retain their original units. The best score among open-source models and Reviso in each column is highlighted in green . Gray closed-source rows are for reference and excluded from ranking.
Variant
Reward
DREAM-1K
CaReBench
KFR
PAR
UTCP
F1 ↑
Action F1 ↑
Object F1 ↑
Base
–
–
–
35.1
25.2
27.2
SFT only
–
–
–
37.6
30.7
30.1
w/o Tool use
–
–
–
35.3
24.8
27.4
GRPO reward variants
w/o KFR
–
✓
✓
38.2
32.9
32.1
Table 2: Ablation of tool use and training. All scores are percentages. Base denotes Qwen3.5-9B. Base and full-model results follow Table 1 . Checkmarks indicate active reward components.
Model
Params.
LVBench
MMVU
MCQ
Open
Overall
Gemini 3.1 Pro [ 12 ]
-
41.2
77.6
46.7
66.1
ARC-Qwen-Video [ 11 ]
7B
48.1
77.6
48.8
66.8
InternVL3.5 [ 35 ]
38B
41.8
69.2
43.7
59.6
Tarsier2 [ 50 ]
7B
45.8
76.2
54.2
67.9
Qwen3.6 [ 25 ]
35B
45.3
77.1
47.2
65.9
Table 3: Caption-to-QA results on two video understanding benchmarks. We report accuracy (%). The best result in each column is shown in bold.
Figure 5: (a) Effect of tool-assisted observation on CaReBench. Bars show the number of videos in each duration group (left axis). The line shows the object F1 difference between tool-assisted and direct captioning with the same Reviso model (right axis, percentage points). (b) Effect of frame sampling rate on DREAM-1K. Direct captioning is evaluated at 2, 4, 6, and 10 FPS. (c) Effect of captioning strategy on DREAM-1K. Direct captioning at 2 FPS is compared with the Codex and Claude Code workflows for five VLMs. The dashed lines in (b) and (c) mark Reviso’s F1 score of 39.5.
Figure 6: Qualitative comparison of Qwen3.5-9B and Reviso. Reviso identifies key information to inspect and uses OCR to track the game state, producing a more accurate caption. Red, green, and blue indicate incorrect facts, key facts, and fine-grained details, respectively.
Despite significant progress, current models struggle to generate comprehensive video captions, constrained primarily by coarse-grained supervision, which often fails to capture spatial-temporal dynamics and the underlying logical intent. To address this issue, we introduce MVRCap, a novel framework that reformulates video captioning as a reinforcement learning task steered by structured verification. Central to our approach is a Fine-Grained Verifiable Reward mechanism, serving as dense supervision to enforce rigorous visual-semantic alignment. Specifically, we construct this reward system anchored by three complementary pillars: a Hierarchical Concept Coverage Reward utilizing our newly constructed MVRCap-QA dataset to ensure multi-dimensional completeness and semantic depth, a Factuality Verification Reward to enhance factual consistency, and a Conciseness Constraint to maximize semantic density. Furthermore, we curate the high-quality MVRCap-SFT dataset, providing structured annotations across six key dimensions, to serve as a robust foundational initialization for the RL policy in this training pipeline. Extensive experiments demonstrate that MVRCap achieves state-of-the-art performance on authoritative benchmarks including VDC, CAPability, and VC4VG.
Video captioning requires fine-grained spatio-temporal understanding of videos, including spatial perception of where objects are located and temporal perception of when events occur. Existing MLLMs usually generate captions directly from video inputs without exposing the perceptual evidence behind descriptions. As a result, mistakes in spatiotemporal perception are only observed in the final caption, making it difficult to identify the underlying perceptual errors directly. To address these issues, we present PercepCap, a perception-aware video captioning framework that makes perceptual evidence explicit before producing the final caption. Specifically, PercepCap follows a perceive-describe generation chain, where the model first produces a spatiotemporal perception trace comprising object trajectories and temporal events, and then generates the final caption conditioned on the perceived evidence. To support this, we design a two-stage training strategy. Perceive-then-Describe Supervised Fine-tuning adapts the model from caption-only generation to the proposed perceive-describe chain, while Perception-Grounded Reinforcement Learning optimizes perception trace and caption quality with joint rewards over perception chain and the final caption. To support our two-stage training, we introduce Caption-Anchored Perception Data Construction. This pipeline builds the SFT and RL training data by first generating a caption-only description, extracting the objects and events it mentions, and grounding them back in the video with boxes and timestamps. This yields caption-aligned perception data that provides solid training ground truth, ensuring that the explicit perception trace and final caption refer to the same objects and events. Across direct caption and caption-to-QA evaluation, PercepCap consistently improves upon the Qwen3-VL baseline and demonstrates leading caption quality.
Yifan Xu, Zihao Wang, Zhixiao Wang +6
State Key Laboratory for Novel Software Technology, Nanjing Univerisity · Kling Team, Kuaishou Technology · Shanghai AI Laboratory
Existing video captioning methods struggle to balance visual fidelity and redundancy: holistic captions are compact but lose fine-grained evidence, whereas segment-wise captions improve coverage but introduce heavy redundancy. We propose CodecCap, a codec-inspired framework for high-fidelity dense video captioning. Analogous to video codecs, CodecCap represents videos using keyframe and residual captions. Keyframe captions exhaustively encode stable visual context, while residual captions capture temporally only localized actions, motions and changes. This effectively preserves fine-grained visual evidence while reducing redundant descriptions. To quantify the fidelity of captions, we introduce VidCapQA, a caption-then-QA benchmark with 1,000 questions across 14 capability dimensions. Results on VidCapQA show that captions directly generated by strong VLMs still miss many visual details, highlighting caption representation as a critical bottleneck. Experiments show that CodecCap significantly surpasses direct captioning with the same underlying VLMs, suggesting keyframe-residual captioning a way for high-fidelity video-language supervision. We further use CodecCap to construct CodecVDC-100K, a large-scale dense captioning dataset with anchor, residual, scene-level, and video-level supervision.
Zihan Lin, Songhe Deng, Shuwei He +6
ERNIE Team, Baidu · College of Artificial Intelligence, Inner Mongolia University