Detailed video captioning requires comprehensive and faithful descriptions of objects, actions, and temporal events. Fixed-frame sampling, however, can miss brief motions, small objects, and other fine-grained evidence, motivating captioning agents that actively seek relevant moments and inspect local regions. Training such behavior is challenging because, unlike video question answering, generic captioning instructions do not specify what visual evidence should be acquired. We propose Perception-Question On-Policy Distillation (PQ-OPD), which uses training-only Perception Questions (PQs) to guide active video perception. PQs specify what should be inspected without revealing the corresponding answers, encouraging the teacher to explore the video more comprehensively and focus on relevant visual evidence. Along trajectories generated by the student, on-policy distillation transfers this guidance to the student's reasoning, tool calls, and captions; no PQs are required at inference. Across three detailed-captioning and two caption-to-QA benchmarks, PQ-OPD consistently improves the corresponding untrained agents across teacher-student scales and input-frame budgets. Under our evaluation protocol, PQ-OPD achieves the strongest primary-metric results among the compared open-source models on DREAM-1K, CaReBench, ShortVidBench, MotionBench, and VidCapBench-AE. Our ablations, together with analysis of temporal coverage and answer leakage, further indicates that the student internalizes the training-time guidance as a more effective video exploration policy.
Figures & tables
Figure 1: Training with privileged Perception Questions. The student generates its own tool-interaction trajectory, while a frozen teacher receives additional Perception Questions and provides token-level supervision. On-policy distillation transfers this guidance to the student, which receives no PQs at inference. The diagram illustrates the 9B teacher–student training.
Model
DREAM-1K
CaReBench
ShortVidBench
MotionBench
Event F1/R/P
Action F1/R/P
Object F1/R/P
Acc.
Acc.
Proprietary models (fixed 32 frames)
Gemini 3 Flash ( Google, 2025 )
37.6/48.5/30.7
42.4/39.6/45.7
40.3/37.8/43.2
78.6
57.6
Gemini 3.1 Pro ( Google, 2026 )
38.2/45.2/33.1
42.0/37.4/47.7
39.3/32.6/49.6
78.4
55.9
Open-source models (fixed 32 frames)
Qwen2.5-VL 7B ( Bai et al., 2025b )
27.2/27.9/26.6
30.9/23.6/45.0
34.1/27.6/44.6
70.0
50.7
Table 1: Evaluation results on DREAM-1K ( Wang et al., 2024a ) , CaReBench ( Xu et al., 2026a ) , ShortVidBench ( Ge et al., 2025 ) , and MotionBench ( Hong et al., 2025 ) . For PQ-OPD, the model size denotes the student; the teacher size is given in parentheses. Scores are reported as percentages. We highlight the best and second-best available open-source results.
Model
VidCapBench-AE
Overall
Video Aesthetics
Video Content
Video Motion
Physical Laws
Acc./Pre./Cov./Con.
Acc./Pre./Cov./Con.
Acc./Pre./Cov./Con.
Acc./Pre./Cov./Con.
Acc./Pre./Cov./Con.
Proprietary models (fixed 32 frames)
GPT-4o ( OpenAI, 2024 )
16.8/57.4/86.0/5.9
14.1/47.6/83.4/4.9
17.5/61.7/87.2/6.1
10.2/41.3/84.0/3.6
27.9/62.1/85.4/9.7
Gemini 3 Flash ( Google, 2025 )
19.3/58.1/90.5/7.3
17.1/47.3/87.5/6.4
19.6/62.8/91.7/7.4
12.9/49.5/89.8/4.9
33.9/58.4/91.7/12.8
Open-source models (fixed 32 frames)
Table 2: Evaluation results on VidCapBench-AE ( Chen et al., 2025 ) . Each group reports Accuracy/Precision/Coverage/Conciseness, as percentages. For PQ-OPD, the model size denotes the student and parentheses specify the teacher. The best and second-best available open-source results are highlighted for each metric.
#
Model Variant
Train PQ
Infer. PQ
CaReBench F1
DREAM-1K F1
Fixed observations (no seek/zoom tools)
#1
Baseline (Qwen3.5 9B)
–
–
37.6
30.3
#2
+ PQ at inference
–
✓
37.4
33.4
#5
+ PQ-OPD
✓
–
38.0
31.9
Interactive observations (seek/zoom tools)
#3
Base agent (Qwen3.5 9B)
–
–
39.8
31.3
Table 3: Ablation study of PQ guidance and training strategies on the 9B backbone. CaReBench F1 is the arithmetic mean of Action F1 and Object F1; DREAM-1K F1 is event F1. “Train PQ” indicates that PQs are used during training, while “Infer. PQ” indicates that video-specific PQs are provided at inference. Rows #5–#11 receive no PQs at inference; #9 uses video-agnostic generic questions ( Section B.2 ).
Teacher → Student
Training
CaReBench F1
DREAM-1K F1
27B → 9B
Vanilla OPD
40.1
31.2
27B → 9B
PQ-OPD
41.2
33.4
9B → 4B
Vanilla OPD
39.4
30.8
9B → 4B
PQ-OPD
40.2
32.6
Table 4: PQ guidance across teacher–student configurations. Both methods use seek/zoom tools; only PQ-OPD conditions the teacher on PQs, and all students are evaluated without PQs. Arrows denote teacher → student sizes. Scores are percentages; CaReBench F1 averages Action and Object F1, and DREAM-1K F1 is event F1. Bold denotes the better score within each teacher–student pair.
Figure 2: Performance across input-frame budgets. F1 under varying frame budgets, with a fixed evaluation cohort along each curve. CaReBench reports the arithmetic mean of Action F1 and Object F1; DREAM-1K reports event F1. Shaded regions show pointwise 95% bootstrap intervals.
Captioner ← Trajectory
CaReBench F1
DREAM-1K F1
Baseline agent ← Baseline agent (#3)
39.8
31.3
Baseline agent ← PQ-OPD
40.3
32.6
PQ-OPD ← Baseline agent
40.3
32.2
PQ-OPD ← PQ-OPD (#11)
40.6
33.3
Table 5: Trajectory-swapping analysis (9B → 9B, 1,000 videos per benchmark). For cross-model swaps, the captioner writes the final description directly from the frozen exploration trajectory collected by the other model, without issuing further tool calls.
Figure 3: Temporal exploration and caption length. Left: PQ interval hit rate by round two (single- seek tIoU ≥0.5 ). Middle: mean best single- seek tIoU across all rounds. Right: final-caption tokens. Temporal scores are averaged within each video and then across videos.
Figure 4: Effect of answer leakage on PQ-OPD. The rate of PQ answers leaked to the teacher during training varies from 0% to 100%; 0% is standard PQ-OPD. Both teacher and student are 9B models. (a) Captioning performance. (b) Mean tool calls per video. (c) Zero-observation answers: the student issues no tool calls and receives no visual input before producing a caption.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
Value
Data and rollout
Training videos
5,979 ( Section 4.1 )
Perception Questions
43,477
Maximum tool-image pixels
460,800
Unique prompts per refresh
64
Rollouts per prompt
4 (256 trajectories per refresh)
Appendix
Table 6: Training hyperparameters and configuration.
Teacher Questions
Question-Only Captioning
Trained Student
Action F1
Object F1
CaReBench F1
DREAM-1K F1
Standard PQs (PQ-OPD)
20.8
14.9
40.6
33.3
L1 (leakier wording)
37.0
36.4
40.3
32.3
L2 (more leaky wording)
43.6
46.5
39.9
32.5
L3 (leakiest wording)
52.7
52.9
37.3
29.4
Appendix
Table 7: Wording content leakage controls. All variants use the 9B → 9B recipe (372 steps) with identical timestamps. “Question-only captioning” reports Action / Object F1 of a text-only model generating captions solely from the questions on a 1,000-video training subset without visual input (for reference, an agent watching the video reaches 48.7 / 55.8). “Trained student” reports standard PQ-free evaluation on the full CaReBench and DREAM-1K benchmarks.
Benchmark
Model
Returned Frames
Tool Calls
Tool Rounds
Visual Tokens
Latency (Median)
CaReBench
Qwen3.5 9B Agent
34.5
11.1
3.7
11.4K
30 s
PQ-OPD
54.6
18.4
4.1
17.6K
41 s
DREAM-1K
Qwen3.5 9B Agent
31.3
9.6
3.3
12.7K
31 s
PQ-OPD
48.8
14.6
3.5
19.8K
42 s
Appendix
Table 8: Per-video inference statistics on CaReBench and DREAM-1K. Evaluated across all 1,000 videos of each benchmark under the standard 16-round, 128-frame protocol (9B → 9B for PQ-OPD). Visual tokens count the image tokens of all returned observations (each capped at 460,800 pixels); latency is the median end-to-end wall-clock time per video under our batched evaluation setting.
Figure 5: Qualitative comparison on DREAM-1K. Qwen3.5-9B receives 32 uniformly sampled frames, while Qwen3.5-9B Agent and PQ-OPD each obtain 35 visual observations through tool use. Circles denote the initial seek observations, triangles denote first-round single-frame inspections, and diamonds denote second-round inspections. The enlarged panels show cropped regions for easier visual comparison. The examples highlight differences in object identification, background details, and the fallen character’s recovery motion.
Figure 6: Player identity and action details on DREAM-1K. Qwen3.5-9B receives 32 uniformly sampled frames, while the two agentic models acquire observations through seek and zoom . Circles denote the initial seek observations, and triangles denote first-round zoom inspections. The comparison highlights jersey-name recognition and the description of the player’s fist-pump celebration and the subsequent cut to the team bench. Enlarged panels show the relevant visual evidence.
Multimodal large language models (MLLMs) can generate detailed video captions, yet they may overemphasize minor details while overlooking essential context and key events. We argue that not all visual evidence deserves equal attention, as the importance of visual information depends on the content of each video. Motivated by this observation, we propose Reviso, an agentic video captioning framework that learns what to observe and how to gather relevant visual evidence. From a low-frame-rate global preview, Reviso constructs a hierarchical Context--Focus--Points (CFP) observation plan that identifies the video context, prioritizes key aspects, and defines concrete observation targets and corresponding tools. Guided by this plan, Reviso performs goal-directed active perception by selectively revisiting temporal segments and combining tools for fine-grained inspection. It interprets the acquired evidence, updates its plan as needed, and integrates global and local information to generate the final caption. To learn this observation strategy, we develop a verification-guided pipeline that constructs high-quality training trajectories through factual verification and teacher-guided refinement. We further train Reviso using Group Relative Policy Optimization (GRPO), incorporating a key-fact reward for caption quality, a Plan Alignment Reward (PAR) for identifying important observation targets, and an Unproductive Tool-Call Penalty (UTCP) that adjusts observation-cost penalties based on caption quality. Experiments across four video captioning benchmarks show that Reviso outperforms the evaluated open-source baselines, while its generated captions better support downstream video question answering. These findings highlight the effectiveness and potential of priority-guided active observation for video captioning.
Video captioning requires fine-grained spatio-temporal understanding of videos, including spatial perception of where objects are located and temporal perception of when events occur. Existing MLLMs usually generate captions directly from video inputs without exposing the perceptual evidence behind descriptions. As a result, mistakes in spatiotemporal perception are only observed in the final caption, making it difficult to identify the underlying perceptual errors directly. To address these issues, we present PercepCap, a perception-aware video captioning framework that makes perceptual evidence explicit before producing the final caption. Specifically, PercepCap follows a perceive-describe generation chain, where the model first produces a spatiotemporal perception trace comprising object trajectories and temporal events, and then generates the final caption conditioned on the perceived evidence. To support this, we design a two-stage training strategy. Perceive-then-Describe Supervised Fine-tuning adapts the model from caption-only generation to the proposed perceive-describe chain, while Perception-Grounded Reinforcement Learning optimizes perception trace and caption quality with joint rewards over perception chain and the final caption. To support our two-stage training, we introduce Caption-Anchored Perception Data Construction. This pipeline builds the SFT and RL training data by first generating a caption-only description, extracting the objects and events it mentions, and grounding them back in the video with boxes and timestamps. This yields caption-aligned perception data that provides solid training ground truth, ensuring that the explicit perception trace and final caption refer to the same objects and events. Across direct caption and caption-to-QA evaluation, PercepCap consistently improves upon the Qwen3-VL baseline and demonstrates leading caption quality.
Yifan Xu, Zihao Wang, Zhixiao Wang +6
State Key Laboratory for Novel Software Technology, Nanjing Univerisity · Kling Team, Kuaishou Technology · Shanghai AI Laboratory
On-policy self-distillation (OPSD) has recently emerged as an effective post-training paradigm that improves policy optimization through dense token-level supervision from a privileged self-teacher. Despite its promise, OPSD remains largely underexplored for Video Large Language Models (Video-LLMs). Existing methods typically construct privileged teachers by augmenting their context with additional information while keeping the primary input unchanged for both teacher and student. Video reasoning, however, offers a distinct source of privileged supervision within the primary input itself: long videos contain substantial temporal redundancy, and only a small subset of frames provides the evidence necessary to answer a question. Building on this observation, we present Video-OPSD, an OPSD framework that exploits privileged visual evidence for both self-teacher construction and knowledge transfer. First, our Evidence-Grounded Self-Teacher conditions the teacher exclusively on annotated evidence frames while the student continues to reason over the complete video. This focused visual input enables the teacher to provide more informative supervision. Second, our Evidence-Guided Token Optimization adaptively weights token-level distillation according to each reasoning token's reliance on privileged visual evidence, thereby emphasizing perceptually grounded reasoning. Experiments across video understanding and reasoning benchmarks show that Video-OPSD consistently improves upon Standard OPSD across multiple backbones and achieves performance comparable to GRPO while requiring substantially less training time, establishing an effective and efficient post-training approach for Video-LLMs.
Ziyue Wang, Shiqi Huang, Weiwen Xu +2
School of Electrical and Electronic Engineering, Nanyang Technological University · The Chinese University of Hong Kong