Teaching What to Look For: Privileged Perception Questions for Agentic On-Policy Distillation in Video Captioning
Organizations: Nanjing University · KlingAI Research
Abstract
Detailed video captioning requires comprehensive and faithful descriptions of objects, actions, and temporal events. Fixed-frame sampling, however, can miss brief motions, small objects, and other fine-grained evidence, motivating captioning agents that actively seek relevant moments and inspect local regions. Training such behavior is challenging because, unlike video question answering, generic captioning instructions do not specify what visual evidence should be acquired. We propose Perception-Question On-Policy Distillation (PQ-OPD), which uses training-only Perception Questions (PQs) to guide active video perception. PQs specify what should be inspected without revealing the corresponding answers, encouraging the teacher to explore the video more comprehensively and focus on relevant visual evidence. Along trajectories generated by the student, on-policy distillation transfers this guidance to the student's reasoning, tool calls, and captions; no PQs are required at inference. Across three detailed-captioning and two caption-to-QA benchmarks, PQ-OPD consistently improves the corresponding untrained agents across teacher-student scales and input-frame budgets. Under our evaluation protocol, PQ-OPD achieves the strongest primary-metric results among the compared open-source models on DREAM-1K, CaReBench, ShortVidBench, MotionBench, and VidCapBench-AE. Our ablations, together with analysis of temporal coverage and answer leakage, further indicates that the student internalizes the training-time guidance as a more effective video exploration policy.
Figures & tables
| Model | DREAM-1K | CaReBench | ShortVidBench | MotionBench | |
| Event F1/R/P | Action F1/R/P | Object F1/R/P | Acc. | Acc. | |
| Proprietary models (fixed 32 frames) | |||||
| Gemini 3 Flash ( Google, 2025 ) | 37.6/48.5/30.7 | 42.4/39.6/45.7 | 40.3/37.8/43.2 | 78.6 | 57.6 |
| Gemini 3.1 Pro ( Google, 2026 ) | 38.2/45.2/33.1 | 42.0/37.4/47.7 | 39.3/32.6/49.6 | 78.4 | 55.9 |
| Open-source models (fixed 32 frames) | |||||
| Qwen2.5-VL 7B ( Bai et al., 2025b ) | 27.2/27.9/26.6 | 30.9/23.6/45.0 | 34.1/27.6/44.6 | 70.0 | 50.7 |
| Model | VidCapBench-AE | ||||
| Overall | Video Aesthetics | Video Content | Video Motion | Physical Laws | |
| Acc./Pre./Cov./Con. | Acc./Pre./Cov./Con. | Acc./Pre./Cov./Con. | Acc./Pre./Cov./Con. | Acc./Pre./Cov./Con. | |
| Proprietary models (fixed 32 frames) | |||||
| GPT-4o ( OpenAI, 2024 ) | 16.8/57.4/86.0/5.9 | 14.1/47.6/83.4/4.9 | 17.5/61.7/87.2/6.1 | 10.2/41.3/84.0/3.6 | 27.9/62.1/85.4/9.7 |
| Gemini 3 Flash ( Google, 2025 ) | 19.3/58.1/90.5/7.3 | 17.1/47.3/87.5/6.4 | 19.6/62.8/91.7/7.4 | 12.9/49.5/89.8/4.9 | 33.9/58.4/91.7/12.8 |
| Open-source models (fixed 32 frames) | |||||
| # | Model Variant | Train PQ | Infer. PQ | CaReBench F1 | DREAM-1K F1 |
| Fixed observations (no seek/zoom tools) | |||||
| #1 | Baseline (Qwen3.5 9B) | – | – | 37.6 | 30.3 |
| #2 | + PQ at inference | – | ✓ | 37.4 | 33.4 |
| #5 | + PQ-OPD | ✓ | – | 38.0 | 31.9 |
| Interactive observations (seek/zoom tools) | |||||
| #3 | Base agent (Qwen3.5 9B) | – | – | 39.8 | 31.3 |
| Teacher Student | Training | CaReBench F1 | DREAM-1K F1 |
| 27B 9B | Vanilla OPD | 40.1 | 31.2 |
| 27B 9B | PQ-OPD | 41.2 | 33.4 |
| 9B 4B | Vanilla OPD | 39.4 | 30.8 |
| 9B 4B | PQ-OPD | 40.2 | 32.6 |
| Captioner Trajectory | CaReBench F1 | DREAM-1K F1 |
| Baseline agent Baseline agent (#3) | 39.8 | 31.3 |
| Baseline agent PQ-OPD | 40.3 | 32.6 |
| PQ-OPD Baseline agent | 40.3 | 32.2 |
| PQ-OPD PQ-OPD (#11) | 40.6 | 33.3 |
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Parameter | Value |
| Data and rollout | |
| Training videos | 5,979 ( Section 4.1 ) |
| Perception Questions | 43,477 |
| Maximum tool-image pixels | 460,800 |
| Unique prompts per refresh | 64 |
| Rollouts per prompt | 4 (256 trajectories per refresh) |
| Teacher Questions | Question-Only Captioning | Trained Student | ||
| Action F1 | Object F1 | CaReBench F1 | DREAM-1K F1 | |
| Standard PQs (PQ-OPD) | 20.8 | 14.9 | 40.6 | 33.3 |
| L1 (leakier wording) | 37.0 | 36.4 | 40.3 | 32.3 |
| L2 (more leaky wording) | 43.6 | 46.5 | 39.9 | 32.5 |
| L3 (leakiest wording) | 52.7 | 52.9 | 37.3 | 29.4 |
| Benchmark | Model | Returned Frames | Tool Calls | Tool Rounds | Visual Tokens | Latency (Median) |
| CaReBench | Qwen3.5 9B Agent | 34.5 | 11.1 | 3.7 | 11.4K | 30 s |
| PQ-OPD | 54.6 | 18.4 | 4.1 | 17.6K | 41 s | |
| DREAM-1K | Qwen3.5 9B Agent | 31.3 | 9.6 | 3.3 | 12.7K | 31 s |
| PQ-OPD | 48.8 | 14.6 | 3.5 | 19.8K | 42 s |