On-policy self-distillation (OPSD) has recently emerged as an effective post-training paradigm that improves policy optimization through dense token-level supervision from a privileged self-teacher. Despite its promise, OPSD remains largely underexplored for Video Large Language Models (Video-LLMs). Existing methods typically construct privileged teachers by augmenting their context with additional information while keeping the primary input unchanged for both teacher and student. Video reasoning, however, offers a distinct source of privileged supervision within the primary input itself: long videos contain substantial temporal redundancy, and only a small subset of frames provides the evidence necessary to answer a question. Building on this observation, we present Video-OPSD, an OPSD framework that exploits privileged visual evidence for both self-teacher construction and knowledge transfer. First, our Evidence-Grounded Self-Teacher conditions the teacher exclusively on annotated evidence frames while the student continues to reason over the complete video. This focused visual input enables the teacher to provide more informative supervision. Second, our Evidence-Guided Token Optimization adaptively weights token-level distillation according to each reasoning token's reliance on privileged visual evidence, thereby emphasizing perceptually grounded reasoning. Experiments across video understanding and reasoning benchmarks show that Video-OPSD consistently improves upon Standard OPSD across multiple backbones and achieves performance comparable to GRPO while requiring substantially less training time, establishing an effective and efficient post-training approach for Video-LLMs.
Figures & tables
Figure 1: (a) Conventional OPSD, where the teacher and student share the same primary input and token-level distillation is applied uniformly. (b) Our evidence-driven Video-OPSD , featuring an Evidence-Grounded Self-Teacher and Evidence-Guided Token Optimization.
Figure 2: Overview of Video-OPSD . The Evidence-Grounded Self-Teacher observes only the evidence frames together with the gold answer, while the student observes the same evidence frames within a broader video context. The student generates a single on-policy trajectory, which is subsequently replayed by the teacher under its privileged context. Evidence-Guided Token Distillation measures each generated token’s attention to evidence visual signals and uses the resulting scores to weight the token-level JSD objective.
Method
Video-Holmes
Video-MMMU
Video-MME
TempCompass
WorldSense
Avg.
Qwen2.5-VL-7B
27.8
48.1
57.6
67.9
36.1
47.5
+ SFT
33.2
48.5
56.4
69.2
36.3
48.7
+ GRPO
35.3
49.9
60.7
70.9
36.6
50.7
+ OPSD
33.4
48.1
57.7
68.5
36.7
48.9
+ Video-OPSD
36.2
49.9
58.6
70.9
39.2
51.0
Qwen3-VL-8B
40.9
63.3
63.7
74.4
40.7
56.6
Table 1: Main results on five video understanding and reasoning benchmarks across three Qwen-VL backbones under the unified video-only evaluation protocol. Comparisons are made within each backbone.
Method
Avg.
Qwen2.5-VL-7B
47.4
OPSD
48.9
+ Evidence-Grounded Input
49.8
+ Evidence-Guided Weighting
50.4
Video-OPSD (Both)
51.0
Table 2: Component ablation of evidence-grounded teacher input and evidence-guided token-wise weighting.
Teacher Visual Input
Gold Answer
Accuracy (%)
Evidence Frames Only
No
70.4
Evidence Frames + 8 Padding Frames
No
64.9
Uniformly Sampled 16 Frames
No
49.8
Evidence Frames Only
Yes
85.3
Evidence Frames + 8 Padding Frames
Yes
83.7
Uniformly Sampled 16 Frames
Yes
78.5
Table 3: Accuracy on the training set to disentangle the sources of privileged teacher information.
Figure 3: Average scores under different token weighting strategies.
Method
TT (h)
SPS
Lat. (s)
FLOPs (T/GPU)
SFT
0.7
2.70
2.9
47.7
GRPO
5.5
0.35
16.3
468.4
OPSD
1.9
0.97
5.3
93.3
Video-OPSD
2.2
0.82
6.1
106.1
Table 4: Training efficiency on NVIDIA H100 GPUs. TT denotes training time, SPS denotes samples per second, and latency and FLOPs are measured per forward pass.
Attention Layers
Avg.
Last layer ( −1 )
50.2
Last two layers ( −1,−2 )
50.4
Last three layers ( −1,−2,−3 )
51.0
Table 5: Effect of attention-layer aggregation on average benchmark performance. Negative indices denote layers counted backward from the final Transformer layer.
Non-Evidence Frames
0
4
8
Average Score
51.0
50.6
50.4
Table 6: Effect of adding contextual frames outside the annotated evidence window.
Figure 4: Scores under different teacher update strategies.
Distillation Objective
Avg.
JSD-30
50.3
JSD-50
51.0
Reverse KL
49.9
Forward KL
49.7
Table 7: Comparison of token-level distillation objectives.
Training VideoLLMs for complex reasoning remains challenging due to sparse sequence level rewards and the lack of fine grained credit assignment over long, temporally grounded reasoning trajectories. While reinforcement learning with verifiable rewards (RLVR) provides reliable supervision, it fails to capture token level contributions, leading to inefficient learning. Conversely, existing self distillation methods offer dense supervision but lack structure and diagnostic specificity, and often interact unstably with reinforcement learning. In this work, we propose VISD, a structured self distillation framework that introduces diagnostically meaningful privileged information for video reasoning. VISD employs a video aware judge model to decompose reasoning quality into multiple dimensions, including answer correctness, logical consistency, and spatio-temporal grounding, and uses this structured feedback to guide a teacher policy for token level supervision. To stably integrate dense supervision with RL, we introduce a direction magnitude decoupling mechanism, where rollout level advantages computed from rewards determine update direction, while structured privileged signals modulate token level update magnitudes. This design enables semantically aligned and fine grained credit assignment, improving both reasoning faithfulness and training efficiency. Additionally, VISD incorporates curriculum scheduling and EMA based teacher stabilization to support robust optimization over long video sequences. Experiments on diverse benchmarks show that VISD consistently outperforms strong baselines, improving answer accuracy and spatio temporal grounding quality. Notably, VISD reaches these gains with nearly 2x faster convergence in optimization steps, highlighting the effectiveness of structured self supervision in improving both performance and sample efficiency for VideoLLMs.
Vision-language models (VLMs) have made substantial progress in long-video understanding, with standard backbone models typically answering questions from frames sampled across the full video. However, as videos become longer, the full-video context inevitably contains more question-irrelevant temporal content, which can distract the model from the evidence needed to answer a specific question. We empirically find that focusing the visual input on short annotated clue intervals containing question-relevant evidence consistently improves prediction accuracy across model scales compared with using the corresponding full videos, while requiring fewer input frames. Based on this finding, we introduce Clue-OPSD, a clue-privileged on-policy self-distillation framework for long-video understanding. During training, a full-video student learns from a self-teacher conditioned on the corresponding clue interval by aligning their next-token distributions along student-generated trajectories. Clue-OPSD thus uses clue intervals as privileged supervision without relying on ground-truth answer labels, while requiring no clue annotations or additional modules at inference time. Extensive experiments across multiple long-video understanding benchmarks and Qwen3.5 model scales demonstrate consistent improvements over the corresponding backbone models and strong performance against supervised post-training baselines.
Kaishen Wang, Dongdi Zhao, Yijun Liang +4
University of Maryland, College Park · Johns Hopkins University
Reinforcement learning (RL) has substantially improved the reasoning ability of multimodal language models through verifiable rewards and increasingly fine-grainedvisual or temporal credit assignment. In video reasoning, however, current RL methods typically train with a fixed sparse frame budget: increasing the number of frames makes autoregressive rollouts expensive, while too few frames may miss temporally localized events and fine-grained visual details. We present Frame Differential On-Policy Self-Distillation (FD-OPSD), which transfers the useful evidence of dense frame observations to a sparse frame policy during RL training. FD-OPSD compares the policy's token level preferences for the same sampled response under sparse and dense views, and distills the resulting frame differential signal without an external teacher or dense autoregressive rollout. The method preserves sparse-frame rollouts and leaves inference unchanged. Across Qwen2.5-VL-7B and Qwen3-VL-4B on six video reasoning benchmarks, FD-OPSD yields higher overall average performance than the strongest corresponding GRPO, T-GRPO, or Video-KTR baselines across the 16, 32, and 64 frame evaluation settings. These results show that dense visual evidence can be transferred selectively during training through token level self-distillation while retaining sparse frame rollouts and unchanged inference.