Redirecting Teacher Supervision: Temporal-Response-Guided Latent Distillation for Video-Language Models

Xinkai Lu; Junli Zha; Haolan Wei; Jiahui Wang; Chao Zhou; Xiang Song

cs.CVOct 9, 2026

Redirecting Teacher Supervision: Temporal-Response-Guided Latent Distillation for Video-Language Models

Authors: Xinkai Lu, Junli Zha, Haolan Wei, Jiahui Wang, Chao Zhou, Xiang Song

Organizations: SF Technology

Abstract

On-policy distillation (OPD) transfers capabilities through teacher predictions on student-generated sequences, as studied by GKD and MiniLLM for language models and Video-OPD for video tasks. We ask how internal supervision can further support temporal learning. Direct hidden-state alignment does not explicitly use the student's temporal responses to choose a supervision direction. We propose Temporal-Response Guidance (TRG), which measures representation responses to local temporal perturbations, adds the teacher correction's projection onto their span back to the original correction, and restores its original norm. The projection is not normalized separately, and correction components outside the response space are retained. With an orthogonal tangent projector and a detached target, TRG changes the supervision direction while preserving the cosine-loss value and aligned-representation gradient norm at target construction. On TempCompass, Video-MME, MVBench, and Video-MMMU, TRG improves the 4B student's macro accuracy by 2.86 percentage points over the reported output-only OPD baseline.

Figures & tables

Figure 1: Output-only OPD and TRG. Both use output supervision on student rollouts. TRG additionally uses student temporal responses to guide the teacher correction in latent space.

Figure 2: TRG training framework. Frames and question: Video-R1-260k, STAR sample #1552 ( Feng et al., 2025 ) . Student rollouts receive output supervision; accepted teacher answers fix the latent context. Responses construct a detached target, while gradients flow through unperturbed student states and alignment maps. Only the trained student is deployed.

Model / method	Frames	TC	V-MME	MVB	V-MMMU	Macro
LongVA-7B †	—	56.9	52.6	—	23.9	—
VILA-1.5-8B †	—	58.8	—	—	20.8	—
VILA-1.5-40B †	—	—	60.1	—	34.0	—
Video-UTR-7B †	—	59.7	52.6	58.8	—	—
LLaVA-OneVision-7B †	—	—	58.2	56.7	33.8	—
Kangaroo-8B †	—	62.5	56.0	61.1	—	—

Table 1: Video-model accuracy (%). Dagger-marked results come from Video-R1, Table 1, under their original protocols. Our 16-frame results are three-run means. Video-MME excludes subtitles. Dashes denote unlisted entries; macro scores cover only our controlled comparisons. TC: TempCompass; V-MME: Video-MME; MVB: MVBench; V-MMMU: Video-MMMU.

Method	TC	V-MME	MVB	V-MMMU	Macro
Plain-AS	74.24	58.70	64.33	64.67	65.48
Random orientation	74.43	59.19	64.80	63.22	65.41
TRG	74.81	60.67	65.03	64.89	66.35

Table 2: Angle-rule-matched direction comparison at 4B. Accuracy (%), averaged over three runs. Shared rows reuse Table 1.

Method	Action (338)	Direction (335)	Speed (317)	Order (302)	Attr. change (288)	Overall
OPD	80.47	71.34	72.56	68.87	72.22	73.23
Plain-AS	81.07	72.54	73.50	70.20	73.26	74.24
Random-input response	80.77	72.54	73.82	69.54	73.26	74.11
TRG	81.07	73.13	74.13	71.19	73.96	74.81
TRG - Plain-AS (pp)	0.00	+0.60	+0.63	+0.99	+0.69	+0.57

Table 3: TempCompass categories, 4B and 16 frames. Accuracy (%) from one evaluation per method, separate from Table 1. Headers give question counts (1,580 total). The last row reports count-based percentage-point gains; complete counts appear in Appendix C.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Value
Video frames / temporal groups / probes	16 / 8 / 7
Updates / effective batch size	300 / 8
Learning rate / weight decay	$10^{-4}$ / 0.01
Attention LoRA rank / scale / dropout	16 / 32 / 0
Student-teacher layer pairs	(16,48), (20,56), (24,64)
Relative SVD threshold / latent weight	$10^{-3}$ / 1

Appendix

Table B1: Distillation settings shared by 4B and 9B. Layer indices are one-based.

Benchmark	Frame sampling	Output limit	Questions
TempCompass	Processor fixed-frame, full video	512	1,580
Video-MME	Uniform, full video	512	2,700
MVBench	Official segment centers and clip bounds	512	4,000
Video-MMMU	Uniform; Adaptation question image	4,096	900

Appendix

Table B2: Evaluation uses 16 frames, greedy decoding, thinking disabled, and no audio or subtitles. Invalid or non-EOS-truncated answers are incorrect.

Variant	TC	V-MME	MVB	V-MMMU	Macro
Plain $\lambda=0.3$	73.61	57.19	64.70	62.33	64.46
Plain $\lambda=3$	74.18	58.81	64.30	64.56	65.46
Unit projection	74.87	59.41	64.75	64.33	65.84
Projection only	73.92	58.52	64.05	63.67	65.04
3 probes	74.62	60.11	64.93	64.56	66.06
5 probes	74.75	60.52	65.00	64.78	66.26

Appendix

Table C1: Single-run variants at 4B; accuracy (%).

Scale / method	TC /1580	V-MME /2700	MVB /4000	V-MMMU /900
4B OPD	1164, 1151, 1156	1532, 1484, 1525	2588, 2581, 2603	538, 545, 534
4B Plain-AS	1171, 1177, 1171	1588, 1584, 1583	2572, 2582, 2565	581, 583, 582
4B Random orientation	1180, 1181, 1167	1618, 1594, 1582	2593, 2594, 2589	570, 571, 566
4B TRG	1186, 1180, 1180	1641, 1639, 1634	2600, 2599, 2605	586, 583, 583
9B OPD	1139, 1139, 1145	1599, 1595, 1600	2594, 2586, 2588	548, 548, 551
9B Plain-AS	1186, 1181, 1182	1672, 1663, 1664	2585, 2586, 2591	589, 586, 589

Appendix

Table C2: Correct counts in run order (1, 2, 3); headers give evaluation totals.

Method	Action /338	Direction /335	Speed /317	Order /302	Change /288	Total /1580
OPD	272	239	230	208	208	1157
Plain-AS	274	243	233	212	211	1173
Random-input response	273	243	234	210	211	1171
TRG	274	245	235	215	213	1182

Appendix

Table C3: Single-evaluation TempCompass correct counts underlying Table 3.

Video	MVBench item	Training record IDs
8LAK1.mp4	action_prediction:0040	184504, 50183
KQYR0.mp4	action_prediction:0105	181881
FQS7O.mp4	action_sequence:0017	27574
ZFQNZ.mp4	action_sequence:0148	229025
3EIRZ.mp4	object_interaction:0047	85670

Appendix

Table D1: Shared media and affected MVBench items.

VLM Distillation

Vision-language model distillation transfers multimodal capabilities to smaller, faster models for efficient deployment.

See all 126 papers

Less Data, Better Timing: Student-Curriculum Coupling for VLM On-Policy Distillation in Temporal Video Grounding
Sep 30, 2026Jiacheng Qiu, Yunsoo Kim, Ruichen Xu +3Temporal Video Grounding VLM Distillation
State University of New York at Stony Brook
Do As I Do, Not As I Say: On-Policy Distillation Inherits the Teacher in Vision-Language Models
Oct 14, 2026Frank F. Yang, Kefei Yan, Yiren Wang +1VLM Distillation On-Policy Distillation
Georgia Institute of Technology, Atlanta, GA, USA
Beyond Single-Sample: Reliable Multi-Sample Distillation for Video Understanding
Mar 12, 2026Songlin Li, Xin Zhu, Zechao Guan +2VLM Distillation Knowledge Distillation
University of Electronic Science and Technology of China · Robotics Center, XPeng Motors

Sep 30, 2026J/K move · Enter open · S save

Less Data, Better Timing: Student-Curriculum Coupling for VLM On-Policy Distillation in Temporal Video Grounding

Jiacheng Qiu, Yunsoo Kim, Ruichen Xu, Jian Luo+2

State University of New York at Stony Brook

Temporal Video Grounding VLM Distillation Curriculum Learning On-Policy Distillation

Open paper AskPDF arXiv

On-policy distillation (OPD) provides dense supervision directly on student-generated trajectories, making it an effective post-training strategy for vision-language models in temporal video grounding (TVG). However, existing pipelines typically construct the training curriculum from a fixed teacher and the initial student state, implicitly assuming that selected examples retain positive supervision value throughout optimization. We show that supervision trustworthiness and supervision necessity are distinct yet coupled: the former concerns target credibility, while the latter varies with the student's current task competence; together, they shape supervision value. Building on this coupled view, we introduce Student-Curriculum Coupling (SCC), a closed-loop framework in which a compact Anchor-Frontier curriculum defines the candidate supervision space and the evolving student dynamically determines its active subset. Supervision can therefore be activated, suspended, or reactivated as competence changes, concentrating teacher computation and optimization on current task-level deficits. Across three TVG benchmarks, SCC achieves a 5.1% relative improvement in mean recall over Video-OPD on its original curriculum, while using 60.0% fewer training examples and reducing training time by 50.4%. Ablations support the complementary roles of capability-structured curriculum design and student-dependent supervision in achieving these gains. Together, these results establish SCC as a data- and compute-efficient framework for TVG post-training, delivering stronger temporal grounding by aligning trustworthy supervision with the student's evolving learning needs.

Abstract

VLM Distillation

Explore similar work

Less Data, Better Timing: Student-Curriculum Coupling for VLM On-Policy Distillation in Temporal Video Grounding

Do As I Do, Not As I Say: On-Policy Distillation Inherits the Teacher in Vision-Language Models

Beyond Single-Sample: Reliable Multi-Sample Distillation for Video Understanding