cs.CVMar 12, 2026

Beyond Single-Sample: Reliable Multi-Sample Distillation for Video Understanding

Authors: Songlin Li, Xin Zhu, Zechao Guan, Peipeng Chen, Jian Yao

Organizations: University of Electronic Science and Technology of China · Robotics Center, XPeng Motors

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Traditional black-box distillation for Large Vision-Language Models (LVLMs) typically relies on a single teacher response per input, which often yields high-variance responses and format inconsistencies in multimodal or temporal scenarios. To mitigate this unreliable supervision, we propose R-MSD (Reliable Multi-Sample Distillation), a framework that explicitly models teacher sampling variance to enhance distillation stability. Rather than relying on a single teacher response, our approach leverages a task-adaptive teacher pool to provide robust supervision tailored to both closed-ended and open-ended reasoning. By integrating quality-aware signal matching with an adversarial distillation objective, our approach effectively filters teacher noise while maximizing knowledge transfer. Extensive evaluations across comprehensive video understanding benchmarks demonstrate that R-MSD consistently outperforms single sample distillation methods. We additionally include an original SFT+RL 4B baseline under the same training budget, which shows only marginal gains, while our method achieves significant improvements. With a 4B student model, our approach delivers gains on VideoMME (+1.5%), Video-MMMU (+3.2%), and MathVerse (+3.6%).

Figures & tables

Figure 1: Teacher sampling variance in video understanding. Left: Temporal QA example: four teacher responses with their temporal IoU to ground truth, showing substantial quality variation across samples of the same question. Right: Global statistics over 200 teacher responses from eight task types: (a) cross-question variance and (b) within-question sampling variance. Both dimensions undermine single-sample distillation and motivate our multi-sample approach.
Figure 2: Overview of the R-MSD framework (Stage 2: RL-based Adversarial Distillation). The pipeline utilizes a multi-sample teacher collection to provide diverse references. A critic-as-discriminator mechanism is employed to perform task-adaptive sampling, matching student rollouts with teacher responses. The student policy is then iteratively refined through adversarial distillation, using a weighted combination of discriminator, format, and task-specific scores as the reward signal.
ModelFramesWhen (Temporal IoU)Where (Visual IoU)
  Chain1Chain2Chain1Chain2
GPT-4o ( Hurst et al., 2024 )-16.712.86.53.0
Gemini-2-Flash ( Comanici et al., 2025 )-24.523.84.62.2
Video-LLaMA3 ( Zhang et al., 2025 )-23.023.10.90.2
LLaVA-Video ( Zhang et al., 2024 )-10.512.21.91.3
VideoChat2 ( Li et al., 2024a )-13.712.52.51.0
Table 2: Results on V-STaR ( Cheng et al., 2025 ) . We report When (temporal IoU) and Where (visual IoU) under the official evaluation. Frames denotes test-time frames. Improvements indicate better spatio-temporal grounding under teacher sampling variance.
SettingKKFilterWeightVideoMMEVideo-MMMU
A (single-sample)1NoNo63.854.4
B (multi-sample)4NoNo64.555.9
C (+filtering)4YesNo65.057.2
D (ours)4YesYes65.358.6
Table 3: Core component ablation on VideoMME and Video-MMMU (accuracy, %). KK : teacher sample count. “Filter” applies threshold τ\tau to remove low-quality teacher samples ( qk<τq_{k}<\tau ). “Weight” enables quality-weighted discriminator updates by qkq_{k} . Setting A reports Stage 1 single-sample SFT; vanilla Qwen3-VL-4B is in Table 1 .
KKVideoMMEVideo-MMMU
264.857.1
465.358.6
865.558.9
Table 4: Sensitivity analyses for teacher sample count KK (left; VideoMME/Video-MMMU accuracy, %) and quality threshold τ\tau (right; valid-sample ratio and accuracy) under the full method. We select τ=0.3\tau=0.3 to balance low-quality removal (72% retention) and teacher-pool diversity.
Task familyGT-based scoringUniform weighting
Closed-ended tasks57.856.2
Open-ended tasks58.459.1
Table 5: Task-adaptive strategy validation: average accuracy (%) over closed-ended and open-ended task subsets across VideoMME and Video-MMMU under GT-based scoring vs. uniform weighting.
Figure 3: Per-task teacher variance and Pass@ kk behavior on Video-MMMU. Left: Teacher quality distribution by task type; box plots over 200 teacher responses across six task families highlight that high-stability closed-ended tasks (e.g., MCQ, numerical) have lower variance, while higher-variance tasks such as visual QA exhibit wider spreads. Right: Pass@ kk accuracy on Video-MMMU as a function of kk (number of sampled student responses per query); our method achieves +3.2% higher Pass@1 accuracy than Qwen3-VL-4B while converging to a similar upper bound as kk increases.
More in this topic

VLM Distillation

Vision-language model distillation transfers multimodal capabilities to smaller, faster models for efficient deployment.

See all 126 papers

Explore similar work

CardsList

  1. Video-MOPD: Multi-Teacher On-Policy Distillation for Video Understanding

    Sep 8, 2026Zhenxin Qin, Peng Shi, Cong Han +3Temporal Video GroundingVideo Understanding

    Tongji University · Bilibili Inc.

  2. VISD: Enhancing Video Reasoning via Structured Self-Distillation

    May 7, 2026Hao Lin, Kunyang Lv, Xu Jiang +5Video-Language ModelsVideo Reasoning

    1HUST · 2Wuhan University · 3Peking University +1

  3. MeSD: Multi-Evidence Self-Distillation for VideoLLM

    Oct 5, 2026Weijie Zhu, Han Fang, Hanyu Fu +13On-Policy Self-DistillationVideo-Language Models

    UCAS · Xingchen AGI Lab, China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd · PKU +2