PIVOT: Perplexity-Informed KD-to-RL Transition Scheduling for Vertical-Domain Few-Shot Distillation

Heng Li; Yong Zhang; Ning Cheng; Zhigen Li; Yun Zhu; Yanmeng Wang; Shaojun Wang; Jing Xiao

cs.LGOct 8, 2026

PIVOT: Perplexity-Informed KD-to-RL Transition Scheduling for Vertical-Domain Few-Shot Distillation

Authors: Heng Li, Yong Zhang, Ning Cheng, Zhigen Li, Yun Zhu, Yanmeng Wang, Shaojun Wang, Jing Xiao

Organizations: Ping An Technology (Shenzhen) Co., Ltd., China · University of Science and Technology of China

Abstract

Vertical-domain few-shot classification remains challenging for small language models, as limited supervision makes it difficult to acquire domain-specific decision knowledge. On-Policy Distillation (OPD) can improve teacher-guided adaptation by supervising student-generated rollouts, while GRPO-based reinforcement learning can further refine downstream predictions. However, existing KD-to-RL pipelines typically rely on globally fixed transition schedules, ignoring that different samples may require different amounts of teacher-guided acquisition before reward-driven refinement. We propose PIVOT (Perplexity-Informed Transition Optimization), a dynamic transition framework that routes samples between OPD and GRPO according to teacher-evaluated sequence perplexity. PIVOT moves low-perplexity samples to GRPO for reward-driven refinement while keeping high-perplexity samples under OPD for continued domain knowledge acquisition. Experiments on Banking77 and HWU64 show that PIVOT consistently outperforms continued OPD and globally synchronized OPD $\rightarrow$ GRPO baselines under the same number of post-warm-up student optimization steps, achieving stronger downstream performance and more stable training dynamics.

Figures & tables

Figure 1: Overview of PIVOT. PIVOT estimates sample-level acquisition states using teacher-side perplexity over on-policy student rollouts, and progressively performs asynchronous KD-to-RL transition by routing low-perplexity samples to GRPO refinement while retaining high-perplexity samples under continued OPD acquisition.

Figure 2: Training dynamics on Banking77 under aligned total optimization steps. For globally synchronized OPD $\rightarrow$ GRPO, the x-axis denotes cumulative optimization steps across the OPD and GRPO stages, and the purple star marks the global transition point. Compared with continued OPD and globally synchronized OPD $\rightarrow$ GRPO, our progressive sample-level transition achieves higher final accuracy with substantially smoother reward, entropy, and completion-length dynamics.

Method	Banking77	HWU64
Teacher	73.25	79.65
SeqKD	58.70	70.17
GRPO	67.66	77.60
OPD	72.47	79.55
OPD $\rightarrow$ GRPO	78.05	83.09
PIVOT	80.26	85.04

Table 1: Main results under 480 post-warm-up student optimization steps. All results report test-set classification accuracy.

Dataset	Setting	Student	Teacher	OPD $\rightarrow$ GRPO	PIVOT	$\Delta$
Banking77	Default	0.5B	Qwen2.5-7B	78.05	80.26	+2.21
Banking77	Larger student	1.5B	Qwen2.5-7B	80.00	82.82	+2.82
Banking77	Stronger teacher	0.5B	DianJin-32B	80.30	81.33	+1.03
HWU64	Default	0.5B	Qwen2.5-7B	83.09	85.04	+1.95
HWU64	Larger student	1.5B	Qwen2.5-7B	82.53	85.97	+3.44
HWU64	Stronger teacher	0.5B	Qwen3-32B	84.39	86.71	+2.32

Table 2: Scaling analysis across different student and teacher model configurations.

\Delta

denotes the improvement of PIVOT over OPD

\rightarrow

GRPO.

Dataset	Low-PPL	Random	High-PPL
Banking77	80.26	78.31	77.27
HWU64	85.04	84.11	83.46

Table 3: Routing-rule ablation under progressive OPD-to-GRPO transition. Column names indicate which samples are preferentially routed to GRPO earlier during training.

Transition Strategy	ACC
Global OPD $\rightarrow$ GRPO	78.05
Linear progressive routing	79.58
Quadratic progressive routing	79.19
Cosine progressive routing	80.26

Table 4: Transition-schedule ablation on Banking77 under the same number of student optimization steps.

Figure 3: Pass@32 performance grouped by teacher-side perplexity buckets during OPD training. High-perplexity samples exhibit substantially larger relative improvement throughout training, whereas low-perplexity samples saturate much earlier, suggesting that the optimal transition timing from OPD to RL varies across samples.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Banking77
SFT-ZeroShot	60.30
DataAug-SFT	71.75
Continued OPD	72.47
OPD $\rightarrow$ GRPO	78.05
PIVOT	80.26

Appendix

Table 7: Comparison between teacher-generated data augmentation and different OPD/GRPO training strategies on Banking77. DataAug-SFT is trained on 5,119 teacher-filtered synthetic examples generated from the same 5-shot split.

Hyperparameter	Value
Total optimization steps	480
Batch size	256
Rollouts / group size	8
Warmup ratio	0.05
Max sequence length	4096
Max new tokens	512

Appendix

Table 9: Training hyperparameters used in all experiments.

Explore similar work

PIVOT: Pivot-Aware On Policy Self Distillation for Multi-Turn VLM Agents
Sep 28, 2026Jiazhou Zhou, Hu Zhou, Yucheng Chen +3On-Policy Self-Distillation Reinforcement Learning with Verifiable Rewards
AI Thrust, The Hong Kong University of Science and Technology (Guangzhou) · International Digital Economy Academy (IDEA) · The Hong Kong Polytechnic University +2
Eliciting Weak-to-Strong Generalization with On-Policy Reverse Distillation
Sep 8, 2026Youngrok Park, Sangmin Bae, Hojung Jung +6Policy Gradient Weak-to-Strong Generalization
KAIST AI · Microsoft · University of Toronto +3
D $^3$ -MOPD: Dynamic Domain ScheDuling for Efficient Multi-Teacher Distillation
Aug 25, 2026Zechen Sun, Zhiwei Zhang, Fei Zhao +7Adaptive Sampling Language Model Distillation
AllSpark Team

Sep 28, 2026, cs.CVJ/K move · Enter open · S save

PIVOT: Pivot-Aware On Policy Self Distillation for Multi-Turn VLM Agents

Jiazhou Zhou, Hu Zhou, Yucheng Chen, Jinyuan Qu+2

AI Thrust, The Hong Kong University of Science and Technology (Guangzhou) · International Digital Economy Academy (IDEA) · The Hong Kong Polytechnic University+2

On-Policy Self-Distillation Reinforcement Learning with Verifiable Rewards RL Post-Training Policy Learning RL for VLMs

Open paper AskPDF arXiv

Reinforcement learning with verifiable rewards (RLVR) via Group-Relative Policy Optimization (GRPO) is widely used for multi-turn VLM agent training, yet it suffers from zero-gradient silence on uniform failures and coarse episode-level credit assignment. While On-Policy Distillation (OPD) and On-Policy Self-Distillation (OPSD) mitigate sparse rewards using hindsight information, their underlying mechanisms remain poorly understood. Through controlled counterfactual rollback probes across five multi-turn VLM agent benchmarks, we reveal that performance gains in OPSD/OPD are largely driven by physical state rollback at the pivot step, defined as the first unrecoverable action without remaining step budget. However, physical state rollbacks are computationally prohibitive and infeasible in real-world environments. To bridge this gap, we present Pivot-Aware Internalized Visual On-Policy Training (PIVOT), an RL framework that internalizes pivot localization and state restoration directly into token-level parameter updates, eliminating environment rollbacks during RL training and additional skill hints at test time. PIVOT unifies three functional roles within a single architecture: a failure Analyzer non-invasively localizes the pivot step and diagnoses failure modes from visual trajectory collages and action logs; a detached Teacher re-scores failed tokens under this privileged diagnostic context; and a Student optimizes joint GRPO and confidence-gated OPD objectives. At test time, both Teacher and Analyzer branches are stripped. Evaluated on five multi-turn VLM agent tasks across cognitive grid puzzles, 3D embodied control and navigation, and generative reasoning, PIVOT achieves 0.90 overall accuracy on Qwen2.5-VL-3B (+8% over SFT+GRPO baseline and +5% over previous SOTA) and scales to 0.92 on Qwen3-VL-2B (+12% over SFT+GRPO baseline).

Abstract

Explore similar work

PIVOT: Pivot-Aware On Policy Self Distillation for Multi-Turn VLM Agents

Eliciting Weak-to-Strong Generalization with On-Policy Reverse Distillation

D3^33-MOPD: Dynamic Domain ScheDuling for Efficient Multi-Teacher Distillation

D $^3$ -MOPD: Dynamic Domain ScheDuling for Efficient Multi-Teacher Distillation