PIVOT: Progressive Intervention via Online Teacher Correction

Zhihao Yang; Boyuan Zheng; Puhui Tan; Yukun Chen; Kuan Zhang; Longze Chen; Jiaming Li; Renke Shan

cs.CLOct 29, 2026

PIVOT: Progressive Intervention via Online Teacher Correction

Authors: Zhihao Yang, Boyuan Zheng, Puhui Tan, Yukun Chen, Kuan Zhang, Longze Chen, Jiaming Li, Renke Shan,

Organizations: University of Chinese Academy of Sciences · School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · Tsinghua University · Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences

Abstract

Reinforcement learning from verifiable rewards (RLVR) is limited by sparse outcome-level feedback on hard reasoning tasks: when all rollouts in a group fail, rewards provide no learning signal. Stronger teachers can provide additional guidance, but this raises a central question: how can such signals be incorporated into RLVR to support successful rollouts while remaining learnable for the student? We distinguish two complementary mechanisms: trajectory-level intervention, which improves access to reward-bearing trajectories, and token-level supervision, which provides dense targets to guide student learning. Our controlled experiments show that either mechanism alone can remain insufficient, motivating their combination. We therefore introduce PIVOT, which inserts teacher-generated corrections into student rollouts and applies token-level KL supervision to these segments alongside the RLVR objective. Beyond their combination, we systematically analyze where to intervene and how to supervise, yielding a recipe for effective guidance. Across hard logic games and mathematical reasoning benchmarks, PIVOT consistently outperforms existing methods, nearly doubling mean avg@8 on logic games from 19.5% for the best-performing baseline to 35.6%.

Figures & tables

Figure 1: Overview of PIVOT . For each query, the policy model samples a group of rollouts, including teacher-assisted trajectories. In a teacher-assisted rollout, the student first generates a prefix, the teacher provides a correction segment, and the student then continues generation. The completed trajectory is optimized by the verifier-based GRPO objective, while the teacher-generated segment provides dense token-level supervision through a teacher-student KL loss.

Figure 2: Prefix supervision vs. prefix-only guidance on Survo. Supervised prefixes improve rewards, while POPE-style prefixes fail to yield comparable gains.

Figure 3: Dense supervision under KDRL and PIVOT on a Survo rollout; insets show the teacher’s top-2 next-token probabilities at supervised positions. Left: on the student’s failed trajectory, the teacher is split between fixing the error ( “Wait” ) and continuing along it ( “Let” , “So” ), giving conflicting targets. Right: the teacher corrects the error itself and continues from there, so its targets are unimodal whether it backtracks ( “Wait” ) or proceeds ( “So” ), and the trajectory reaches the reward.

Intervention	Survo		Minesweeper		Kakurasu
Intervention	avg@8	pass@8	avg@8	pass@8	avg@8	pass@8
Prefix ( $\ell_{u}=0$ )	45.90 +0.00	87.00 +0.00	14.81 +0.00	42.50 +0.00	0.57 +0.00	3.20 +0.00
Middle ( $\ell_{u}=512$ )	50.70 +4.80	90.40 +3.40	19.25 +4.44	55.50 +13.00	3.52 +2.95	16.60 +13.40
Middle ( $\ell_{u}=1024$ )	49.88 +3.98	86.40 -0.60	17.94 +3.13	55.00 +12.50	3.77 +3.20	15.20 +12.00
Middle ( $\ell_{u}=2048$ )	51.55 +5.65	87.20 +0.20	12.50 -2.31	43.50 +1.00	3.32 +2.75	14.00 +10.80

Table 1: Effect of the intervention position

\ell_{u}

on hard logic games.

\ell_{u}=0

is prefix intervention.

Figure 5: Prefix vs. middle intervention. Guidance is more effective when the teacher can build on the student’s partial solution and correct errors along the way.

Figure 6: Reverse KL favors low initial accuracy; forward KL has a modest advantage at higher accuracy. Dashed lines mark base accuracy. Panels use different scales; labels show gains over the other KL, colored by the better KL.

Figure 7: Calibrating teacher guidance on hard logic games under middle intervention with forward KL. Each column varies one factor from the default setting, which is shaded. Solid lines denote avg@8 and dashed lines denote pass@8. Full numbers are given in Tab. 6 in the appendix.

Method	Hard Logic Games (Qwen3-1.7B)			Math Benchmarks (Qwen3-4B)
Method	Survo	Minesweeper	Kakurasu	Math-ID	AIME 2024	AIME 2025	MATH500
Base model	0.05	0.06	0.00	5.04	60.89	49.17	88.68
Baselines
RL only	0.05	1.94 +1.88	0.00	20.70 +15.66	60.62 -0.27	49.74 +0.57	94.30 +5.62
SFT+RL	39.77 +39.72	13.44 +13.38	1.38 +1.38	8.82 +3.78	60.94 +0.05	49.21 +0.04	89.98 +1.30
LUFFY	1.30 +1.25	0.81 +0.75	0.00	16.66 +11.62	60.68 -0.21	46.93 -2.24	90.60 +1.92

Table 2: Main results on hard logic games (Qwen3-1.7B) and math (Qwen3-4B). Math-ID is in-distribution; other math benchmarks are OOD. The base model uses thinking mode. Metrics are avg@8, or avg@64 for AIME. Colored boxes show absolute changes over the base model.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Method	avg@8	pass@8
POPE w/ noisy prefix	0.03	0.20
POPE w/ oracle prefix	0.14	0.91
Prefix-Fwd	59.05	94.20

Appendix

Table 3: Prefix-only guidance and supervised prefix training on Survo. Oracle prefixes use verifier-filtered teacher rollouts.

Method	Survo		Minesweeper		Kakurasu
Method	avg@8	pass@8	avg@8	pass@8	avg@8	pass@8
Base model	0.05	0.40	0.06	0.50	0.00	0.00
Baselines
RL only	0.05	0.40	1.94 +1.88	8.00 +7.50	0.00	0.00
SFT+RL	39.77 +39.72	86.40 +86.00	13.44 +13.38	41.50 +41.00	1.38 +1.38	7.60 +7.60
LUFFY	1.30 +1.25	9.00 +8.60	0.81 +0.75	5.00 +4.50	0.00	0.00

Appendix

Table 4: Full results on hard logic games (Qwen3-1.7B; base model in thinking mode). Signed values show changes from the base model. Shading identifies PIVOT ; bold indicates the best result in each column.

Method	Math-ID		MATH500
Method	avg@8	pass@8	avg@8	pass@8
Base model	5.04	22.20	88.68	92.20
Baselines
RL only	20.70 +15.66	37.00 +14.80	94.30 +5.62	98.00 +5.80
SFT+RL	8.82 +3.78	28.20 +6.00	89.98 +1.30	94.20 +2.00
LUFFY	16.66 +11.62	33.00 +10.80	90.60 +1.92	97.20 +5.00

Appendix

Table 5: Full mathematical reasoning results (Qwen3-4B, thinking mode). Signed values show changes from the base model. The two panels use different sampling budgets; bold identifies the best result for each benchmark and metric.

Setting	Survo		Minesweeper		Kakurasu
Setting	avg@8	pass@8	avg@8	pass@8	avg@8	pass@8
Teacher capacity
Qwen3-4B-Thinking $\ast$	50.70	90.40	19.25	55.50	3.52	16.60
Qwen3-30B-A3B-Thinking	53.75	88.80	40.12	61.40	17.27	50.20
Fixed teacher budget ( $G_{T}\times\ell_{v}=8$ k)
$2\times 4$ k $\ast$	50.70	90.40	19.25	55.50	3.52	16.60

Appendix

Table 6: Teacher-guidance ablations with middle intervention and forward KL. Shaded rows repeat the default setting ( ∗ ). Bold values identify the best result within each block.

RL for Language Model Reasoning

Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.

See all 1,103 papers

Residual Advantage: Student-Relative Teacher Guidance for RL with Verifiable Rewards
Oct 8, 2026Xiaobing Chen, Zhiqi PangRL for Language Model Reasoning RL Post-Training
Harbin Engineering University · Tencent · Harbin Institute of Technology
PIVOT: Pivot-Aware On Policy Self Distillation for Multi-Turn VLM Agents
Sep 28, 2026Jiazhou Zhou, Hu Zhou, Yucheng Chen +3On-Policy Self-Distillation Reinforcement Learning with Verifiable Rewards
AI Thrust, The Hong Kong University of Science and Technology (Guangzhou) · International Digital Economy Academy (IDEA) · The Hong Kong Polytechnic University +2
MENTOR: Learning from Failure with On-Policy Pedagogical Intervention
Oct 9, 2026Guanlan Yue, Hang Yu, Xiaoyong Huang +4RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards

Oct 8, 2026J/K move · Enter open · S save

Residual Advantage: Student-Relative Teacher Guidance for RL with Verifiable Rewards

Xiaobing Chen, Zhiqi Pang

Harbin Engineering University · Tencent · Harbin Institute of Technology

RL for Language Model Reasoning RL Post-Training Credit Assignment in RL Reinforcement Learning with Verifiable Rewards On-Policy Distillation

Open paper AskPDF arXiv

Reinforcement learning with verifiable rewards (RLVR) and on-policy distillation (OPD) have become two main paradigms for post-training reasoning models. RLVR gives each response a single outcome label, leaving the steps inside it without separate credit. OPD provides token-level guidance at student-visited prefixes, but its pointwise signal does not directly reflect the pattern of teacher--student disagreement across the vocabulary. Dense, unbounded log-ratio supervision can amplify the teacher's influence, yet a strong solver is not necessarily a suitable guide when the student's solution paths depart from the teacher's. We propose Residual Advantage (\RA{}), which treats the teacher--student probability residual as a bounded one-step reward, subtracts the corresponding state value under the student policy to form a standard advantage, and centers the result within each response before adding it to the verifier advantage. The guidance term has zero mean within each response, so the verifier advantage remains the response's mean label and the teacher only redistributes credit among the steps within it. \CoRA{} further updates a teacher LoRA with verifier advantages on the same scored student batch and uses the updated teacher in the next iteration's residual, adapting guidance to the student's attempts. With Qwen3-1.7B-Base and Qwen3-4B-Base students and a Qwen3-8B teacher, \RA{} combined with GRPO or REINFORCE++ improves the underlying sequence-advantage algorithm in all 24 comparisons on three mathematical benchmarks, raising macro Avg@8 by 1.7--3.6 points and Pass@8 by 3.9--6.3 points. Both combinations surpass teacher-only OPD, and \CoRA{} adds a further 1.0--1.5 Avg@8 points.

Abstract

RL for Language Model Reasoning

Explore similar work

Residual Advantage: Student-Relative Teacher Guidance for RL with Verifiable Rewards

PIVOT: Pivot-Aware On Policy Self Distillation for Multi-Turn VLM Agents

MENTOR: Learning from Failure with On-Policy Pedagogical Intervention