Training large language models for complex reasoning tasks relies on two complementary signals: sparse, outcome-grounded reinforcement learning (RL) for correctness, and dense, token-level distillation for guidance. However, complementarity in supervision does not guarantee compatibility in optimization; imitation-focused distillation can conflict with the outcome-driven RL objective, and existing hybrids lack a principled resolution. We propose GIFT (Gradient-Informed FilTering for Distillation-Augmented Reasoning RL), a novel method that directly addresses the allocation problem of which teacher updates to apply. GIFT performs a token-level geometric analysis, decomposing the distillation gradient into a component parallel to the RL gradient, which measures first-order conflict and provides a safety filter, and a component orthogonal to it, which quantifies novel directional guidance and provides a measure of value. This decomposition yields a principled scoring rule to rank token-level guidance. We combine this with a progressive coverage annealing schedule that shrinks the budget of tokens receiving distillation, ensuring that the most conflicting updates are discarded first while the most complementary ones are retained at full strength. This facilitates a smooth transition from dense, geometrically-filtered guidance to a pure, outcome-grounded objective at convergence. Extensive experiments on mathematical and logical reasoning benchmarks show that GIFT consistently outperforms strong baselines across multiple model scales, achieving relative gains of up to 9.5% on difficult mathematical reasoning tasks. Code: https://github.com/lilifree/GIFT
Figures & tables
Figure 1: GIFT decomposes the distillation gradient into conflict ( ai ) and complement ( bi ) components relative to the RL gradient, ranks tokens, and anneals the selection budget from full to zero.
Size
Bench.
BASE
GRPO
OPSD
SDPO
RLSD
RLCSD
OPDVR
LUFFY
UFT
GIFT
1.7B
AMC23
75.50
75.70
75.60
75.30
75.30
75.80
76.51
75.40
74.60
75.80 (+0.40%)
AIME24
44.44
45.28
45.56
44.72
44.72
46.94
46.67
48.33
48.33
48.61 (+9.38%)
AIME25
35.00
35.56
33.33
36.67
34.72
34.72
35.83
34.17
36.67
38.33 (+9.51%)
Avg.
51.65
52.18
51.50
52.23
51.58
52.49
53.00
52.63
53.20
54.25 (+5.03%)
4B
AMC23
88.45
89.16
88.25
88.96
88.86
90.26
88.45
89.16
89.66
88.55 (+0.11%)
AIME24
71.67
74.17
73.06
70.83
73.61
72.22
73.61
76.11
70.56
73.33 (+2.32%)
Table 1: Mathematical reasoning across Qwen3 scales (mean@12, %). Avg. is the mean over AMC23, AIME24, and AIME25. Green denotes relative gains over BASE; bold and underlined values indicate the best and second-best distinct scores. Bench. denotes benchmark.
Benchmark
BASE
GRPO
OPSD
SDPO
RLSD
RLCSD
OPDVR
LUFFY
UFT
GIFT
In-distribution (ID)
4–8 roles
72.40
73.40
72.80
75.20
73.60
74.00
73.00
74.60
74.20
73.40 (+1.38%)
Out-of-distribution (OOD)
9 roles
69.00
72.00
75.00
73.00
71.00
69.00
71.00
67.00
71.00
73.00 (+5.80%)
10 roles
59.00
56.00
59.00
56.00
55.00
59.00
57.00
59.00
56.00
61.00 (+3.39%)
11 roles
42.00
46.00
41.00
40.00
41.00
44.00
44.00
43.00
40.00
44.00 (+4.76%)
Table 2: Logical reasoning on Knights-and-Knaves (pass@1, %). The ID split contains 4–8 roles, while the OOD splits contain 9, 10, and 11 roles. Green denotes relative gains over BASE.
Method
Mathematical Reasoning (mean@12)
Logical Reasoning (pass@1)
AMC23
AIME24
AIME25
Avg.
ID (4–8)
OOD-9
OOD-10
OOD-11
Avg.
All Tokens
89.96
76.11
66.39
77.49
73.80
69.00
59.00
45.00
61.70
Random Tokens
89.66
75.28
66.67
77.20
73.60
71.00
58.00
46.00
62.15
Entropy-KL Selection
89.36
75.28
66.11
76.92
71.80
63.00
54.00
52.00
60.20
Gradient-cosine Selection
89.56
77.50
65.83
77.63
72.80
68.00
61.00
39.00
60.20
Full Model
89.66
76.11
71.11
78.96
73.40
73.00
61.00
44.00
62.85
Table 3: Ablation of token-selection strategies. Mathematical reasoning uses Qwen3-8B and is evaluated by mean@12. Logical reasoning uses Qwen3-4B and is evaluated by pass@1.
Table 5
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Notation
Type
Meaning
i,I
index / set
Token position i and the set I of all valid token positions.
zi,pi
vectors
Student logits and predictive distribution, where pi=softmax(zi) .
q~i
distribution
Privileged teacher distribution at position i .
ui,eui
token / vector
Sampled token and its corresponding one-hot vector.
wi,Ai,ρi,hi
scalars
Token weight, advantage, importance ratio, and active-branch indicator in the clipped GRPO objective.
ℓiGRPO
loss
Token-level clipped GRPO loss providing outcome-grounded supervision.
Appendix
Table 6: Summary of the main notations used in GIFT.
Figure 2: Results on OLMo3-7B, reported as mean@12. Avg. is the equally weighted average over the two benchmarks. Green numbers indicate absolute gains over Base.
Training schedule
AMC23
AIME24
AIME25
Avg.
OPSD → GRPO
89.06
75.28
67.78
77.37
Weight annealing
89.66
72.22
66.39
76.09
GIFT
89.66
76.11
71.11
78.96
Appendix
Table 7: Teacher-withdrawal schedules on Qwen3-8B (mean@12, %). Avg. equally weights the three benchmarks. Bold marks the best score in each column, including ties.
Schedule
Hyperparameters
Evaluation
H
Texit
AMC23
AIME24
AIME25
Avg.
Linear (GIFT)
10
78
89.66
76.11
71.11
78.96
Linear
0
78
89.36
75.56
69.17
78.03
Linear
20
78
89.56
75.28
66.94
77.26
Linear
10
50
90.16
76.39
65.83
77.46
Linear
10
90
89.86
75.00
68.06
77.64
Appendix
Table 8: Preliminary sensitivity to the full-coverage period H , teacher exit step Texit , and decay shape on Qwen3-8B (mean@12, %). Avg. equally weights the three benchmarks. Bold marks the best score in each evaluation column.
Figure 3: Example solutions to a combinatorial counting problem. The displayed GIFT solution is correct, while the GRPO and OPSD solutions contain counting errors. Correct-sample counts summarize 12 generations per method.
Figure 4: Token-level distillation selection in a reasoning excerpt. Red marks selected tokens and gray marks unselected tokens. The number of selected positions decreases from 69 to 40 to 17 across the displayed steps; counts refer only to the shown excerpt.
Figure 5: Token retention under coverage annealing. The horizontal and vertical axes show the signed parallel component ai and orthogonal residual norm bi of the distillation gradient relative to the GRPO direction. Colors indicate the fraction of eligible tokens retained within each bin. Annotations summarize retention for conflicting ( ai<0 ) and non-conflicting ( ai≥0 ) tokens with usable GRPO reference directions. Bottom bars separately report retention for neutral tokens without such a direction.
Reasoning distillation from powerful teacher models to smaller students faces the Gap Curse: as teachers grow more sophisticated, their complex distributions increasingly diverge from what students can approximate, causing performance degradation. Existing mitigation strategies either filter out challenging examples through data selection or introduce weaker intermediate assistant models, inherently compromising supervision coverage or quality. We propose Teacher Alignment, which directly adapts the teacher toward the student's distribution without discarding data or degrading reasoning quality. However, naive alignment through standard knowledge distillation triggers catastrophic collapse of the teacher's reasoning capabilities. To address this, we reformulate teacher alignment as reinforcement learning and introduce TeacherGRPO, built on Group Relative Policy Optimization with two key innovations: (i) Curriculum Selective Alignment applies dual token- and distribution-level curricula to focus rewards on high-signal reasoning gaps while filtering noise from trivial tokens and uncertain tail distributions, and (ii) Importance-Adaptive Length Regularization selectively penalizes verbose redundancy while preserving pedagogically critical reasoning steps. The aligned teacher then distills knowledge to students via standard pipelines. Extensive experiments show TeacherGRPO significantly outperforms baselines across diverse reasoning benchmarks and distillation methods. Our code is available at https://github.com/LzyFischer/TeacherGRPO.
Zhenyu Lei, Zihan Chen, Yaochen Zhu +5
University of Virginia · Netflix · University of Washington +2
Reinforcement learning (RL) post-training has recently driven major gains in long chain-of-thought reasoning large language models (LLMs), but the high inference cost of such models motivates distillation into smaller students. Most existing knowledge distillation (KD) methods are designed for supervised fine-tuning (SFT), relying on fixed teacher traces or teacher-student Kullback-Leibler (KL) divergence-based regularization. When combined with RL, these approaches often suffer from distribution mismatch and objective interference: teacher supervision may not align with the student's evolving rollout distribution, and the KL regularizer can compete with reward maximization and require careful loss balancing. To address these issues, we propose RL-aware distillation (RLAD), which performs selective imitation during RL -- guiding the student toward the teacher only when it improves the current policy update. Our core component, Trust Region Ratio Distillation (TRRD), replaces the teacher-student KL regularizer with a PPO/GRPO-style likelihood-ratio objective anchored to a teacher--old-policy mixture, yielding advantage-aware, trust-region-bounded distillation on student rollouts and naturally balancing exploration, exploitation, and imitation. Across diverse logic reasoning and math benchmarks, RLAD consistently outperforms offline distillation, standard GRPO, and KL-based on-policy teacher-student knowledge distillation.
Reasoning distillation aims to transfer multi-step reasoning capabilities from large language models to smaller, more efficient ones. While recent methods have shown promising gains, they typically rely on static teacher-student hierarchies and frame distillation as trajectory imitation. This is misaligned with the structure of reasoning, where intermediate steps are often locally under-specified: global correctness constrains the final answer, but does not uniquely determine each intermediate move. We propose validity-calibrated reasoning distillation, a framework that treats reasoning distillation as a problem of local learning-signal allocation rather than path alignment. Instead of enforcing token-level imitation, we compare the student's and teacher's proposed next-step actions under the same prefix and use their relative local validity to modulate the strength of the distillation update. This yields a dynamic, context-dependent supervision mechanism that preserves the teacher's structural guidance while adapting update strength to local reasoning quality. Across mathematical reasoning, code generation, and instruction-following benchmarks, our method consistently outperforms strong distillation baselines. These results indicate that effective LLM reasoning distillation is governed not by rigid trajectory imitation, but by principled, locally calibrated allocation of learning signal.
Khouloud Saadi, Di Wang
Department of Computer Science, KAUST, Thuwal, Saudi Arabia