GIFT: Gradient Informed Filtering for Distillation Augmented Reasoning RL

Shijie Li; Yilin Gao; Tieyuan Chen; Yun Luo; Weiyao Lin; Hang Yu

cs.LGOct 22, 2026

GIFT: Gradient Informed Filtering for Distillation Augmented Reasoning RL

Authors: Shijie Li, Yilin Gao, Tieyuan Chen, Yun Luo, Weiyao Lin, Hang Yu

Organizations: Shanghai Jiao Tong University · Tencent

Abstract

Training large language models for complex reasoning tasks relies on two complementary signals: sparse, outcome-grounded reinforcement learning (RL) for correctness, and dense, token-level distillation for guidance. However, complementarity in supervision does not guarantee compatibility in optimization; imitation-focused distillation can conflict with the outcome-driven RL objective, and existing hybrids lack a principled resolution. We propose GIFT (Gradient-Informed FilTering for Distillation-Augmented Reasoning RL), a novel method that directly addresses the allocation problem of which teacher updates to apply. GIFT performs a token-level geometric analysis, decomposing the distillation gradient into a component parallel to the RL gradient, which measures first-order conflict and provides a safety filter, and a component orthogonal to it, which quantifies novel directional guidance and provides a measure of value. This decomposition yields a principled scoring rule to rank token-level guidance. We combine this with a progressive coverage annealing schedule that shrinks the budget of tokens receiving distillation, ensuring that the most conflicting updates are discarded first while the most complementary ones are retained at full strength. This facilitates a smooth transition from dense, geometrically-filtered guidance to a pure, outcome-grounded objective at convergence. Extensive experiments on mathematical and logical reasoning benchmarks show that GIFT consistently outperforms strong baselines across multiple model scales, achieving relative gains of up to 9.5% on difficult mathematical reasoning tasks. Code: https://github.com/lilifree/GIFT

Figures & tables

Figure 1: GIFT decomposes the distillation gradient into conflict ( $a_{i}$ ) and complement ( $b_{i}$ ) components relative to the RL gradient, ranks tokens, and anneals the selection budget from full to zero.

Size	Bench.	BASE	GRPO	OPSD	SDPO	RLSD	RLCSD	OPDVR	LUFFY	UFT	GIFT
1.7B	AMC23	75.50	75.70	75.60	75.30	75.30	75.80	76.51	75.40	74.60	75.80 (+0.40%)
	AIME24	44.44	45.28	45.56	44.72	44.72	46.94	46.67	48.33	48.33	48.61 (+9.38%)
	AIME25	35.00	35.56	33.33	36.67	34.72	34.72	35.83	34.17	36.67	38.33 (+9.51%)
	Avg.	51.65	52.18	51.50	52.23	51.58	52.49	53.00	52.63	53.20	54.25 (+5.03%)
4B	AMC23	88.45	89.16	88.25	88.96	88.86	90.26	88.45	89.16	89.66	88.55 (+0.11%)
	AIME24	71.67	74.17	73.06	70.83	73.61	72.22	73.61	76.11	70.56	73.33 (+2.32%)

Table 1: Mathematical reasoning across Qwen3 scales (mean@12, %). Avg. is the mean over AMC23, AIME24, and AIME25. Green denotes relative gains over BASE; bold and underlined values indicate the best and second-best distinct scores. Bench. denotes benchmark.

Benchmark	BASE	GRPO	OPSD	SDPO	RLSD	RLCSD	OPDVR	LUFFY	UFT	GIFT
In-distribution (ID)
4–8 roles	72.40	73.40	72.80	75.20	73.60	74.00	73.00	74.60	74.20	73.40 (+1.38%)
Out-of-distribution (OOD)
9 roles	69.00	72.00	75.00	73.00	71.00	69.00	71.00	67.00	71.00	73.00 (+5.80%)
10 roles	59.00	56.00	59.00	56.00	55.00	59.00	57.00	59.00	56.00	61.00 (+3.39%)
11 roles	42.00	46.00	41.00	40.00	41.00	44.00	44.00	43.00	40.00	44.00 (+4.76%)

Table 2: Logical reasoning on Knights-and-Knaves (pass@1, %). The ID split contains 4–8 roles, while the OOD splits contain 9, 10, and 11 roles. Green denotes relative gains over BASE.

Method	Mathematical Reasoning (mean@12)				Logical Reasoning (pass@1)
Method	AMC23	AIME24	AIME25	Avg.	ID (4–8)	OOD-9	OOD-10	OOD-11	Avg.
All Tokens	89.96	76.11	66.39	77.49	73.80	69.00	59.00	45.00	61.70
Random Tokens	89.66	75.28	66.67	77.20	73.60	71.00	58.00	46.00	62.15
Entropy-KL Selection	89.36	75.28	66.11	76.92	71.80	63.00	54.00	52.00	60.20
Gradient-cosine Selection	89.56	77.50	65.83	77.63	72.80	68.00	61.00	39.00	60.20
Full Model	89.66	76.11	71.11	78.96	73.40	73.00	61.00	44.00	62.85

Table 3: Ablation of token-selection strategies. Mathematical reasoning uses Qwen3-8B and is evaluated by mean@12. Logical reasoning uses Qwen3-4B and is evaluated by pass@1.

Table 5

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Notation	Type	Meaning
$i,\mathcal{I}$	index / set	Token position $i$ and the set $\mathcal{I}$ of all valid token positions.
$z_{i},\ p_{i}$	vectors	Student logits and predictive distribution, where $p_{i}=\operatorname{softmax}(z_{i})$ .
$\tilde{q}_{i}$	distribution	Privileged teacher distribution at position $i$ .
$u_{i},\ e_{u_{i}}$	token / vector	Sampled token and its corresponding one-hot vector.
$w_{i},\ A_{i},\ \rho_{i},\ h_{i}$	scalars	Token weight, advantage, importance ratio, and active-branch indicator in the clipped GRPO objective.
$\ell_{i}^{\mathrm{GRPO}}$	loss	Token-level clipped GRPO loss providing outcome-grounded supervision.

Appendix

Table 6: Summary of the main notations used in GIFT.

Figure 2: Results on OLMo3-7B, reported as mean@12. Avg. is the equally weighted average over the two benchmarks. Green numbers indicate absolute gains over Base.

Training schedule	AMC23	AIME24	AIME25	Avg.
OPSD $\rightarrow$ GRPO	89.06	75.28	67.78	77.37
Weight annealing	89.66	72.22	66.39	76.09
GIFT	89.66	76.11	71.11	78.96

Appendix

Table 7: Teacher-withdrawal schedules on Qwen3-8B (mean@12, %). Avg. equally weights the three benchmarks. Bold marks the best score in each column, including ties.

Schedule	Hyperparameters		Evaluation
Schedule	$H$	$T_{\mathrm{exit}}$	AMC23	AIME24	AIME25	Avg.
Linear (GIFT)	10	78	89.66	76.11	71.11	78.96
Linear	0	78	89.36	75.56	69.17	78.03
Linear	20	78	89.56	75.28	66.94	77.26
Linear	10	50	90.16	76.39	65.83	77.46
Linear	10	90	89.86	75.00	68.06	77.64

Appendix

Table 8: Preliminary sensitivity to the full-coverage period

H

, teacher exit step

T_{\mathrm{exit}}

, and decay shape on Qwen3-8B (mean@12, %). Avg. equally weights the three benchmarks. Bold marks the best score in each evaluation column.

Figure 3: Example solutions to a combinatorial counting problem. The displayed GIFT solution is correct, while the GRPO and OPSD solutions contain counting errors. Correct-sample counts summarize 12 generations per method.

Figure 4: Token-level distillation selection in a reasoning excerpt. Red marks selected tokens and gray marks unselected tokens. The number of selected positions decreases from 69 to 40 to 17 across the displayed steps; counts refer only to the shown excerpt.

Figure 5: Token retention under coverage annealing. The horizontal and vertical axes show the signed parallel component $a_{i}$ and orthogonal residual norm $b_{i}$ of the distillation gradient relative to the GRPO direction. Colors indicate the fraction of eligible tokens retained within each bin. Annotations summarize retention for conflicting ( $a_{i}<0$ ) and non-conflicting ( $a_{i}\geq 0$ ) tokens with usable GRPO reference directions. Bottom bars separately report retention for neutral tokens without such a direction.

RL for Language Model Reasoning

Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.

See all 1,094 papers

TeacherGRPO: Closing the Capacity Gap in Reasoning Distillation via Teacher Alignment
Sep 27, 2026Zhenyu Lei, Zihan Chen, Yaochen Zhu +5Reasoning Distillation RL for Language Model Reasoning
University of Virginia · Netflix · University of Washington +2
Reinforcement-aware Knowledge Distillation for LLM Reasoning
Feb 26, 2026Zhaoyang Zhang, Shuli Jiang, Yantao Shen +6Knowledge Distillation RL for Knowledge Distillation
AWS Agentic AI
Validity-Calibrated Reasoning Distillation
Apr 14, 2026Khouloud Saadi, Di WangReasoning Distillation Language Model Distillation
Department of Computer Science, KAUST, Thuwal, Saudi Arabia

Sep 27, 2026J/K move · Enter open · S save

TeacherGRPO: Closing the Capacity Gap in Reasoning Distillation via Teacher Alignment

Zhenyu Lei, Zihan Chen, Yaochen Zhu, Shangbin Feng+4

University of Virginia · Netflix · University of Washington+2

Reasoning Distillation RL for Language Model Reasoning Language Model Distillation Teacher-Student Learning

Open paper AskPDF arXiv

Reasoning distillation from powerful teacher models to smaller students faces the Gap Curse: as teachers grow more sophisticated, their complex distributions increasingly diverge from what students can approximate, causing performance degradation. Existing mitigation strategies either filter out challenging examples through data selection or introduce weaker intermediate assistant models, inherently compromising supervision coverage or quality. We propose Teacher Alignment, which directly adapts the teacher toward the student's distribution without discarding data or degrading reasoning quality. However, naive alignment through standard knowledge distillation triggers catastrophic collapse of the teacher's reasoning capabilities. To address this, we reformulate teacher alignment as reinforcement learning and introduce TeacherGRPO, built on Group Relative Policy Optimization with two key innovations: (i) Curriculum Selective Alignment applies dual token- and distribution-level curricula to focus rewards on high-signal reasoning gaps while filtering noise from trivial tokens and uncertain tail distributions, and (ii) Importance-Adaptive Length Regularization selectively penalizes verbose redundancy while preserving pedagogically critical reasoning steps. The aligned teacher then distills knowledge to students via standard pipelines. Extensive experiments show TeacherGRPO significantly outperforms baselines across diverse reasoning benchmarks and distillation methods. Our code is available at https://github.com/LzyFischer/TeacherGRPO.

Abstract

RL for Language Model Reasoning

Explore similar work

TeacherGRPO: Closing the Capacity Gap in Reasoning Distillation via Teacher Alignment

Reinforcement-aware Knowledge Distillation for LLM Reasoning

Validity-Calibrated Reasoning Distillation