Validity-Calibrated Reasoning Distillation

Khouloud Saadi; Di Wang

cs.LGApr 14, 2026

Validity-Calibrated Reasoning Distillation

Authors: Khouloud Saadi, Di Wang

Organizations: Department of Computer Science, KAUST, Thuwal, Saudi Arabia

Abstract

Reasoning distillation aims to transfer multi-step reasoning capabilities from large language models to smaller, more efficient ones. While recent methods have shown promising gains, they typically rely on static teacher-student hierarchies and frame distillation as trajectory imitation. This is misaligned with the structure of reasoning, where intermediate steps are often locally under-specified: global correctness constrains the final answer, but does not uniquely determine each intermediate move. We propose validity-calibrated reasoning distillation, a framework that treats reasoning distillation as a problem of local learning-signal allocation rather than path alignment. Instead of enforcing token-level imitation, we compare the student's and teacher's proposed next-step actions under the same prefix and use their relative local validity to modulate the strength of the distillation update. This yields a dynamic, context-dependent supervision mechanism that preserves the teacher's structural guidance while adapting update strength to local reasoning quality. Across mathematical reasoning, code generation, and instruction-following benchmarks, our method consistently outperforms strong distillation baselines. These results indicate that effective LLM reasoning distillation is governed not by rigid trajectory imitation, but by principled, locally calibrated allocation of learning signal.

Figures & tables

Figure 1: Distribution of the reward ratio $r_{s}/r_{t}$ between the student Qwen2.5-Math-1.5B ( $r_{s}$ ) and the teacher Qwen2.5-Math-7B-Instruct ( $r_{t}$ ) policies. $r_{s}$ and $r_{t}$ are computed with Skywork-o1-OpenPRM-Qwen-2.5-1.5B. While the ratio is expected to concentrate below 1, a substantial fraction of probability mass (28.8%) lies in the region $\geq 1$ , indicating frequent cases where the student attains higher reward than the teacher.

Figure 2: Overview of VCRD. Rather than enforcing uniform trajectory imitation, VCRD allocates token-level learning signal based on the relative local validity of teacher and student proposals under the same prefix. By modulating update strength rather than direction, the method preserves teacher guidance while adapting supervision to locally under-specified reasoning steps.

Qwen2-Math-7B-Inst ( $\mathcal{M}_{T}$ ) $\rightarrow$ Qwen2-Math-1.5B ( $\mathcal{M}_{S}$ )
Method	GSM8K	MATH	SVAMP	Minerva	Gaokao	Olympiad	AMC23	SAT-Math	AVG.
$\mathcal{M}_{T}$	88.40	74.90	94.30	30.90	64.70	37.60	60.00	100.0	68.85
$\mathcal{M}_{S}$	26.20	20.50	21.70	6.60	16.10	7.40	10.00	81.20	23.71
KD	80.10	61.00	84.30	26.50	51.90	23.90	27.50	40.60	49.48
GKD	81.00	61.10	84.70	25.00	52.70	23.70	25.00	78.10	53.91
DistilLLM	80.90	61.00	85.80	27.20	50.90	23.70	37.50	68.80	54.47

Table 1: Pass@1 results for distilling Qwen2-Math-7B-Instruct (

\mathcal{M}_{T}

) into Qwen2-Math-1.5B (

\mathcal{M}_{S}

) on eight math-reasoning benchmarks. AVG is the average over all datasets.

Qwen2.5-Math-7B-Inst ( $\mathcal{M}_{T}$ ) $\rightarrow$ Qwen2.5-Math-1.5B ( $\mathcal{M}_{S}$ )
Method	GSM8K	MATH	Gaokao	Olympiad	CMATH	AIME24	AMC23	AVG.
$\mathcal{M}_{T}$	85.10	76.30	66.50	37.00	89.80	16.70	60.00	61.63
$\mathcal{M}_{S}$	79.10	51.10	42.10	16.90	62.00	3.30	22.50	39.57
KD	85.50	73.70	62.90	32.40	88.20	10.00	47.50	57.17
GKD	85.50	74.10	60.00	33.30	88.80	6.70	50.00	56.91
DistilLLM	84.90	74.40	60.30	33.50	89.30	13.30	47.50	57.60

Table 2: Pass@1 results for distilling Qwen2.5-Math-7B-Instruct (

\mathcal{M}_{T}

) into Qwen2.5-Math-1.5B (

\mathcal{M}_{S}

) across seven math-reasoning benchmarks. AVG denotes the average score across all datasets.

Qwen2.5-Coder-7B-Inst ( $\mathcal{M}_{T}$ ) $\rightarrow$ Qwen2.5-Coder-1.5B ( $\mathcal{M}_{S}$ )
Method	HEval	HEval $+$	MBPP	MBPP $+$	AVG
$\mathcal{M}_{T}$	91.50	86.00	82.00	70.40	82.47
$\mathcal{M}_{S}$	69.50	64.00	73.00	61.40	66.97
KD	68.90	61.00	72.80	61.90	66.15
GKD	68.30	62.80	73.30	63.20	66.90
DistilLLM	68.30	62.80	73.30	62.70	66.77

Table 3: Pass@1 results on HumanEval and MBPP benchmarks for two Qwen2.5-Coder distillation settings. AVG denotes the average Pass@1 across the four tasks.

Method	Qwen2.5-7B-Inst ( $\mathcal{M}_{T}$ ) $\rightarrow$ Qwen2.5-1.5B ( $\mathcal{M}_{S}$ )				Qwen2-7B-Inst ( $\mathcal{M}_{T}$ ) $\rightarrow$ Qwen2-1.5B ( $\mathcal{M}_{S}$ )
Method	AlpacaEval	Evol-Inst	UltraFeed	AVG.	AlpacaEval	Evol-Inst	UltraFeed	AVG.
$\mathcal{M}_{T}$	91.43	69.67	77.16	79.42	88.54	59.75	64.86	71.05
$\mathcal{M}_{S}$	60.09	30.73	42.94	44.59	56.11	19.49	38.20	37.93
KD	72.14	54.59	62.60	63.11	60.81	32.45	49.25	47.50
SeqKD	67.92	46.33	52.89	55.71	56.58	28.32	39.10	41.33
DistilLLM	75.11	55.38	65.92	65.47	64.13	35.66	51.26	50.35

Table 4: Win Rate (WR%) on instruction-following datasets for Qwen2.5-7B-Instruct

\rightarrow

Qwen2.5-1.5B and Qwen2-7B-Instruct

\rightarrow

Qwen2-1.5B. AVG is the average WR across all benchmarks.

Figure 3: Qwen2.5-Math-7B-Inst $\rightarrow$ Qwen2.5-Math-1.5B (left) and Qwen2.5-Coder-7B-Inst $\rightarrow$ Qwen2.5-Coder-1.5B (right).

Qwen2.5-Math-7B-Instruct ( $\mathcal{M}_{T}$ ) $\rightarrow$ Qwen2.5-Math-1.5B ( $\mathcal{M}_{S}$ )
Method	GSM8K	MATH	Gaokao	Olympiad	CMATH	AIME24	AMC23	AVG
LV-SKL ( $w^{T}=1$ )	84.50	73.80	62.30	34.70	86.30	6.70	47.50	56.54
LV-SRKL ( $w^{S}=1$ )	85.70	72.90	59.00	33.90	88.00	6.70	47.50	56.24
LV-SKL+LV-SRKL ( $w^{T}=w^{S}=1$ )	85.50	73.20	61.30	34.10	90.20	13.30	47.50	57.87
LV-SKL	83.70	73.80	62.60	34.70	87.50	10.10	60.00	58.91
LV-SRKL	86.60	72.80	61.00	35.10	88.70	10.00	42.50	56.67

Table 5: Contribution of validity calibration across mathematical reasoning and code generation. SRKL (

w^{S}=1

) denotes pure on-policy distillation with student rollouts, per-token reverse-KL supervision, and no validity modulation. SKL+SRKL (

w^{T}=w^{S}=1

) is the constant-weight mixed-prefix baseline.

Method	HEval	HEval $+$	AVG
$\mathcal{M}_{T}$	77.40	70.70	74.05
$\mathcal{M}_{S}$	35.40	29.30	32.35
KD	42.10	37.20	39.65
SeqKD	41.50	36.00	38.75
GKD	40.09	36.00	38.05
DistilLLM	42.10	38.40	40.25

Table 6: Pass@1 on HumanEval for DS-Coder-6.9B

\rightarrow

DS-Coder-1.3B.

Method	GSM8K	MATH	Gaokao	Olympiad	AVG
Distillm-2	85.70	73.20	61.60	34.10	63.65
VCRD-Prob	84.80	73.90	64.20	33.60	64.13

Table 7: Pass@1 results on four math-reasoning benchmarks. AVG is the mean across the four tasks. VCRD-Prob represents the PRM-free approach. Qwen-2.5-Math-7B-Instruct

\rightarrow

Qwen-2.5-Math-1.5B

Shared prefix	Teacher continuation	Student continuation	$\boldsymbol{r_{T}}$	$\boldsymbol{r_{S}}$	Ratio	Better Local continuation
The cost of printing is $7$ copies $\times$ $25$ pages $\times$ $0.10/page$ =$17$	$550$	$.50$	0.085	0.589	6.97	Student . The correct amount is $17.50; the student completes the decimal.
Tim buys 3 goats for $400 each. Tim also buys twice as many llamas as goats, so he buys	$2\times 2=6$ llamas	$3\times 2=6$ llamas	0.152	0.847	5.57	Student . The multiplier should be the number of goats, 3, so $3\times 2=6$ .
Cost per liter is $3 and the pool volume is$ 25\times 60x $. Thus,$ 3\times 25\times 60x=90000 $. Simplify:$ 3\times$	$15\times 60x$	$25\times 60x$	0.575	0.801	1.39	Student . The coefficient 25 should be preserved from $3\times 25\times 60x$ .
Find $1^{234}+4^{6}\div 4^{4}$ . First, we simplify	$1^{6}\div 4^{4}$	$4^{6}\div 4^{4}$	0.601	0.877	1.46	Student . The expression being simplified is $4^{6}\div 4^{4}$ , not $1^{6}\div 4^{4}$ .
James drives at 30 mph for half an hour. Then he drives at twice the speed for twice as long, which means he drives at a speed of	$3\times 30=6\cdots$	$2\times 30=6\cdots$	0.360	0.827	2.30	Student . “Twice the speed” requires multiplying by 2, giving $2\times 30=60$ mph.

Table 8: Qualitative examples where the student Qwen2.5-Math-1.5B continuation receives higher local validity than the teacher Qwen2.5-Math-7B-Instruct continuation under the same prefix.

r_{T}

and

r_{S}

denote the PRM (Skywork-o1-OpenPRM-Qwen-2.5-1.5B) rewards for the teacher and student first token continuations, respectively, and the ratio is

r_{S}/r_{T}

. The “Better Local Continuation” column indicates which continuation is locally correct or a valid better alternative under the shared prefix between teacher and student models.

Hyperparameter	Value
Finetuning method	LoRA (rank = 16, $\alpha$ = 128, dropout = 0.05)
LoRA target modules	all self-attention and MLP layers
Learning rate	$5\times 10^{-5}$
Max sequence length	1024
Max prompt length	512
Effective batch size	128

Table 9: Hyperparameter values used in VCRD (distillation stage) experiments across all task families.

Qwen2.5-Math-7B-Instruct ( $\mathcal{M}_{T}$ ) $\rightarrow$ Qwen2.5-Math-1.5B ( $\mathcal{M}_{S}$ )
Method	GSM8K	MATH	Gaokao	Olympiad	CMATH	AIME24	AMC23	AVG
LV-Joint- $r_{s}$	85.6	73.8	60.5	32.7	90.0	6.7	47.5	56.68
LV-Joint-( $r_{s}-r_{t}$ )	85.2	73.6	61.8	34.2	89.2	10.0	52.5	58.03
VCRD	85.0	73.7	62.1	35.4	90.2	13.3	60.0	59.96

Table 10: Ablation of validity-weighting rules on Qwen2.5-Math-7B-Instruct

\rightarrow

Qwen2.5-Math-1.5B. AVG is the average over all datasets.

Method	Time (s/iteration)
KD	30.19
Distillm-2	32.84
VCRD (PRM-based)	33.84
VCRD-Prob (PRM-free)	32.31

Table 11: Average training time per iteration on Qwen2.5-Math-7B-Instruct (

\mathcal{M}_{T}

)

\rightarrow

Qwen2.5-Math-1.5B (

\mathcal{M}_{S}

PRM	GSM8K	MATH	SVAMP	Minerva	Gaokao	Olympiad	AVG
VCRD (Skywork-o1-Open-PRM-1.5B )	81.0	61.8	85.9	25.7	53.2	24.7	55.38
VCRD (Qwen2.5-Math-PRM-7B)	81.6	60.8	85.5	27.2	53.5	24.3	55.48
SKL+SRKL	81.6	61.0	85.9	25.7	51.4	21.9	54.58
SRKL (on-policy KD)	78.0	57.6	83.6	17.3	47.5	21.9	50.98

Table 12: PRM sensitivity results.Qwen2.5-Math-7B-Instruct (

\mathcal{M}_{T}

)

\rightarrow

Qwen2.5-Math-1.5B (

\mathcal{M}_{S}

Reasoning Distillation

Reasoning distillation transfers step-by-step problem-solving skills from powerful teachers to smaller, more efficient models.

See all 22 papers

Beyond Scaling Law: A Data-Efficient Distillation Framework for Reasoning
Aug 13, 2025Xiaojun Wu, Xiaoguang Jiang, Huiyang Li +11CoT Distillation Language Model Distillation
1Zhongxing Telecom Equipment(ZTE), China · 2China Mobile Communications Group Co Ltd
Collaborative Reasoning Distillation via Cross-Feedback and Coherent Curation
Oct 7, 2026Taehoon Kim, Seunggeun Cho, Dongsu HanReasoning Distillation LLM Reasoning
KAIST
Distribution Corrected Offline Data Distillation for Large Language Models
May 13, 2026Yumeng Zhang, Zhengbang Yang, Yevin Nikhel Goonatilake +1CoT Distillation Online Knowledge Distillation
George Mason University, Fairfax, VA, USA

Aug 13, 2025J/K move · Enter open · S save

Beyond Scaling Law: A Data-Efficient Distillation Framework for Reasoning

Xiaojun Wu, Xiaoguang Jiang, Huiyang Li, Jucai Zhai+10

1Zhongxing Telecom Equipment(ZTE), China · 2China Mobile Communications Group Co Ltd

CoT Distillation Language Model Distillation Algorithmic Reasoning LLM Reasoning Data-Efficient Learning

Open paper AskPDF arXiv

Large language models (LLMs) demonstrate remarkable reasoning capabilities in tasks such as algorithmic coding and mathematical problem-solving. Recent methods have improved reasoning through expanded corpus and multistage training combining reinforcement learning and supervised fine-tuning. Although some methods suggest that small but targeted dataset can incentivize reasoning via only distillation, a reasoning scaling laws is still taking shape, increasing computational costs. To address this, we propose a data-efficient distillation framework (DED) that optimizes the Pareto frontier of reasoning distillation. Inspired by the on-policy learning and diverse roll-out strategies of reinforcement learning, the key idea of our approach is threefold: (1) We identify that benchmark scores alone do not determine an effective teacher model. Through comprehensive comparisons of leading reasoning LLMs, we develop a method to select an optimal teacher model. (2) While scaling distillation can enhance reasoning, it often degrades out-of-domain performance. A carefully curated, smaller corpus achieves a balanced trade-off between in-domain and out-of-domain capabilities. (3) Diverse reasoning trajectories encourage the student model to develop robust reasoning skills. We validate our method through evaluations on mathematical reasoning (AIME 2024/2025, MATH-500) and code generation (LiveCodeBench), achieving state-of-the-art results with only 0.8k carefully curated examples, bypassing the need for extensive scaling. Our systematic analysis demonstrates that DED outperforms existing methods by considering factors beyond superficial hardness, token length, or teacher model capability. This work offers a practical and efficient pathway to advanced reasoning while preserving general capabilities.

Abstract

Reasoning Distillation

Explore similar work

Beyond Scaling Law: A Data-Efficient Distillation Framework for Reasoning

Collaborative Reasoning Distillation via Cross-Feedback and Coherent Curation

Distribution Corrected Offline Data Distillation for Large Language Models