Weaken to Strengthen: Weakening-Induced On-Policy Self-Distillation for Large Reasoning Models

Yecheng Wu; Song Han; Han Cai

cs.AIOct 28, 2026

Weaken to Strengthen: Weakening-Induced On-Policy Self-Distillation for Large Reasoning Models

Authors: Yecheng Wu, Song Han, Han Cai

Organizations: Massachusetts Institute of Technology · NVIDIA

Abstract

Self-distillation offers a way to improve reasoning models when a suitable external teacher is unavailable. However, common approaches rely on privileged information to construct a stronger self-teacher, limiting their use when only unlabeled prompts are available. In this paper, we introduce Weakening-Induced On-Policy Self-Distillation (WISD), which instead constructs a weaker counterpart of the original model and uses the contrast between the two as a training signal without requiring privileged information. We explore two ways to construct the weaker model: low-bit quantization and magnitude pruning. We evaluate WISD across diverse architectures and scales on mathematical and coding reasoning tasks. Across these benchmarks, quantization and pruning improve average accuracy over the original models by 4.0 and 5.5 percentage points, respectively, achieving state-of-the-art performance even without privileged information. Notably, pruning raises Qwen3.5-4B's accuracy from 57.7% to 70.5% on HMMT 2025 and from 39.1% to 56.7% on LiveCodeBench v6. Quantization raises Qwen3.6-35B-A3B's accuracy from 68.5% to 74.6% and from 61.8% to 69.5% on the same benchmarks, respectively. These results establish WISD as a novel and effective self-distillation framework for large reasoning models.

Figures & tables

Figure 1: Accuracy gains from self-distillation against a weakened model. By constructing a weakened copy of the original model and using the contrast between them as a training signal, WISD achieves consistent accuracy improvements across diverse reasoning models and benchmarks. Full results are reported in Table 2 .

Method	Dense Learning Signal	No External Teacher	No Reference Solutions	No Environment Feedback
GRPO ( Shao et al., 2024 )	✗	✓	✓	✗
OPD ( Agarwal et al., 2024 )	✓	✗	✓	✓
OPSD ( Zhao et al., 2026 )	✓	✓	✗	✓
SDPO ( Hübotter et al., 2026 )	✓	✓	✓	✗
WISD (ours)	✓	✓	✓	✓

Table 1: Comparison of training methods for reasoning. WISD provides dense supervision without external teachers, reference solutions, or environment feedback. Reference solutions are externally supplied answers or worked solutions used as teacher context. Environment feedback includes outcome verification and textual responses. GRPO uses verifiable outcome rewards. SDPO uses textual feedback or reward-verified self-generated solutions.

Figure 2: Overview of WISD. (a) Prior self-distillation. Privileged information

c

, such as solutions, demonstrations, or feedback, strengthens the original model into a self-teacher. (b) WISD. We instead retain the original model as the stronger member and construct a weaker copy through training-free quantization or pruning. The frozen pair scores the same self-generated trajectories, and its token-level log-probability contrast provides cached supervision for a student initialized from the original model, without privileged information.

Method	Math Reasoning				Code Generation			Avg.
	AIME24	AIME25	HMMT25	Avg.	LCB v5	LCB v6	Avg.
Qwen3-4B
Original	73.4	65.6	41.1	60.07	53.0	46.8	49.86	55.98
OPSD ( Zhao et al., 2026 )	75.2	66.7	43.9	61.92	42.7	35.1	38.88	52.70
SDPO ( Hübotter et al., 2026 )	70.6	62.8	40.8	58.06	40.9	34.4	37.61	49.88
PW-OPSD ( Liu et al., 2026 )	76.2	67.8	43.3	62.44	—	—	—	—

Table 2: Accuracy (%) across mathematics and code benchmarks. Training and evaluation settings are described in Section 4.1 . Bold and underlined values denote the best and second-best reported results per model. A dash denotes an unavailable result. Math Avg. averages three benchmarks, Code Avg. averages two, and the final Avg. weights all five equally. WISD achieves consistent improvements across models and benchmarks and remains competitive with other on-policy self-distillation methods without using any privileged information.

Figure 3: Ablations of weak-model construction on Qwen3.5-4B, evaluated on HMMT25. (a) Quantization bit width. (b) Pruning threshold. Each pair shows accuracy (Avg@32, left plot) and average generated length in thousands of tokens (right plot). Dashed lines denote Base accuracy and average generated length. Outlined diamonds mark the default settings

b=3

and

\tau=1/6

. Both constructions achieve their highest accuracy at intermediate weakening strengths. This pattern highlights the importance of balancing contrast strength and signal quality for effective self-distillation.

Method	AIME24	LCB v5
Qwen3-4B	73.4	53.0
Quantization
GPTQ (3-bit) ( Frantar et al., 2023 )	72.0	53.4
W4A4	70.0	53.0
Smooth W4A4 ( Xiao et al., 2023 )	75.3	55.2
RTN (3-bit)	75.4	56.4

Table 3: Qwen3-4B student accuracy (%) with weak-model constructions and simple baselines on AIME24 (Avg@32) and LCB v5 (Avg@4). Multiple quantization and pruning constructions improve accuracy on both tasks. Our default RTN and magnitude-pruning constructions outperform both simple baselines, supporting the value of structured contrast beyond confidence reinforcement. Bold marks the highest displayed accuracy within the quantization and pruning families.

Method	Read	Analyze	Plan	Implement	Explore	Verify	Monitor	Answer	Formatting
All tokens	9.1	24.3	6.6	28.2	14.8	8.1	1.4	1.8	5.6
Quantization	7.6	27.1	8.4	16.0	22.3	11.2	2.4	2.0	3.0
Pruning	7.8	26.7	8.3	15.9	22.1	11.4	2.5	2.1	3.2

Table 4: Distribution of initial training signals across reasoning roles (%). The first row gives token frequencies. Remaining rows give shares of logit-gradient L1 magnitude. Bold marks shares above token frequency. Quantization and pruning allocate about 22% of the signal to exploration and 11% to verification, exceeding these roles’ token shares of 14.8% and 8.1%. Both constructions thus place greater training weight on exploring solution paths and checking intermediate reasoning.

Figure 4: Token-level effects of WISD. Original-model trajectories illustrate a coverage check with quantization (a) and a zero-divisor check with pruning (b). Blue and orange indicate increased and decreased target probability relative to the behavior distribution. Quantization reinforces tokens supporting the coverage calculation and the conclusion that ten gallons are insufficient, while pruning reinforces the intermediate zero used to exclude an invalid root. These examples illustrate how weak-model contrasts provide local supervision for numerical checks and the logical decisions they support.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Rollout parameter	Value	Training parameter	Value
Prompts per domain	3,200	Distillation coefficient $\alpha$	2
Responses per prompt	4	Learning rate	$10^{-6}$ , constant
Trajectories per domain	12,800	Global batch size	64
Temperature / top- $p$	1.0 / 1.0	Optimizer updates	200
Default response limit	2,048 tokens	Student precision	BF16
Rollout seed	42	Training seed	1234

Appendix

Table 5: Training configuration for WISD. Mathematics and code students use separate prompt sets and cached trajectories.

Quantization			Pruning
Construction	Setting	Acc. (%)	Construction	Setting	Acc. (%)
Qwen3-4B	—	73.4	Qwen3-4B	—	73.4
RTN	6-bit, $g=128$	73.0	Magnitude pruning	20% per row	74.3
RTN	4-bit, $g=128$	72.0	Magnitude pruning	40% per row	74.4
RTN	4-bit, $g=32$	71.0	Wanda	20% per row	72.3
GPTQ	3-bit, $g=128$	72.0	Wanda	40% per row	74.4

Appendix

Table 6: Alternative constructions on Qwen3-4B and AIME24 (Avg@32). Default RTN and magnitude pruning (shaded) achieve the highest accuracy in their families.

g

denotes group size. Alternatives include GPTQ ( Frantar et al., 2023 ) , Smooth W4A4 ( Xiao et al., 2023 ) , Wanda ( Sun et al., 2024 ) , and SparseGPT ( Frantar and Alistarh, 2023 ) .

Construction	Qwen3-4B	Qwen3.5-4B
Construction	AIME25	HMMT25
Pruning w/o row-norm restoration	65.4	70.0
Pruning w/ row-norm restoration	66.0	70.5

Appendix

Table 7: Effect of row-norm restoration on the pruning construction. Accuracy (%) is measured with Avg@32. The default with restoration (shaded) gives a small additional gain on both models.

Role	Function
Read	Restates the given facts, conditions, or requested result.
Analyze	Explains a concept, builds a representation, or makes a logical deduction.
Plan	Commits to an action before carrying it out.
Implement	Executes calculations, substitutions, or other concrete operations.
Explore	Proposes a tentative approach or uncertain possibility.
Verify	Checks an existing result or approach against the problem’s conditions.

Appendix

Table 8: Contextual roles used for the training-signal analysis. The eight reasoning roles follow ThinkARM ( Li et al., 2026a ) . Labels reflect each segment’s function in the response.

S=-972+2=-970

Appendix

Figure 5: Reasoning improvements after self-distillation. On HMMT25, the quantization-trained Qwen3.5-4B student counts each symmetric divisor pair once, correcting the original model’s double counting (a). The pruning-trained student models the door constraint as an Eulerian circuit, allowing repeated visits to cells while traversing each selected door once (b). Excerpts are taken from the original and trained models’ responses to the same questions, with omissions marked by […].

for m in range(1, K_max + 1):

max_gap = 0

for g_list in gaps:

gap = g_list[m-1]

if gap > max_gap:

max_gap = gap

current_T += max_gap

Appendix

Figure 6: Code excerpts before and after self-distillation on LiveCodeBench v6. On Gravity (a), quantization-based training replaces an incorrect accumulation of inter-block gaps with the maximum initial height at each rank. On Switch Seats (b), pruning-based training preserves both nonadjacency conditions when translating the problem into code. Both post-training programs pass the corresponding test suites. The examples illustrate improved algorithmic reasoning and constraint checking.

Figure 7: Weak-model contrasts reinforce local reasoning steps. Blue and orange indicate increased and decreased target probability relative to the behavior distribution, with underlines marking the tokens discussed in the text. Quantization reinforces the negative sign in

p=-5

and the rejection of the resulting nonprime candidates (a). Pruning reinforces the zero used when substituting

x_{2}=x_{3}=1/2

and all other variables equal to zero, together with the comparison identifying a uniform four-variable allocation as worse (b). The highlighted steps connect numerical calculations to candidate rejection and comparisons between feasible assignments.

On-Policy Self-Distillation

On-policy self-distillation trains a model on its own generated trajectories using a teacher view to provide dense supervision.

See all 162 papers

Better Starts, Better Ends: Bootstrapped Iterative Self-Reasoning Distillation for Compressed Reasoning
Jul 17, 2026Leichao Dong, Dongxu Zhang, Yiding Sun +4CoT Distillation Efficient Language Model Reasoning
1Xi’an Jiaotong University · 2Peking University
Does self-distillation improve reasoning?
Oct 13, 2026Jason Ken Adhinarta, Raina Wu, Ashutosh Sharma +5Self-Distillation On-Policy Self-Distillation
Massachusetts Institute of Technology · MIT-IBM Computing Research Lab
CRISP: Compressed Reasoning via Iterative Self-Policy Distillation
Mar 5, 2026Hejian Sang, Yuanda Xu, Zhengze Zhou +3CoT Distillation Efficient Language Model Reasoning

Jul 17, 2026J/K move · Enter open · S save

Better Starts, Better Ends: Bootstrapped Iterative Self-Reasoning Distillation for Compressed Reasoning

Leichao Dong, Dongxu Zhang, Yiding Sun, Qirui Wang+3

1Xi’an Jiaotong University · 2Peking University

CoT Distillation Efficient Language Model Reasoning On-Policy Distillation LLM Reasoning

Open paper AskPDF arXiv

Large reasoning models often solve problems through long chain-of-thought (CoT) traces, yet much of this computation is spent on redundant derivations, repeated self-verification, and detours that do not improve the final answer. Existing on-policy self-distillation methods reduce this cost by matching a student model to a concise copy of itself on prefixes sampled from the student's own rollouts. We show that this objective has an initialization bottleneck. Since supervision is applied only to visited prefixes, training from a verbose base model places the KL loss on contexts that are often noisy, redundant, or already off track. In such regions, a concise teacher can provide only local corrections, while the student continues to explore trajectories that an efficient reasoner should avoid. In this paper, we propose BIRD(Bootstrapped Iterative Self-Reasoning Distillation), a two-stage self-reasoning distillation method that improves the rollout distribution before on-policy training. BIRD first samples concise solutions from the base model under a brevity instruction, keeps only answer-correct traces, and performs a lightweight prompt-switch SFT step. The traces are generated with the brevity instruction but learned under the original task prompt, turning instruction-induced conciseness into a default reasoning behavior. Starting from this warm model, BIRD then applies on-policy reverse-KL distillation with a concise self-teacher, now on cleaner and more informative prefixes. Across Qwen3 series models, BIRD achieves a stronger accuracy-efficiency trade-off than prompting and cold-start on-policy distillation on MATH-500 and AIME benchmarks. On Qwen3-8B, it improves MATH-500 accuracy from 86.2% to 92.0% while reducing the average response length from 3,099 to 1,115 tokens. These results highlight prefix support as a central factor in efficient reasoning distillation.

Abstract

On-Policy Self-Distillation

Explore similar work

Better Starts, Better Ends: Bootstrapped Iterative Self-Reasoning Distillation for Compressed Reasoning

Does self-distillation improve reasoning?

CRISP: Compressed Reasoning via Iterative Self-Policy Distillation