cs.AIOct 28, 2026

Weaken to Strengthen: Weakening-Induced On-Policy Self-Distillation for Large Reasoning Models

Authors: Yecheng Wu, Song Han, Han Cai

Organizations: Massachusetts Institute of Technology · NVIDIA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Self-distillation offers a way to improve reasoning models when a suitable external teacher is unavailable. However, common approaches rely on privileged information to construct a stronger self-teacher, limiting their use when only unlabeled prompts are available. In this paper, we introduce Weakening-Induced On-Policy Self-Distillation (WISD), which instead constructs a weaker counterpart of the original model and uses the contrast between the two as a training signal without requiring privileged information. We explore two ways to construct the weaker model: low-bit quantization and magnitude pruning. We evaluate WISD across diverse architectures and scales on mathematical and coding reasoning tasks. Across these benchmarks, quantization and pruning improve average accuracy over the original models by 4.0 and 5.5 percentage points, respectively, achieving state-of-the-art performance even without privileged information. Notably, pruning raises Qwen3.5-4B's accuracy from 57.7% to 70.5% on HMMT 2025 and from 39.1% to 56.7% on LiveCodeBench v6. Quantization raises Qwen3.6-35B-A3B's accuracy from 68.5% to 74.6% and from 61.8% to 69.5% on the same benchmarks, respectively. These results establish WISD as a novel and effective self-distillation framework for large reasoning models.

Figures & tables

Figure 1: Accuracy gains from self-distillation against a weakened model. By constructing a weakened copy of the original model and using the contrast between them as a training signal, WISD achieves consistent accuracy improvements across diverse reasoning models and benchmarks. Full results are reported in Table 2 .
MethodDense Learning SignalNo External TeacherNo Reference SolutionsNo Environment Feedback
GRPO ( Shao et al., 2024 )✗✓✓✗
OPD ( Agarwal et al., 2024 )✓✗✓✓
OPSD ( Zhao et al., 2026 )✓✓✗✓
SDPO ( Hübotter et al., 2026 )✓✓✓✗
WISD (ours)✓✓✓✓
Table 1: Comparison of training methods for reasoning. WISD provides dense supervision without external teachers, reference solutions, or environment feedback. Reference solutions are externally supplied answers or worked solutions used as teacher context. Environment feedback includes outcome verification and textual responses. GRPO uses verifiable outcome rewards. SDPO uses textual feedback or reward-verified self-generated solutions.
Figure 2: Overview of WISD. (a) Prior self-distillation. Privileged information cc , such as solutions, demonstrations, or feedback, strengthens the original model into a self-teacher. (b) WISD. We instead retain the original model as the stronger member and construct a weaker copy through training-free quantization or pruning. The frozen pair scores the same self-generated trajectories, and its token-level log-probability contrast provides cached supervision for a student initialized from the original model, without privileged information.
MethodMath ReasoningCode GenerationAvg.
 AIME24AIME25HMMT25Avg.LCB v5LCB v6Avg. 
Qwen3-4B
Original73.465.641.160.0753.046.849.8655.98
OPSD ( Zhao et al., 2026 )75.266.743.961.9242.735.138.8852.70
SDPO ( Hübotter et al., 2026 )70.662.840.858.0640.934.437.6149.88
PW-OPSD ( Liu et al., 2026 )76.267.843.362.44————
Table 2: Accuracy (%) across mathematics and code benchmarks. Training and evaluation settings are described in Section 4.1 . Bold and underlined values denote the best and second-best reported results per model. A dash denotes an unavailable result. Math Avg. averages three benchmarks, Code Avg. averages two, and the final Avg. weights all five equally. WISD achieves consistent improvements across models and benchmarks and remains competitive with other on-policy self-distillation methods without using any privileged information.
Figure 3: Ablations of weak-model construction on Qwen3.5-4B, evaluated on HMMT25. (a) Quantization bit width. (b) Pruning threshold. Each pair shows accuracy (Avg@32, left plot) and average generated length in thousands of tokens (right plot). Dashed lines denote Base accuracy and average generated length. Outlined diamonds mark the default settings b=3b=3 and τ=1/6\tau=1/6 . Both constructions achieve their highest accuracy at intermediate weakening strengths. This pattern highlights the importance of balancing contrast strength and signal quality for effective self-distillation.
MethodAIME24LCB v5
Qwen3-4B73.453.0
Quantization
GPTQ (3-bit) ( Frantar et al., 2023 )72.053.4
W4A470.053.0
Smooth W4A4 ( Xiao et al., 2023 )75.355.2
RTN (3-bit)75.456.4
Table 3: Qwen3-4B student accuracy (%) with weak-model constructions and simple baselines on AIME24 (Avg@32) and LCB v5 (Avg@4). Multiple quantization and pruning constructions improve accuracy on both tasks. Our default RTN and magnitude-pruning constructions outperform both simple baselines, supporting the value of structured contrast beyond confidence reinforcement. Bold marks the highest displayed accuracy within the quantization and pruning families.
MethodReadAnalyzePlanImplementExploreVerifyMonitorAnswerFormatting
All tokens9.124.36.628.214.88.11.41.85.6
Quantization7.627.18.416.022.311.22.42.03.0
Pruning7.826.78.315.922.111.42.52.13.2
Table 4: Distribution of initial training signals across reasoning roles (%). The first row gives token frequencies. Remaining rows give shares of logit-gradient L1 magnitude. Bold marks shares above token frequency. Quantization and pruning allocate about 22% of the signal to exploration and 11% to verification, exceeding these roles’ token shares of 14.8% and 8.1%. Both constructions thus place greater training weight on exploring solution paths and checking intermediate reasoning.
Figure 4: Token-level effects of WISD. Original-model trajectories illustrate a coverage check with quantization (a) and a zero-divisor check with pruning (b). Blue and orange indicate increased and decreased target probability relative to the behavior distribution. Quantization reinforces tokens supporting the coverage calculation and the conclusion that ten gallons are insufficient, while pruning reinforces the intermediate zero used to exclude an invalid root. These examples illustrate how weak-model contrasts provide local supervision for numerical checks and the logical decisions they support.
Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Rollout parameterValueTraining parameterValue
Prompts per domain3,200Distillation coefficient α\alpha2
Responses per prompt4Learning rate10−610^{-6} , constant
Trajectories per domain12,800Global batch size64
Temperature / top- pp1.0 / 1.0Optimizer updates200
Default response limit2,048 tokensStudent precisionBF16
Rollout seed42Training seed1234
Appendix
Table 5: Training configuration for WISD. Mathematics and code students use separate prompt sets and cached trajectories.
QuantizationPruning
ConstructionSettingAcc. (%)ConstructionSettingAcc. (%)
Qwen3-4B—73.4Qwen3-4B—73.4
RTN6-bit, g=128g=12873.0Magnitude pruning20% per row74.3
RTN4-bit, g=128g=12872.0Magnitude pruning40% per row74.4
RTN4-bit, g=32g=3271.0Wanda20% per row72.3
GPTQ3-bit, g=128g=12872.0Wanda40% per row74.4
Appendix
Table 6: Alternative constructions on Qwen3-4B and AIME24 (Avg@32). Default RTN and magnitude pruning (shaded) achieve the highest accuracy in their families. gg denotes group size. Alternatives include GPTQ ( Frantar et al., 2023 ) , Smooth W4A4 ( Xiao et al., 2023 ) , Wanda ( Sun et al., 2024 ) , and SparseGPT ( Frantar and Alistarh, 2023 ) .
ConstructionQwen3-4BQwen3.5-4B
AIME25HMMT25
Pruning w/o row-norm restoration65.470.0
Pruning w/ row-norm restoration66.070.5
Appendix
Table 7: Effect of row-norm restoration on the pruning construction. Accuracy (%) is measured with Avg@32. The default with restoration (shaded) gives a small additional gain on both models.
RoleFunction
ReadRestates the given facts, conditions, or requested result.
AnalyzeExplains a concept, builds a representation, or makes a logical deduction.
PlanCommits to an action before carrying it out.
ImplementExecutes calculations, substitutions, or other concrete operations.
ExploreProposes a tentative approach or uncertain possibility.
VerifyChecks an existing result or approach against the problem’s conditions.
Appendix
Table 8: Contextual roles used for the training-signal analysis. The eight reasoning roles follow ThinkARM ( Li et al., 2026a ) . Labels reflect each segment’s function in the response.
 S=−972+2=−970S=-972+2=-970 
Appendix
Figure 5: Reasoning improvements after self-distillation. On HMMT25, the quantization-trained Qwen3.5-4B student counts each symmetric divisor pair once, correcting the original model’s double counting (a). The pruning-trained student models the door constraint as an Eulerian circuit, allowing repeated visits to cells while traversing each selected door once (b). Excerpts are taken from the original and trained models’ responses to the same questions, with omissions marked by […].
for m in range(1, K_max + 1):
max_gap = 0
for g_list in gaps:
gap = g_list[m-1]
if gap > max_gap:
max_gap = gap
current_T += max_gap
Appendix
Figure 6: Code excerpts before and after self-distillation on LiveCodeBench v6. On Gravity (a), quantization-based training replaces an incorrect accumulation of inter-block gaps with the maximum initial height at each rank. On Switch Seats (b), pruning-based training preserves both nonadjacency conditions when translating the problem into code. Both post-training programs pass the corresponding test suites. The examples illustrate improved algorithmic reasoning and constraint checking.
Figure 7: Weak-model contrasts reinforce local reasoning steps. Blue and orange indicate increased and decreased target probability relative to the behavior distribution, with underlines marking the tokens discussed in the text. Quantization reinforces the negative sign in p=−5p=-5 and the rejection of the resulting nonprime candidates (a). Pruning reinforces the zero used when substituting x2=x3=1/2x_{2}=x_{3}=1/2 and all other variables equal to zero, together with the comparison identifying a uniform four-variable allocation as worse (b). The highlighted steps connect numerical calculations to candidate rejection and comparisons between feasible assignments.
More in this topic

On-Policy Self-Distillation

On-policy self-distillation trains a model on its own generated trajectories using a teacher view to provide dense supervision.

See all 161 papers

Explore similar work

CardsList
Jul 17, 2026

Better Starts, Better Ends: Bootstrapped Iterative Self-Reasoning Distillation for Compressed Reasoning

Large reasoning models often solve problems through long chain-of-thought (CoT) traces, yet much of this computation is spent on redundant derivations, repeated self-verification, and detours that do not improve the final answer. Existing on-policy self-distillation methods reduce this cost by matching a student model to a concise copy of itself on prefixes sampled from the student's own rollouts. We show that this objective has an initialization bottleneck. Since supervision is applied only to visited prefixes, training from a verbose base model places the KL loss on contexts that are often noisy, redundant, or already off track. In such regions, a concise teacher can provide only local corrections, while the student continues to explore trajectories that an efficient reasoner should avoid. In this paper, we propose BIRD(Bootstrapped Iterative Self-Reasoning Distillation), a two-stage self-reasoning distillation method that improves the rollout distribution before on-policy training. BIRD first samples concise solutions from the base model under a brevity instruction, keeps only answer-correct traces, and performs a lightweight prompt-switch SFT step. The traces are generated with the brevity instruction but learned under the original task prompt, turning instruction-induced conciseness into a default reasoning behavior. Starting from this warm model, BIRD then applies on-policy reverse-KL distillation with a concise self-teacher, now on cleaner and more informative prefixes. Across Qwen3 series models, BIRD achieves a stronger accuracy-efficiency trade-off than prompting and cold-start on-policy distillation on MATH-500 and AIME benchmarks. On Qwen3-8B, it improves MATH-500 accuracy from 86.2% to 92.0% while reducing the average response length from 3,099 to 1,115 tokens. These results highlight prefix support as a central factor in efficient reasoning distillation.

Leichao Dong, Dongxu Zhang, Yiding Sun +4

1Xi’an Jiaotong University · 2Peking University

CoT DistillationEfficient Language Model ReasoningOn-Policy DistillationLLM Reasoning
AskPDF
Oct 13, 2026

Does self-distillation improve reasoning?

Self-distillation is an emerging approach that trains a model to imitate its own predictions when conditioned on privileged information, such as correct answers or environmental feedback. Recent methods have reported better sample efficiency and generalization than reinforcement learning (RL), but it remains unclear whether these gains extend beyond in-distribution evaluations and whether they stem from privileged distillation. In this paper, we revisit two representative methods, Self-Distillation Policy Optimization (SDPO) and On-Policy Self-Distillation (OPSD), across code, science, tool use, and math reasoning tasks. We find that their reported gains do not consistently generalize to unseen problems and are largely explained by overfitting or transferring existing reasoning behavior, rather than effectively using privileged information. Moreover, self-distillation gradients show weak alignment with policy gradients that directly optimize expected reward, questioning whether they provide an effective direction for policy improvement. Our experiments across Qwen3 models (1.7B-14B) and OLMo-3-7B suggest that effectively leveraging privileged information to improve reasoning remains an open challenge. Our code is available at https://github.com/sleepypandu/does-sd-improve-reasoning

Jason Ken Adhinarta, Raina Wu, Ashutosh Sharma +5

Massachusetts Institute of Technology · MIT-IBM Computing Research Lab

Self-DistillationOn-Policy Self-DistillationKnowledge DistillationRL for Language Model Reasoning
AskPDF
Mar 5, 2026

CRISP: Compressed Reasoning via Iterative Self-Policy Distillation

Reasoning models often generate far more tokens than a task requires, which raises inference cost and can compound errors. We introduce CRISP (Compressed Reasoning via Iterative Self-Policy Distillation), an on-policy self-distillation method that teaches a model to reason more concisely by distilling its own concise behavior back into itself. The method uses a single idea: condition the same model on a "be concise" instruction to obtain teacher logits, then minimize the per-token reverse KL divergence between the student and this teacher on the student's own rollouts. It requires no ground-truth answers, no token budgets, and no difficulty estimators. The reverse-KL objective is naturally difficulty-adaptive: it compresses easy problems aggressively while preserving the reasoning steps that hard problems require. On Qwen3-14B, CRISP cuts reasoning length by up to 56% on MATH-500 and 38% on the harder AIME 2024, while improving MATH-500 accuracy by up to 3.3 points over the base model and holding AIME 2024 accuracy within about one point. This behavior generalizes across model sizes and families: Qwen3-8B shows the same compression with accuracy preserved, and DeepSeek-R1-Distill-Llama-8B improves accuracy on all five benchmarks while shortening its responses. General capabilities are preserved across all three models. Code is available at https://github.com/HJSang/OPSD_Reasoning_Compression.

Hejian Sang, Yuanda Xu, Zhengze Zhou +3

CoT DistillationEfficient Language Model ReasoningOn-Policy DistillationCoT CompressionLLM Reasoning
AskPDF