Steer, Don't Tell: Breaking Reasoning Plateaus with Reachability-Aligned Self-Distillation
Authors: Zijian Wu, Lingkai Kong, Lin Yang, Haiteng Zhao, Kuikun Liu, Kai Chen, Dahua Lin, Wenwei Zhang
Organizations: Department of Information Engineering, The Chinese University of Hong Kong · Shanghai Artificial Intelligence Laboratory · Shanghai Jiao Tong University
Post training for large reasoning models remains insufficient on challenging problems, whose correct trajectories are hard to sample. Though prior works address this difficulty by integrating supervised fine tuning on expert trajectories with reinforcement learning, such external experts are not always available. Existing self improvement methods instead use reference solutions to create a privileged self-teacher. Yet privileged teacher correctness does not imply student reachability. Solution conditioning shifts the teacher distribution toward actions unlikely under the unprivileged student, producing seemingly "correct" trajectories that are unreachable or even mal-formed learning targets. Accordingly, we identify reachability alignment as the key to self constructed supervision and propose Self-Steered Reasoner (SSR). SSR realizes this principle through student support projection, which transforms the privileged teacher distribution into a directional signal only over reasoning actions locally reachable under the unguided student. Verification then retains correct trajectories sampled from this reachable space. To further boost the sampling efficiency, an iterative refinement framework is adopted, using diagnostic guidance based on current trajectories. On problems with pass@64 = 0 even after extensive RL, SSR recovers correct and learnable trajectories for 74.6% without parameter updates. Fine tuning on these trajectories followed by resumed reinforcement learning improves aggregate accuracy by 4.9 percentage points on AIME25/26 and HMMT26. These results show that privileged guidance becomes effective self-improvement data when aligned with the reasoning actions the student can reach.
Figures & tables
Figure 1: RLVR can extend the model’s ability in its comfort zone, but it meets obstacles when coming to anxiety zone where correct trajectories are hard to sample. Privileged-teacher can provide aid signals on anxiety zone, but it can also select actions outside the student’s reachable region, producing unlearnable traces. Directly imitating these actions can corrupt the student’s reasoning ability(e.g., learning to hallucinate a non-existent hint). By projecting privileged preferences into the consensus region, SSR retains the learnable steering signals within the privileged preferences, effectively expands the student’s learning boundary.
Figure 2: Overview of the SSR framework. 1. Given a very hard question Q, we sample failed rollouts from the student model. The failed rollout is then summarized and diagnosed against the reference solution to produce a self-steer. The self-steer describes the error and provide fixing directions while avoiding directly offering the answer. 2. The self-steer is then used to construct the privileged self-teacher. The KL-divergence between teacher and student is calculated on the failed rollout to determine the proper intervention point. Everything after that point is discarded. 3. After the intervention point, for every decoding prefix, the student first calculate its support-set by retaining every token with a student possibility above a certain threshold δ . The teacher then rewights these tokens and choose the next token. For each new trajectory generated, it will be stored for further training if correct, and send back to a new round of refinement if incorrect. After the iterative refinement succeeds, the saved positive trajectory can be used to improve the student model.
Per-benchmark accuracy, avg@8
Overall
Method
AIME2025
AIME2026
HMMT Feb.
Aggregate
Pass@8
Δ
Qwen3-4B-Instruct-2507
Initial model
46.7
60.0
31.8
46.2
59.7
−6.0
Vanilla RLVR
55.5
63.3
37.9
52.2
66.0
0.0
DASD
52.5
59.2
34.1
48.6
67.5
−3.6
DAIL
47.5
53.3
33.1
44.6
61.8
−7.6
Table 1: Peak post-plateau performance , reported in percent. Aggregate is the unweighted mean over AIME2025, AIME2026, and HMMT February 2026. Both model blocks use eight samples per problem. Δ is the aggregate difference from Vanilla RLVR.
Decoder
Mean@16 ↑
Pass@16 ↑
Lex. ↓
Leakage ↓
Prob. ↓
Shared Prob. ↓
Raw guided decoding
68.62%
92.89%
76.46%
25.44
10.94%
8.41% (5.20%)
Student-support projected decoding
5.71%
31.98%
41.97%
0.02
4.25%
5.58% (3.36%)
Table 2: Controlled effect of student-support projection on 197 Qwen3-8B hard problems. Both decoders share all inputs and differ only in the continuation distribution. Shared Prob. reports the mean and median, in parentheses, on the 60 problems solved by both decoders.
Diagnostic steer with no retained trajectory prefix
17
7
Table 4: Search-design ablations. All variants use student-support projected decoding and 16 continuations per problem.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
δ
Verified correct ↑
Prob. anchoring ↓
Lex. anchoring ↓
Avg. leakage ↓
0.1
6/197(3.05%)
4.77%
59.56%
0.041
0.02
31/197(15.74%)
5.41%
62.69%
0.159
0.01
46/197(23.35%)
6.06%
64.03%
0.328
0.001
119/197(60.41%)
14.02%
71.33%
2.015
Appendix
Table 5: Selection of the student-support threshold. The passrate, degree of anchoring and leakage of different threshold values. Higher verified correctness is better, while lower anchoring and leakage are better.
Figure 4: One SSR correction round. A failed trajectory is summarized and diagnosed, continued under student-support projection, and retained after verification.
Setting
Qwen3-8B/Qwen-3-4B-Instruct-2507
Training data
Offline trajectories; the user prompt is the student prompt without the reference solution.
Number of trajectories
147×4 / 590×4
Epochs
1 epoch
Global batch size
1
Maximum sequence length
40,960 tokens for sample truncation and sequence packing
Optimizer
AdamW, with foreach=False
Appendix
Table 6: Supervised fine-tuning configuration.
Setting
Value
Optimization
Objective
GRPO-style clipped policy gradient
Advantage estimator
leave-one-out RLOO group advantage, entropy shaped
On-policy self-distillation, where a student is pulled toward a copy of itself conditioned on privileged context (e.g., a verified solution or feedback), offers a promising direction for advancing reasoning capability without a stronger external teacher. Yet in math reasoning the gains are inconsistent, even when the same approach succeeds elsewhere. A pointwise mutual information analysis traces the failure to the privileged context itself: it inflates the teacher's confidence on tokens already implied by the solution (structural connectives, verifiable claims) and deflates it on deliberation tokens ("Wait", "Let", "Maybe") that drive multi-step search. We propose Anti-Self-Distillation (AntiSD), which ascends a divergence between student and teacher rather than descending it: this reverses the per-token sign and yields a naturally bounded advantage in one step. An entropy-triggered gate disables the term once the teacher entropy collapses, completing a drop-in replacement for default self-distillation. Across five models from 4B to 30B parameters on math reasoning benchmarks, AntiSD reaches the GRPO baseline's accuracy in 2 to 10x fewer training steps and improves final accuracy by up to 11.5 points. AntiSD opens a path to scalable self-improvement, where a language model bootstraps its own reasoning through its training signal.
Guobin Shen, Xiang Cheng, Chenxiao Zhao +4
1Xiaohongshu Inc. · Institute of Automation, Chinese Academy of Sciences
Self-improvement training enables the large reasoning models (LRMs) to improve themselves by self-generating reasoning trajectories as training data without external supervision. However, we find that this method often falls short in complex reasoning tasks and even leads to model collapse. Through a series of preliminary analyses, we reveal two problems: (1) data imbalance, where most training samples are simple, but the challenging yet crucial samples are scarce; (2) overthinking, where many undesired samples with redundant reasoning steps are used for self-training. To this end, we propose HSIR, which effectively Harnesses Self-Improvement in large Reasoning models via two simple-yet-effective approaches. Specifically, HSIR introduces a verify-then-exit sampling strategy to mitigate data imbalance by efficiently collecting more accurate solutions for difficult queries, and designs an Intrinsic Diversity score to quantify overthinking and filter out the undesired solutions. We apply HSIR to various post-training paradigms, among which we further propose H-GRPO, an enhanced GRPO algorithm that leverages the intrinsic diversity as an external reward to encourage concise and diverse reasoning via reinforcement learning. Extensive results show that HSIR not only effectively enhances the reasoning performance, i.e., bringing up to +10.9% average performance gains, but also significantly improves the reasoning efficiency by reducing up to 42.4% relative inference overhead.
Qihuang Zhong, Liang Ding, Juhua Liu +3
1Nanyang Technological University, Singapore · 2Alibaba Group · School of Computer Science, Wuhan University, China +1
Self-distillation is an emerging approach that trains a model to imitate its own predictions when conditioned on privileged information, such as correct answers or environmental feedback. Recent methods have reported better sample efficiency and generalization than reinforcement learning (RL), but it remains unclear whether these gains extend beyond in-distribution evaluations and whether they stem from privileged distillation. In this paper, we revisit two representative methods, Self-Distillation Policy Optimization (SDPO) and On-Policy Self-Distillation (OPSD), across code, science, tool use, and math reasoning tasks. We find that their reported gains do not consistently generalize to unseen problems and are largely explained by overfitting or transferring existing reasoning behavior, rather than effectively using privileged information. Moreover, self-distillation gradients show weak alignment with policy gradients that directly optimize expected reward, questioning whether they provide an effective direction for policy improvement. Our experiments across Qwen3 models (1.7B-14B) and OLMo-3-7B suggest that effectively leveraging privileged information to improve reasoning remains an open challenge. Our code is available at https://github.com/sleepypandu/does-sd-improve-reasoning
Jason Ken Adhinarta, Raina Wu, Ashutosh Sharma +5
Massachusetts Institute of Technology · MIT-IBM Computing Research Lab