Steer, Don't Tell: Breaking Reasoning Plateaus with Reachability-Aligned Self-Distillation
Authors: Zijian Wu, Lingkai Kong, Lin Yang, Haiteng Zhao, Kuikun Liu, Kai Chen, Dahua Lin, Wenwei Zhang
Organizations: Department of Information Engineering, The Chinese University of Hong Kong · Shanghai Artificial Intelligence Laboratory · Shanghai Jiao Tong University
Post training for large reasoning models remains insufficient on challenging problems, whose correct trajectories are hard to sample. Though prior works address this difficulty by integrating supervised fine tuning on expert trajectories with reinforcement learning, such external experts are not always available. Existing self improvement methods instead use reference solutions to create a privileged self-teacher. Yet privileged teacher correctness does not imply student reachability. Solution conditioning shifts the teacher distribution toward actions unlikely under the unprivileged student, producing seemingly "correct" trajectories that are unreachable or even mal-formed learning targets. Accordingly, we identify reachability alignment as the key to self constructed supervision and propose Self-Steered Reasoner (SSR). SSR realizes this principle through student support projection, which transforms the privileged teacher distribution into a directional signal only over reasoning actions locally reachable under the unguided student. Verification then retains correct trajectories sampled from this reachable space. To further boost the sampling efficiency, an iterative refinement framework is adopted, using diagnostic guidance based on current trajectories. On problems with pass@64 = 0 even after extensive RL, SSR recovers correct and learnable trajectories for 74.6% without parameter updates. Fine tuning on these trajectories followed by resumed reinforcement learning improves aggregate accuracy by 4.9 percentage points on AIME25/26 and HMMT26. These results show that privileged guidance becomes effective self-improvement data when aligned with the reasoning actions the student can reach.
Figures & tables
Figure 1: RLVR can extend the model’s ability in its comfort zone, but it meets obstacles when coming to anxiety zone where correct trajectories are hard to sample. Privileged-teacher can provide aid signals on anxiety zone, but it can also select actions outside the student’s reachable region, producing unlearnable traces. Directly imitating these actions can corrupt the student’s reasoning ability(e.g., learning to hallucinate a non-existent hint). By projecting privileged preferences into the consensus region, SSR retains the learnable steering signals within the privileged preferences, effectively expands the student’s learning boundary.
Figure 2: Overview of the SSR framework. 1. Given a very hard question Q, we sample failed rollouts from the student model. The failed rollout is then summarized and diagnosed against the reference solution to produce a self-steer. The self-steer describes the error and provide fixing directions while avoiding directly offering the answer. 2. The self-steer is then used to construct the privileged self-teacher. The KL-divergence between teacher and student is calculated on the failed rollout to determine the proper intervention point. Everything after that point is discarded. 3. After the intervention point, for every decoding prefix, the student first calculate its support-set by retaining every token with a student possibility above a certain threshold δ . The teacher then rewights these tokens and choose the next token. For each new trajectory generated, it will be stored for further training if correct, and send back to a new round of refinement if incorrect. After the iterative refinement succeeds, the saved positive trajectory can be used to improve the student model.
Per-benchmark accuracy, avg@8
Overall
Method
AIME2025
AIME2026
HMMT Feb.
Aggregate
Pass@8
Δ
Qwen3-4B-Instruct-2507
Initial model
46.7
60.0
31.8
46.2
59.7
−6.0
Vanilla RLVR
55.5
63.3
37.9
52.2
66.0
0.0
DASD
52.5
59.2
34.1
48.6
67.5
−3.6
DAIL
47.5
53.3
33.1
44.6
61.8
−7.6
Table 1: Peak post-plateau performance , reported in percent. Aggregate is the unweighted mean over AIME2025, AIME2026, and HMMT February 2026. Both model blocks use eight samples per problem. Δ is the aggregate difference from Vanilla RLVR.
Decoder
Mean@16 ↑
Pass@16 ↑
Lex. ↓
Leakage ↓
Prob. ↓
Shared Prob. ↓
Raw guided decoding
68.62%
92.89%
76.46%
25.44
10.94%
8.41% (5.20%)
Student-support projected decoding
5.71%
31.98%
41.97%
0.02
4.25%
5.58% (3.36%)
Table 2: Controlled effect of student-support projection on 197 Qwen3-8B hard problems. Both decoders share all inputs and differ only in the continuation distribution. Shared Prob. reports the mean and median, in parentheses, on the 60 problems solved by both decoders.
Diagnostic steer with no retained trajectory prefix
17
7
Table 4: Search-design ablations. All variants use student-support projected decoding and 16 continuations per problem.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
δ
Verified correct ↑
Prob. anchoring ↓
Lex. anchoring ↓
Avg. leakage ↓
0.1
6/197(3.05%)
4.77%
59.56%
0.041
0.02
31/197(15.74%)
5.41%
62.69%
0.159
0.01
46/197(23.35%)
6.06%
64.03%
0.328
0.001
119/197(60.41%)
14.02%
71.33%
2.015
Appendix
Table 5: Selection of the student-support threshold. The passrate, degree of anchoring and leakage of different threshold values. Higher verified correctness is better, while lower anchoring and leakage are better.
Figure 4: One SSR correction round. A failed trajectory is summarized and diagnosed, continued under student-support projection, and retained after verification.
Setting
Qwen3-8B/Qwen-3-4B-Instruct-2507
Training data
Offline trajectories; the user prompt is the student prompt without the reference solution.
Number of trajectories
147×4 / 590×4
Epochs
1 epoch
Global batch size
1
Maximum sequence length
40,960 tokens for sample truncation and sequence packing
Optimizer
AdamW, with foreach=False
Appendix
Table 6: Supervised fine-tuning configuration.
Setting
Value
Optimization
Objective
GRPO-style clipped policy gradient
Advantage estimator
leave-one-out RLOO group advantage, entropy shaped