Distilling long-chain reasoning trajectories from large teacher models provides an effective way to improve the reasoning ability of small language models. However, existing methods typically select a fixed training set before distillation, overlooking that the usefulness of a reasoning example changes as the student evolves. This mismatch becomes especially costly when only a small number of teacher trajectories can be used for training. Besides, examples already mastered provide little new supervision, while examples far beyond the capability of student may be difficult to learn from. Training data should therefore adapt as model capability evolves, yet reliable data selection is difficult when the student is initially weak. To address these challenges, our key idea is to first establish the basic reasoning capability and then adapt the training data to the evolving capability boundary of the student. We propose BoundaryEvo, a two-stage data--model co-evolution framework. In Capability Priming, we directly distill a small set of teacher-generated long reasoning trajectories without using a separate model to judge and filter them, allowing the student to acquire an initial reasoning capability before capability-aware data selection. In Capability-Boundary Co-Evolution, we repeatedly select questions that the current student has not mastered but the expert can reliably solve, and update the training data as the student evolves. With only 1,000 training examples, BoundaryEvo consistently outperforms static and dynamic data selection baselines across 11 reasoning benchmarks, raising the original Qwen3.5-4B accuracy from 62.9 to 75.3 on IMO-AnswerBench and from 26.8 to 46.0 on AMO-Bench. The code and data will be made publicly available.
Figures & tables
Figure 1: BoundaryEvo evolves supervision along the capability boundary of the student model. (a-c) Comparison with static distillation and error-driven acquisition, BoundaryEvo dynamically adapts supervision according to the evolving capability boundary of the student model. (d) BoundaryEvo-4B consistently outperforms strong baselines across various reasoning benchmarks.
Figure 2: Overview of BoundaryEvo. Stage 1 performs judge-free teacher distillation to obtain the capability-primed checkpoint. In Stage 2, responses from the current student and a fixed expert are used to judge the candidate QAs, from which replay and novel records are selected for multiple teacher rollouts. The resulting trajectories are then independently verified against reference answers, and only those that pass verification supervise the next student.
Model
Params
Data
Mathematical Reasoning
General Reasoning
AIME
AIME
HMMT
HMMT
BruMO
IMO
AMO
Arxiv
Zebra
GPQA
Super
25
26
25
26
25
-Ans
-Bench
Math
Logic
-D
GPQA
Small reasoning models
SmolLM3-3B
3B
–
36.7
41.0
26.0
36.4
49.2
28.7
5.3
7.8
26.0
41.7
29.2
VibeThinker-3B
3B
–
91.4
94.3
89.3
80.3
93.8
76.4
41.0
27.2
74.7
70.2
34.8
DASD-4B-Thinking
4B
448K
83.3
88.3
80.0
72.7
85.0
61.6
26.5
9.7
69.0
68.4
42.9
Table 1: Comparison with Reasoning Models under Different Scales and Training Budgets. Scores are percentages. Data denotes the number of publicly disclosed reasoning post-training examples and – indicates unavailable information.
Method
Mathematical Reasoning
General Reasoning
Avg.
AIME
AIME
HMMT
HMMT
BruMO
IMO
AMO
Arxiv
Zebra
GPQA
Super
Math
General
25
26
25
26
25
-Ans
-Bench
Math
Logic
-D
GPQA
Qwen3.5-4B
+ Zero-shot
79.8
84.0
73.8
68.9
83.5
62.9
26.8
25.2
92.4
76.2
52.9
63.1
73.8
+ Stage 1
84.2 +4.4
88.3 +4.3
81.7 +7.9
69.7 +0.8
85.0 +1.5
66.3 +3.4
38.5 +11.7
28.2 +3.0
92.5 +0.1
79.8 +3.6
53.4 +0.5
67.7 +4.6
75.2 +1.4
Stage 2 with Random Selection
Table 2: Qwen3.5-4B results across three evolution turns. All Stage 2 strategies start from the same Stage 1 checkpoint. Scores are percentages, with Math and General averaged over eight and three benchmarks. Superscripts indicate absolute changes from zero-shot. Bold and underlining mark the highest and second-highest values in each column.
Method
Mathematical Reasoning
General Reasoning
Avg.
AIME
AIME
HMMT
HMMT
BruMO
IMO
Zebra
GPQA
Math
General
25
26
25
26
25
-Ans
Logic
-D
Qwen3.5-2B
+ Zero-shot
34.2
38.8
22.9
22.7
35.0
22.0
44.9
50.0
29.3
47.5
+ Stage 1
38.3 +4.1
50.0 +11.2
26.7 +3.8
24.5 +1.8
43.3 +8.3
35.2 +13.2
50.5 +5.6
50.5 +0.5
36.3 +7.0
50.5 +3.0
+ Turn 1
40.0 +5.8
46.7 +7.9
30.0 +7.1
33.3 +10.6
46.7 +11.7
35.7 +13.7
63.7 +18.8
52.5 +2.5
38.7 +9.4
58.1 +10.6
Table 3: Transfer across model size and architecture. Each backbone is primed before three sequential Stage 2 turns. Scores are percentages. Superscripts indicate absolute changes from zero-shot. Bold/underline denote the top two values, and blue shading marks the best checkpoint.
Figure 3: Training and selection dynamics across the Stage 2. (a) SFT loss for representative runs. (b) Mean student and expert scores on each turn. Annotations show the score gap.
Figure 4: Ablation study on IMO-AnswerBench performance. The curves compare BoundaryEvo with verified Stage 1 and no-replay variants, while reference lines show zero-shot and static SFT on 1K examples.
Method
Math AVG
General AVG
BoundaryEvo
76.4
77.4
⊢ Stage 1 (w/ verification)
73.1
73.9
⊢ w/o Stage 1
73.5
74.0
Table 4: Ablation study on effect of Stage 1 on final Qwen3.5-4B performance. Math and General average eight and three benchmarks.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Ablated factor
E
R
τg
Stage 2 data
Math / General
Reference
5
3
0.20
900
76.4 / 77.4
Epochs
1
3
0.20
900
72.6 / 75.9
Epochs
3
3
0.20
900
73.5 / 76.1
Epochs
10
3
0.20
900
74.3 / 77.0
Evolution turns
5
1
0.20
300
71.2 / 75.5
Evolution turns
5
2
0.20
600
73.3 / 76.9
Appendix
Table 5: Core hyperparameter ablations on Qwen3.5-4B. E is the number of SFT epochs per update, R is the number of Stage 2 turns, and τg is the minimum expert–student capability gap. Stage 2 data count is 300R and counts replayed examples whenever they are reused. Each non-reference row changes one factor only.
Setting
Qwen3.5-4B / 2B
Gemma-4-E4B-it
Training objective
Response-only SFT
Response-only SFT
Fine-tuning method
LoRA
LoRA
LoRA rank / scaling factor
64 / 128
64 / 128
LoRA target modules
All linear modules
q,k,v,o , gate, up, and down projections
Trainable parameters
129.9M / 67.3M
139.5M
Conversation template
qwen3
gemma4
Appendix
Table 6: Training configuration used in experiments. The effective batch size is the product of the per-device batch size, gradient accumulation steps, and the number of data-parallel workers.
Figure 5: Reasoning-length distributions on matched evaluation questions. Curves show smoothed densities of per-question reasoning-token counts, and vertical lines mark medians. The final panel reports the relative change in median length from Qwen3.5-4B to BoundaryEvo-4B. Repeated rollouts of the same question are summarized by their median before plotting.
Benchmark
n
Median
Δ
90th percentile
Base
Ours
Base
Ours
AIME25
30
42.8
20.5
−52.2%
99.4
52.1
AIME26
30
36.9
17.4
−52.9%
74.9
54.9
HMMT25
30
46.1
27.3
−40.8%
102.4
74.3
HMMT26
31
45.4
34.6
−23.9%
71.3
74.6
ArxivMath
103
56.4
52.2
−7.4%
82.1
71.2
Appendix
Table 7: Reasoning-length statistics for the matched questions in Figure 5 . Token counts are reported in thousands. Δ denotes the relative change in median length from the base model to BoundaryEvo-4B.
Figure 6: GPQA-D performance versus model scale. The horizontal axis is logarithmic, bubble area encodes parameter count, and color denotes the parameter range. BoundaryEvo-4B is highlighted with a white star. Only models with both a reported parameter count and GPQA-D score in Table 1 are included.
Figure 7: GPQA-D performance versus publicly reported reasoning post-training data. The horizontal axis is logarithmic and bubble area encodes model parameters. Right-pointing arrows mark reported lower bounds rather than exact dataset sizes. Only systems with disclosed data amounts and GPQA-D scores in Table 1 are shown.
College of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics, Nanjing, China · 2The Key Laboratory of Brain-Machine Intelligence Technology, Ministry of Education, Nanjing, China