Large language models increasingly rely on extended reasoning traces, yet the training signals used to shape this behavior remain coarse. Supervised fine-tuning on full trajectories forces the model to imitate an entire demonstration, overconstraining its output distribution, while trajectory-level reinforcement learning provides only weak credit assignment, rewarding or penalizing whole completions without identifying where they succeed or fail. We propose edit-based training, a general framework for learning from a model's own trajectories via localized textual corrections. Given a sampled completion, an editor identifies the first point of failure under a specified criterion and proposes a minimal replacement span. The model is then supervised only on these edited spans. By design, this concentrates supervision where the model's reasoning first breaks down and aims to preserve its existing reasoning patterns elsewhere. The method applies both offline and online, with the latter continually adapting supervision to the model's evolving failure modes. We instantiate this approach on reasoning instruction following with Qwen3-32B on the ReasonIF benchmark. Compared to full-trajectory imitation and trajectory-level reinforcement learning baselines, edit-based training substantially improves compliance with the reasoning instruction while largely preserving the model's skills on untargeted tasks. These results suggest that localized correction is an effective complement to these approaches for shaping how LLMs reason.
Figures & tables
Figure 1: Edit-based training within one example. The instruction asks the model to reason in Chinese; the final answer is to remain in English. Each column shows the state of the trajectory after one editing round. The prompt and instruction are fixed across rounds. A red circle marks the first remaining instruction violation at the current round. A green circle is the editor’s minimal replacement span and is used as a supervised training target. Gray circles are model-generated tokens that are uncorrected in the current round. White circles are tokens accepted as part of the final trajectory but not used as edit targets. In the sampled rollout, the model begins reasoning in English, so the first reasoning step is flagged as the first violation. The editor makes a minimal Chinese replacement, and generation resumes from the corrected prefix. The editor applies replacements until the model continues reasoning in Chinese. Once the model adopts the requested mode, it may overshoot and emit the final answer in Chinese as well, which is itself a violation. The last edit then restores an English answer. The trajectory is now compliant.
Evaluation
Base
SFT
SFT-GPT
GRPO
Offline
Online
ReasonIF (in-domain)
13
40.3 ± 0.4
92 ± 2
64 ± 6
87 ± 1
97 ± 1
Out-of-domain prompts
21
57.1 ± 0.2
96.8 ± 0.4
80 ± 3
96.6 ± 0.2
95 ± 2
Unseen reasoning instructions
2
5 ± 1
63 ± 2
30 ± 10
68.4 ± 0.4
34 ± 9
ReasonIF answer accuracy
96
96.9 ± 0.3
94 ± 1
91 ± 2
95 ± 1
95 ± 1
IMO-AnswerBench
61
61 ± 1
43 ± 2
60 ± 2
57 ± 1
59 ± 2
Table 1: Overview of the main results (higher is better). The first three rows report overall reasoning instruction-following accuracy (%) in three settings: in-domain, on out-of-domain prompts, and under unseen reasoning instructions. The last two rows measure preservation of capabilities not targeted by training: final-answer accuracy (%) on the ReasonIF tasks and on IMO-AnswerBench, both evaluated with the reasoning instruction removed. Trained systems report mean ± standard deviation across three training runs; among them, the best result in each row is shown in bold, together with any result whose standard deviation covers it. Per-category breakdowns of the instruction-following results are given in Tables 3 , 5 , and 6 .
Categories
Constraints
Base
Online -
Online +
Seen
Single
85
89
96
Seen
Multiple
81
85
85
Unseen
Single
93
93
91
Unseen
Multiple
90
89
88
Table 2: IFEval constraint accuracy (%), macro-averaged over constraint categories. Online - and Online + evaluate the edit-trained model without and with the draft-review-revise reasoning instruction, respectively; Base is evaluated without it. Seen comprises the three constraint categories represented during training and Unseen the remaining categories; Constraints indicates whether the prompt imposes a single constraint or multiple.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Category
Base
SFT
SFT-GPT
GRPO
Offline
Online
Uppercase only
0
33 ± 3
93 ± 2
38 ± 29
70 ± 2
96 ± 1
JSON formatting
0
66 ± 1
91 ± 8
0 ± 0
75 ± 1
95 ± 1
Multilinguality
44
78 ± 2
94.0 ± 0.3
100 ± 0
98 ± 2
99 ± 1
Word limit
0
7 ± 2
94 ± 2
65 ± 14
99.9 ± 0.1
99 ± 1
Remove commas
26
48 ± 2
99.0 ± 0.2
97 ± 2
94 ± 1
99 ± 1
Disclaimer
7
9 ± 1
83 ± 3
77 ± 2
79 ± 2
94 ± 4
Appendix
Table 3: Qwen3-32B reasoning instruction-following accuracy (%) on the ReasonIF benchmark. Each training run is evaluated six times; 1800 responses per training run.
Category
Base
SFT
SFT-GPT
GRPO
Offline
Online
Uppercase only
0
36 ± 1
90 ± 2
0 ± 0
29 ± 3
91 ± 1
JSON formatting
0
83 ± 6
91 ± 2
0 ± 0
90 ± 3
99.5 ± 0.3
Multilinguality
27
73 ± 1
93 ± 1
27.1 ± 0.3
92 ± 2
94.6 ± 0.5
Word limit
13
16 ± 2
89 ± 1
8 ± 6
87 ± 7
93 ± 2
Remove commas
4
46 ± 1
97.7 ± 0.4
36 ± 3
97 ± 1
100 ± 0
Disclaimer
3
45 ± 2
78 ± 2
2 ± 2
71 ± 18
97 ± 0
Appendix
Table 4: Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 reasoning instruction-following accuracy (%) on the ReasonIF benchmark. Each training run is evaluated six times; 1800 responses per training run.
Category
Base
SFT
SFT-GPT
GRPO
Offline
Online
Uppercase only
0
83 ± 1
94 ± 1
83 ± 16
91.2 ± 0.4
94 ± 3
JSON formatting
0
6 ± 3
94 ± 3
0 ± 0
94.1 ± 0.8
96 ± 3
Multilinguality
5
33 ± 2
99.3 ± 0.3
99.7 ± 0.3
98.1 ± 0.6
93 ± 1
Word limit
81
85 ± 1
99.1 ± 0.1
99.7 ± 0.5
99.1 ± 0.3
91 ± 5
Remove commas
16
87 ± 2
97 ± 1
100 ± 0
100 ± 0
99.7 ± 0.1
Disclaimer
22
48 ± 1
98 ± 1
98 ± 1
97 ± 1
98.8 ± 0.6
Appendix
Table 5: Qwen3-32B reasoning instruction-following accuracy (%) on a dataset constructed from the HuggingFace H4 Multilingual-Thinking dataset [ 28 ] with ReasonIF instructions appended. Each training run is evaluated twice; 2000 responses per training run.
Category
Base
SFT
SFT-GPT
GRPO
Offline
Online
Answer draft
1
0.9 ± 0.4
43 ± 5
34 ± 7
60 ± 1
33 ± 14
Sections
2
8 ± 2
63 ± 1
24 ± 10
53 ± 2
30 ± 9
Two approaches
4
6 ± 1
82 ± 1
30 ± 13
89 ± 2
38 ± 5
Overall
2
5 ± 1
63 ± 2
30 ± 10
68.4 ± 0.4
34 ± 9
Appendix
Table 6: Qwen3-32B reasoning instruction-following accuracy (%) on the ReasonIF benchmark with the original instructions replaced by unseen reasoning-process instructions. Each training run is evaluated three times; 900 responses per training run.
Evaluation
With reg.
Without reg.
ReasonIF (in-domain)
87 ± 1
85 ± 2
Out-of-domain prompts
96.6 ± 0.2
94 ± 1
Unseen reasoning instructions
68.4 ± 0.4
30 ± 8
Appendix
Table 7: Qwen3-32B Offline reasoning instruction-following accuracy (%) with and without regularization. Both settings report mean ± standard deviation across three training runs.