Large language models increasingly rely on extended reasoning traces, yet the training signals used to shape this behavior remain coarse. Supervised fine-tuning on full trajectories forces the model to imitate an entire demonstration, overconstraining its output distribution, while trajectory-level reinforcement learning provides only weak credit assignment, rewarding or penalizing whole completions without identifying where they succeed or fail. We propose edit-based training, a general framework for learning from a model's own trajectories via localized textual corrections. Given a sampled completion, an editor identifies the first point of failure under a specified criterion and proposes a minimal replacement span. The model is then supervised only on these edited spans. By design, this concentrates supervision where the model's reasoning first breaks down and aims to preserve its existing reasoning patterns elsewhere. The method applies both offline and online, with the latter continually adapting supervision to the model's evolving failure modes. We instantiate this approach on reasoning instruction following with Qwen3-32B on the ReasonIF benchmark. Compared to full-trajectory imitation and trajectory-level reinforcement learning baselines, edit-based training substantially improves compliance with the reasoning instruction while largely preserving the model's skills on untargeted tasks. These results suggest that localized correction is an effective complement to these approaches for shaping how LLMs reason.
Figures & tables
Figure 1: Edit-based training within one example. The instruction asks the model to reason in Chinese; the final answer is to remain in English. Each column shows the state of the trajectory after one editing round. The prompt and instruction are fixed across rounds. A red circle marks the first remaining instruction violation at the current round. A green circle is the editor’s minimal replacement span and is used as a supervised training target. Gray circles are model-generated tokens that are uncorrected in the current round. White circles are tokens accepted as part of the final trajectory but not used as edit targets. In the sampled rollout, the model begins reasoning in English, so the first reasoning step is flagged as the first violation. The editor makes a minimal Chinese replacement, and generation resumes from the corrected prefix. The editor applies replacements until the model continues reasoning in Chinese. Once the model adopts the requested mode, it may overshoot and emit the final answer in Chinese as well, which is itself a violation. The last edit then restores an English answer. The trajectory is now compliant.
Evaluation
Base
SFT
SFT-GPT
GRPO
Offline
Online
ReasonIF (in-domain)
13
40.3 ± 0.4
92 ± 2
64 ± 6
87 ± 1
97 ± 1
Out-of-domain prompts
21
57.1 ± 0.2
96.8 ± 0.4
80 ± 3
96.6 ± 0.2
95 ± 2
Unseen reasoning instructions
2
5 ± 1
63 ± 2
30 ± 10
68.4 ± 0.4
34 ± 9
ReasonIF answer accuracy
96
96.9 ± 0.3
94 ± 1
91 ± 2
95 ± 1
95 ± 1
IMO-AnswerBench
61
61 ± 1
43 ± 2
60 ± 2
57 ± 1
59 ± 2
Table 1: Overview of the main results (higher is better). The first three rows report overall reasoning instruction-following accuracy (%) in three settings: in-domain, on out-of-domain prompts, and under unseen reasoning instructions. The last two rows measure preservation of capabilities not targeted by training: final-answer accuracy (%) on the ReasonIF tasks and on IMO-AnswerBench, both evaluated with the reasoning instruction removed. Trained systems report mean ± standard deviation across three training runs; among them, the best result in each row is shown in bold, together with any result whose standard deviation covers it. Per-category breakdowns of the instruction-following results are given in Tables 3 , 5 , and 6 .
Categories
Constraints
Base
Online -
Online +
Seen
Single
85
89
96
Seen
Multiple
81
85
85
Unseen
Single
93
93
91
Unseen
Multiple
90
89
88
Table 2: IFEval constraint accuracy (%), macro-averaged over constraint categories. Online - and Online + evaluate the edit-trained model without and with the draft-review-revise reasoning instruction, respectively; Base is evaluated without it. Seen comprises the three constraint categories represented during training and Unseen the remaining categories; Constraints indicates whether the prompt imposes a single constraint or multiple.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Category
Base
SFT
SFT-GPT
GRPO
Offline
Online
Uppercase only
0
33 ± 3
93 ± 2
38 ± 29
70 ± 2
96 ± 1
JSON formatting
0
66 ± 1
91 ± 8
0 ± 0
75 ± 1
95 ± 1
Multilinguality
44
78 ± 2
94.0 ± 0.3
100 ± 0
98 ± 2
99 ± 1
Word limit
0
7 ± 2
94 ± 2
65 ± 14
99.9 ± 0.1
99 ± 1
Remove commas
26
48 ± 2
99.0 ± 0.2
97 ± 2
94 ± 1
99 ± 1
Disclaimer
7
9 ± 1
83 ± 3
77 ± 2
79 ± 2
94 ± 4
Appendix
Table 3: Qwen3-32B reasoning instruction-following accuracy (%) on the ReasonIF benchmark. Each training run is evaluated six times; 1800 responses per training run.
Category
Base
SFT
SFT-GPT
GRPO
Offline
Online
Uppercase only
0
36 ± 1
90 ± 2
0 ± 0
29 ± 3
91 ± 1
JSON formatting
0
83 ± 6
91 ± 2
0 ± 0
90 ± 3
99.5 ± 0.3
Multilinguality
27
73 ± 1
93 ± 1
27.1 ± 0.3
92 ± 2
94.6 ± 0.5
Word limit
13
16 ± 2
89 ± 1
8 ± 6
87 ± 7
93 ± 2
Remove commas
4
46 ± 1
97.7 ± 0.4
36 ± 3
97 ± 1
100 ± 0
Disclaimer
3
45 ± 2
78 ± 2
2 ± 2
71 ± 18
97 ± 0
Appendix
Table 4: Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 reasoning instruction-following accuracy (%) on the ReasonIF benchmark. Each training run is evaluated six times; 1800 responses per training run.
Category
Base
SFT
SFT-GPT
GRPO
Offline
Online
Uppercase only
0
83 ± 1
94 ± 1
83 ± 16
91.2 ± 0.4
94 ± 3
JSON formatting
0
6 ± 3
94 ± 3
0 ± 0
94.1 ± 0.8
96 ± 3
Multilinguality
5
33 ± 2
99.3 ± 0.3
99.7 ± 0.3
98.1 ± 0.6
93 ± 1
Word limit
81
85 ± 1
99.1 ± 0.1
99.7 ± 0.5
99.1 ± 0.3
91 ± 5
Remove commas
16
87 ± 2
97 ± 1
100 ± 0
100 ± 0
99.7 ± 0.1
Disclaimer
22
48 ± 1
98 ± 1
98 ± 1
97 ± 1
98.8 ± 0.6
Appendix
Table 5: Qwen3-32B reasoning instruction-following accuracy (%) on a dataset constructed from the HuggingFace H4 Multilingual-Thinking dataset [ 28 ] with ReasonIF instructions appended. Each training run is evaluated twice; 2000 responses per training run.
Category
Base
SFT
SFT-GPT
GRPO
Offline
Online
Answer draft
1
0.9 ± 0.4
43 ± 5
34 ± 7
60 ± 1
33 ± 14
Sections
2
8 ± 2
63 ± 1
24 ± 10
53 ± 2
30 ± 9
Two approaches
4
6 ± 1
82 ± 1
30 ± 13
89 ± 2
38 ± 5
Overall
2
5 ± 1
63 ± 2
30 ± 10
68.4 ± 0.4
34 ± 9
Appendix
Table 6: Qwen3-32B reasoning instruction-following accuracy (%) on the ReasonIF benchmark with the original instructions replaced by unseen reasoning-process instructions. Each training run is evaluated three times; 900 responses per training run.
Evaluation
With reg.
Without reg.
ReasonIF (in-domain)
87 ± 1
85 ± 2
Out-of-domain prompts
96.6 ± 0.2
94 ± 1
Unseen reasoning instructions
68.4 ± 0.4
30 ± 8
Appendix
Table 7: Qwen3-32B Offline reasoning instruction-following accuracy (%) with and without regularization. Both settings report mean ± standard deviation across three training runs.
Large language models (LLMs) often exhibit flawed reasoning ability that undermines reliability. Existing approaches to improving reasoning typically treat it as a general and monolithic skill, applying broad training which is inefficient and unable to target specific reasoning errors. We introduce Reasoning Editing, a paradigm for selectively modifying specific reasoning patterns in LLMs while preserving other reasoning pathways. This task presents a fundamental trade-off between Generality, the ability of an edit to generalize across different tasks sharing the same reasoning pattern, and Locality, the ability to preserve other reasoning capabilities. Through systematic investigation, we uncover the Circuit-Interference Law: Edit interference between reasoning patterns is proportional to the overlap of their neural circuits. Guided by this principle, we propose REdit, the first framework to actively reshape neural circuits before editing, thereby modulating interference between reasoning patterns and mitigating the trade-off. REdit integrates three components: (i) Contrastive Circuit Reshaping, which directly addresses the generality-locality trade-off by disentangling overlapping circuits; (ii) Meta-Contrastive Learning, which extends transferability to novel reasoning patterns; and (iii) Dual-Level Protection, which preserves preexisting abilities by constraining reshaping update directions and regularizing task-level predictions. Extensive experiments with Qwen-2.5-3B on propositional logic reasoning tasks across three difficulty levels demonstrate that REdit consistently achieves superior generality and locality compared to baselines, with additional validation in mathematics showing broader potential. Our code is available at https://github.com/LzyFischer/REdit.
Zhenyu Lei, Qiong Wu, Jianxiong Dong +4
University of Virginia · AT&T Chief Data Office · Florida State University
Large language models (LLMs) are often post-trained on pre-collected reasoning trajectories to improve their reasoning capability. Such trajectories tend to be long due to complex, interwoven paths, which often include detours on the path toward the answer. However, it has been underexplored whether LLMs indeed benefit from learning complete trajectories in post-training, such as supervised fine-tuning (SFT). Starting from our pilot study, we find that full trajectories provide only limited benefit, while partial trajectories are effective even under heavy truncation. We analyze redundancy in reasoning trajectories through attention-based analyses and controlled token-removal studies, both of which show that intermediate tokens contribute minimally to final reasoning quality. This suggests that avoiding redundant information may allow LLMs to internally infer coherent alternatives by inferring missing steps from their internal knowledge, given known trajectory endpoints. Furthermore, we show that training LLMs using endpoints leads to consistent changes in reasoning behavior, and that it also benefits post-training methods based on reinforcement learning or on-policy distillation, highlighting the need to revisit complete reasoning traces. Code is available at https://github.com/naver-ai/revisiting-trace.
Large language models struggle to reliably identify and correct errors in their own reasoning. Unlike prior training-based approaches that fine-tune an LLM to generate revisions or feedback, we learn where to intervene in failed rollouts and determine how to redirect them. To choose where to intervene, we train a 130K-parameter MLP head on a frozen encoder using only final-answer correctness. For redirection, we find that under a fixed generation budget, the tested inline injections fail to reliably redirect reasoning, whereas presenting the same instruction in a new user turn changes the model's course. The resulting correction framework, UPTURN (User-Prompt TURN injection at a learned position), requires no LLM fine-tuning: it retains a selected prefix, injects an instruction as a new user turn, and regenerates the suffix. Across four reasoners and six math benchmarks, UPTURN improves average accuracy in every setting over a 32-rollout baseline. It outperforms self-correction and reflection-token injection baselines under matched correction budgets, transfers across reasoners and to GPQA-Diamond without retraining, and complements GRPO and self-correction methods.
Jiwon Jeon, Myungsik Cho, Jeonghye Kim +1
Korea Advanced Institute of Science and Technology (KAIST)