Autoregressive decoding is less suited to non-causally ordered data, e.g., cross-references or future dependencies, since their outputs cannot be revised post hoc, unlike iterative human reasoning. Diffusion LLMs partially mitigate this limitation but still cannot flexibly insert or remove arbitrary numbers of tokens from a pre-generated context. We design AutoIndexer as a generative model with a novel training and decoding mechanism that simulates a Chain of Edits, allowing the model to flexibly modify its prior outputs in hindsight via token insertions, substitutions, and deletions. The model learns to predict marker tokens for opening and closing edits, simultaneously moving a cursor to positions in the existing token sequence, analogous to a text editor. Training sequences are obtained by perturbing ground-truth labels with inverse chained edit operations, and the model learns to edit by undoing those inverses. The perturbation training allows AutoIndexer to be robust to erroneous context and adaptable to editing tasks such as code correction. We demonstrate that our method exhibits post-generation context-editing capabilities that most autoregressive and diffusion models lack without compromising performance on standard LLM benchmarks. The code is available at https://github.com/AutodeskAILab/autoindexer.
Figures & tables
Figure 1: Overview of the AutoIndexer method. Analogous to using a text editor, AutoIndexer can edit its prior outputs by predicting edit-start/-end cursor positions. The positional encodings of the tokens are modified after each edit and efficiently computed by our specialized attention kernel.
Autoregressive
Discrete Diffusion
Chain of Edits (Ours)
Generation order of tokens
Autoregressive
Non-autoregressive
Autoregressive
Display order of tokens
Fixed-order
Fixed-order
Decoupled (flexible)
Prediction target
Next token
Token demasking
Undo perturbation
KV-caching
Exact
Approximate
Exact
Amendable context
No
Limited
Yes
Table 1: Comparison between decoding strategies of transformers.
Figure 3: A simulated edit sequence for training. The ground-truth clean sequence (top) in display order is randomly perturbed to obtain a target sequence (bottom) with edit/return markers, which the model learns to generate. A sequence parser recovers the intended display order of the tokens.
Model
Qwen3 8B
Qwen3 8B LoRA ∗
LLaDA 8B
Dream 7B
AutoIndexer 8B LoRA ∗
Decoding strategy
AR
AR
Diffusion
Diffusion
CoE (Chain of Edits)
Editable context
No
No
Limited
Limited
Yes
General Tasks
MMLU (5)
76.7
77.0
65.9
69.5
76.9
ARC-Challenge (0)
56.7
60.8
45.3
59.7
59.6
Hellaswag (0)
78.7
77.7
69.4
72.5
77.5
Table 2: Pre-training benchmark results. All models were evaluated under the same protocol. Numbers in parentheses are the number of few-shot examples used for in-context learning. The models with ∗ were finetuned with 100K additional CPT training samples ( ∼ 200M tokens). Bold shows the best score out of flexible-order decoding models, and the underline shows the best overall.
Model
Qwen3 8B
Qwen3 8B LoRA ∗
LLaDA 8B Instruct
Dream 7B Instruct
AutoIndexer 8B LoRA ∗
Decoding strategy
AR
AR
Diffusion
Diffusion
CoE (Chain of Edits)
Editable context
No
No
Limited
Limited
Yes
General Tasks
ARC-Challenge (0)
90.61
90.70
86.26
88.23
90.61
MMLU-Pro (0)
42.38
47.90
32.65
44.39
47.90
Mathematics & Science
Table 3: Post-training (instruction-tuned) benchmark results. All models were evaluated instruct-formatted, under the same protocol. Numbers in parentheses are the number of few-shot examples used for in-context learning. The models with ∗ were supervised-finetuned with the same 100K. Bold shows the best score out of flexible-order decoding models, and the underline shows the best overall.
Outcome / 100
Start cursor
End cursor
Insert
Delete
Substitute
No modification
Repaired exactly
✓
✓
10
53
18
–
No edit needed (true negative)
–
–
–
–
–
89
No edit emitted (false negative)
–
–
57
10
8
–
Correct span, improved
✓
✓
12
9
29
–
Correct span, neutral
✓
✓
2
1
11
–
Correct span, worse
✓
✓
2
6
8
–
Table 4: Free-generation per-outcome counts (100 examples per category, marker head sampled with the default settings of temperature Tmark=1.0 and marker top- p=1.0 ).
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A.1 : Relationship between generation order and display order of tokens. The left-hand-side shows the generated tokens taking the display order as reference, whereas the right-hand-side shows the relative positions of the tokens, taking the generation order as reference.
Source
Dataset / subset
Weight
DCLM-baseline
mlfoundations/dclm-baseline-1.0-parquet
30
FineWeb-Edu
HuggingFaceFW/fineweb-edu
15
SlimPajama
AlppAI/SlimPajama-chunked
25
OpenWebMath
open-web-math/open-web-math
5
StarCoderData Python
bigcode/starcoderdata/python
7
StarCoderData JavaScript
bigcode/starcoderdata/javascript
4
Appendix
Table A.1 : CPT data mix used for the matched Qwen3 LoRA and AutoIndexer LoRA runs reported in Table 2 .
Source
Dataset / subset
Weight
General-purpose instructions
Alpaca
tatsu-lab/alpaca
15
Dolly-15k
databricks/databricks-dolly-15k
10
WizardLM Evol-Instruct-70k
WizardLMTeam/WizardLM_evol_instruct_70k
20
UltraChat-200k (multi-turn)
HuggingFaceH4/ultrachat_200k
25
Coding instructions
Appendix
Table A.2 : SFT instruction data mix used for the matched Qwen3 LoRA and AutoIndexer LoRA runs reported in Table 3 .
Table A.3 : Teacher-forced edit execution with oracle-placed markers (window of 512 tokens). Measures the increase in P(\textscedit) at the synthetic trigger position against the same position in a clean sequence, as well as the edit start and end localization, content improvement, and edit termination accuracy.
Figure A.2 : Visualization of attention weights averaged over the attention heads. The left-hand-side shows the full attention weights, while the right-hand-side shows a close-up around the edit block ( [EDIT] … [RETURN] ). [EDIT] and [RETURN] are masked out upon the completion of the edit, as well as the deleted tokens. It can be seen that attention is focused before and after the erroneous token region during the edit so that the newly inserted tokens are coherent with the context.
Figure A.3 : Visualization of the cursor pointer network heads for edit start and edit end. The edit start head attends to all tokens in the previous context to identify the beginning of a potential erroneous content. The edit end head attends to all tokens following the edit start cursor to identify where the edit should end.