Causal Edit Serialization

Louis Robinson

cs.CLOct 16, 2026

Causal Edit Serialization

Authors: Louis Robinson

Organizations: Independent Researcher

Abstract

Standard autoregressive language models generate text sequentially, but revision requires referring back to tokens in the input context or those already generated. Causal Edit Serialization (CES) is introduced as a framework for expressing such revisions within the same causal token stream. Special pointer tokens are paired with predicted source addresses, allowing insertion, deletion, replacement, copying and movement to be localized explicitly. A lightweight localization head and dual-index rotary position encoding support both address prediction and conditioning on previously emitted pointers, while retaining a standard causal attention mask. A pretraining procedure combines ordinary language modeling with serialized repairs derived from random corruptions, model-generated edits and repeated text. Across model scales and editing baselines, CES models demonstrate complete applied edits with modestly higher clean-text bits per byte than clean autoregressive models. Supervised and reinforcement-learning post-training provide a grammatical-error-correction proof of concept, producing more beneficial than harmful changes under reference-blind model judging. Correct repair programs also improve subsequent text prediction while the original context and its cached states are retained. CES provides a common interface for generation and localized revision, with potential extensions to iterative refinement and additional pointer-addressed transformations.

Figures & tables

Figure 1: Causal edit programs and CES architecture. Left: edit programs and their applied results. Numbers denote stream positions; @ labels give predicted source addresses. Dotted borders mark masked pretraining vocabulary targets; pointer addresses remain supervised. Right: dual-index RoPE and the localization head, with CES additions in red.

C

denotes model width.

Figure 2: Complete editing, language modeling and representation cost across methods. A/B: clean ClimbMix BPB and CORE performance. C–F: controlled ClimbMix repair. A–F use pretrained checkpoints, showing means and sample SDs over three training seeds; repair suites contain 1,000 inputs per run (250 for adapted CES). C: successful program closure, parsing and application, or text termination for regeneration/FIM. D: exact target recovery. E/F: frequency and mean magnitude of character-similarity improvement. Invalid or unterminated outputs return the source and remain in the denominators; oracle FIM receives gold spans. Clean AR appears only in A/B. G: lengths on 485 CoEdIT pairs. Filled bars encode references; outlines count observed d24 seed-42 outputs after supervised CoEdIT fine-tuning, including failed attempts. Stacked bars combine prompts (grey) and minimum reference completions (striped colour), using fixed templates and token boundaries and excluding COPY/MOVE. The iterative editor’s striped reference and prompt bars measure sequence storage, not autoregressive output. Short observed outputs need not indicate successful editing; Appendix I defines the accounting.

Method	Mechanism and successful output for She walk to school. $\to$ She walks to school.
CES	Predicted source addresses and dual RoPE. $\mathcal{R},\mathcal{R}_{\mathrm{S}}^{3},\mathcal{R}_{\mathrm{E}}^{3},\texttt{\textvisiblespace walks},\mathcal{R}_{\mathrm{EOS}},\mathcal{E}_{\mathrm{E}}$
AR-DSL	Ordinary BPE text encodes operation, numeric coordinates, and payload. R 2 3 " walks" , then </edit-script>
Flat-relative	Dedicated backward-distance vocabulary tokens; ordinary RoPE. R <\|back_0012\|> <\|back_0013\|> " walks" , then script closure.
Regeneration	Rewrite the complete target, including unchanged text. She walks to school. , then the text terminator.
Oracle-span FIM	Given the target-derived hole between She and ␣to school. : emit ␣walks , then the text terminator.
Predicted-span FIM	Predict boundaries $(2,3)$ through a pointer head; then fill the span: emit ␣walks , then the text terminator.

Table 1: Baseline designs with a shared worked repair, not sampled outputs. Opening controls are supplied; ␣ marks a leading space. The source includes BOS at index 0: DSL/FIM spans are half-open, so walk occupies

[2,3)

. CES adds a boundary marker and uses inclusive addresses, hence

(3,3)

. Appendix F gives full formats. Oracle FIM receives gold locations; adapted CES has an additional adaptation budget.

Figure 3: CES correction with and without post-generation filtering. The same d24 checkpoint and 5,000 CoEdIT inputs, plus 5,000 nominal-clean counterparts, underlie both evaluations. A–C: unfiltered threshold sweep; dashed lines show independently decoded plain greedy and the dotted line marks threshold 0.5. B shows judged counts; C shows all-input improvement minus worsening. All 2,983 distinct changed outputs have reference-blind GPT-5 Mini judgments. D–E: conservative acceptance policy at threshold 0.5. D conditions on 880 retained changes; E uses all-input denominators and includes the human-reference control. CES and human-reference coverage in E are 17.6% and 100%. Intervals are 95% example bootstraps. Separate judge runs make the filtered/unfiltered contrast descriptive. F: manually selected plain-greedy examples without a gate or post-generation filters (5 improved, 3 neutral, 4 worsened). Red strike-through marks deletions and green bold marks insertions; indented continuation lines belong to the same example. Category counts are illustrative, not representative frequencies.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	d12	d18	d24
CES parameters (millions)	287.44	704.55	1,388.84
Causal pretraining updates	26,880	39,600	74,800
Serialized token slots (millions)	880.80	1,297.61	2,451.05
Iterative editor updates	8,960	13,200	24,933
Clean AR reference updates	26,880	39,600	4,675
Additional CES adaptation updates	1,000	1,000	1,000

Appendix

Table 2: Training budgets and optimization settings for the method comparison. Context length is 2,048. Nominal token-slot budgets and clean-token exposure are distinct quantities. Warmup is in updates; LR decay occupies the final stated fraction and ends at 0.05 of the peak rate. ∗ Adaptation weight decay is disabled during its initial frozen-backbone stage. Iterative Levenshtein uses three encoder passes per step; pass-count matching approximates computation, not equal FLOPs or wall-clock cost.

Figure 4: Operation and edit-length distributions in GPT-generated, random, and copy-mined training data. GPT-generated edits are replacement-heavy and longer than random edits; copy mining supplies long repeated spans. Appendix D compares training with these sources.

Figure 5: Position-head weight selection over three paired depth-12 seeds. The selected

\lambda_{\mathrm{pos}}=0.01

is the retention/editing Pareto knee. Error bars are two-sided 95% Student-

t

intervals over training seeds; faint lines show individual seeds.

Figure 6: Dataset-source comparison over three paired depth-18 seeds. Validity must be read together with source copying and nontrivial editing. Error bars are 95% Student-

t

intervals over the three paired seeds.

Figure 7: Retention and transfer as a function of measured edit mass

\rho

, before and after CoEdIT fine-tuning. Points show means and error bars one sample standard deviation over three training seeds.

Policy	Valid	Exact	$\Delta S$	$F_{0.5}$	Closer	Farther
Greedy	0.8870	0.0570	+0.00877	0.01990	0.0536	0.7031
$T=0.2$	0.8960	0.0543	+0.00818	0.01934	0.0605	0.6907
$T=0.5$	0.9163	0.0440	+0.00643	0.02184	0.0770	0.6660
$T=0.8$	0.9163	0.0363	+0.00271	0.02153	0.0674	0.6701
$T=1.0$	0.9100	0.0307	-0.00121	0.01631	0.0488	0.7079
Nucleus $p=0.90$	0.9180	0.0357	+0.00221	0.01923	0.0591	0.6873

Appendix

Table 3: Decoding sensitivity for the depth-24 seed-42 CES checkpoint. Stochastic rows are means over three decoding seeds; the deterministic greedy row is a single evaluation. Bold marks the best value in each column.

Figure 8: CES temperature sensitivity across model depth. Error bars for stochastic settings are one sample standard deviation over three decoding seeds; greedy is deterministic.

Figure 9: Depth-24 policy sensitivity for all methods. Each stochastic cell is the mean of three decoding seeds from one trained checkpoint. A dash marks a not-applicable or unevaluated cell. Zero harm for methods that copy the source must not be read as useful safety; it accompanies zero correction coverage.

Method	BPB	CORE	Valid (%)	Exact (%)	Copy (%)
Clean AR	$0.7484\pm 0.0006$	$0.227\pm 0.007$	–	–	–
CES	$0.7629\pm 0.0015$	$0.201\pm 0.009$	$90.53\pm 1.85$	$5.27\pm 0.45$	$9.53\pm 1.85$
AR-DSL	$0.7634\pm 0.0004$	$0.196\pm 0.017$	$0.00\pm 0.00$	$0.10\pm 0.00$	$100.00\pm 0.00$
Regeneration	$0.9365\pm 0.1079$	$0.148\pm 0.004$	$7.13\pm 3.86$	$0.20\pm 0.10$	$92.93\pm 3.88$
Oracle FIM †	$1.6816\pm 1.0851$	$0.096\pm 0.059$	$35.10\pm 0.44$	$0.10\pm 0.00$	$64.90\pm 0.44$
Predicted FIM	$4.6474\pm 0.1787$	$-0.018\pm 0.005$	$91.23\pm 13.23$	$0.03\pm 0.06$	$16.67\pm 10.48$

Appendix

Table 4: Depth-24 pretrained results, means and sample SDs over three training seeds. BPB is measured on validation text. Controlled metrics use 1,000 examples per seed, except adapted CES (250). Copy includes invalid-as-source fallback. † Oracle FIM receives the reference span. Clean AR uses the same nominal token budget; no edit capability is evaluated. Iterative Levenshtein has no comparable causal BPB or program-validity diagnostic.

Method	Changed (%)	$F_{0.5}$	Closer (%)	Farther (%)	Ctrl. $\Delta S$
CES	$90.0\pm 2.6$	$0.0155\pm 0.0051$	$5.4\pm 1.2$	$68.5\pm 2.7$	$0.0092\pm 0.0035$
AR-DSL	$0.0\pm 0.0$	$0.0000\pm 0.0000$	$0.0\pm 0.0$	$0.0\pm 0.0$	$0.0000\pm 0.0000$
Regeneration	$72.9\pm 2.2$	$0.0733\pm 0.0235$	$15.3\pm 3.8$	$53.9\pm 6.9$	$-0.6819\pm 0.0370$
Oracle FIM †	$97.9\pm 1.0$	$0.0148\pm 0.0011$	$0.4\pm 0.4$	$97.3\pm 1.3$	$-0.2228\pm 0.0062$
Predicted FIM	$37.0\pm 12.2$	$0.0009\pm 0.0004$	$0.8\pm 0.3$	$34.8\pm 12.5$	$-0.0919\pm 0.0685$
Flat-relative	$0.1\pm 0.2$	$0.0000\pm 0.0000$	$0.0\pm 0.0$	$0.1\pm 0.1$	$0.0000\pm 0.0000$

Appendix

Table 5: After supervised CoEdIT fine-tuning, three training seeds. Changed, ERRANT, and reference-distance outcomes use 485 GEC pairs; controlled direction uses a stratified 250-example subset.

F_{0.5}

is on a 0–1 scale. Oracle FIM is information-advantaged. Zero harm with zero changes represents abstention, not effective correction.

Depth	Oracle input	Pre. valid	Pre. exact	FT $F_{0.5}$	Coverage	Closer	Farther
12	Gold pointers	0.288	0.049	0.0303	0.214	0.078	0.039
12	Gold span + ops	0.994	0.077	0.0460	0.726	0.045	0.631
18	Gold pointers	0.337	0.052	0.0275	0.231	0.087	0.052
18	Gold span + ops	0.996	0.083	0.0591	0.635	0.062	0.520
24	Gold pointers	0.399	0.073	0.0245	0.223	0.076	0.052
24	Gold span + ops	0.994	0.084	0.0981	0.868	0.074	0.740

Appendix

Table 6: Single-seed target-aware Oracle CES diagnostics. These protocols are not deployable end-to-end systems and should not be ranked with the main table.

Metric	Adapted CES (1,000 steps)	Mature CES (74,800 steps)
Clean empty BPB	0.8215	0.8681
Corrupt empty BPB	0.8241	0.8706
Sham BPB	0.8250	0.8661
Repair BPB	0.8242	0.8642
Gain vs. empty [95% CI]	$-0.44\ [-2.50,1.80]$	$+21.23\ [18.68,24.22]$
Gain vs. sham [95% CI]	$+2.69\ [0.99,4.77]$	$+6.41\ [3.91,9.56]$

Appendix

Table 7: Held-out natural-text continuation. Gains are millinats per tail token with paired 95% document-bootstrap intervals. The mature checkpoint lowers NLL relative to both empty and non-correcting sham programs. Recovery above 100% is not an accuracy; see the interpretation in the text.

Figure 10: Natural-text continuation likelihood. Left: excess clean-tail NLL relative to clean plus empty program. Right: mature- CES repair gain for every operation, span, and distance aggregate; error bars are paired 95% bootstrap intervals.

Figure 11: Excess tail NLL by distance after edit-end, shown as an eight-token moving mean with pointwise 95% uncertainty bands. The mature checkpoint’s gold-repair curve remains below the corrupt empty-program curve throughout most of the continuation.

Context	Rebuild	Cached continuation	Extra time (%)
(tokens)	(ms)	32 tokens (ms)	32 tokens	256 tokens
128	5.9	172.4	3.4	0.6
512	13.9	172.6	8.0	1.1
1,792	33.0	171.6	19.5	2.3

Appendix

Table 8: Full cache rebuilding before continuation, d24 on RTX 5080. Cached continuation measures generation from an already-prefilled source, excluding prefill and rebuilding. Rebuild measures full prefill of the edited context. Times are medians over three case medians at each length; the first two timing columns use the 32-token continuation condition. Extra time is the mean paired percentage increase of rebuild-plus-edited-text continuation over cached source continuation, computed from per-case median times.

Stage	Updates	Data mixture
Balanced SFT	800	40% human edit programs, 40% matched clean no-ops, 10% synthetic repairs, 10% clean continuations.
Broad RL	100	50% model-generated corruptions, 30% random corruptions, 20% clean no-ops.
GEC RL	200	Localized GPT-5 Mini corruptions of clean CoEdIT training targets.

Appendix

Table 9: Post-training curriculum. SFT uses 40 examples per global update; each RL update uses 32 prompts with eight sampled programs per prompt. RL data mixtures describe rollout prompts; auxiliary supervised losses are specified below.

Metric	Role-specific	Shared token
Validation BPB	0.905563	0.904688
Synthetic pointer accuracy	48.64%	45.81%
Controlled program validity	87.2%	88.0%
Controlled exact recovery ( $n=250$ )	2.4% (6)	3.6% (9)
CoEdIT pointer accuracy	23.12%	23.34%

Appendix

Table 10: Shared-pointer ablation. Three extra exact repairs do not establish a robust advantage.

Pointer Networks

See all 10 papers

Jul 23, 2026

Reviser: Writing and Editing Text with Autoregressive Cursor Actions

Language models write text once, left to right, and models that write in other orders, such as diffusion LMs, pay for it with many full-sequence passes. We introduce Reviser, a decoder-only Transformer that writes and edits text with cursor actions (insert, move, delete, stop) that a deterministic executor applies to a canvas. Reviser is autoregressive over its own actions, so a step costs the same as an autoregressive (AR) step. We train Reviser to continue text, including a self-correction recipe that types a wrong token and deletes it, and Reviser-Edit to revise a draft, and compare each with a same-size AR model trained on the same data, using three LLM judges on continuations cut to the same length. Reviser-Edit beats its matched AR rewriter on CoEdIT SARI (+7.2 at 100M, +4.5 at 300M; two seeds agree) with a third of the decoding steps and latency, though the AR rewriter corrects grammar better. In continuation, Reviser beats diffusion LMs of similar size (62-74% win rate) at a fraction of their compute; two of three judges prefer its AR twin, and one slightly prefers Reviser at 100M. Reviser fills gaps it left and backspaces a just-typed repeated token with probability 0.85-0.87.

Sean Diab

Independent Researcher

Text Generation Language Model Decoding

Ask PDF

Oct 8, 2026

AutoIndexer: Flexible-Order Decoding by Training Causal Models on Chains of Edits

Autoregressive decoding is less suited to non-causally ordered data, e.g., cross-references or future dependencies, since their outputs cannot be revised post hoc, unlike iterative human reasoning. Diffusion LLMs partially mitigate this limitation but still cannot flexibly insert or remove arbitrary numbers of tokens from a pre-generated context. We design AutoIndexer as a generative model with a novel training and decoding mechanism that simulates a Chain of Edits, allowing the model to flexibly modify its prior outputs in hindsight via token insertions, substitutions, and deletions. The model learns to predict marker tokens for opening and closing edits, simultaneously moving a cursor to positions in the existing token sequence, analogous to a text editor. Training sequences are obtained by perturbing ground-truth labels with inverse chained edit operations, and the model learns to edit by undoing those inverses. The perturbation training allows AutoIndexer to be robust to erroneous context and adaptable to editing tasks such as code correction. We demonstrate that our method exhibits post-generation context-editing capabilities that most autoregressive and diffusion models lack without compromising performance on standard LLM benchmarks. The code is available at https://github.com/AutodeskAILab/autoindexer.

Shu Ishida, Aliasghar Khani, Tianyu Zhang +3

Autodesk Research

Language Model Decoding Text Generation Autoregressive Decoding

Ask PDF

Sep 17, 2026

CaLR: Causal Latent Revision for Robust Diffusion Reasoning

Autoregressive (AR) models suffer from local greediness, while diffusion language models (DLMs) often lack the strict causal structure required for reasoning. To combine the advantages and overcome the drawbacks of the dual, we propose Causal Latent Revision (CaLR), a framework that reformulates reasoning as constrained latent optimization. By adopting a causal topology matrix (CTM) from an expert model and implicit differentiation, CaLR performs gradient-guided ``thought revision" to enforce logical consistency, enabling dynamic self-correction of intermediate steps during parallel generation. Empirically, CaLR achieves SOTA DLM performance on complex benchmarks, surpassing strong AR baselines and demonstrating superior robustness in constrained tasks like Sudoku.

Wei Cai, Jian Zhao, Yuchen Yuan +1

Peking University

Reasoning Consistency in Language Models Diffusion Language Models Causal Reasoning in Language Models Language Model Robustness

Ask PDF

Abstract

Pointer Networks

Explore similar work

Reviser: Writing and Editing Text with Autoregressive Cursor Actions

AutoIndexer: Flexible-Order Decoding by Training Causal Models on Chains of Edits

CaLR: Causal Latent Revision for Robust Diffusion Reasoning