Editing as Language: Teaching LLM Agents a Grammar for Native Context Editing
Authors: Yunlong Deng, Yan Li, Xing Hu, Yida Zhu, Sean Du, Weiyang Liu, Guangyi Chen
Organizations: The Chinese University of Hong Kong · Meituan LongCat Team · Nanyang Technological University · Mohamed bin Zayed University of Artificial Intelligence · Carnegie Mellon University
A synopsis in novels folds a story into a few lines; a function in code gathers many steps with a single execution. Context editing is native to language itself. In this paper, we propose the Native Context Editing (NCE) framework, which gives language model agents a grammar for natively editing their own context. By encoding context-editing operations in a grammar of dedicated special tokens, we teach the agents to invoke edits through the language itself. We train agents on iteratively constructed trajectory segments, with each teacher edit conditioned on the evolving context. This establishes context editing as an atomic and inherent capability of LLM agents, enabling them to dynamically modify their context as they generate responses over long-horizon reasoning and interaction. Compared with context-editing methods that learn to generate tool-use code within specific external harness environments, our approach integrates context editing as an internal mechanism of the model's native generation process, enabling better generalization across diverse real-world workflows. On four long-horizon agentic benchmarks (Terminal-Bench 2.0, SWE-bench Verified, SWE-bench Multilingual, and SWE-bench Pro), NCE improves the average task success rate over the base model by 5.6 percentage points for LongCat-Flash-Lite-Sparse and by 4.5 for Qwen3.6-35B-A3B, while reducing average token usage by 36% and 15%, respectively.
Figures & tables
Figure 1: Overview of NCE. (a) The agent writes edits inline with its output, each naming a target block by its icon: drop removes a file it no longer needs, rewrite condenses a failing test log to one line; the runtime applies them before the agent continues. (b) Success vs. tokens per task for LongCat-Flash-Lite-Sparse (69B-A3B) on Terminal-Bench 2.0 (Table 2 ). NCE succeeds most with the fewest tokens at every budget.
Figure 2: An abridged excerpt of an NCE training trajectory on a Go repository. (a) The rendered context at turn t ; (b) the model’s output, a rationale followed by three edits and an action; (c) the blocks after execution. More detailed examples can be found in Appendix E .
Figure 3: Editing supervision from two teachers. Left: the offline teacher reads the whole trajectory and releases a block only by citing the event that ended its use ( bi⊆bj : bj supersedes or consumes bi ). Right: the online teacher reads only up to b4 and edits released blocks, here rewriting b2 and dropping b3 .
Figure 4: Distillation with evolving contexts. Each row supervises one turn (solid box) on the prefix left by all earlier edits (dotted box). Each editing turn closes a training sequence, so the five rows form four sequences (rows 1, 2–3, 4, and 5). Dropped blocks are drawn only for alignment.
LongCat-Flash-Lite-Sparse (69B-A3B)
Qwen3.6-35B-A3B
Success (%) ↑
Tokens ( ×106 ) ↓
Success (%) ↑
Tokens ( ×106 ) ↓
Benchmark
rules
tools
native
rules
tools
native
rules
native
rules
native
Comp.
Tools
NCE
Comp.
Tools
NCE
Comp.
NCE
Comp.
NCE
TB 2.0
32.6
32.6
38.2 +5.6
6.62
6.38
4.60 − 31%
38.2
43.8 +5.6
2.98
3.04 +2%
SWE Verified
65.2
62.4
70.0 +4.8
4.07
3.40
2.70 − 34%
69.8
74.0 +4.2
2.92
2.19 − 25%
SWE Multiling.
56.0
54.3
62.3 +6.3
10.83
10.93
6.12 − 44%
58.0
64.3 +6.3
5.12
4.13 − 19%
Table 1: Results at a 256K budget for harness compaction (Comp.), tool-call editing (Tools), and NCE ; TB 2.0 is Terminal-Bench 2.0, SWE is SWE-bench, and tokens are per task. Bold marks the best per base model. Small numbers give the success gain of NCE over the best baseline and its token change relative to harness compaction.
Success (%) ↑
Tokens ( ×106 ) ↓
Budget
Comp.
Tools
NCE
Comp.
Tools
NCE
256K
32.6
32.6
38.2 +5.6
6.62
6.38
4.60 − 31%
128K
25.8
30.3
34.8 +4.5
6.08
4.96
3.70 − 39%
64K
27.0
29.2
32.6 +3.4
3.42
3.58
2.52 − 26%
48K
20.2
27.0
30.3 +3.3
3.40
3.17
2.15 − 37%
Table 2: LongCat on Terminal-Bench 2.0 as the context budget shrinks .
Figure 5: Context management on Terminal-Bench 2.0 with LongCat under a 128K budget. (a) Success rate vs. tokens per task. (b) Context length vs. unmanaged history length (median and interquartile range); the drop of NCE at 128K is the harness compaction it keeps.
Figure 6: Building editing supervision. (a, c) Share of the 40 trajectories with at least one edit of each kind (95% bootstrap intervals). (b) Share of the original history no longer in the agent’s context at each editing decision, over 4,250 training trajectories (median and interquartile range).
Figure 7: Operator use of NCE with LongCat at 256K. (a) Claude Code pools the three SWE-bench sets, and Terminus-2 runs Terminal-Bench 2.0; append removes no tokens. (b) Each operator’s removed tokens by the tool that produced the block.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Table 3: The dedicated tokens of NCE. Operators are shown with the chips used throughout the paper; their token strings are <op_drop> , <op_rewrite> , and <op_append> .
Figure 8: The beginning of a rendered context. Each block opens with its address; the system prompt carries the lock marker. Actions, set in italics here, are assistant turns without an address. Contents are abridged with […].
Figure 9: One editing turn, executed. Left: the output of NCE at turn 22 of a SWE-bench Multilingual task under Claude Code, with no reminder pending; the edits are typeset with the chips of Table 3 . Right: every block of the context before the turn and its size after it. Blocks 10, 12, 14, and 16 are earlier reads of the function that block 19 shows in its fixed form; blocks 13, 15, 17, and 18 confirm intermediate edits to it. Rewritten blocks keep their addresses; dropped addresses are retired.
Figure 10: The annotation pipeline. Only the online teacher writes edits; the offline teacher decides which blocks it may touch, and the audit removes edits that fail a mechanical check.
Figure 11: Teacher prompts, abridged ( […] ). Placeholders filled by the annotation driver are highlighted. The online teacher sees no turn after t , no release point, and no token count.
Figure 12: Annotation of one trajectory. (a) Release points of four blocks; kb is the turn of the cited event and λb the last turn at which a rare literal of the block reappears. (b) The online teacher’s rationale and edits at a decision point, in the NCE grammar; block n has address <sid_hi_00> <sid_lo_ n > .
Statistic
Value
Segments per trajectory
≈ 2.9
Decision points per trajectory
≈ 2.0
milestone / soft / hard
60 / 38 / 2%
Segments without edits
31%
Edited blocks: rewrite / drop
35 / 65%
Appended notes, share of operations
2%
Appendix
Table 4: Statistics of one annotation batch. A drop that names several blocks counts once per block.
Figure 13: The same two edits in NCE and in tool-call editing. NCE edits vanish once executed, while each tool call stays in the history with its result message.
Context (K)
Method
Solved
Success (%)
Tokens (M)
Mean
Peak
Reductions
Turns
Harness compaction
23
25.8
6.08
38.3
79.7
0.6
119
Sliding window
23
25.8
4.62
43.6
71.5
1.7
92
Fixed memory state
26
29.2
5.59
44.0
72.9
1.9
95
Prompt compression
29
32.6
4.68
38.3
66.0
14.0
95
NCE
31
34.8
3.70
30.5
61.1
3.1
99
Appendix
Table 5: Five ways to manage context on Terminal-Bench 2.0 with LongCat under a 128K budget. Solved counts are out of 89 tasks; tokens, context, reductions, and turns are per task.
Trajectories (%)
One pass
Original
NCE
Uses future information
30.0
12.5
5.0
Deletes evidence used later
72.5
40.0
37.5
Edits a removed block
0.0
95.0
0.0
Restates removed text
7.5
85.0
2.5
Any defect
80.0
100.0
37.5
Defective edits (%)
4.5
22.5
1.1
Appendix
Table 6: Defects of three ways to build supervision on 40 trajectories. Original: two teachers on the original history.
Figure 14: Editing at inference. (a) Context NCE maintains when the unedited history first fills the budget on Terminal-Bench, as a share of the budget (median and interquartile range). (b) Operation mix at 256K and share of edits executed as written. (c) Cumulative share of editing turns by re-prefill time over idle time at 9.5K tokens/s; turns left of 1 hide the re-prefill entirely.
Table 7: Editing behavior of NCE at 256K, in percent except the two per-task counts and the tokens removed per editing turn. On Terminal-Bench, peak context reduction covers the turns before the first harness compaction.
Prefill rate
Terminus-2
Claude Code
5K tokens/s
92
62
9.5K tokens/s
98
81
20K tokens/s
100
93
40K tokens/s
100
98
Time hidden (9.5K)
98
81
Appendix
Table 8: Share of editing turns (%) whose re-prefill fits entirely in the idle time, at four prefill rates ; bold marks the rate measured from our traces. The last row gives the share of re-prefill time hidden.
Figure 15: Re-prefill time against idle time for every editing turn at 9.5K tokens/s. Points in the shaded region are hidden entirely.
Figure 16: NCE keeps a Terminal-Bench task inside a 48K budget by folding finished experiments into notes. Left: context length per turn. The dashed line adds back the tokens removed by earlier edits; the shaded gap is what NCE keeps out of the context. Numbered markers are editing turns, colored by the operation that removes the most tokens. Right: rationale (italics) and edits at each editing turn, with the net tokens the turn removes. Each failed configuration becomes one line with its settings and error, and the raw logs go once a note covers them. Terminus-2 counts the unedited history and hands off at turn 40 (dotted line).
Figure 17: The same model edits its context the same way under Claude Code. Both editing turns come without a reminder. The first folds a finished reproduction phase into one note with the dead ends and the root cause, then drops the superseded scripts and failing runs; the second rewrites spent file reads and edits into one-line records, keeping which fix was replaced and which one is final. Plot and excerpt conventions follow Figure 16 .
Recent advances in large language models (LLMs) have enabled agents to tackle long-horizon tasks across diverse environments. To further improve agent performance, existing language world models typically predict environment observations, yet reconstructing high-entropy, execution-dependent tool responses offers limited value when real feedback is available. Meanwhile, agents suffer from task-state contamination, where unsupported assumptions and outdated plans persist in history and distort subsequent decisions. We propose the Agent-Editing World Model (AEWM), which models how reasoning and actions shape future task progress rather than simulating tool responses. AEWM combines Action Judge to distinguish Critical, Exploratory, and Noisy decisions with State Revision to edit noisy reasoning--action continuations from the same observed history. EditAct integrates these capabilities with real execution, directly changing the state underlying subsequent decisions rather than merely providing critiques. We train AEWM across Search, Terminal, and Software Engineering through mid-training and supervised fine-tuning. AEWM achieves 70.5% macro-F1 on our Action Judge benchmark, exceeding the strongest frontier baseline by 10.6 points. Across six benchmarks and three agent backbones, EditAct improves average scores by 3.2--6.7 points over the strongest baseline. Furthermore, rejection sampling fine-tuning on verified EditAct trajectories, termed AEWM-RFT, improves over Self-RFT by 2.2--2.6 points across three domains without online AEWM guidance.
Shuang Sun, Guoxin Chen, Fanzhe Meng +6
Gaoling School of Artificial Intelligence, Renmin University of China
The increasing complexity of agentic tasks has led to rapidly growing trajectory lengths, which poses significant challenges for large language model (LLM) based agents with fixed context windows. Existing context management techniques, such as truncation and summarization, suffer from inherent inflexibility and irreversibility: once information is discarded or compressed, it cannot be recovered even when it becomes critically relevant in later decision steps. To address these limitations, we propose the Adaptive Context Elasticizer (ACE), a plug-and-play module that elastically orchestrates historical step information into the agent's context at each decision step. ACE maintains a lossless message maintenance layer that stores both raw messages and compressed abstractions for each historical step, while a context orchestration layer adaptively assigns each step an elastic type as raw, abstract, or drop, at every decision step based on the current task state. This reversible design ensures that the main LLM always receives a compact yet information-rich context. We adapt ACE to four diverse agent frameworks, including ReAct, DeepAgent, WebThinker, and MiroFlow, without training or architectural modifications. Experiments show that ACE consistently outperforms truncation and summarization baselines, and brings consistent performance gains across all four agent frameworks.
Although existing model editing methods perform well in recalling exact edit facts, they often struggle in complex scenarios that require deeper semantic understanding rather than mere knowledge regurgitation. Leveraging the strong contextual reasoning abilities of large language models (LLMs), in-context learning (ICL) becomes a promising editing method by comprehending edit information through context encoding. However, this method is constrained by the limited context window of LLMs, leading to degraded performance and efficiency as the number of edits increases. To overcome this limitation, we propose InComeS, a flexible framework that enhances LLMs' ability to process editing contexts through explicit compression and selection mechanisms. Specifically, InComeS compresses each editing context into the key-value (KV) cache of a special gist token, enabling efficient handling of multiple edits without being restricted by the model's context window. Furthermore, specialized cross-attention modules are added to dynamically select the most relevant information from the gist pools, enabling adaptive and effective utilization of edit information. We conduct experiments on diverse model editing benchmarks with various editing formats, and the results demonstrate the effectiveness and efficiency of our method.
Shuaiyi Li, Zhisong Zhang, Yang Deng +6
The Chinese University of Hong Kong · City University of Hong Kong · Singapore Management University +1