cs.AISep 29, 2026
SaveContext Language Models
Organizations: University of Washington · Meta Superintelligence Labs · MIT · Trillium Labs
Abstract
We introduce Context Language Models (CLMs), language models that natively manage their own context. We implement this by treating the context as a file and allowing the model to make unrestricted updates to this file. This allows the model to learn what is most important to maintain in context, and naturally extends to multi-agent systems where multiple agent contexts coexist as files. Building CLMs zero-shot with existing models outperforms SOTA context management strategies across a variety of tasks: 11.4% higher accuracy with 21.5% fewer FLOPs on BrowseComp-Plus, 5% higher scores with 59% fewer FLOPs on 12-hour EdgeBench, and 65% greater improvement with the same compute on a 24-hour multi-repository agent-swarm task. Moreover, by shifting context management from external harness control to intrinsic model behavior, CLMs naturally enable both in-context and parametric learning of context-management strategies. We show that CLMs can be steered with natural-language instructions evolved through a standard skill-optimization loop, improving held-out accuracy by up to 35.9 points on a context-management task while reducing compute. We also introduce an online reinforcement learning method for CLMs, improving Qwen3.5-9B performance on BrowseComp-Plus by 47.6% while using 12% fewer FLOPs. Finally, we co-design Suffix Cache Reuse for CLM serving, further reducing server-side compute by 35% relative to standard SGLang at matched performance.
Figures & tables
Figure 1: Context Language Models (CLMs) natively manage their own context by treating context as a file. CLMs work out of the box and can be further improved through in-context learning and reinforcement learning. (a) Qualitative examples of creative context-management behaviors introduced by CLM. (b) Out of the box, CLMs improve performance at lower cost on BrowseComp-Plus, a deep-research benchmark, and Software World, where an agent swarm jointly optimizes six interdependent repositories. (c) CLMs can follow textual instructions to adopt corresponding context-management strategies (top) or evolve better strategies through a skill-evolution loop on ContextBench (bottom). (d) CLMs can explore and internalize context-management strategies through online reinforcement learning. By using a success-gated efficiency advantage for stepwise GRPO, we improve both accuracy and efficiency for CLMs simultaneously.
Figure 2: Illustration of the four tasks in ContextBench and a performance comparison of CLM against baselines using GPT-5.4 with a 32K context limit.
Figure 4: Comparison of standard serving and Suffix Cache Reuse (SCR). Standard serving reuses only prefix-matched cache, while SCR reuses cached states for all surviving tokens, reducing re-prefilling.
Figure 6: EdgeBench-10 single-repository optimization.
Figure 11: Comparison of Suffix Cache Reuse and standard SGLang serving on BCP with Qwen3.6-27B. Left : Task accuracy and prefix-reuse FLOPs per question. Right : Server-side compute decomposition, showing the fraction of prompt tokens by compute type across all turns and turns following context edits.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 12: Suffix Cache Reuse for full attention layers and linear attention layers.
Figure 13: Small-scale sensitivity study of relocated spans per edit, . Results on 64 BrowseComp-Plus questions with Qwen3.6-27B. Left: task accuracy. Middle: prefix-reuse PFLOPs per question. Right: relocated tokens per edited request. Dashed lines show standard SGLang; hollow markers show a repeat run; the shaded column marks , used elsewhere. Error bars show standard error across questions; the gray band shows standard error for standard SGLang.
Figure 14: Suffix Cache Reuse also benefits standard chat serving. On BrowseComp-Plus, SCR reuses cached states not only after model-driven context edits but also after reasoning tokens are stripped from prior turns in standard chat endpoints. Results use Qwen3.6-27B on 830 questions with .
Figure 15: Remaining re-prefill under Suffix Cache Reuse on BrowseComp-Plus. SCR removes most re-prefilling of unchanged suffix tokens, while substantial unchanged-prefix prefill remains due to the current hybrid-model caching behavior in SGLang. Same runs as Figure 14 .
Figure 16: FLOPs of one Qwen3.6-27B turn for illustrative lengths ( , ) and three reusable-prefix lengths . Colors split the FLOPs actually computed by operation; gray shows the additional computation that would be required without prefix caching.
| Task | Input stream | Queries | Metric |
|---|---|---|---|
| Needle Retention | 4K-token chunks, each with 2–8 needle lines and 140 filler lines | none | needle lines retained verbatim in the final context |
| Sudoku Sketchpad | one move per turn on a board | current board after each move | board versions reproduced exactly |
| KV Store | batches of 100 SET operations with random 24-word values | 24 GET queries | exact-value accuracy |
| Log Triage | batches of 14–54 service log lines | 24 lookup and count queries | exact-answer accuracy |
Table 3: The four ContextBench tasks. All metrics are computed from the agent’s context; answers held only in files are not credited.
Figure 17: Context pressure of every ContextBench level. Each row varies one generator setting (unit at right); the label on each point is its value. Total input is every message the environment pushes over the episode, including the task instruction and, for Sudoku, the board once per move, counted in o200k tokens; pressure is total input divided by 32,768.
| Method | Excerpt of the KV Store skill |
|---|---|
| CLM | Mechanism: Your context is mirrored to a file you can edit; changing that file changes what you are holding. Each batch: On the SAME turn you read a SET-BATCH, move the whole block out of your context and onto disk with this one command [a python3 heredoc that moves the block to /tmp/ctx_offload/ and leaves one placeholder line]. Each GET: Recover the value with grep -h ’SET <key> =’ /tmp/ctx_offload/* and then print the ANSWER block. Check: No note, or a token count that did not drop, means the batch is still in your context. |
| ACM | Mechanism: manage_context takes everything since your previous manage_context call … and replaces it with one message [summary_id: N] <summary>. Each batch: Read the gauge after every batch; when it is above 22,000, call manage_context on your very next turn, before you release another batch. Each GET: Spend one turn on query_memory(N, “the SET line for key K00084, verbatim”) with N the summary whose range covers the key. Check: What confirms a compression is the [summary_id: N] message and a [context: ~N/M tokens] readout lower than the turn before. |
| RLM | Mechanism: The REPL namespace persists: every variable and function you define survives from one operation to the next. That persistence is your store. Each batch: On the first operation define the store and the handler once; on every later operation only call them [a dictionary filled by a regular expression over the SET lines]. Each GET: A GET is answered by placing the ANSWER block in answer["content"] . Check: print(out) shows stored N keys so far growing by one batch per SET operation. |
| Context Folding | Mechanism: A branch begins from a copy of a context that already holds every batch and cannot delete anything from MAIN, so MAIN is as full after it as before. Each batch: On a SET-BATCH, run one command in MAIN and nothing else: echo READY_FOR_NEXT_OP. Do not open a branch. Each GET: Answer in MAIN … by finding the SET <key> = line in the batch in front of you and printing its value. Check: The [context: ~N/M tokens] readout should rise by the size of each batch and by almost nothing else. |
| Self-Compact | Mechanism: Compression happens only when C1=Y, C2=Y, C3=Y, N1=N. Then your whole history is replaced by … your summary. Each batch: On the SAME turn a SET-BATCH arrives, copy the whole block … to its own file. Answer [the probe] so that compression fires as soon as everything you have seen is on disk. Each GET: grep -h " ˆ SET K00084 = " /tmp/store/*.txt , then print the value in an ANSWER block. Check: What confirms a store is the count grep -c prints back, equal to 100. |
| Summary | Mechanism: At three quarters of the budget (about 24.6K of 32,768 tokens) … everything except the system prompt and this task message is then replaced by one message. Each batch: On the SAME turn a SET-BATCH arrives, copy the whole block … to its own file. Write the summary as a pointer, not an inventory. Each GET: grep -h " ˆ SET K00084 = " /tmp/store/*.txt , then print the value in an ANSWER block. Check: What confirms the store is the grep -c count coming back as 100. |
Table 4: Excerpts of each method’s KV Store skill. Every skill gives the same four kinds of guidance for its own mechanism: how the mechanism works, what to do with each batch, how to answer a query, and how to confirm that a step worked. Blue marks the concrete instruction for using the method’s own tools; … marks omitted text and square brackets paraphrase code.
| Task | Category | Language | What the verifier scores | Start: files / LOC |
|---|---|---|---|---|
| ad_placement_optimization | Combinatorial optimization | C++ | solution score over judge cases | 1 / 20 |
| apple_incremental_game | Combinatorial optimization | Python | solution score over judge cases | 3 / 101 |
| graph_node_classification | Science & ML | Python | held-out accuracy (CPU-only judge) | 2 / 418 |
| grid_turing_robot | Combinatorial optimization | Python | solution score (lower is better) | 3 / 440 |
| juliet_vulnerability_analyzer | Software engineering | Python | hidden evaluator on the Juliet suite | 1 / 9 |
| openrct2_theme_park_ai | Games & simulators | JavaScript plugin | park value in a headless OpenRCT2 run | 6 / 5,950 |
Table 5: EdgeBench-10 tasks. Scores are rescaled to 0–100 by the EdgeBench judge. Start: files and lines of code in the working directory before any edit.
Figure 19: Qwen3.5-9B and Qwen3.6-27B. Accuracy against compute for Qwen3.5-9B (top) and Qwen3.6-27B (bottom) on BrowseComp-Plus, TerminalBench 2.1 and TBLite with a 32K context limit. Cost is prefix-reuse PFLOPs per question; dashed lines indicate the Pareto frontier.
Figure 20: Circle packing ( ).
Figure 25: Score against time.
Figure 28: Evolving context-management skills on ContextBench (32K budget), all four tasks. Assisted evolution (top): Qwen3.6-27B serves as the agent, while Claude Fable 5.1 proposes the skills; the x-axis shows prefix-reuse PFLOPs per task over solved runs. Self-evolution (bottom): Opus 5 serves as both the agent and the skill proposer; the x-axis shows gateway cost per task in USD. Skills are proposed based on training-split rollouts and never use the held-out evaluation set.
Figure 29: RL training curves. Qwen3.5-9B trained on OpenResearcher and evaluated on BrowseComp-Plus with a 32K context limit, through training step 70. (a) Accuracy and prefix-reuse PFLOPs per question against training step. (b) Accuracy against compute, shaded from light to dark by training step.
Figure 30: Context length awareness. Each point compares a model’s estimated context length with the provider-reported prompt length for the same 50 inputs. Panels vary the available hint: none or a token-count anchor at 25%, 50%, or 75% of the input. The dashed line indicates perfect calibration; legend values report mean absolute error (MAE) in tokens.