cs.LGOct 22, 2026

Benchmarks Are Surprisingly Compaction-Friendly

Authors: Jeanne Salle, Hardik Bhatnagar, Matthias Bethge, Sahar Abdelnabi, Maksym Andriushchenko

Organizations: ELLIS Institute Tübingen · Max Planck Institute for Intelligent Systems · Tübingen AI Center · University of Tübingen

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Long horizon agentic tasks are now pushing LLM agents past their context window. To continue the task, the agent scaffold has to free up the context window, a process known as context compaction. Compaction is generally lossy by design: agents can silently lose task constraints, instructions, or their reasoning behind earlier decisions. Although compaction is a standard component of coding agent scaffolds, it has not been studied systematically with frontier models on realistic, long horizon agentic tasks. Previous work evaluate smaller open weight models on toy or synthetic trajectories. We fix the agent scaffold (OpenCode), implement five compaction strategies inside it, and evaluate three frontier models on three long horizon benchmarks (PaperBench, ProgramBench, PostTrainBench). For the two 1M context models, the compaction threshold does not measurably affect the task performance, whereas GPT-5.3-Codex, the model with the smallest context window, performs worse at lower thresholds on PaperBench and, less clearly, on PostTrainBench. No compaction method outperforms default self-summarization, and methods that discard more content cost some models up to 20 points while leaving other models the same. Compaction can, however, silently remove task constraints: on ProgramBench, the first compaction summary already drops about one third of the task's anticheating rules on average, and this rises monotonically with the number of compactions. On real user-agent sessions from SWE-chat, instructions about how the agent should work and what it should not do are dropped around 6 times more than specifications of what to build. We argue that current benchmarks are compaction-friendly: i.e., the objective is fixed and the agent's progress stays on disk, so most of what compaction discards can be recovered, except constraints that don't exist anywhere else.

Figures & tables

Figure 1: Compaction keeps what the agent can re-read from disk and drops some of the rules it cannot. Left: overview of a ProgramBench run. The task prompt states the goal and anti-cheating rules: no copying or vendoring the reference, no internet access, and an original implementation that does not derive from or wrap the reference. At each compaction the agent replaces its context with a summary. The goal and the agent’s progress survive, because the agent can re-read its files, the rules exist only in the context, so a rule the summary discards is lost. On average, around one of the three rules is missing after the first compaction and around two after the tenth. Right: (1) for the 1M context models, task scores barely change when compaction is forced at thresholds from 160K to 380K tokens; (2) rules are dropped more often as compactions accumulate on ProgramBench, and on real user sessions from SWE-chat, compaction drops process instructions (“how to work”) and scope exclusions (“don’t touch X”) far more often than feature requirements (“what to build”).
Figure 2: The compaction threshold does not affect 1M context models; GPT-5.3-Codex performs worse at lower thresholds on PaperBench. Accuracy conditional on producing a valid submission as a function of the number of tokens before compaction, for each benchmark (columns) and model. All models are run at the same absolute compaction thresholds. Error bars are bootstrap 95% confidence intervals over runs.
Figure 3: Accuracy conditional on producing a valid submission by compaction method, for each benchmark (rows) and model (columns), at a fixed compaction threshold (160K).
Figure 4: Compaction preferentially drops constraints about how the agent should work and what it should not do. (a) On ProgramBench, the task’s anti-cheating rules disappear from the summaries progressively over a run. (b) On real user-agent sessions, process instructions and scope exclusions are dropped much more than feature requirements.
ModelProviderContext window (tokens)API model stringReasoning effort
Claude Opus 4.6Anthropic1Mclaude-opus-4-6max
GPT-5.3-CodexOpenAI400Kgpt-5.3-codexmedium
GPT-5.4OpenAI1,050Kgpt-5.4-2026-03-05medium
Gemini 3.1 ProGoogle1Mgemini-3.1-pro-previewhigh
Table 1: Models evaluated in this study. All models are accessed through OpenCode using the providers’ APIs.
Figure 5: GPT-5.3-Codex accuracy drops after many compactions, Claude Opus 4.6 and GPT-5.4 don’t. Accuracy of each run, relative to the mean accuracy of its task, as a function of the number of compactions in that run. All three benchmarks and all thresholds are included here. Accuracy is conditional on producing a valid submission.
Figure 6: Accuracy as a function of the compaction threshold (160K to 380K tokens). The compaction threshold does not significantly affect accuracy. However, Gemini 3.1 Pro Preview rarely compacts (see Appendix 6.8 ). Figure 8 shows that on average it compacts once per run at the 160K threshold. So the above results are not informative about the effects of compaction.
Figure 7: Failure rate by compaction strategy . Fraction of runs in which the agent failed to produce a valid result, for ProgramBench and PostTrainBench, per model. For each task, the failure rate is the number of failures divided by the number of attempts; bars corresponds to the mean over tasks, so every task has equal weight (ProgramBench: Canop/broot and jarun/nnn ; PostTrainBench: GPQA, GSM8K and BFCL, each on two base models). Error bars are bootstrap 95% confidence intervals. Runs were launched until enough succeeded, so the number of attempts per task differs across tasks and conditions. Pooling all attempts would weight each task by how often it was retried, and since harder tasks fail more and are rerun more, this would inflate the weight of exactly the tasks that fail most. We therefore compute each rate per task and average, so that every task contributes equally and strategies are compared on the same task mix. Tasks with few attempts will thus have noisier rates, which the bootstrap intervals reflect. All runs were obtained at a fixed compaction thresholds (160K tokens before triggering compaction).
Figure 8: Number of compactions as a function of the compaction threshold. Average number of compactions per run for each model, in the baseline condition (self-summarization).
Tokens before    
compactionClaude Opus 4.6Gemini 3.1 Pro PreviewGPT-5.4GPT-5.3-Codex
160k4.90±2.234.90\pm 2.231.00±0.561.00\pm 0.5615.08±6.5815.08\pm 6.5827.71±17.0027.71\pm 17.00
180k6.09±4.456.09\pm 4.450.50±0.330.50\pm 0.3311.08±6.1211.08\pm 6.1210.25±6.8810.25\pm 6.88
200k4.95±3.104.95\pm 3.100.40±0.400.40\pm 0.406.33±4.936.33\pm 4.935.80±2.705.80\pm 2.70
260k3.18±2.053.18\pm 2.050.50±0.500.50\pm 0.504.19±2.714.19\pm 2.711.90±1.141.90\pm 1.14
320k1.82±1.231.82\pm 1.230.00±0.000.00\pm 0.002.68±1.532.68\pm 1.531.29±0.651.29\pm 0.65
Table 2: Number of compactions (mean ±\pm 95% CI) as a function of the tokens-before-compaction threshold, across benchmarks.
ModelMean dropped (of 3)≥\geq 1 droppedAll 3 dropped
GPT-5.3 Codex2.7399%80%
GPT-5.41.7690%25%
Claude Opus 4.60.5143%0%
Table 3: Constraint erosion by model, on ProgramBench. “Dropped” counts, per compaction, how many of the task’s three categories of prohibited behavior are absent from that compaction’s summary (Section 4.2 ).
Figure 9: Mean number of constraint categories dropped per compaction summary as a function of the compaction index within a ProgramBench run, split by model (cf. Figure 4(a) ). Each line is drawn out up to the index where at least five runs have a compaction that deep. From the first compaction, the GPT models already drop more than one of the three categories on average.
Figure 10: Reward hack mechanisms used by each model on ProgramBench. One agent trajectory can use several of these mechanisms at once, so bars count tags, not runs.
Figure 11: Per-category drop rate under two LLM judges (Claude Sonnet 5 and GPT-5.6 Sol), using each judge’s majority verdict over three runs. The categorization is fixed. Error bars are Wilson-score 95% confidence intervals. Dashed lines correspond to each judge’s overall drop rate. Hatched bars indicate categories containing fewer than 10 samples. One constraint with no majority verdict for Sonnet is excluded for both judges (268 constraints in total).
More in this topic

Context Compaction for LLM Agents

See all 32 papers

Explore similar work

CardsList
Jul 6, 2026

CompactionRL: Reinforcement Learning with Context Compaction for Long-Horizon Agents

Long-horizon agentic LLMs are increasingly limited by finite context windows, as extended interaction trajectories can exceed the maximum context length before a task is completed. Context compaction offers a natural solution by summarizing previous interaction states and continuing the rollout under a compressed context, but incorporating compaction into reinforcement learning remains underexplored. We propose CompactionRL, a reinforcement learning strategy to train long-horizon agentic LLMs with context compaction. Our approach jointly optimizes task execution and summary generation with token-level loss normalization and cross-segment generalized advantage estimation. This design enables the LLM agents to learn from compacted long-horizon trajectories. We train CompactionRL on top of open models and observe consistent performance gains on agentic coding tasks. CompactionRL enables the open GLM-4.5-Air model (106B-A12B) to achieve Pass@1 scores of 66.4% on SWE-bench Verified and 26.2% on Terminal-Bench 2.0, exceeding the base model under inference-time compaction by 6.6 and 4.9 points, respectively. Built upon GLM-4.7-Flash (30B-A3B), CompactionRL improves Pass@1 by 5.5 and 6.7 points against the base model, reaching 56.0% on SWE-bench Verified and 20.2% on Terminal-Bench 2.0. CompactionRL is thus deployed in the RL pipeline for training the open GLM-5.2 model (750B-A40B).

Yujiang Li, Zhenyu Hou, Yi Jing +2

Work done while YL, ZH, and YJ interned at Z.AI. · Tsinghua University

Software Engineering AgentsLong-Horizon LLM AgentsRL for LLM AgentsContext CompressionContext Compaction for LLM Agents
AskPDF
Oct 22, 2026

Compact-Bench: Measuring What Context Compaction Costs Long-Horizon Coding Agents

Context compaction lets long-horizon coding agents continue beyond their context windows, but can erase the state needed to finish a task. Prior evaluations measure efficiency, retained information, or end-to-end success, leaving the effect of a specific history replacement difficult to isolate. Compact-Bench pairs full-history and summary-only continuations from a shared repository state, and evaluates recurring details and successive production compactions. It covers 35 SWE-bench repairs, 146 agentic sessions, and 106 production chains. Across 14 summarizer backbones, stable damage ranges from 5-64% of 22 full-history successes: Claude Opus 4.8 preserves the most repairs, while DeepSeek-Flash damages the most. With Kimi-K3 as summarizer, the Kimi Code prompt preserves the most task continuations (19/22), whereas Claude Code retains the most recurring details (15.0%). LLM summaries retain at most 16.0% of recurring details, compared with 79.1% for mechanical extraction; production summaries retain 15.7% at the first boundary. Failures recur across methods: one repair is damaged by 13/14 backbones, and another by all 5 prompts. These results show that reliable compaction requires preserving the progress needed to complete a task, and that retaining relevant information alone does not ensure successful continuation. Code is available at https://github.com/cai-jianfeng/compact-bench and data is available at https://huggingface.co/datasets/caijanfeng/compact-bench.

Ruopei Sun, Jianfeng Cai, Kangwen Zhao +2

University of Science and Technology of China

Context Compaction for LLM AgentsLLM Context CompressionSoftware Engineering AgentsLLM Agent MemoryAI Agent BenchmarksLong-Horizon Agent Evaluation
AskPDF
Oct 18, 2026

Bud ContextCompact 2.0: Context Compaction for Agent Success and Accuracy with Small Language Models and Constrained Context Windows

Agents that work for many turns outgrow the context window of the model that serves them, and on small language models (SLMs) and constrained inference systems that window is small. Context compaction replaces older turns with a summary so a session can continue, but it is lossy, costs a model call, and can help or hurt depending on when it runs. We describe Bud ContextCompact 2.0, the compaction built into the Bud agent runtime, and evaluate it in two settings. On recall questions over recorded agent sessions, compaction to a 16K budget raised a 1.2B-parameter model's accuracy to 0.335, from 0.247 with truncation and 0.176 with its full 32K history (p < 0.001), and raised a 9B model from 0.515 (truncation) to 0.577 (p = 0.04). On 100 SWE-bench Verified tasks run end to end by a 27B reasoning model on a 49K-token window (556 graded sessions), compaction raised the resolve rate on long sessions from 41% to 53% (+12.3 points, 95% CI +1.6 to +23.0, McNemar p = 0.043), left short sessions unchanged, and used 13% fewer tokens per solved task at equal total cost. The main failure without compaction was not the history overflowing but a reasoning turn exhausting the room the history left. The gain has limits: it vanished under a 45-minute session budget, because each summary took a median of 161 s, and in a threshold study it appeared only when compaction triggered early enough to leave room. An 8B model submitted after about ten turns at 5-8K tokens and never needed compaction. We argue that the accuracy results transfer across hardware, including CPU inference, for an equivalent model and window, while the time cost scales with the hardware and must be budgeted for.

Jithin VG, Ditto PS

Bud Ecosystem

LLM Context ManagementLLM Context CompressionContext CompressionSmall Language Models
AskPDF