Compact-Bench: Measuring What Context Compaction Costs Long-Horizon Coding Agents

Ruopei Sun; Jianfeng Cai; Kangwen Zhao; Wengang Zhou; Houqiang Li

cs.SEOct 22, 2026

Compact-Bench: Measuring What Context Compaction Costs Long-Horizon Coding Agents

Authors: Ruopei Sun, Jianfeng Cai, Kangwen Zhao, Wengang Zhou, Houqiang Li

Organizations: University of Science and Technology of China

Abstract

Context compaction lets long-horizon coding agents continue beyond their context windows, but can erase the state needed to finish a task. Prior evaluations measure efficiency, retained information, or end-to-end success, leaving the effect of a specific history replacement difficult to isolate. Compact-Bench pairs full-history and summary-only continuations from a shared repository state, and evaluates recurring details and successive production compactions. It covers 35 SWE-bench repairs, 146 agentic sessions, and 106 production chains. Across 14 summarizer backbones, stable damage ranges from 5-64% of 22 full-history successes: Claude Opus 4.8 preserves the most repairs, while DeepSeek-Flash damages the most. With Kimi-K3 as summarizer, the Kimi Code prompt preserves the most task continuations (19/22), whereas Claude Code retains the most recurring details (15.0%). LLM summaries retain at most 16.0% of recurring details, compared with 79.1% for mechanical extraction; production summaries retain 15.7% at the first boundary. Failures recur across methods: one repair is damaged by 13/14 backbones, and another by all 5 prompts. These results show that reliable compaction requires preserving the progress needed to complete a task, and that retaining relevant information alone does not ensure successful continuation. Code is available at https://github.com/cai-jianfeng/compact-bench and data is available at https://huggingface.co/datasets/caijanfeng/compact-bench.

Figures & tables

Figure 1: Paired Task Preservation replay. (a) Phase A fixes history

H

and repository snapshot

D

. (b) PRE receives

H

directly; only POST receives the summary

S=C(H)

. Each run restores

D

and uses matched agent, tool, environment, and step-budget conditions. (c) Repository FAIL_TO_PASS tests classify the paired outcomes; rows denote PRE and columns denote POST.

Evaluation	Size	Composition
Task Preservation	35 instances	Localized SWE-bench repairs from 7 repositories: pytest (18), sympy (6), xarray (3), pylint (3), requests (2), sphinx (2), and flask (1).
Recurring-Detail Retention	146 sessions	143 real and 3 simulator-synthesized sessions; clients: Kimi Code (73), Claude Code (50), opencode (14), and other clients (9); 9,330 recurring details.
Production-Chain Retention	106 sessions	224 production compaction boundaries; median 2 and maximum 4 boundaries per session.

Table 1: Datasets for the three Compact-Bench evaluations.

Summarizer	Round 1	Stable	95% CI
Claude Opus 4.8	$5\%$ (1)	$5\%$ (1)	$1-22$
Kimi-K3	$18\%$ (4)	$18\%$ (4)	$7-39$
GPT-5.6-Luna	$18\%$ (4)	$18\%$ (4)	$7-39$
GPT-5.6-Sol	$27\%$ (6)	$27\%$ (6)	$13-48$
Claude Fable 5	$41\%$ (9)	$36\%$ (8)	$20-57$
Claude Fable 5.1	$36\%$ (8)	$36\%$ (8)	$20-57$

Table 2: Task Preservation damage under the Claude Code prompt and Kimi-K3 continuation agent. Entries give the rounded percentage and count among

22

validated PRE-pass cases. Stable damage requires POST failure in at least two of three runs; intervals are Wilson

95\%

confidence intervals for that rate.

Prompt	POST pass $\uparrow$	Detail retention $\uparrow$
Kimi Code	19/22 = 86%	8.2%
Claude Code	18/22 = 82%	15.0%
Gemini CLI	17/22 = 77%	7.2%
Codex	16/22 = 73%	9.5%
opencode	16/22 = 73%	9.5%

Table 3: Task Preservation POST-pass counts among 22 validated PRE-pass instances and Recurring-Detail Retention mean scores over the separate 146-session set, both using Kimi-K3 as summarizer.

Backbone	Detail-retention score $\uparrow$
DeepSeek-Flash	16.0%
Kimi-K3	15.0%
GPT-5.6-Sol	14.8%
GPT-5.6-Luna	13.3%
DeepSeek-V4-Pro	11.9%
GLM-5.3	9.5%

Table 4: Recurring-Detail Retention score by summarizer backbone (Claude Code prompt). Per-backbone valid

n

is 144–146 of 146 sessions. API or parse errors are excluded, while empty or whitespace summaries score 0 and remain included.

Figure 3: Mean retention

R_{k}

per compaction index across 106 production compaction chains. Sample sizes shrink quickly with

k

Damaged cell	Target file in $S$	Fix evidence in $S$
pytest-10081	yes (1/1)	8/12
pytest-7490	yes (1/1)	10/12
pytest-7571	yes (1/1)	5/5
sphinx-9230	yes (1/1)	4 listed strings

Table 5: Retained repair evidence in the four stably damaged cells under Kimi-K3 summaries. The target file appears in all four; fractions give retained gold-patch tokens for the pytest cases. The sphinx row lists four strings from the worked example and has no exhaustive token denominator.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Instance	K3	Luna	Sol	Astra	Terra	Opus	F5	F5.1	O5.5	Qwen	GLM	V4Pro	Hy4	Flash
flask-5014	P	P	P	D	D	P	D	P	P	P	P	D	D	D
req-5414	P	P	P	P	P	P	P	P	P	P	P	P	P	P
req-6028	U	–	–	–	–	–	–	–	–	–	–	–	–	–
xa-3677	P	P	D	P	P	P	D	P	P	P	P	P	D	D
xa-4629	P	P	P	P	P	P	P	P	P	P	P	P	P	P
xa-6721	P	D	P	P	P	P	P	P	P	D	P	P	P	P

Appendix

Table 6: First-round outcome of every Task Preservation cell under the fourteen backbone summarizers (column headers use short forms: K3 = Kimi-K3, Luna = GPT-5.6-Luna, Sol = GPT-5.6-Sol, Terra = GPT-5.6-Terra, Opus = Claude Opus 4.8, F5 = Claude Fable 5, F5.1 = Claude Fable 5.1, O5.5 = Claude Opus 5.5, Hy4 = Hy4-Preview, Astra = GPT-6-Astra, Qwen = Qwen3.8-Max, GLM = GLM-5.3, V4Pro = DeepSeek-V4-Pro, Flash = DeepSeek-Flash). P: preserved (both conditions pass); D: damage (PRE passes, POST fails); R: rescue (PRE fails, POST passes); U: unsolved (both fail); X: excluded PRE-failing instance; –: not evaluated (the cell is outside the cross-backbone PRE-pass comparison).

Model	Instance	$r_{1}$	$r_{2}$	$r_{3}$	Result
K3	pt-10081	D	P	D	SD
K3	pt-7490	D	P	D	SD
K3	pt-7571	D	P	D	SD
K3	sp-9230	D	U	D	SD
Luna	xa-6721	D	D	D	SD
Luna	pt-10081	D	D	D	SD

Appendix

Table 7: Per-seed verification of first-round damaged Task Preservation cells. P: POST passes; D: POST fails with a passing PRE; U: PRE fails (possible only for Kimi-K3, which reruns both conditions); –: no run. Other backbones resample the summary and POST while reusing the first-seed PRE baseline. SD denotes POST failure in at least two of three seeds; V denotes seed variance and is excluded from stable-damage counts.

Table 10

Instance	v1	v2	v3	Observation
sympy-13480	U	P	P	environment fix; preserved in v3
sympy-13091	R	U	U	v1 rescue not reproduced
xarray-6721	–	D	P	v2 path-leak damage
requests-5414	R	–	P	v1 rescue not reproduced
sphinx-9230	–	D	D	damage persists in v3
pytest-7521	–	D	P	v2 path-leak damage

Appendix

Table 11: Interpreter and directory validation (v1: system interpreter; v2: instance interpreter, separate directories; v3: single directory with snapshot reset). Outcome codes as in Table 6 .

Instance	Repo	#F2P	Environment
flask-5014	flask	1	werkzeug<3
req-5414	requests	1	urllib3<2
req-6028	requests	2	urllib3<2
pt-10051	pytest	1	setuptools_scm shim
pt-10081	pytest	1	setuptools_scm shim
pt-10356	pytest	1	setuptools_scm shim

Appendix

Table 12: Task Preservation instances with FAIL_TO_PASS test counts and environment recipe notes. All environments use pytest

<

8 and a per-instance venv.

Instance	Claude	Codex	Gemini	opc	Kimi
flask-5014	P	P	P	P	P
req-5414	P	F	P	P	F
xa-3677	P	P	P	F	P
xa-4629	P	P	P	P	P
xa-6721	P	P	P	P	P
pl-6903	P	P	P	F	P

Appendix

Table 13: First-round Task Preservation outcomes under the five production compaction prompts with Kimi-K3 as the summarizer. Codes: P = POST passes; F = POST fails; X = excluded PRE-failing instance. All non-excluded listed cells pass the PRE control. Prompt headers abbreviate Claude Code, Codex, Gemini CLI, opencode, and Kimi Code.

#	tool	PRE (full history)	POST (summary only)
1	c / r	sed -n 290,340p sphinx/domains/python.py	sphinx/util/docfields.py
2	c	cd /tmp && rm -rf reprotest && mkdir reprotest && cd reprotest && cat > index.rst <<’…	cd /testbed && git log –oneline -3 && git status
3	c	cd /tmp/reprotest && python - <<’EOF’ import re html = open(’_build/index.html’).read…	pwd && ls && git log –oneline -3 2>/dev/null \| head
4	c	cd /tmp/reprotest && python - <<’EOF’ import re html = open(’_build/index.html’).read…	sed -n 190,320p sphinx/util/docfields.py
5	c	sed -n 295,310p sphinx/util/docfields.py	grep -n "split\|import\|def " sphinx/util/docfields.py \| head -40
6	c	cd /home/user/sphinx 2>/dev/null \|\| pwd; python - <<’EOF’ import re path = ’sphinx/ut…	sed -n 890,1010p tests/test_domain_py.py

Appendix

Table 14: Paired sphinx-9230 action traces (

r_{1}

, Kimi-K3 summarizer), truncated to 85 characters per action. Tools: c = run_command , r = read_file ; differing tools are listed as PRE/POST. Rows align action order. PRE ends after 8 actions and POST after 13, within the eight-step continuation budget; the harness judges FAIL_TO_PASS tests afterward. PRE applies the repair, while POST continues inspecting tests and reproduction code.

Cell	PRE redo-reads	POST redo-reads
pytest-10081	1	0
pytest-7490	1	0
pytest-7571	0	1
sphinx-9230	1	1

Appendix

Table 15: Repeated read_file calls targeting files read in Phase A for the four stably damaged cells, Kimi-K3 summarizer.

Context Compaction for LLM Agents

See all 33 papers

Prune the Summary, Not the History: Source-Judged Compaction for Long-Horizon Agents
Oct 14, 2026Guanghui Min, Chen ChenContext Compaction for LLM Agents LLM Agent Reliability
Department of Computer Science, University of Virginia
Benchmarks Are Surprisingly Compaction-Friendly
Oct 22, 2026Jeanne Salle, Hardik Bhatnagar, Matthias Bethge +2Context Compaction for LLM Agents LLM Context Compression
ELLIS Institute Tübingen · Max Planck Institute for Intelligent Systems · Tübingen AI Center +1
Parallel Context Compaction for Long-Horizon LLM Agent Serving
May 22, 2026Musa Cim, Burak Topcu, Chita Das +1LLM Inference Efficiency LLM Agent Workflow Optimization
The Pennsylvania State University

Oct 14, 2026J/K move · Enter open · S save

Prune the Summary, Not the History: Source-Judged Compaction for Long-Horizon Agents

Guanghui Min, Chen Chen

Department of Computer Science, University of Virginia

Context Compaction for LLM Agents LLM Agent Reliability LLM Agent Memory Text Summarization

Open paper AskPDF arXiv

Long-horizon LLM agents rely on context compaction: when the interaction history outgrows the context window, a compressor replaces it with a natural-language summary. We show that these summaries contain more than a record of what happened. Compaction prompts ask for decisions and next steps, so each summary mixes facts established by the environment with reasoning written by the compressor, and the agent cannot tell the two apart afterwards. In an analysis of a widely used structured compressor, 83% of failed runs trace back to an action that carries out a specific summary entry, most of which no environment output supports. We propose Source-Judged Compaction (SJC), a training-free post-processing step that scores each summary entry for whether it has a source in the environment outputs or the previous summary, and removes the entries that do not. It changes neither the agent nor the compressor and adds one inexpensive judge request per compaction. The judge is Jev, a non-generative decision model that answers the source question with a calibrated probability instead of text. On AppWorld and OfficeBench with two agent models, SJC improves cross-run reliability over the underlying compressor by 2.1%-8.9% without increasing the agent's token usage.

Abstract

Context Compaction for LLM Agents

Explore similar work

Prune the Summary, Not the History: Source-Judged Compaction for Long-Horizon Agents

Benchmarks Are Surprisingly Compaction-Friendly

Parallel Context Compaction for Long-Horizon LLM Agent Serving