Maximising Transparency in Context Compaction Improves Retention of Facts and Commitments for Lightweight Coding Agents

Felix Woolford

cs.AIOct 14, 2026

Maximising Transparency in Context Compaction Improves Retention of Facts and Commitments for Lightweight Coding Agents

Authors: Felix Woolford

Organizations: Center for Human Nature, Artificial Intelligence, and Neuroscience (CHAIN), Hokkaido University

Abstract

Long coding agent sessions outgrow the model's context window, so agents compact: older history is replaced by something shorter, usually a model-written summary. A summary selects and rewords, and the agent cannot see what was lost. Existing work has explored a range of compaction techniques, but comparatively little attention has been paid to their effectiveness with lightweight models, including those deployed locally or used for high-volume workloads. Here, we present transparent-compact, a simple deterministic method for the open-source Pi coding agent, designed specifically for use with such models and with the purpose of maximising the transparency of the agent's view of its own compacted context. It guarantees that user and assistant prose in its context is unchanged by a model or algorithm, and explicitly flags where compaction has stripped information. We compare it with Pi's default model-written summary compaction and with pi-vcc, a popular deterministic method with similar methods but without verbatim retention of prose. Our benchmarks run on four models of varying performance levels, with emphasis on Qwen3.8-27B: a lightweight model at the top end of what consumer hardware can serve locally. On sets probing commitments (rules the user set which exist only in the conversation) and recall of facts, transparent-compact matched or outperformed both alternatives. In particular, it showed an advantage in avoiding confident blunders, i.e. assertion of false information as opposed to recognizing that something was not recalled. Even with a frontier model, its advantage was present but smaller. We argue that transparent-compact spares lightweight models from reasoning or searching their way around what compaction lost, and results in improved recall and application of information as a result, without the need for task or context-specific structuring of the session history.

Figures & tables

	transparent-compact	pi-vcc	pi-default
Written by	code (deterministic)	code (deterministic)	the model
User messages	word for word	extracted: goal lines, pattern-matched preferences, one-line earlier turns, brief transcript	reworded into template sections
Assistant prose	word for word	one-line turns, brief transcript	reworded
Tool calls	one-line signatures, file lists	one-liners in the brief transcript, file and commit lists	summarised, file lists
Tool outputs, thinking	removed, declared	errors extracted, rest removed	summarised or dropped
Recovery	context_lookup subagent (+ 3 fallback tools), recovery protocol appended in context	vcc_recall search	none

Table 1: What each method keeps. Compaction times are means per compaction with Qwen3.8-27B in the planted-fact study.

	Commitments	Planted facts
Material	18 real sessions	18 synthetic sessions
Compaction	replay at 48k, many compactions	3 compactions at $\sim$ 95k tokens
Probes	27 (14 constraints, 13 decisions)	17 facts + 3 lures after each compaction
Scoring	blind LLM judge; failures labelled confident or not	by rule
Models (reps)	Qwen, Luna, Gemma, Sol (3; 5 in the first Qwen run)	Qwen, Luna, Gemma, Sol (1)

Table 2: Probe sets. “Reps”: independent replies per probe and method.

Figure 1: The three methods on both probe sets. Commitments : 27 real-session probes

\times

3 repetitions on each model; Sol is a frontier model, the others are lightweight. Planted facts : 18 synthetic sessions, each probed after three compactions; the Sol group is an exploratory update with two methods and is not part of the ANOVA. pi-vcc was not run on Sol because of cost– extrapolating from earlier data we expect that it would approximately match transparent-compact’s near perfect performance with that model. Statistics are in Table 3 .

Commitments (27 probes; Qwen, Luna, Gemma, Sol)
	Effect	$F$ (df)	$p$	$\eta_{p}^{2}$
Omnibus	model	15.12 (3, 78)	$<$ 0.001	0.37
	method	5.37 (2, 52)	0.008	0.17
	model $\times$ method	1.43 (6, 156)	0.22	0.05
transparent-compact vs pi-vcc	method	9.98 (1, 26)	0.004	0.28
	model $\times$ method	1.35 (3, 78)	0.27	0.05

Table 3: Model

\times

method repeated-measures ANOVAs. The omnibus test uses all three methods; each pairwise ANOVA uses one pair of methods, and its interaction tests whether that pair’s gap differs between models.

p

-values are Greenhouse–Geisser corrected.

	Pass rate			Failed confidently
Model	transparent- compact	pi-default	pi-vcc	transparent- compact	pi-default	pi-vcc
Qwen3.8-27B	74.1	64.2	53.1	14.8	22.2	40.7
GPT-6 Luna	46.9	30.9	28.4	34.6	46.9	51.9
Gemma 4 31B	39.5	35.8	32.1	37.0	48.1	55.6
GPT-6.1 Sol (frontier)	69.1	74.1	58.0	23.5	23.5	33.3

Table 4: Commitments: pass rate and confident failures (%, 27 probes

\times

3 repetitions per model; failed turns and missing cells count as failures). Pass rates are the ANOVA’s cell means.

Figure 2: How commitment probes failed. Each bar divides all probe repetitions into passes, failures that flagged doubt (a question, warning or hedge) and confident failures; 27 probes

\times

3 repetitions per model.

Model	transparent-compact	pi-default	pi-vcc
Qwen3.8-27B	97.8	75.3	98.2
GPT-6 Luna	95.1	55.0	45.6
Gemma 4 31B	93.3	51.5	67.5
GPT-6.1 Sol (frontier) †	99.3	75.6	—

Table 5: Planted facts: recall (%, 18 sessions, three compactions). Qwen, Luna and Gemma are the ANOVA’s cell means; Sol † is an exploratory update with two methods and is not part of the ANOVA.

	Qwen		Gemma
	transparent- compact	pi-vcc	transparent- compact	pi-vcc
Prompt tokens	57,300	21,389	34,576	20,822
Output tokens	5,975	5,656	1,419	1,440
Recovery calls (total)	186	1,238	27	65
Turns	139	139	79	80

Table 6: Token cost per commitment probe turn (transparent-compact against pi-vcc; local models, measured at the server). “Prompt tokens”: mean prompt tokens processed per turn, excluding cache hits and including transparent-compact’s lookup subagent.

	transparent-compact	pi-default	pi-vcc
Commitment present	22 of 26, word for word	13 of 26 in “Constraints & Preferences” or “Key Decisions”, plus 3 in the goal or kept tail	9 of 26 intact; 7 truncated or only a trace of an action; 10 absent
Pass rate when present	54/66	31/39 (in its constraint sections)	19/27 (when intact)
Median summary size	31k chars	34k chars	9.5k chars
On the 21 probes left after removing 4 that no method kept and 1 flawed probe
Pass rate	83%	78%	62%
Confident failures	6	10	22

Table 7: Where the commitment stood in each method’s context (Qwen, 26 commitment probes; case review).

	Values from tool output and reasoning	User rules and decisions	Recovery of what was dropped	Significantly behind transparent-compact on
transparent-compact	removed but declared; recovered by lookup	kept word for word	one delegated lookup	no probe set
pi-default	compressed or dropped; look-alikes confused	kept in template sections, sometimes reworded	none	planted facts
pi-vcc	removed; recoverable by search	truncated, a trace, or absent	search; depends on the model’s persistence	commitments; planted facts on Luna and Gemma

Table 8: What each method keeps of the information the probes need , summarising Section 5.1 .

Figure 3: transparent-compact’s lead over each alternative, by model (percentage points; differences between the ANOVA’s cell means). Planted facts: 18 sessions; commitments: 27 probes

\times

3 repetitions. The dashed line separates the frontier model. The hatched Sol bar for planted facts is the exploratory update and is not part of the ANOVA.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Exploratory results comparing the two main versions of pi-vcc Same parameters as in Figure 1 .

Context Compaction for LLM Agents

See all 32 papers

Oct 22, 2026

Benchmarks Are Surprisingly Compaction-Friendly

Long horizon agentic tasks are now pushing LLM agents past their context window. To continue the task, the agent scaffold has to free up the context window, a process known as context compaction. Compaction is generally lossy by design: agents can silently lose task constraints, instructions, or their reasoning behind earlier decisions. Although compaction is a standard component of coding agent scaffolds, it has not been studied systematically with frontier models on realistic, long horizon agentic tasks. Previous work evaluate smaller open weight models on toy or synthetic trajectories. We fix the agent scaffold (OpenCode), implement five compaction strategies inside it, and evaluate three frontier models on three long horizon benchmarks (PaperBench, ProgramBench, PostTrainBench). For the two 1M context models, the compaction threshold does not measurably affect the task performance, whereas GPT-5.3-Codex, the model with the smallest context window, performs worse at lower thresholds on PaperBench and, less clearly, on PostTrainBench. No compaction method outperforms default self-summarization, and methods that discard more content cost some models up to 20 points while leaving other models the same. Compaction can, however, silently remove task constraints: on ProgramBench, the first compaction summary already drops about one third of the task's anticheating rules on average, and this rises monotonically with the number of compactions. On real user-agent sessions from SWE-chat, instructions about how the agent should work and what it should not do are dropped around 6 times more than specifications of what to build. We argue that current benchmarks are compaction-friendly: i.e., the objective is fixed and the agent's progress stays on disk, so most of what compaction discards can be recovered, except constraints that don't exist anywhere else.

Jeanne Salle, Hardik Bhatnagar, Matthias Bethge +2

ELLIS Institute Tübingen · Max Planck Institute for Intelligent Systems · Tübingen AI Center +1

Context Compaction for LLM Agents LLM Context Compression LLM Agent Memory LLM Agent Evaluation Long-Horizon Agent Tasks Agent Benchmarks

Ask PDF

Oct 22, 2026

Compact-Bench: Measuring What Context Compaction Costs Long-Horizon Coding Agents

Context compaction lets long-horizon coding agents continue beyond their context windows, but can erase the state needed to finish a task. Prior evaluations measure efficiency, retained information, or end-to-end success, leaving the effect of a specific history replacement difficult to isolate. Compact-Bench pairs full-history and summary-only continuations from a shared repository state, and evaluates recurring details and successive production compactions. It covers 35 SWE-bench repairs, 146 agentic sessions, and 106 production chains. Across 14 summarizer backbones, stable damage ranges from 5-64% of 22 full-history successes: Claude Opus 4.8 preserves the most repairs, while DeepSeek-Flash damages the most. With Kimi-K3 as summarizer, the Kimi Code prompt preserves the most task continuations (19/22), whereas Claude Code retains the most recurring details (15.0%). LLM summaries retain at most 16.0% of recurring details, compared with 79.1% for mechanical extraction; production summaries retain 15.7% at the first boundary. Failures recur across methods: one repair is damaged by 13/14 backbones, and another by all 5 prompts. These results show that reliable compaction requires preserving the progress needed to complete a task, and that retaining relevant information alone does not ensure successful continuation. Code is available at https://github.com/cai-jianfeng/compact-bench and data is available at https://huggingface.co/datasets/caijanfeng/compact-bench.

Ruopei Sun, Jianfeng Cai, Kangwen Zhao +2

University of Science and Technology of China

Context Compaction for LLM Agents LLM Context Compression Software Engineering Agents LLM Agent Memory AI Agent Benchmarks Long-Horizon Agent Evaluation

Ask PDF

Oct 1, 2026

AutoCompact: Learning When to Compact Context in Long-Horizon Coding Agents

Coding agents solve repository-level software engineering tasks through long trajectories of code inspection, search, editing, and testing. As a task progresses, earlier exploration becomes stale, so managing context is more than avoiding overflow: an agent must decide when to compact, what working state to preserve, and how to continue from it. We introduce AutoCompact, which trains a coding agent to make these decisions as part of its policy. To collect training data, we run the base agent on coding tasks and use a judge to review its compaction decisions, summaries, and actions after compaction. Flawed outputs are replaced with corrected ones before being executed in the environment, so each trajectory continues from the corrected decisions. We use these trajectories for supervised fine-tuning, then jointly optimize coding and compaction through reinforcement learning with task-success rewards. Experiments on SWE-bench Verified and SWE-PolyBench Verified show that AutoCompact improves pass rates over the base model by an absolute 9.2% and 5.0%, respectively. The improvements hold across all evaluated inference budgets, with a 256K context window that never overflows and with a 16K window whose overflow triggers fallback compaction.

Xuan Zhang, Longtao Zheng, Cunxiao Du +2

Singapore Management University · Nanyang Technological University · Harvard University

Software Engineering Agents LLM Agent Memory Context Compaction for LLM Agents

Ask PDF

Abstract

Context Compaction for LLM Agents

Explore similar work

Benchmarks Are Surprisingly Compaction-Friendly

Compact-Bench: Measuring What Context Compaction Costs Long-Horizon Coding Agents

AutoCompact: Learning When to Compact Context in Long-Horizon Coding Agents