Maximising Transparency in Context Compaction Improves Retention of Facts and Commitments for Lightweight Coding Agents

Felix Woolford

cs.AIOct 14, 2026

Maximising Transparency in Context Compaction Improves Retention of Facts and Commitments for Lightweight Coding Agents

Authors: Felix Woolford

Organizations: Center for Human Nature, Artificial Intelligence, and Neuroscience (CHAIN), Hokkaido University

Abstract

Long coding agent sessions outgrow the model's context window, so agents compact: older history is replaced by something shorter, usually a model-written summary. A summary selects and rewords, and the agent cannot see what was lost. Existing work has explored a range of compaction techniques, but comparatively little attention has been paid to their effectiveness with lightweight models, including those deployed locally or used for high-volume workloads. Here, we present transparent-compact, a simple deterministic method for the open-source Pi coding agent, designed specifically for use with such models and with the purpose of maximising the transparency of the agent's view of its own compacted context. It guarantees that user and assistant prose in its context is unchanged by a model or algorithm, and explicitly flags where compaction has stripped information. We compare it with Pi's default model-written summary compaction and with pi-vcc, a popular deterministic method with similar methods but without verbatim retention of prose. Our benchmarks run on four models of varying performance levels, with emphasis on Qwen3.8-27B: a lightweight model at the top end of what consumer hardware can serve locally. On sets probing commitments (rules the user set which exist only in the conversation) and recall of facts, transparent-compact matched or outperformed both alternatives. In particular, it showed an advantage in avoiding confident blunders, i.e. assertion of false information as opposed to recognizing that something was not recalled. Even with a frontier model, its advantage was present but smaller. We argue that transparent-compact spares lightweight models from reasoning or searching their way around what compaction lost, and results in improved recall and application of information as a result, without the need for task or context-specific structuring of the session history.

Figures & tables

	transparent-compact	pi-vcc	pi-default
Written by	code (deterministic)	code (deterministic)	the model
User messages	word for word	extracted: goal lines, pattern-matched preferences, one-line earlier turns, brief transcript	reworded into template sections
Assistant prose	word for word	one-line turns, brief transcript	reworded
Tool calls	one-line signatures, file lists	one-liners in the brief transcript, file and commit lists	summarised, file lists
Tool outputs, thinking	removed, declared	errors extracted, rest removed	summarised or dropped
Recovery	context_lookup subagent (+ 3 fallback tools), recovery protocol appended in context	vcc_recall search	none

Table 1: What each method keeps. Compaction times are means per compaction with Qwen3.8-27B in the planted-fact study.

	Commitments	Planted facts
Material	18 real sessions	18 synthetic sessions
Compaction	replay at 48k, many compactions	3 compactions at $\sim$ 95k tokens
Probes	27 (14 constraints, 13 decisions)	17 facts + 3 lures after each compaction
Scoring	blind LLM judge; failures labelled confident or not	by rule
Models (reps)	Qwen, Luna, Gemma, Sol (3; 5 in the first Qwen run)	Qwen, Luna, Gemma, Sol (1)

Table 2: Probe sets. “Reps”: independent replies per probe and method.

Figure 1: The three methods on both probe sets. Commitments : 27 real-session probes

\times

3 repetitions on each model; Sol is a frontier model, the others are lightweight. Planted facts : 18 synthetic sessions, each probed after three compactions; the Sol group is an exploratory update with two methods and is not part of the ANOVA. pi-vcc was not run on Sol because of cost– extrapolating from earlier data we expect that it would approximately match transparent-compact’s near perfect performance with that model. Statistics are in Table 3 .

Commitments (27 probes; Qwen, Luna, Gemma, Sol)
	Effect	$F$ (df)	$p$	$\eta_{p}^{2}$
Omnibus	model	15.12 (3, 78)	$<$ 0.001	0.37
	method	5.37 (2, 52)	0.008	0.17
	model $\times$ method	1.43 (6, 156)	0.22	0.05
transparent-compact vs pi-vcc	method	9.98 (1, 26)	0.004	0.28
	model $\times$ method	1.35 (3, 78)	0.27	0.05

Table 3: Model

\times

method repeated-measures ANOVAs. The omnibus test uses all three methods; each pairwise ANOVA uses one pair of methods, and its interaction tests whether that pair’s gap differs between models.

p

-values are Greenhouse–Geisser corrected.

	Pass rate			Failed confidently
Model	transparent- compact	pi-default	pi-vcc	transparent- compact	pi-default	pi-vcc
Qwen3.8-27B	74.1	64.2	53.1	14.8	22.2	40.7
GPT-6 Luna	46.9	30.9	28.4	34.6	46.9	51.9
Gemma 4 31B	39.5	35.8	32.1	37.0	48.1	55.6
GPT-6.1 Sol (frontier)	69.1	74.1	58.0	23.5	23.5	33.3

Table 4: Commitments: pass rate and confident failures (%, 27 probes

\times

3 repetitions per model; failed turns and missing cells count as failures). Pass rates are the ANOVA’s cell means.

Figure 2: How commitment probes failed. Each bar divides all probe repetitions into passes, failures that flagged doubt (a question, warning or hedge) and confident failures; 27 probes

\times

3 repetitions per model.

Model	transparent-compact	pi-default	pi-vcc
Qwen3.8-27B	97.8	75.3	98.2
GPT-6 Luna	95.1	55.0	45.6
Gemma 4 31B	93.3	51.5	67.5
GPT-6.1 Sol (frontier) †	99.3	75.6	—

Table 5: Planted facts: recall (%, 18 sessions, three compactions). Qwen, Luna and Gemma are the ANOVA’s cell means; Sol † is an exploratory update with two methods and is not part of the ANOVA.

	Qwen		Gemma
	transparent- compact	pi-vcc	transparent- compact	pi-vcc
Prompt tokens	57,300	21,389	34,576	20,822
Output tokens	5,975	5,656	1,419	1,440
Recovery calls (total)	186	1,238	27	65
Turns	139	139	79	80

Table 6: Token cost per commitment probe turn (transparent-compact against pi-vcc; local models, measured at the server). “Prompt tokens”: mean prompt tokens processed per turn, excluding cache hits and including transparent-compact’s lookup subagent.

	transparent-compact	pi-default	pi-vcc
Commitment present	22 of 26, word for word	13 of 26 in “Constraints & Preferences” or “Key Decisions”, plus 3 in the goal or kept tail	9 of 26 intact; 7 truncated or only a trace of an action; 10 absent
Pass rate when present	54/66	31/39 (in its constraint sections)	19/27 (when intact)
Median summary size	31k chars	34k chars	9.5k chars
On the 21 probes left after removing 4 that no method kept and 1 flawed probe
Pass rate	83%	78%	62%
Confident failures	6	10	22

Table 7: Where the commitment stood in each method’s context (Qwen, 26 commitment probes; case review).

	Values from tool output and reasoning	User rules and decisions	Recovery of what was dropped	Significantly behind transparent-compact on
transparent-compact	removed but declared; recovered by lookup	kept word for word	one delegated lookup	no probe set
pi-default	compressed or dropped; look-alikes confused	kept in template sections, sometimes reworded	none	planted facts
pi-vcc	removed; recoverable by search	truncated, a trace, or absent	search; depends on the model’s persistence	commitments; planted facts on Luna and Gemma

Table 8: What each method keeps of the information the probes need , summarising Section 5.1 .

Figure 3: transparent-compact’s lead over each alternative, by model (percentage points; differences between the ANOVA’s cell means). Planted facts: 18 sessions; commitments: 27 probes

\times

3 repetitions. The dashed line separates the frontier model. The hatched Sol bar for planted facts is the exploratory update and is not part of the ANOVA.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Exploratory results comparing the two main versions of pi-vcc Same parameters as in Figure 1 .

Context Compaction for LLM Agents

See all 34 papers

Benchmarks Are Surprisingly Compaction-Friendly
Oct 22, 2026Jeanne Salle, Hardik Bhatnagar, Matthias Bethge +2Context Compaction for LLM Agents LLM Context Compression
ELLIS Institute Tübingen · Max Planck Institute for Intelligent Systems · Tübingen AI Center +1
Compact-Bench: Measuring What Context Compaction Costs Long-Horizon Coding Agents
Oct 22, 2026Ruopei Sun, Jianfeng Cai, Kangwen Zhao +2Context Compaction for LLM Agents LLM Context Compression
University of Science and Technology of China
AutoCompact: Learning When to Compact Context in Long-Horizon Coding Agents
Oct 1, 2026Xuan Zhang, Longtao Zheng, Cunxiao Du +2Software Engineering Agents LLM Agent Memory
Singapore Management University · Nanyang Technological University · Harvard University

Oct 22, 2026J/K move · Enter open · S save

Benchmarks Are Surprisingly Compaction-Friendly

Jeanne Salle, Hardik Bhatnagar, Matthias Bethge, Sahar Abdelnabi+1

ELLIS Institute Tübingen · Max Planck Institute for Intelligent Systems · Tübingen AI Center+1

Context Compaction for LLM Agents LLM Context Compression LLM Agent Memory LLM Agent Evaluation Long-Horizon Agent Tasks Agent Benchmarks

Open paper AskPDF arXiv

Long horizon agentic tasks are now pushing LLM agents past their context window. To continue the task, the agent scaffold has to free up the context window, a process known as context compaction. Compaction is generally lossy by design: agents can silently lose task constraints, instructions, or their reasoning behind earlier decisions. Although compaction is a standard component of coding agent scaffolds, it has not been studied systematically with frontier models on realistic, long horizon agentic tasks. Previous work evaluate smaller open weight models on toy or synthetic trajectories. We fix the agent scaffold (OpenCode), implement five compaction strategies inside it, and evaluate three frontier models on three long horizon benchmarks (PaperBench, ProgramBench, PostTrainBench). For the two 1M context models, the compaction threshold does not measurably affect the task performance, whereas GPT-5.3-Codex, the model with the smallest context window, performs worse at lower thresholds on PaperBench and, less clearly, on PostTrainBench. No compaction method outperforms default self-summarization, and methods that discard more content cost some models up to 20 points while leaving other models the same. Compaction can, however, silently remove task constraints: on ProgramBench, the first compaction summary already drops about one third of the task's anticheating rules on average, and this rises monotonically with the number of compactions. On real user-agent sessions from SWE-chat, instructions about how the agent should work and what it should not do are dropped around 6 times more than specifications of what to build. We argue that current benchmarks are compaction-friendly: i.e., the objective is fixed and the agent's progress stays on disk, so most of what compaction discards can be recovered, except constraints that don't exist anywhere else.

Abstract

Context Compaction for LLM Agents

Explore similar work

Benchmarks Are Surprisingly Compaction-Friendly

Compact-Bench: Measuring What Context Compaction Costs Long-Horizon Coding Agents

AutoCompact: Learning When to Compact Context in Long-Horizon Coding Agents