Prune the Summary, Not the History: Source-Judged Compaction for Long-Horizon Agents

Guanghui Min; Chen Chen

cs.CLOct 14, 2026

Prune the Summary, Not the History: Source-Judged Compaction for Long-Horizon Agents

Authors: Guanghui Min, Chen Chen

Organizations: Department of Computer Science, University of Virginia

Abstract

Long-horizon LLM agents rely on context compaction: when the interaction history outgrows the context window, a compressor replaces it with a natural-language summary. We show that these summaries contain more than a record of what happened. Compaction prompts ask for decisions and next steps, so each summary mixes facts established by the environment with reasoning written by the compressor, and the agent cannot tell the two apart afterwards. In an analysis of a widely used structured compressor, 83% of failed runs trace back to an action that carries out a specific summary entry, most of which no environment output supports. We propose Source-Judged Compaction (SJC), a training-free post-processing step that scores each summary entry for whether it has a source in the environment outputs or the previous summary, and removes the entries that do not. It changes neither the agent nor the compressor and adds one inexpensive judge request per compaction. The judge is Jev, a non-generative decision model that answers the source question with a calibrated probability instead of text. On AppWorld and OfficeBench with two agent models, SJC improves cross-run reliability over the underlying compressor by 2.1%-8.9% without increasing the agent's token usage.

Figures & tables

Method	Summary	Train-Free	Drop-in
ACE ( Zhang et al., 2026 )	✗	✓	✗
Mem1 ( Zhou et al., 2025 )	✗	✗	✗
ACM ( Li et al., 2026 )	✗	✗	✗
Fast-Jev ( Tran, 2026 )	✗	✓	✓
ReSum 1 ( Wu et al., 2025 )	✓	✗	✗
SUPO ( Lu et al., 2025 )	✓	✗	✗

Table 1: Comparison of context-management approaches. Summary denotes a natural-language history checkpoint; Train-Free requires no context-management policy training; and Drop-in requires neither agent modification nor task-specific adaptation.

Section	Entries	Unsupported (%)
Exact identifiers	14,444	3.4
Constraints/Rules	4,183	11.1
Decisions	4,270	21.0
Open TODOs	2,600	67.7
Pending user asks	1,384	86.1
All	26,881	17.9

Table 2: Entries of OpenClaw summaries on the AppWorld training split, by section. An entry is unsupported when Jev finds no environment output or previous summary that establishes it.

Figure 1: Overview of SJC . (1) A host compressor produces a structured candidate summary. (2) SJC splits it into entries, retrieves relevant environment evidence from the history span, and scores all entries with a source judge in a single request. (3) Supported entries are retained unchanged, while unsupported entries are removed before the committed summary is shown to the agent and passed to the next compression boundary.

Agent: GPT-5.6 Luna / Compressor: GPT-5.6 Luna
Method	Average (168)				Easy (57)			Medium (48)			Hard (63)
Method	Acc. $\uparrow$	Pass 3 $\uparrow$	Steps $\downarrow$	Peak $\downarrow$	Acc. $\uparrow$	Pass 3 $\uparrow$	Steps $\downarrow$	Acc. $\uparrow$	Pass 3 $\uparrow$	Steps $\downarrow$	Acc. $\uparrow$	Pass 3 $\uparrow$	Steps $\downarrow$
No compression	84.1 $\pm$ 0.9	78.6	12.1	12.53	97.7 $\pm$ 1.0	93.0	9.6	86.8 $\pm$ 2.4	81.2	12.2	69.8 $\pm$ 0.0	63.5	14.4
FIFO	53.2 $\pm$ 1.2	45.2	30.0	7.14	95.9 $\pm$ 2.0	93.0	9.6	47.9 $\pm$ 4.2	37.5	33.7	18.5 $\pm$ 3.3	9.5	42.8
LLMLingua	39.5 $\pm$ 1.9	29.2	34.1	8.59	83.6 $\pm$ 2.7	68.4	16.9	25.7 $\pm$ 4.3	12.5	38.9	10.1 $\pm$ 1.8	6.3	46.0
LLMLingua-2	71.4 $\pm$ 2.3	61.3	19.2	10.53	97.7 $\pm$ 2.7	94.7	8.7	77.8 $\pm$ 2.4	66.7	17.6	42.9 $\pm$ 6.3	27.0	30.0

Table 3: Results on the AppWorld test-normal split by difficulty, over three independent runs. Acc.: mean task success, with the across-run standard deviation as a subscript. Pass 3 : fraction of tasks solved in all three runs. Steps and Peak: mean number of interaction steps and peak input length (

10^{3}

tokens). No compression is the uncompressed reference. SJC post-processes the summaries of Prompting with Jev as the source judge. Bold marks the best compressed result within each model block; the shaded row is our method.

Agent: GPT-5.6 Luna / Compressor: GPT-5.6 Luna
Method	Average (95)				1-APP (42)			2-APP (22)			3-APP (31)
Method	Acc. $\uparrow$	Pass 3 $\uparrow$	Steps $\downarrow$	Peak $\downarrow$	Acc. $\uparrow$	Pass 3 $\uparrow$	Steps $\downarrow$	Acc. $\uparrow$	Pass 3 $\uparrow$	Steps $\downarrow$	Acc. $\uparrow$	Pass 3 $\uparrow$	Steps $\downarrow$
No compression	82.1 $\pm$ 4.8	75.8	11.2	7.02	92.9 $\pm$ 4.1	88.1	7.6	81.8 $\pm$ 4.5	77.3	10.9	67.7 $\pm$ 8.5	58.1	16.2
FIFO	62.1 $\pm$ 1.8	54.7	23.2	3.48	81.7 $\pm$ 1.4	73.8	10.3	56.1 $\pm$ 2.6	50.0	28.2	39.8 $\pm$ 3.7	32.3	37.0
LLMLingua	58.2 $\pm$ 2.2	46.3	20.2	4.16	75.4 $\pm$ 6.0	61.9	9.8	59.1 $\pm$ 4.5	45.5	20.0	34.4 $\pm$ 4.9	25.8	34.4
LLMLingua-2	76.1 $\pm$ 1.2	68.4	14.4	4.14	89.7 $\pm$ 1.4	83.3	8.0	81.8 $\pm$ 0.0	77.3	14.0	53.8 $\pm$ 4.9	41.9	23.2

Table 4: Results on OfficeBench , grouped by the number of applications involved in each task. Metrics and notation follow Table 3 .

Figure 2: Pass 3 versus mean cumulative LLM tokens per task (agent plus LLM compressor). The dotted line marks the cost of uncompressed execution. Jev calls are excluded and reported in Table 5 .

	AppWorld		OfficeBench
	GPT	M3	GPT	M3
Prompting tokens (K)	112.8	142.6	51.0	57.5
SJC tokens (K)	103.8	138.5	50.4	53.5
$\Delta$ (%)	$-8.0$	$-2.9$	$-1.3$	$-7.0$
Entries pruned (%)	18.5	13.6	21.3	13.0
Summary length (%)	$-22.9$	$-13.7$	$-26.7$	$-17.3$

Table 5: Overhead of SJC . LLM tokens are means per task;

\Delta

is relative to Prompting. Summary length is the change in summary tokens after pruning.

Figure 3: Left: distribution of Jev source scores, with forward-looking sections highlighted. Right: fraction of entries pruned as

\tau

varies (AW: AppWorld; OB: OfficeBench). Dotted lines mark

\tau=0.5

	AppWorld		OfficeBench
	GPT	M3	GPT	M3
Prompting	64.3	55.4	68.4	74.7
+SJC (Self)	75.6	50.0	60.0	70.5
+SJC (Jev)	73.2	62.5	73.7	76.8

Table 6: Pass 3 (%) with the compressor itself as the source judge (Self) and with Jev.

Figure 4: An AppWorld task with GPT-5.6 Luna. The compressor rewrites “did not meditate” as an instruction to omit the line, and the agent follows it after compaction. SJC prunes the corresponding entry.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Agent and Environment
Symbol	Description
$\pi$	LLM agent that solves a task by interacting with an environment
$P$	Fixed prompt containing the system prompt and the task instruction; never compressed
$a_{i}$	Action taken by the agent at step $i$
$o_{i}$	Environment output received at step $i$
$\rho=(a_{1},o_{1},\ldots,a_{T},o_{T})$	Rollout of $T$ interaction steps

Appendix

Table 7: Summary of notations used in this paper.

System	Commit	Facts	Decisions	Plans	Intent
OpenClaw (default)	d2f8755	✓	✓	✓	✓
OpenClaw (safeguard)	d2f8755	✓	$\circ$	✓	✓
Hermes Agent	302c5d9	✓	✓	–	✓
AG2 v1.0	e1c8857	✓	$\circ$	–	–
AG2 Classic	86fa5cc	–	–	–	–
Codex CLI	5fe2152	✓	$\circ$	✓	✓

Appendix

Table 8: Content requested by 29 compaction prompts. Facts: observed state and results. Decisions: decisions made. Plans: next steps or pending work. Intent: the user’s goals or constraints. ✓: requested;

\circ

: requested in a weaker form (decisions without their rationale, or a task goal not attributed to the user); –: not requested.

Boundary	$N$	All	Reas.	Count
		Unsupported (%)
1st	243	29.2	54.9	4.3
2nd	152	16.6	39.6	3.9
3rd+	272	12.6	29.6	3.8

Appendix

Table 9: Unsupported entries by compression boundary.

N

: number of summaries. All and Reas.: share of unsupported entries among all entries and among reasoning entries (those under Decisions and Open TODOs ). Count: mean number of unsupported reasoning entries per summary.

First seen as	$d{=}1$	$d{=}2$	$d{=}3$	$d{=}4$
Unsupported	20.5	10.9	5.3	4.5
Supported	22.6	14.4	10.1	6.6

Appendix

Table 10: Share of reasoning entries (%) still present

d

summaries after they first appear.

		Failure pattern					Support of executed entry
Scenario	Failed	RC	ID	CS	Other	Unrelated	Unsupported	Contradicted	Supported
22cc237	3		2			1	2
29caf6f	9	9					9
2a163ab	2			1	1		1	1
60d0b5b	1		1				1
6104387	3					3

Appendix

Table 11: Failed runs of the Prompting baseline on the AppWorld training split, by scenario, after adjudication. Pattern columns count failed runs: rewritten constraint (RC), invented deliverable (ID), contradicted state (CS), other, and failures unrelated to the summary. Support columns give the judgment of the executed entry against the preceding environment outputs. The excluded scenario b0a8eae (nine failed runs) is not shown.

Judge	Provenance	Correctness
Jev	0.905 [0.857, 0.950]	0.733 [0.635, 0.868]
GPT-5.6 Luna	0.792 [0.732, 0.855]	0.639 [0.536, 0.787]

Appendix

Table 12: AUROC of each judge against annotated labels, with

95\%

bootstrap confidence intervals over task scenarios. Only

24

of the

298

entries with a correctness label are incorrect, so the correctness intervals are wide.

Context Compaction for LLM Agents

See all 33 papers

Compact-Bench: Measuring What Context Compaction Costs Long-Horizon Coding Agents
Oct 22, 2026Ruopei Sun, Jianfeng Cai, Kangwen Zhao +2Context Compaction for LLM Agents LLM Context Compression
University of Science and Technology of China
Parallel Context Compaction for Long-Horizon LLM Agent Serving
May 22, 2026Musa Cim, Burak Topcu, Chita Das +1LLM Inference Efficiency LLM Agent Workflow Optimization
The Pennsylvania State University
Slipstream: Trajectory-Grounded Compaction Validation for Long-Horizon Agents
May 9, 2026Zhuofu Chen, Rui Pan, Yinwei Dai +1LLM Agent Memory Long-Horizon LLM Agents
Princeton University

Oct 22, 2026J/K move · Enter open · S save

Compact-Bench: Measuring What Context Compaction Costs Long-Horizon Coding Agents

Ruopei Sun, Jianfeng Cai, Kangwen Zhao, Wengang Zhou+1

University of Science and Technology of China

Context Compaction for LLM Agents LLM Context Compression Software Engineering Agents LLM Agent Memory AI Agent Benchmarks Long-Horizon Agent Evaluation

Open paper AskPDF arXiv

Context compaction lets long-horizon coding agents continue beyond their context windows, but can erase the state needed to finish a task. Prior evaluations measure efficiency, retained information, or end-to-end success, leaving the effect of a specific history replacement difficult to isolate. Compact-Bench pairs full-history and summary-only continuations from a shared repository state, and evaluates recurring details and successive production compactions. It covers 35 SWE-bench repairs, 146 agentic sessions, and 106 production chains. Across 14 summarizer backbones, stable damage ranges from 5-64% of 22 full-history successes: Claude Opus 4.8 preserves the most repairs, while DeepSeek-Flash damages the most. With Kimi-K3 as summarizer, the Kimi Code prompt preserves the most task continuations (19/22), whereas Claude Code retains the most recurring details (15.0%). LLM summaries retain at most 16.0% of recurring details, compared with 79.1% for mechanical extraction; production summaries retain 15.7% at the first boundary. Failures recur across methods: one repair is damaged by 13/14 backbones, and another by all 5 prompts. These results show that reliable compaction requires preserving the progress needed to complete a task, and that retaining relevant information alone does not ensure successful continuation. Code is available at https://github.com/cai-jianfeng/compact-bench and data is available at https://huggingface.co/datasets/caijanfeng/compact-bench.

Abstract

Context Compaction for LLM Agents

Explore similar work

Compact-Bench: Measuring What Context Compaction Costs Long-Horizon Coding Agents

Parallel Context Compaction for Long-Horizon LLM Agent Serving

Slipstream: Trajectory-Grounded Compaction Validation for Long-Horizon Agents