What Does Context Compression Cost an Agent? Interaction Costs Unrevealed by Task-Completion Metrics

Shuyu Liu

cs.AIAug 17, 2026

What Does Context Compression Cost an Agent? Interaction Costs Unrevealed by Task-Completion Metrics

Authors: Shuyu Liu

Organizations: Chongqing University of Science and Technology

Abstract

Task completion is the standard metric for evaluating context compression, yet it is incomplete: compression can increase an agent's interaction cost by forcing it to reacquire dropped state while leaving completion statistically unchanged. We introduce a controlled runtime measurement protocol for reacquisition cost in a bounded-horizon tool-using agent. The agent acts in a deterministic planning environment under a fixed 24-turn horizon. We vary compression severity, compare a dropping operator with a fact-preserving operator, restore dropped state through controlled oracle interventions, and decompose tool calls into retrieval and execution. We evaluate three models across two task regimes. Retrieval calls increase in all six model-regime comparisons and account for almost all added interaction; five of six remain significant after Holm correction. At the prespecified 5x comparison point, completion changes are not significant in any cell. DeepSeek shows a significant completion drop only at 10x compression. GPT-5.5 is the clearest case: completion changes from 80% to 85% (p = 1.0) while retrieval increases from 21.0 to 63.9 calls (p = .002). Retention interventions further separate state quantity, state type, and content validity. Random selection is comparable to an offline hindsight oracle, while replacing retained D-state with semantically irrelevant content increases retrieval by 57% (p < .001) without a significant completion change. In a second environment, ALFWorld, sliding compression produces no retrieval surge, showing that the reacquisition signature is environment-dependent rather than intrinsic to shortening context. Overall, compression can impose hidden interaction costs when execution-relevant state becomes absent and must be reacquired, while completion alone may not expose those costs.

Figures & tables

Work	Measures	Manipulates	Decomposes	What completion can miss	Our difference
CostBench	cost-optimal planning; cost gaps	tool costs & availability (blocking events)	cost by event type	cost-suboptimal but valid plans	intact context; we price reacquiring dropped state
CompressAgent	reliability & success vs compression	compression retention level	failures: tool-execution vs reasoning	tool failures precede success collapse	no causal state manipulation; we add oracle attribution
Token Red. $\neq$ Cost	billed cost; token vs cost deltas	token-reduction strategies	cost by billing component (cache $\approx$ 87%)	token savings $\neq$ cost savings	provider-specific billing; we use a model-agnostic budget
Tool-Use Tax	accuracy tax of the tool protocol	tool availability (no-op / oracle probes)	accuracy gap: $\Delta$ cmp + $\Delta$ frc + $\Delta$ sty	protocol tax swamps tool gains	accuracy over intact context; we meter cost of lost state

Table 1: Closest evaluation-perspective contrasts.

Figure 1: Measurement protocol overview: full / fact-preserving vs. sliding (compressed) context; missing execution state diverges into retrieval (reacquisition) vs. execution (task work); a parallel oracle branch (compressed +

D^{*}

R^{*}

\to

injected dropped state

\to

retrieval cost drops) makes recoverability a manipulated variable.

Condition	Context
Full	the complete trajectory (baseline)
Sliding	a token budget of $r^{-1}\times$ the trajectory ( $r=1.7\times$ , $2.5\times$ , $5\times$ , $10\times$ ); greedily keep the most recent turns within budget; drop early turns outright
Summary	the same budget, but dropped early turns are replaced by a deterministic fact digest [Summary of earlier trajectory --- observed facts] (extractive; keeps observed state facts, discards the model’s reasoning text); zero extra LLM calls
Oracle	Full or Sliding context with a specific state string injected as a trailing message (same position in every condition)

Table 2: Context conditions.

Intervention	Selection rule
Sliding	recency baseline; no digest (recent turns only)
RAR-D	retain D-type atoms only
RAR-R	retain R-type atoms only
RAR-All	retain all atoms in observation order ( $\approx$ Summary control)
RAR-TypeAware	retain all atoms, R-first (recoverability-prioritized)
Random	random subset of atoms (seeded)

Table 3: Retention interventions.

ratio	completion	tools	retrieval	execute	turns
$1\times$ (Full)	83%	39.5	22.2	17.3	22.9
$1.7\times$	87%	42.7	25.4	17.3	23.3
$2.5\times$	81%	49.7	33.3	16.4	22.8
$5\times$	72%	71.4	55.1	16.3	23.1
$10\times$	66%	77.1	63.2	13.9	23.6

Table 4: DeepSeek, High regime: severity sweep.

Figure 2: Compression raises reacquisition cost before completion may degrade. Panel A: completion vs. compression ratio; Panel B/C: tool / retrieval–execution decomposition.

condition ( $5\times$ , High)	completion	tools	retrieval
Full	80%	37.4	19.0
Sliding	72%	71.4	55.1
Extractive Summary	83%	39.0	19.5

Table 5: Operator contrast at

5\times

, High regime (DeepSeek).

condition ( $5\times$ , High)	completion	tools	$\Delta$ tools vs sliding
Full	86%	38.8	—
Sliding	66%	72.9	—
Sliding + $R^{*}$	78%	69.0	$-3.9$
Sliding + $D^{*}$	80%	35.8	$-37.1$
Full + $R^{*}$ (sanity)	85%	32.1	—

Table 6: Oracle intervention at

5\times

, High regime (DeepSeek).

Model	regime	completion Full $\to 5\times$	$\Delta$ (p)	tools Full $\to 5\times$	retrieval $\Delta$ (p)	execute $\Delta$
DeepSeek	High	83 $\to$ 72	$-11$ (0.25)	39 $\to$ 71	+32.9 (.002) †	$-1.0$
Qwen	High	75 $\to$ 66	$-9$ (0.13)	34 $\to$ 37	+2.9 (.088)	$\approx 0$
GPT-5.5	High	80 $\to$ 85	+5 (1.0)	39 $\to$ 82	+42.9 (.002) †	$\approx 0$
DeepSeek	Low	100 $\to$ 100	0	33 $\to$ 55	+22.6 (.004) †	+0.1
Qwen	Low	100 $\to$ 94	$-6$ (0.25)	28 $\to$ 36	+5.3 (.023) †	+2.7
GPT-5.5	Low	100 $\to$ 100	0	26 $\to$ 49	+22.4 (.002) †	+0.7

Table 7: Three models, Full vs. Sliding (

5\times

), both regimes.

Figure 3: Two axes, not one. (a) Phase map of the six model–regime cells in

(\Delta\text{retrieval},\Delta\text{completion})

space; (b) tool decompositions.

policy ( $B=265$ )	$\Delta C_{R}$ vs Sliding	p
RAR-All	$-26.0\%$	$<$ 0.001
RAR-TypeAware	$-24.5\%$	$<$ 0.001
Hindsight	$-22.4\%$	0.001
Random	$-22.1\%$	0.001
Recent	$-20.4\%$	0.002
RAR-D	$-17.8\%$	0.002

Table 8: Selection null at

B=265

(DeepSeek, High).

metric	TypeAware (real D)	D-Irrelevant	$\Delta$	95% CI	p
$C_{R}$ (retrieval)	20.4	31.9	+11.5	[9.2, 13.9]	$<$ 0.001
tools	37.6	48.2	+10.7	[8.1, 13.4]	$<$ 0.001
execute	17.2	16.4	$-0.8$	$[-1.9,+0.2]$	n.s.
Q (completion)	75.0	81.0	+6.0 pp	$[-2.5,+16.0]$	0.41

Table 9: D-Irrelevant content intervention at

B=265

(DeepSeek, High).

Figure 4: Retention interventions. Digest budget

B

decreases from left to right in (a)–(b). (a)

\Delta C_{R}

vs. Sliding (%): TypeAware (real D) and D-Irrelevant are highlighted against the faded remaining policies; the

\sim

57% retrieval gap at

B{=}265

leaves completion statistically unchanged. (b) task completion (

\approx

flat across policies and budgets). (c) coverage dose–response (

\mathrm{cov}_{D}\to\Delta C_{R}

) with the selection null (Random

\approx

Hindsight) and the D-Irrelevant point at

B{=}265

Environment	Compression	$\Delta$ retrieval-like actions	$\Delta$ completion
IRBench (this work)	Sliding	large positive (2–3 $\times$ )	model / regime dependent
ALFWorld	Sliding	$\approx$ 0 (paired, symmetric)	small / ceiling-bound

Table 10: External environment boundary.

Figure 5: External environment boundary: paired

\Delta

retrieval-like actions under the same sliding operator are large and positive in IRBench and

\approx 0

(symmetric) in ALFWorld.

Figure 6: Oracle conditions and the interaction budget: retrieval / execution decomposition, termination (all_complete vs. max_turns), and within-budget diagnostics.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Model	Regime	$\rho$	p	95% bootstrap CI
DeepSeek	High	$-0.40$	.25	$[-0.97,+0.49]$
DeepSeek	Low	—	—	completion invariant across seeds
Qwen	High	$-0.32$	.37	$[-0.75,+0.29]$
Qwen	Low	$+0.16$	.67	—
GPT-5.5	High	$-0.64$	.045	$[-0.84,-0.18]$
GPT-5.5	Low	—	—	completion invariant across seeds

Appendix

Table 11: Seed-level association between per-seed

\Delta C_{R}

and

\Delta Q

Context Compression

Context compression reduces the information or memory a model must carry while preserving what it needs to answer questions and act.

See all 90 papers

An Empirical Cost Attribution of Context-Compression Gateways in Multi-Turn Coding Agents
Aug 19, 2026Luzhuo Chen, Jiayu ShiContext Compression LLM Agent Workflow Optimization
Paritok
Beyond Token Savings: A Systematic Study of Context Compression in LLM Agents
Sep 26, 2026Ritul Satish, Prasoon Sinha, Akiho Kawada +1Context Compaction for LLM Agents Context Compression
The University of Texas at Austin
FOCUS: Training-Free Decision-Preserving Context Compression for LLM Agents
Sep 29, 2026Shantanu Dixit, Anson Bastos, Xuchao Zhang +2Long-Horizon LLM Agents LLM Context Management
M365 Research, Microsoft

Aug 19, 2026J/K move · Enter open · S save

An Empirical Cost Attribution of Context-Compression Gateways in Multi-Turn Coding Agents

Luzhuo Chen, Jiayu Shi

Paritok

Context Compression LLM Agent Workflow Optimization Token Compression Cost-Aware Inference LLM Inference Efficiency AI Coding Agents

Open paper AskPDF arXiv

Context compression is widely proposed as a way to cut the token bill of LLM coding agents, and public benchmarks report that aggressive compression preserves task-solving quality. These two facts do not imply the third one commonly assumed: that compressing file reads saves money in a real multi-turn agent. We instrument a production compression gateway (Paritok) between coding agents (Claude Code, Codex) and frontier LLMs (Claude Sonnet, GPT-5), and decompose the token bill of real sessions into three independent levers: tool-schema filtering, content compression of file reads and tool output, and history summarization. Measured in isolation under controlled A/B runs, the three save at fundamentally different rates. Tool-schema filtering removes a fixed block every turn, roughly 21K-57K tokens on a typical turn; it is linear in the turn count N and the only unambiguously and reproducibly positive lever. Content compression saves only about 2% of the cache-priced prefix per turn, but compressed reads accumulate in history and are re-sent on every later turn, so its cumulative saving grows quadratically, about 3350*N^2 tokens (measured), overtaking the fixed tool-filter saving within roughly 6 turns until the context window caps it. A non-destructive gateway lets the agent pull original bytes back on demand; each recall re-sends exactly the one segment just compressed away, so its cost is fixed and bounded rather than a multiplicative blowup, and heavy recall spends the accumulated saving back one segment at a time. Finally, a strong single-shot compression benchmark - 86.5% of SWE-bench quality retained at a 25.7% compression rate, achieved by the model this gateway deploys (Paritok-4B, reported separately) - is orthogonal to multi-turn agent cost and must not be cited as a cost-saving argument. We distill the results into an actionable recipe for where token-saving effort pays off.

Abstract

Context Compression

Explore similar work

An Empirical Cost Attribution of Context-Compression Gateways in Multi-Turn Coding Agents

Beyond Token Savings: A Systematic Study of Context Compression in LLM Agents

FOCUS: Training-Free Decision-Preserving Context Compression for LLM Agents