Coding agents repeatedly send their conversation history to a language model. Adding useful code may reduce search, while removing old tool outputs may shorten later prompts. But do these changes reduce the cost of solving a task? We study this as a measurement problem in more than 20,000 runs of open models on SWE-bench Verified, SWE-bench and Loc-Bench. We account separately for uncached input, cached input and output tokens. Three findings emerge. First, in our main evidence-package experiments, retrieved code reduces read and search commands by 18--34%, but increases cost by 33--42% when cached input costs one tenth of uncached input, without a measurable improvement in resolution. Second, removing old observations can either save money or increase cost: the same approach saves 19--36% without a cache discount but increases cost by 38--116% with a tenth-price cache. Third, shorter prompts do not guarantee cheaper runs because agents may respond by taking more steps. We also find that concurrent experiments can share cached prefixes and distort cost comparisons. These results show why context-management methods should be evaluated using both token prices and the agent's subsequent behavior.
Figures & tables
Figure 1: Adding and removing context can change both the bill and the patch. Retrieved code adds evidence to the prompt; masking, eviction or summarization removes old observations. Both interventions can affect the agent’s later actions. We separate direct cache effects under fixed actions from behavioral responses in new runs, and evaluate patch quality offline. The issue and code labels are illustrative.
Lever
Comparison (setting)
Read+search
Cost, c=0.1
Cost, c=1
Quality
Add
evidence package vs. none (Verified, 80 tasks)
−32%
+36%
+25%
resolved +3.8 official
Add
evidence package vs. none, agent can run tests (183 tasks)
−9%
+21%
–
resolved −0.3 official
Add
retrieval appended to the agent’s localization (184 tasks, 3 models)
–
0%
0%
function Acc@10 +14 to +20
Remove
rolling masking vs. none (201 fresh tasks; 27B / Coder)
–
+116% / +87%
−19% / −25%
−0.5 / +2.0 official
Remove
break-even masking ( c=0.1 ) vs. none (same)
–
+2.8% / +1.8%
−26% / −21%
equivalent, +1.3 official
Remove
rolling / break-even masking vs. none, SWE-agent (115 tasks; 27B)
–
+101% / +9.6%
−29% / −25%
−0.9 / −1.7 local
Table 1: Adding and removing context at a glance. Entries are paired point estimates over tasks. Quality entries are percentage-point changes in resolution unless marked Acc@10; localization accuracy and resolution are different outcomes and should not be compared directly. Cost follows Equation 1 with p=4 , counting only cache hits a run could have produced itself ( Section 2 ). “Official” is the SWE-bench harness; “local” runs the same hidden tests in our executable sandbox.
Figure 2: Deleting an old output B shortens later prompts but breaks the cache match for the following suffix S . In this illustrative fixed-action example, B=1,000 , S=5,000 and c=0.1 . Ten remaining requests save 1,000 fresh-token cost units but incur 4,500 units to rebuild the suffix; breaking even requires 45 requests. Prompt lengths are schematic, and patch quality is evaluated separately.
Figure 3: Cost of three masking policies relative to no masking as the price of a cached token varies, on 201 fresh Verified tasks in mini-SWE-agent. Rolling masking doubles cost at c=0.1 and saves at c=1 ; break-even masking, set for c=0.1 , stays near no masking when caching is cheap and saves when it is not.
Qwen3.6-27B
Qwen3-Coder
Qwen3.8-27B
Method
c=0.1
c=1
c=0.1
c=1
c=0.1
c=1
MM10 rolling masking
+116%
−19%
+87%
−25%
+76%
−24%
MB batched masking
+5%
−22%
+12%
−4%
−14%
−33%
MAR break-even
+3%
−26%
+2%
−21%
+7%
−21%
MS summarization
+22%
−14%
+28%
+3%
−19%
−53%
MD reflection
+36%
+4%
+31%
+7%
+29%
+2%
Table 2: Cost of five context-management methods relative to no management ( MR ), on 201 fresh Verified tasks in mini-SWE-agent, one run per cell, at two prices of a cached token ( p=4 ). The last row gives the mean cost of MR in thousands of tokens.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Experiment
Primary test
Expected
Verdict
SWE-bench test, Qwen3.6-27B
E5 − E0 read+search
<0
supported
SWE-bench test, Qwen3.6-27B
EL − E0 read+search
<0
supported
Verified, Qwen3.6-27B
E5 − E2 prompt tokens
<0
supported
Verified, Qwen3.6-27B
E5 − E0 , EL − E0 read+search
<0
supported
SWE-bench test, Qwen3-Coder
E5 − E0 read+search
<0
supported
Loc-Bench, Qwen3.6-27B
E5 − E0 read+search
<0
supported
Appendix
Table 3: Preregistered primary tests reported in the paper. One-sided tests use paired bootstrap over tasks; families are Holm-corrected within each experiment.
Setting
Read+search
Cost, c=0.1
Cost, c=1
On target
Submitted
SWE-bench test, Qwen3.6-27B
−5.97
+42%
+33%
+0.01
+0.14
Verified, Qwen3.6-27B
−6.64
+36%
+25%
+0.04
+0.14
Loc-Bench, Qwen3.6-27B
−4.52
+36%
+28%
+0.02
+0.07
SWE-bench test, Qwen3-Coder
−2.97
+95%
+68%
+0.03
−0.01
Verified, Qwen3.5-4B
−7.54
+25%
–
+0.08
+0.19
Verified, Qwen3.5-9B
−7.27
+33%
–
+0.03
+0.11
Appendix
Table 4: E5 − E0 on each benchmark and model: paired mean differences. Cost is relative to the E0 mean. Cost at c=0.1 counts only cache hits a run could have produced itself ( Appendix K ). SWE-bench test uses the batch of the main end-to-end experiment; Figure 5 uses a later batch with EI and ED , where the corresponding costs are +25% ( c=1 ) and +32% ( c=0.1 ).
Figure 4: Change in read and search commands per run relative to no evidence ( E0 ), with 95% bootstrap CIs over tasks. The full package ( E5 ) reduces exploration on every benchmark and model, though much less or not at all in mini-SWE-agent and with Gemma-4; random code of the same length ( E5r ) does not.
Figure 5: Run cost relative to no evidence for each condition, with cached tokens at full price ( c=1 , open circles) and at 10% of the price ( c=0.1 , filled), Qwen3.6-27B. Cost follows Equation 1 , counting only cache hits a run could have produced itself.
Figure 6: Share of runs in which removing evidence at the trigger pays off under Equation 2 at c=0.1 , using each run’s remaining turns (bars) or the condition’s median (ticks). Labels give the median tokens removed ( B ) and recomputed ( X ) and the net saving.
Figure 7: Change in the official resolved rate (percentage points) against each condition’s baseline, paired by task, with 95% bootstrap CIs. Labels give the number of tasks that improved and worsened and an exact two-sided sign test. Tasks whose gold patch fails in our harness are excluded.
Figure 8: (a) Package lines per task by when the agent without evidence first displayed them in the same task. (b) Re-read commands per run: re-reads of package code by the agent with evidence, by why the code was not simply in view, and re-reads of code already in context (including the package while it is shown) with and without evidence. Loc-Bench, Qwen3.6-27B.
Method
Commands
Cost, c=0.1
Cost, c=1
Resolved
None
51.9
–
–
0.632
Truncation
60.1
+3.6%
+5.6%
0.612
Narrow reads
57.9
+11.2%
+16.3%
0.649
Estimated reuse
53.4
−0.7%
−10.2%
0.610
Appendix
Table 5: Further removal methods on 178 tasks, two runs each. Cost relative to no removal; local resolution.
Figure 9: Fixed-action replay. Replaying the unmasked run’s actions under the masking rule gives the direct component; the rest of the observed difference is the residual. The example is the digest confirmation of Appendix H .
Setting
Rule
Total
Direct
Behavior
Steps
mini, Qwen3.6-27B
Rolling
+116%
+90%
+26%
+5.3
Batched
+4.9%
−3.4%
+8.2%
+2.9
Break-even
+2.8%
−3.7%
+6.5%
+2.7
mini, Coder
Rolling
+87%
+86%
+1.2%
−0.1
Batched
+12%
+2.2%
+9.8%
+1.8
Break-even
+1.8%
+3.4%
−1.6%
−0.4
Appendix
Table 6: Masking cost relative to no masking at c=0.1 , split into the direct effect (actions fixed) and the agent’s reaction. mini: 201 tasks; SWE-agent: 115 and 78 tasks.
Figure 10: Base Qwen3.5-9B versus the same model fine-tuned on efficient Qwen3.6-27B trajectories, on SWE-bench Verified. (a) Cost per run at c=0.1 ; (b) share of runs that submit; (c) official resolved rate; (d) script or test runs after the first edit, among submitted runs.
9.8% fewer hits than the token count (784-token blocks); rolling masking +173% instead of +116% at c=0.1
Reuse of the previous reply
never for Qwen3.6-27B (0 of 27,389 isolated requests, history re-rendered); 83% of requests for Qwen3-Coder; counting it moves rolling masking from +87% to +98%
Evaluation noise, cost
relative standard error of one evaluation’s mean cost 3–4% (201 tasks, one run each); two evaluations differ by 4–5%
Evaluation noise, resolution
two evaluations of one agent differ by 1.6 points (178 tasks, two runs each)
Appendix
Table 7: Measurement biases in cost experiments, with their size in our runs.
Coding agents resend a growing context on every model call, and context policies are usually judged by the prompt tokens they save. Executed work also depends on call count and prefix reuse: a caching runtime prefills only the tokens after the longest cached prefix. A shorter prompt can therefore cost more if it rewrites cached history. Our quality-gated protocol splits executed prefill into logical prompt minus reused prefix and measures KV residency, time, and energy. On 450 localization cases derived from SWE-rebench V2, preloading ten retrieved blocks (Eager) and adding anchored history compaction (Compact) both pass a five-point non-inferiority gate. In a separate 900-cell study, Eager cuts model calls by 35.7% and the GPU-energy proxy by 33.2% against on-demand search (Grep); Compact saves nothing further. Enabling prefix caching leaves Eager's prompt volume similar (+8.3%; its 95% CI spans zero) but cuts its uncached prefill by 67.7%. Deleting history can turn a token saving into extra work. With mini-SWE-agent on 100 SWE-bench Verified instances, keeping only the last five observations (Mask-5) cuts Eager's prompt volume by 33.3% but loses 135.9k reusable tokens per task. Uncached prefill rises by 608.5%, the energy proxy by 125.6%, and the policy misses the quality gate. Policies thus save physical work by shortening the trajectory and can multiply it by editing reusable history.
Zhongming Yu, Yitong Ma, Hengjia Yu +4
University of California, San Diego · University of Massachusetts Amherst
Context-reduction layers for API-based coding agents, including command-output compressors, retrieval rankers, and API-boundary proxies, are commonly evaluated by how much context or tool output they remove. We ask a different question: which interventions actually reduce end-to-end billed cost while preserving task success? Our primary evidence is a pre-specified, hash-frozen, paired campaign of 2,908 provider-billed Claude Code runs, of which 2,848 were analyzed, covering 103 tasks, seven repositories, and three models. The campaign compared a baseline with two generations of hook-based compression and an API-boundary proxy within a broader measured program of roughly 5,500 billed executions. Three findings emerge. First, prompt-cache traffic dominated cost composition, accounting for about 87% of reconstructed four-component cost (about 80% of the actual bill), with an 8.7% dollar-weighted residual not attributable from retained telemetry. Second, local payload reduction was not a reliable predictor of end-to-end billed cost. An arm that removed 38% of estimated raw tool-output tokens incurred 6.8% higher paired cost (95% CI: +2.8% to +11.3%), while per-task reduction showed only a weak association with cost change (Pearson r = 0.15). Third, aggressive compression can remove action-critical evidence: on SWE-bench-derived Go tasks, compression reduced successful patch application from 27/40 to 15/40 by corrupting verbatim edit anchors. We propose evaluating context-reduction systems by success-adjusted billed cost rather than token reduction alone.
Modern coding agents expose multiple tool surfaces -- IDE primitives, bash, and Model Context Protocol (MCP) code-execution -- and the field has shipped three contradictory claims about which one matters. We run the missing crossed comparison: an integrity-clean three-arm ablation (baseline / bash_only / code_only) on synthetic computation tasks and SWE-bench Mini modification tasks, holding model, harness, and prompts fixed, with two agents (Claude Code, OpenAI Codex CLI) so the comparison spans both regime and agent-design axes. Across the four resulting (regime, agent) cells, restricting the agent to a single execute_code MCP tool is cheaper than -- or statistically tied with -- its cheapest tool-rich rival in three cells (significantly on Artifact/Claude and SWE-bench/Codex; directionally on Artifact/Codex), with pass rates statistically tied within each cell. The lone exception is SWE-bench/Claude, where code_only is directionally costlier (+14.4%, not significant); a conditional-cost analysis localizes that gap to failure-cost on doomed-run trajectories, not a per-edit tax on successful runs. Two implications: the cheapest tool surface is jointly determined by task regime and agent design rather than by either axis alone, and the headline cost signal lives in cache-adjusted cost -- not pass rate, which is invariant across surfaces at the model sizes we evaluate. The benchmark harness, task suite, and analysis code are available at https://github.com/hyang0129/onlycodes.
Hong Yang, Qi Yu, Travis Desell
Rochester Institute of Technology Rochester, New York, USA