In CodeAct, language-model agents write Python that calls tools and use execution feedback to choose actions. Persistent runtimes preserve Python variables between actions; stateless runtimes clear them without resetting task progress. Training traces demonstrate task-solving strategies and runtime-specific ways to store and recover intermediate results. We study runtime transfer: whether agents trained under one contract remain effective under the other, a dependence that fixed-runtime evaluations can conceal. Across three tasks requiring a working record built from tool feedback, we fine-tune separate Qwen3-8B agents per task and runtime on instance-paired persistent and stateless traces and evaluate all four training-deployment combinations. We vary the per-turn tool-call cap: how many calls one Python action may execute. In our primary task, agents inspect hidden item attributes and select a high-value subset under a weight limit. At 80 calls per turn, a persistent-trained agent deployed statelessly scores 0.61 of optimal value against 0.77 under matched persistent deployment, but uses 4.4 times as many tokens rebuilding its record. Tightening the cap to 25 reduces mismatched quality to 0.07 while matched quality remains 0.66. Trace replay shows that the cap interrupts inspection, the reset erases the partial record, and rebuilding consumes calls needed for progress. The widening gap replicates across training seeds, a second rollout, and two additional base models. Across the settings studied, cap tightening produces a mismatch-specific collapse only where interruptions are frequent and matched execution resumes while mismatched execution restarts. Evaluations of trace-fine-tuned agents should specify episode limits, test runtime transfer across caps, and report cost alongside quality; otherwise, the same mismatch may appear as redundant computation or near-total quality loss.
Figures & tables
Figure 1: The same runtime mismatch costs computation under a slack cap and quality under a binding cap. We fix a persistent-trained agent on the stateless runtime and vary the per-turn cap. At c=80 the complete 61 -call sweep drawn here fits in one turn, so the agent rebuilds after the reset and retains much of matched quality at 4.4× the token cost. At c=25 the cap truncates the sweep, the reset erases the partial working record, and rebuilding the prefix consumes the next turn. Bars are mean normalized optimality over n=25 paired tasks (Table 4 ). Blue markers are matched persistent deployment. A representative excerpt is in App. D.1 .
Training runtime
Persistent deployment
Stateless deployment
Persistent
P→P : matched
P→S : main mismatch
Stateless
S→P : reverse transfer
S→S : matched
Table 1: The four conditions. We write X→Y for an X -trained agent deployed on a Y runtime, X,Y∈{P,S} .
Task
Working record
What a reset disrupts
Opaque Knapsack
Table of inspected items
A reset erases the table left by a cut
Navigation
Map of locks, traps and keys
A reset erases a partly checked map
Rule Diagnosis
Probes and a revised hypothesis
A reset breaks the gathered evidence
Table 2: Three tasks built so that the runtime matters. Each needs a record assembled from tool feedback that later actions depend on, and each is solvable under either runtime.
Limit
What it controls
Per-turn cap c
Tool calls inside one action block
Task budget
How much hidden task information the agent may reveal
Turn limit Tmax=40
Model calls an episode may attempt
Token and context limit
Maximum transcript length
Wall-clock limit
Maximum execution time
Table 3: The limits an evaluation sets, and what each one controls. The intervention changes the first row only. A failed or rejected model call spends a turn of the horizon without executing an action block, so attempted turns and executed blocks are not the same count (App. Table 20 ).
P -trained
S -trained
P→S cost
Cap
P→P
P→S
S→P
S→S
Runtime gap GP
Tokens/ep.
Cap-hit
c=25 (binding)
0.66
0.07
0.12
0.26
+0.59 [+0.46,+0.71]
498k
71%
c=80 (slack)
0.77
0.61
0.23
0.51
+0.15 [+0.00,+0.32]
162k
4%
Table 4: The 2×2 experiment at binding and slack per-turn caps. Entries are mean normalized optimality over the same n=25 paired instances (seed-0). GP=QPP−QPS carries a task-paired bootstrap 95% interval. Each row carries both training regimes’ runtime gaps, so the train × runtime interaction D is the difference between them. The last two columns summarize operational cost in the P→S condition. Per-condition intervals and the termination taxonomy are in App. Table 10 .
Figure 2: Cap tightening selectively harms the mismatched condition. (a) Mean normalized optimality across 8 caps on n=12 paired tasks, unsmoothed, with paired bootstrap 95% intervals. The matched condition, swept on the same tasks, does not collapse. (b) Distinct items gained after turn 0 against cumulative inspect calls at c=25 on the main evaluation’s n=25 paired tasks, measured from a common turn-0 baseline of 23.6 items on average: the mismatched agent issues 681 calls to add only 2.1 items, where matched deployment adds 13.2 in 89 , over the same instances.
Figure 3: The runtime gap increases when the cap is tightened from 80 to 25 in all four replications. Points are GP(25)−GP(80) over the same 25 instances, and bars are paired bootstrap 95% intervals. Replications 1–2 are two rollouts of the same adapters and differ only in sampling. Replications 3–4 retrain the same recipe on Mistral-7B-v0.3 and Llama-3.1-8B. The full factorial, the model-selection difference and the scope of each replication are in App. F.2 .
Figure 4: In the four settings studied, cap tightening amplifies mismatch only where truncation is common and matched execution resumes. Same four settings, same rows in both panels. (a) Exposure E25 for the mismatched P→S condition: the share of its slack-cap action blocks a cap of 25 would split. (b) Post-cap restart rate under matched and mismatched deployment, meaning the agent re-runs its listing call before acquiring anything new. Knapsack meets both conditions. With batch-limited Navigation both runtimes restart, so both deteriorate and we do not detect additional mismatch-specific amplification. Plain Navigation ( 17 post-cap turns) is too sparse for (b), and Rule Diagnosis has none (App. C.2 ).
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
Task
Latent information
Working record
Verifier
Key anti-collapse constraint
Opaque Knapsack
Item attributes and the allowed-class set
Inspected-item table
Exact DP optimum over allowed items
Non-degenerate optimum: optimal set ≥3 items, no item above 40% of optimal value
Navigation
Graph mechanics: traps, keys, locks
Map, keys, traps, locks
Goal / shortest-path progress score
Exactly 1 locked gate into the goal, its key reachable trap-free
Rule Diagnosis
Hidden piecewise rule
Probe map and hypothesis
Exact latent rule
≥3 intervals of length ≥6 , at least two internal breakpoints
Appendix
Table 5: The three families as working-record tasks. The anti-collapse constraint is enforced per instance by construction, except for Navigation, where it is best-effort : if the generator exhausts its candidate pool it returns the closest candidate with a recorded warning, so a minority of Navigation instances may be easier than targeted.
Call → return
Budget / constraints
list_items() → JSON array of all opaque item IDs
Free (no budget cost).
inspect(id) → JSON {"weight","value","class"}
Consumes one unit of B on the first inspection of an ID, and repeat inspections are cached and free. Errors if B is exhausted or the ID is unknown.
take_item(id) → None
Adds the item. Errors if already taken, of a disallowed class, or if it would exceed C . Every rejection is checked before any state change, and there is no drop tool, so an accepted take is irreversible.
finish() → None
Terminates the episode (runtime-provided).
Appendix
Table 6: Opaque Knapsack tool API. Tool return values are available to the executing Python block. The next observation carries the captured stdout and any uncaught execution error, so a return value is not added to the transcript automatically and must be printed to stay visible after a reset. Every tool call counts against the per-turn cap, including the calls below that spend no inspection budget. Budget B counts distinct inspected IDs.
Component
Persistent condition
Stateless condition
Changes with cap?
Interpreter bindings
Survive the turn boundary
Cleared each turn
No
active_globals
Prior names shown as active
Empty
No
State rule
“globals persist; never re-paste code”
“state resets; redefine each step”
No
Few-shot behavior
Reuses variables across turns
Prints and rebuilds state
No
Tool-cap instruction
Same wording with N substituted
Same
Yes
Appendix
Table 7: What the two conditions change, and what the cap changes. Within a runtime condition, changing c changes its announced value and its enforcement, and every other prompt component and resource limit stays fixed (App. D.1 ). The persistent-vs-stateless comparison is different: it bundles the top four rows with the interpreter’s actual reset behavior, which is why we read every cross-adapter quantity as bundled and say so in the Limitations.
Setting
Value
Teacher
Gemini 3 Flash ( gemini/gemini-3-flash-preview )
Teacher runtime
Runtime-matched (one rollout per instance per runtime)
Teacher cap / turns (Opaque Knapsack)
25 / 40
Candidate pairs
1,900
Dual-valid pairs
1,109 (58.4% paired yield)
Retained pairs
1,000 per training runtime (seeded shuffle, seed 42 )
Appendix
Table 8: Training-data construction (the candidate and dual-valid counts are Opaque Knapsack’s, and the two control families have their own paired pools, retained at the same size by the same recipe and evaluated only on their own family). A trace is valid only if it scores ≥0.5 , calls finish() within its last three steps, contains no near-duplicate recovery loop (character match ratio >0.9 against any of the previous four assistant messages, from Python’s standard-library sequence matcher at its default settings), and has bad-error density ≤0.1 , counting interpreter errors but excluding tool-contract errors (an exhausted inspection budget, say) and “ Tool call limit exceeded ”, which are legitimate feedback.
Table 9: Fine-tuning and inference. Settings are shared across all replications unless a column says otherwise. The two base-model replications retrain the same recipe on a different base checkpoint (App. F.2 ). The compute totals cover different numbers of adapters: 6 for Qwen3-8B and two for each base-model replication, with the 4 control-task adapters adding a further ≈10.71 GPU-h outside this table. All arms are served with vLLM [ Kwon et al., 2023 ] . Serving compute was never metered and is not reported.
Cap
Condition
Quality (95% CI)
Steps
Listing calls
Tokens
Cap-hit%
NameErr
Fin/turn/to
c=25
P→P (matched)
0.66[0.55,0.77]
9.8
1.1
113k
41
0.04
19/6/0
P→S (mismatch)
0.07[0.01,0.16]
35.2
30.8
498k
71
5.44
2/23/0
S→P
0.12[0.03,0.22]
18.6
1.0
288k
36
0.00
6/16/3
S→S (matched)
0.26[0.17,0.35]
11.2
1.0
172k
15
0.00
6/10/9
c=80
P→P
0.77[0.66,0.86]
4.6
1.0
37k
0
0.00
24/1/0
P→S
0.61[0.45,0.77]
13.0
7.9
162k
4
3.80
18/7/0
Appendix
Table 10: The 2×2 experiment at a binding ( c=25 ) and slack ( c=80 ) cap (Opaque Knapsack). Quality is normalized optimality on the main evaluation (seed-0, n=25 paired instances) with task-paired bootstrap 95% CIs (resampling task instances, from a single temperature- 0.2 rollout per condition with no sampling seed, not decoding variability). Operational columns are per-episode means over the same instances. Steps counts logged steps, which are the model calls that returned a usable completion, not the model calls the episode attempted: a call that fails or is rejected spends a turn of the horizon without logging a step (App. Table 20 ). Listing calls counts list_items() occurrences in the executed code, including the opening listing every episode makes, so a condition that never re-reconstructs reads 1.0 . Cap-hit is the mean per-episode fraction of logged steps reaching the cap and NameErr counts steps raising a missing-variable error. Tokens sums input and output tokens across accepted model calls, including transcript tokens processed again on later calls, which dominate the total: input is 95 – 97% of it in every cell here. Termination is finish/turn-limit/timeout counts, and timed-out episodes retain their partial score rather than zero. The collapse condition ( P→S @ 25 ) records 30.8 listing calls per episode yet inspects fewer distinct items, spends the most tokens, and terminates by turn-limit. The binding-cap interaction replicates across three training seeds (Sec. 4 ).
Quantity
Definition
Cap hit
The logged step’s interpreter error reports the per-turn tool-call limit exceeded, and a cap-rejected call does not decrement the task inspection budget (App. D.1 )
Missing binding ( NameErr )
The logged step’s interpreter error contains NameError , the signature of a reset destroying a binding the policy assumed persisted
Post-cap restart
The next executed turn after a cap hit reissues the task’s index or listing operation ( list_items , neighbors ) before acquiring anything new. This is conditional on a cap hit, unlike the unconditional listing-call count in Table 10
Termination
Exactly one of: finish() called, maximum turns reached, or episode wall-clock timeout
Exposure Ec
The fraction of a condition’s slack-cap action blocks whose replayed call count exceeds c (App. D.2 )
Appendix
Table 11: Trace-derived quantities.
Seed
Cap
QPP
QPS
QSP
QSS
D
Primary evaluation, seed-0, n=25
0
25
0.662
0.075
0.115
0.262
+0.734
0
80
0.766
0.615
0.232
0.505
+0.426
Sign-replication set, common 8 task IDs
0
25
0.722
0.039
0.166
0.109
+0.625
0
80
0.796
0.752
0.112
0.326
+0.257
Appendix
Table 12: Per-seed condition means (normalized optimality) and the interaction D . The first row is the primary seed-0 evaluation at n=25 . The rest are all three seeds on the same common 8 task IDs (generation indices 0 – 7 , pre-specified and outcome-independent). Seed-0’s common-set conditions differ from its primary row because that is the smaller sample. Per-seed T=D(25)−D(80) is positive in every seed.
Cap c
10
20
25
30
40
50
80
100
P→S (mismatch)
0.00
0.00
0.05
0.05
0.23
0.23
0.64
0.55
P→P (matched)
0.40
0.54
0.60
0.57
0.60
0.73
0.64
0.61
Appendix
Table 13: Dense cap sweep, marginal means (Opaque Knapsack, n=12 per cap, the same paired task IDs at every cap). Scores are normalized optimality from a single temperature- 0.2 rollout per cap, with no sampling seed. Fig. 2 a plots these with their intervals.
Setting
Inspect calls
Unique items
Replay frac.
Coverage
Decision
Post-cap restart
P→P , c=25 (matched)
89
36.8
0.59
0.83
0.80
0.00
P→S , c=25 (mismatch)
681
25.7
0.96
0.71
0.09
0.84
P→S , c=40
660
38.2
0.94
0.81
0.33
—
Appendix
Table 14: Trace mechanism decomposition (trace-conditioned replay, per-episode means). Unique is the number of distinct items inspected and replay frac. the share of inspect calls that are cached re-inspections. Coverage and decision are defined in the text. Post-cap restart is the fraction of turns following a cap hit that reissue list_items before acquiring anything new (App. C.2 ), counted over those turns rather than over cap hits, so a truncation the episode never answers is excluded. Fig. 4 b plots the restart column for both runtimes, and it is the difference between them that carries the continuation claim. The c=40 row is a replay condition only and has no restart measurement. The cross-task restart values are in Table 15 .
Setting
Tool organization
E25
Restart P→P
Restart P→S
Interpretation
Opaque Knapsack
Long batched inspection
0.53
0.00
0.84
Mismatch-specific failure
Navigation
Broad batched topology
0.04
Too few events
Too few events
Cap rarely exposes mismatch
Navigation b=2
Narrow topology batches
0.42
0.98
0.94
Both runtimes deteriorate
Rule Diagnosis
Short adaptive probe cycles
0.00
—
—
No exposure
Appendix
Table 15: Cross-task exposure and continuation. E25 is the fraction of slack-cap action blocks whose replayed call count exceeds the binding cap. Restart is the post-cap re-derivation fraction of Table 14 . Plain Navigation has too few post-cap turns to estimate either restart fraction ( n=6 and 11 ), and Rule Diagnosis has none at all.
Task
Condition
Q(c=25)
Q(c=80)
Tokens @ 25
Tokens @ 80
Navigation
P→P (matched)
0.55
0.68
142k
155k
P→S (mismatch)
0.41
0.38
352k
366k
S→P
0.32
0.44
292k
228k
S→S (matched)
0.53
0.33
200k
256k
Rule Diagnosis
P→P (matched)
0.70
0.69
102k
61k
P→S (mismatch)
0.33
0.41
77k
108k
Appendix
Table 16: Control-task outcomes: quality and cost for the full train × runtime factorial at both caps ( n=16 paired instances per cell, one seed, one rollout). Quality is each family’s own score, phased shortest-path progress for Navigation and, for Rule Diagnosis, the composite rule score defined in App. E.1 . Tokens are per-episode means of recorded usage for accepted completions.
Task
GP(25)
GP(80)
AP
GS(25)
GS(80)
AS
Navigation
+0.13
+0.30
−0.17[−0.50,+0.15]
+0.21
−0.11
+0.32[−0.12,+0.81]
Rule Diagnosis
+0.38
+0.27
+0.11[−0.17,+0.39]
−0.16
−0.01
−0.14[−0.38,+0.10]
Appendix
Table 17: Cap amplification of each runtime gap on the control tasks, over the same 16 paired instances. GP(c)=QPP(c)−QPS(c) is the persistent-trained agent’s runtime gap and GS(c)=QSS(c)−QSP(c) is the stateless-trained agent’s. Amplification is the paired difference G(25)−G(80) , with a 95% bootstrap CI over task instances. Every interval spans zero.
Quantity
P→S
+ Carry
P→P
Carry (re-run)
+ Announced
Quality
0.038
0.132
0.575
0.081
0.080
NameError turns/ep
9.6
1.3
0.1
6.0
5.0
… of them on carried turns
6.2
0.0
0.1
0.0
0.0
Re-derives after a truncation
0.76
0.76
0.00
0.86
0.31
Distinct items
24.2
28.2
34.1
28.0
29.6
Decision achieved/OPT( Q )
0.08
0.23
0.72
0.11
0.11
Appendix
Table 18: Cap-boundary carryover evaluations, with trace-derived behavioral metrics ( n=25 per condition). The first three conditions are one batch and the last two another, so read quality down each block, not across them. “Carried” turns are those following a truncated one. In conditions without carryover those are the turns a checkpoint would have been carried into.
Tmax=40
Tmax=80
Tmax=160
P→P (matched)
0.66
0.59
0.66
P→S (mismatch)
0.076
0.058
0.088
P→S overflow
3/25
13/20
13/20
P→P overflow
4/25
—
—
Appendix
Table 19: Episode-horizon sweep at c=25 (Opaque Knapsack, normalized optimality), on the same common set of 20 instances, with the per-turn context behavior that accompanies it. The 80/160 configs raise both the turn limit Tmax and the episode wall-clock ( 3,000→4,000 s). “Overflow” counts episodes with at least one context-window overflow. The runtime hard-errors rather than silently truncating once a turn’s input plus the 8192 -token output reservation would exceed the 40960 window. Overflow denominators are each condition’s own full episode set, not the common set the quality rows use.
Cap
Condition
Attempted
Steps
Failed
Context
Rejected
c=25
P→P (matched)
13.2
9.8
85
85/85
0
P→S (mismatch)
37.3
35.2
52
52/52
0
S→P
27.8
18.6
229
227/229
0
S→S (matched)
21.6
11.2
260
260/260
0
c=80
P→P
4.6
4.6
0
0/0
0
P→S
14.5
13.0
37
37/37
0
Appendix
Table 20: Horizon accounting for the main evaluation (Opaque Knapsack, seed-0, n=25 episodes per condition, Tmax=40 ). Attempted and Steps are per-episode means of model calls attempted and steps logged. Failed and Rejected are totals over the 25 episodes, the first counting calls that raised and the second counting completions rejected for not stopping on their own. Context is the share of the failed calls that were serving-window rejections. Termination counts are in Table 10 , and every episode that ended at the horizon satisfies attempted = steps + failed + rejected.
Qwen3-8B
+2nd rollout
Mistral-7B-v0.3
Llama-3.1-8B
GP(25)
+0.59 [+0.46,+0.71]
+0.63 [+0.50,+0.75]
+0.64 [+0.50,+0.78]
+0.58 [+0.43,+0.72]
GP(80)
+0.15 [+0.00,+0.32]
+0.08 [-0.06,+0.21]
+0.14 [-0.01,+0.28]
+0.18 [+0.01,+0.35]
Cap amplification
+0.44 [+0.22,+0.65]
+0.55 [+0.38,+0.71]
+0.51 [+0.32,+0.71]
+0.40 [+0.16,+0.63]
M(80)
+0.11 [-0.08,+0.29]
-0.02 [-0.15,+0.09]
-0.18 [-0.32,-0.04]
-0.23 [-0.42,-0.04]
T
+0.31 [+0.04,+0.57]
+0.40 [+0.24,+0.55]
+0.41 [+0.09,+0.74]
+0.34 [+0.03,+0.65]
Appendix
Table 21: Replications. GP is the persistent model’s runtime gap and GP(25)−GP(80) its cap amplification. T is the change in the train × runtime interaction across the caps, and M=QPS−QSS is the secondary model-selection difference under stateless deployment. The runtime gap and its amplification are present within every replication. M(80) is the quantity whose sign the main text declines to claim: it is never positive with an interval that excludes zero, and is reliably negative in the two base-model settings.
Artifact
Released
Contents
Code
Yes
Task generators, both runtimes, agent harness, training and serving scripts, every condition’s benchmark config, analysis scripts, and the verbatim prompts
Evaluation traces
Yes
Per-task rollouts of every reported condition: main evaluation, dense sweep, horizon control, cross-task controls, both interventions, replications
Opaque Knapsack adapters
Yes
6 Qwen3-8B (persistent/stateless × 3 seeds) + 4 replication adapters, one card each
Control-task adapters
Yes
The 4 Navigation and Rule Diagnosis adapters, same recipe and card format
Teacher traces
Yes
The filtered, instance-paired SFT data of App. B , as fed to the fine-tuning script. The raw cached teacher responses it was built from are not released
Appendix
Table 22: Released artifacts.
Runtime contracts and conditions
P , S
The persistent runtime, under which the variables an agent creates stay available on later turns, and the stateless runtime, under which each turn begins with a fresh set of them. Task and environment state, including the tool objects, survives under both (Sec. 3 )
Runtime contract
The interpreter semantics alone: whether agent-created bindings survive the turn boundary (Sec. 3 )
Runtime condition
The contract together with the prompt package that announces it: a mode instruction, a runtime-state banner and runtime-specific few-shot examples. Cross-runtime comparisons switch the whole package, not the semantics in isolation (Sec. 3 , App. A.3 )
X→Y
An X -trained agent deployed on a Y runtime. P→P and S→S are the matched references, P→S the main mismatch and S→P the reverse transfer (Table 1 )
Condition
One of the four X→Y combinations, evaluated at one cap (Table 1 )
Runtime transfer
Whether an agent trained under one runtime still solves the task when deployed under the other (Sec. 1 )
Appendix
Table 23: Notation and terminology: runtimes, conditions and evaluation settings.
Outcome and reported quantities
Normalized optimality
The outcome: achieved value over the exact optimum on allowed-class items, clamped to 0 on any capacity overflow (Sec. 3 , App. A.2 )
QXY(c)
Mean task quality for an adapter trained under runtime X and evaluated under runtime Y at cap c : normalized optimality for Opaque Knapsack, and each control family’s own score (Sec. 3 , App. E.1 )
A multi-agent pipeline with N agents typically issues N LLM calls per run. Merging agents into fewer calls (compound execution) promises token savings, but naively merged calls silently degrade quality through tool loss and prompt compression. We present Agent Capsules, an adaptive execution runtime that treats multi-agent pipeline execution as an optimization problem with empirical quality constraints. The runtime instruments coordination overhead per group, scores composition opportunity, selects among three compound execution strategies, and gates every mode switch on rolling-mean output quality. A controlled negative result confirms that injecting more context into a merged call worsens compression rather than relieving it, so the framework's escalation ladder (standard, then two-phase, then sequential) recovers quality by moving toward per-agent dispatch rather than by rewriting merged prompts. On LLM-judged quality, the controller matches a hand-tuned oracle on every measured (model, group, mode) cell: routing compound whenever the oracle would, and reverting to fine whenever quality would fail the floor, without per-model configuration. Against a hand-crafted LangGraph implementation of a 14-agent competitive intelligence pipeline, Agent Capsules uses 51% fewer fine-mode input tokens and 42% fewer compound-mode input tokens, at +0.020 and +0.017 quality respectively. Against a DSPy implementation of a 5-agent due diligence pipeline, the framework uses 19% fewer tokens than uncompiled DSPy at quality parity, and 68% fewer tokens than MIPROv2 at +0.052 quality. Even before compound mode fires, the runtime delivers efficiency through automatic policy resolution, cache-aligned prompts, and topology-aware context injection, matching both hand-tuned and compile-time baselines without training data or per-pipeline engineering.
Modern coding agents expose multiple tool surfaces -- IDE primitives, bash, and Model Context Protocol (MCP) code-execution -- and the field has shipped three contradictory claims about which one matters. We run the missing crossed comparison: an integrity-clean three-arm ablation (baseline / bash_only / code_only) on synthetic computation tasks and SWE-bench Mini modification tasks, holding model, harness, and prompts fixed, with two agents (Claude Code, OpenAI Codex CLI) so the comparison spans both regime and agent-design axes. Across the four resulting (regime, agent) cells, restricting the agent to a single execute_code MCP tool is cheaper than -- or statistically tied with -- its cheapest tool-rich rival in three cells (significantly on Artifact/Claude and SWE-bench/Codex; directionally on Artifact/Codex), with pass rates statistically tied within each cell. The lone exception is SWE-bench/Claude, where code_only is directionally costlier (+14.4%, not significant); a conditional-cost analysis localizes that gap to failure-cost on doomed-run trajectories, not a per-edit tax on successful runs. Two implications: the cheapest tool surface is jointly determined by task regime and agent design rather than by either axis alone, and the headline cost signal lives in cache-adjusted cost -- not pass rate, which is invariant across surfaces at the model sizes we evaluate. The benchmark harness, task suite, and analysis code are available at https://github.com/hyang0129/onlycodes.
Hong Yang, Qi Yu, Travis Desell
Rochester Institute of Technology Rochester, New York, USA
Long-horizon agents can fail even when their underlying models can solve the constituent steps. They may lose track of mutable state, fail to reactivate lessons from earlier executions, skip known procedures, or stop prematurely. We bet on harness scaling to improve the execution system around an agent without changing its model weights. We introduce StateM, an agent-native runtime that organizes execution around durable states, phase-local context, checked transitions, recoverable runbooks, and versioned procedural practices that agents and users can inspect together. On Terminal-Bench 2.1, StateM raises GPT-5.5 xhigh to 92.1%, versus 83.1% reference and GPT-5.6 Sol Ultra at 91.9%. The runbook transfers unchanged to GPT-5.6. With GPT-5.6 Sol xhigh, StateM reaches 95.3% raw accuracy across 445 trials and succeeds on all 89 tasks at least once. The frozen profile raises GPT-5.6 Luna from 76.7 to 85.4%, above the 84.9% Sol xhigh reference. Using the same runtime, runbook structure, and golden rules, less than $38 of adaptation raises DeepSeek-V4 Flash from 82.7 to 88.1% under standard timeouts and to 89.1% on an 88-task common core. Extending only the remaining latency-sensitive task matches the reported 88.8% GPT-5.6 Sol max result. Final-score API usage is about $15 versus $574.68 for the GPT reference; total DeepSeek expenditure is $52.22. On BusinessBench, family-specific runbooks built on development sets yield held-out gains of 0.55 macro and 1.34 micro points; two mechanism-matched families improve by 10.04 points. Concrete rules generalize when tasks share execution structure, while the control methodology applies broadly. StateM turns selected postmortem findings into persistent, executable preconditions and practices, making learned controls explicit and enforceable through stateful controls. Code at github.com/henryqin1997/statem.