Evaluating Agents Across Runtime Contracts: When Mismatch Costs Efficiency or Quality

Victor May; Van Khue Nguyen; Aaditya Salgarkar; Yishan Wang; Diganta Misra; Huu Nguyen

cs.AIMar 1, 2026

Evaluating Agents Across Runtime Contracts: When Mismatch Costs Efficiency or Quality

Authors: Victor May, Van Khue Nguyen, Aaditya Salgarkar, Yishan Wang, Diganta Misra, Huu Nguyen

Organizations: Ontocord · École Polytechnique · Carnegie Mellon University · MPI-IS Tübingen ·

Abstract

In CodeAct, language-model agents write Python that calls tools and use execution feedback to choose actions. Persistent runtimes preserve Python variables between actions; stateless runtimes clear them without resetting task progress. Training traces demonstrate task-solving strategies and runtime-specific ways to store and recover intermediate results. We study runtime transfer: whether agents trained under one contract remain effective under the other, a dependence that fixed-runtime evaluations can conceal. Across three tasks requiring a working record built from tool feedback, we fine-tune separate Qwen3-8B agents per task and runtime on instance-paired persistent and stateless traces and evaluate all four training-deployment combinations. We vary the per-turn tool-call cap: how many calls one Python action may execute. In our primary task, agents inspect hidden item attributes and select a high-value subset under a weight limit. At 80 calls per turn, a persistent-trained agent deployed statelessly scores 0.61 of optimal value against 0.77 under matched persistent deployment, but uses 4.4 times as many tokens rebuilding its record. Tightening the cap to 25 reduces mismatched quality to 0.07 while matched quality remains 0.66. Trace replay shows that the cap interrupts inspection, the reset erases the partial record, and rebuilding consumes calls needed for progress. The widening gap replicates across training seeds, a second rollout, and two additional base models. Across the settings studied, cap tightening produces a mismatch-specific collapse only where interruptions are frequent and matched execution resumes while mismatched execution restarts. Evaluations of trace-fine-tuned agents should specify episode limits, test runtime transfer across caps, and report cost alongside quality; otherwise, the same mismatch may appear as redundant computation or near-total quality loss.

Figures & tables

Figure 1: The same runtime mismatch costs computation under a slack cap and quality under a binding cap. We fix a persistent-trained agent on the stateless runtime and vary the per-turn cap. At

c{=}80

the complete

61

-call sweep drawn here fits in one turn, so the agent rebuilds after the reset and retains much of matched quality at

4.4\times

the token cost. At

c{=}25

the cap truncates the sweep, the reset erases the partial working record, and rebuilding the prefix consumes the next turn. Bars are mean normalized optimality over

n{=}25

paired tasks (Table 4 ). Blue markers are matched persistent deployment. A representative excerpt is in App. D.1 .

Training runtime	Persistent deployment	Stateless deployment
Persistent	$P\!\to\!P$ : matched	$P\!\to\!S$ : main mismatch
Stateless	$S\!\to\!P$ : reverse transfer	$S\!\to\!S$ : matched

Table 1: The four conditions. We write

X\!\to\!Y

for an

X

-trained agent deployed on a

Y

runtime,

X,Y\in\{P,S\}

Task	Working record	What a reset disrupts
Opaque Knapsack	Table of inspected items	A reset erases the table left by a cut
Navigation	Map of locks, traps and keys	A reset erases a partly checked map
Rule Diagnosis	Probes and a revised hypothesis	A reset breaks the gathered evidence

Table 2: Three tasks built so that the runtime matters. Each needs a record assembled from tool feedback that later actions depend on, and each is solvable under either runtime.

Limit	What it controls
Per-turn cap $c$	Tool calls inside one action block
Task budget	How much hidden task information the agent may reveal
Turn limit $T_{\max}{=}40$	Model calls an episode may attempt
Token and context limit	Maximum transcript length
Wall-clock limit	Maximum execution time

Table 3: The limits an evaluation sets, and what each one controls. The intervention changes the first row only. A failed or rejected model call spends a turn of the horizon without executing an action block, so attempted turns and executed blocks are not the same count (App. Table 20 ).

Cap	$P\!\to\!P$	$P\!\to\!S$	$S\!\to\!P$	$S\!\to\!S$	Runtime gap $G_{P}$	Tokens/ep.	Cap-hit
	$P$ -trained		$S$ -trained			$P\!\to\!S$ cost
$c{=}25$ (binding)	0.66	$\mathbf{0.07}$	0.12	0.26	+0.59 [+0.46,+0.71]	498k	71%
$c{=}80$ (slack)	0.77	0.61	0.23	0.51	+0.15 [+0.00,+0.32]	162k	4%

Table 4: The

2\times 2

experiment at binding and slack per-turn caps. Entries are mean normalized optimality over the same

n{=}25

paired instances (seed-0).

G_{P}=Q_{PP}-Q_{PS}

carries a task-paired bootstrap

95\%

interval. Each row carries both training regimes’ runtime gaps, so the train

\times

runtime interaction

D

is the difference between them. The last two columns summarize operational cost in the

P\!\to\!S

condition. Per-condition intervals and the termination taxonomy are in App. Table 10 .

Figure 2: Cap tightening selectively harms the mismatched condition. (a) Mean normalized optimality across

8

caps on

n{=}12

paired tasks, unsmoothed, with paired bootstrap

95\%

intervals. The matched condition, swept on the same tasks, does not collapse. (b) Distinct items gained after turn 0 against cumulative inspect calls at

c{=}25

on the main evaluation’s

n{=}25

paired tasks, measured from a common turn-0 baseline of

23.6

items on average: the mismatched agent issues

681

calls to add only

2.1

items, where matched deployment adds

13.2

89

, over the same instances.

Figure 3: The runtime gap increases when the cap is tightened from 80 to 25 in all four replications. Points are

G_{P}(25)-G_{P}(80)

over the same

25

instances, and bars are paired bootstrap

95\%

intervals. Replications 1–2 are two rollouts of the same adapters and differ only in sampling. Replications 3–4 retrain the same recipe on Mistral-7B-v0.3 and Llama-3.1-8B. The full factorial, the model-selection difference and the scope of each replication are in App. F.2 .

Figure 4: In the four settings studied, cap tightening amplifies mismatch only where truncation is common and matched execution resumes. Same four settings, same rows in both panels. (a) Exposure

E_{25}

for the mismatched

P\!\to\!S

condition: the share of its slack-cap action blocks a cap of

25

would split. (b) Post-cap restart rate under matched and mismatched deployment, meaning the agent re-runs its listing call before acquiring anything new. Knapsack meets both conditions. With batch-limited Navigation both runtimes restart, so both deteriorate and we do not detect additional mismatch-specific amplification. Plain Navigation (

17

post-cap turns) is too sparse for (b), and Rule Diagnosis has none (App. C.2 ).

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Latent information	Working record	Verifier	Key anti-collapse constraint
Opaque Knapsack	Item attributes and the allowed-class set	Inspected-item table	Exact DP optimum over allowed items	Non-degenerate optimum: optimal set $\geq 3$ items, no item above $40\%$ of optimal value
Navigation	Graph mechanics: traps, keys, locks	Map, keys, traps, locks	Goal / shortest-path progress score	Exactly $1$ locked gate into the goal, its key reachable trap-free
Rule Diagnosis	Hidden piecewise rule	Probe map and hypothesis	Exact latent rule	$\geq 3$ intervals of length $\geq 6$ , at least two internal breakpoints

Appendix

Table 5: The three families as working-record tasks. The anti-collapse constraint is enforced per instance by construction, except for Navigation, where it is best-effort : if the generator exhausts its candidate pool it returns the closest candidate with a recorded warning, so a minority of Navigation instances may be easier than targeted.

Call $\rightarrow$ return	Budget / constraints
list_items() $\rightarrow$ JSON array of all opaque item IDs	Free (no budget cost).
inspect(id) $\rightarrow$ JSON {"weight","value","class"}	Consumes one unit of $B$ on the first inspection of an ID, and repeat inspections are cached and free. Errors if $B$ is exhausted or the ID is unknown.
take_item(id) $\rightarrow$ None	Adds the item. Errors if already taken, of a disallowed class, or if it would exceed $C$ . Every rejection is checked before any state change, and there is no drop tool, so an accepted take is irreversible.
finish() $\rightarrow$ None	Terminates the episode (runtime-provided).

Appendix

Table 6: Opaque Knapsack tool API. Tool return values are available to the executing Python block. The next observation carries the captured stdout and any uncaught execution error, so a return value is not added to the transcript automatically and must be printed to stay visible after a reset. Every tool call counts against the per-turn cap, including the calls below that spend no inspection budget. Budget

B

counts distinct inspected IDs.

Component	Persistent condition	Stateless condition	Changes with cap?
Interpreter bindings	Survive the turn boundary	Cleared each turn	No
active_globals	Prior names shown as active	Empty	No
State rule	“globals persist; never re-paste code”	“state resets; redefine each step”	No
Few-shot behavior	Reuses variables across turns	Prints and rebuilds state	No
Tool-cap instruction	Same wording with $N$ substituted	Same	Yes

Appendix

Table 7: What the two conditions change, and what the cap changes. Within a runtime condition, changing

c

changes its announced value and its enforcement, and every other prompt component and resource limit stays fixed (App. D.1 ). The persistent-vs-stateless comparison is different: it bundles the top four rows with the interpreter’s actual reset behavior, which is why we read every cross-adapter quantity as bundled and say so in the Limitations.

Setting	Value
Teacher	Gemini 3 Flash ( gemini/gemini-3-flash-preview )
Teacher runtime	Runtime-matched (one rollout per instance per runtime)
Teacher cap / turns (Opaque Knapsack)	$25$ / $40$
Candidate pairs	1,900
Dual-valid pairs	1,109 (58.4% paired yield)
Retained pairs	1,000 per training runtime (seeded shuffle, seed $42$ )

Appendix

Table 8: Training-data construction (the candidate and dual-valid counts are Opaque Knapsack’s, and the two control families have their own paired pools, retained at the same size by the same recipe and evaluated only on their own family). A trace is valid only if it scores

\geq 0.5

, calls finish() within its last three steps, contains no near-duplicate recovery loop (character match ratio

>0.9

against any of the previous four assistant messages, from Python’s standard-library sequence matcher at its default settings), and has bad-error density

\leq 0.1

, counting interpreter errors but excluding tool-contract errors (an exhausted inspection budget, say) and “ Tool call limit exceeded ”, which are legitimate feedback.

	Qwen3-8B	Mistral-7B-v0.3	Llama-3.1-8B
Base checkpoint	Qwen/Qwen3-8B	mistralai/	meta-llama/
		Mistral-7B-v0.3	Llama-3.1-8B
Adaptation	QLoRA, 4-bit NF4, double quantization, bfloat16 compute
Rank / $\alpha$ / dropout	$r{=}64$ / $\alpha{=}128$ / $0.05$
Modules	All attention and MLP projections ( $q,k,v,o,\text{gate},\text{up},\text{down}$ )
Epochs / optimizer / LR	$3$ / AdamW, cosine, peak $1\times 10^{-4}$ , warmup $0.03$ , decay $0.01$

Appendix

Table 9: Fine-tuning and inference. Settings are shared across all replications unless a column says otherwise. The two base-model replications retrain the same recipe on a different base checkpoint (App. F.2 ). The compute totals cover different numbers of adapters: 6 for Qwen3-8B and two for each base-model replication, with the 4 control-task adapters adding a further

\approx 10.71

GPU-h outside this table. All arms are served with vLLM [ Kwon et al., 2023 ] . Serving compute was never metered and is not reported.

Cap	Condition	Quality (95% CI)	Steps	Listing calls	Tokens	Cap-hit%	NameErr	Fin/turn/to
$c{=}25$	$P\!\to\!P$ (matched)	$0.66\ [0.55,0.77]$	9.8	1.1	113k	41	0.04	19/6/0
	$P\!\to\!S$ (mismatch)	$\mathbf{0.07}\ [0.01,0.16]$	35.2	30.8	498k	71	5.44	2/23/0
	$S\!\to\!P$	$0.12\ [0.03,0.22]$	18.6	1.0	288k	36	0.00	6/16/3
	$S\!\to\!S$ (matched)	$0.26\ [0.17,0.35]$	11.2	1.0	172k	15	0.00	6/10/9
$c{=}80$	$P\!\to\!P$	$0.77\ [0.66,0.86]$	4.6	1.0	37k	0	0.00	24/1/0
	$P\!\to\!S$	$\mathbf{0.61}\ [0.45,0.77]$	13.0	7.9	162k	4	3.80	18/7/0

Appendix

Table 10: The

2\times 2

experiment at a binding (

c{=}25

) and slack (

c{=}80

) cap (Opaque Knapsack). Quality is normalized optimality on the main evaluation (seed-0,

n{=}25

paired instances) with task-paired bootstrap

95\%

CIs (resampling task instances, from a single temperature-

0.2

rollout per condition with no sampling seed, not decoding variability). Operational columns are per-episode means over the same instances. Steps counts logged steps, which are the model calls that returned a usable completion, not the model calls the episode attempted: a call that fails or is rejected spends a turn of the horizon without logging a step (App. Table 20 ). Listing calls counts list_items() occurrences in the executed code, including the opening listing every episode makes, so a condition that never re-reconstructs reads

1.0

. Cap-hit is the mean per-episode fraction of logged steps reaching the cap and NameErr counts steps raising a missing-variable error. Tokens sums input and output tokens across accepted model calls, including transcript tokens processed again on later calls, which dominate the total: input is

95

–

97\%

of it in every cell here. Termination is finish/turn-limit/timeout counts, and timed-out episodes retain their partial score rather than zero. The collapse condition (

P\!\to\!S

25

) records

30.8

listing calls per episode yet inspects fewer distinct items, spends the most tokens, and terminates by turn-limit. The binding-cap interaction replicates across three training seeds (Sec. 4 ).

Quantity	Definition
Cap hit	The logged step’s interpreter error reports the per-turn tool-call limit exceeded, and a cap-rejected call does not decrement the task inspection budget (App. D.1 )
Missing binding ( NameErr )	The logged step’s interpreter error contains NameError , the signature of a reset destroying a binding the policy assumed persisted
Post-cap restart	The next executed turn after a cap hit reissues the task’s index or listing operation ( list_items , neighbors ) before acquiring anything new. This is conditional on a cap hit, unlike the unconditional listing-call count in Table 10
Termination	Exactly one of: finish() called, maximum turns reached, or episode wall-clock timeout
Exposure $E_{c}$	The fraction of a condition’s slack-cap action blocks whose replayed call count exceeds $c$ (App. D.2 )

Appendix

Table 11: Trace-derived quantities.

Seed	Cap	$Q_{PP}$	$Q_{PS}$	$Q_{SP}$	$Q_{SS}$	$D$
Primary evaluation, seed-0, $n{=}25$
0	25	0.662	0.075	0.115	0.262	$+0.734$
0	80	0.766	0.615	0.232	0.505	$+0.426$
Sign-replication set, common $8$ task IDs
0	25	0.722	0.039	0.166	0.109	$+0.625$
0	80	0.796	0.752	0.112	0.326	$+0.257$

Appendix

Table 12: Per-seed condition means (normalized optimality) and the interaction

D

. The first row is the primary seed-0 evaluation at

n{=}25

. The rest are all three seeds on the same common

8

task IDs (generation indices

0

–

7

, pre-specified and outcome-independent). Seed-0’s common-set conditions differ from its primary row because that is the smaller sample. Per-seed

T{=}D(25){-}D(80)

is positive in every seed.

Cap $c$	10	20	25	30	40	50	80	100
$P\!\to\!S$ (mismatch)	0.00	0.00	0.05	0.05	0.23	0.23	0.64	0.55
$P\!\to\!P$ (matched)	0.40	0.54	0.60	0.57	0.60	0.73	0.64	0.61

Appendix

Table 13: Dense cap sweep, marginal means (Opaque Knapsack,

n{=}12

per cap, the same paired task IDs at every cap). Scores are normalized optimality from a single temperature-

0.2

rollout per cap, with no sampling seed. Fig. 2 a plots these with their intervals.

Setting	Inspect calls	Unique items	Replay frac.	Coverage	Decision	Post-cap restart
$P\!\to\!P$ , $c{=}25$ (matched)	89	36.8	0.59	0.83	0.80	0.00
$P\!\to\!S$ , $c{=}25$ (mismatch)	681	25.7	0.96	0.71	0.09	0.84
$P\!\to\!S$ , $c{=}40$	660	38.2	0.94	0.81	0.33	—

Appendix

Table 14: Trace mechanism decomposition (trace-conditioned replay, per-episode means). Unique is the number of distinct items inspected and replay frac. the share of inspect calls that are cached re-inspections. Coverage and decision are defined in the text. Post-cap restart is the fraction of turns following a cap hit that reissue list_items before acquiring anything new (App. C.2 ), counted over those turns rather than over cap hits, so a truncation the episode never answers is excluded. Fig. 4 b plots the restart column for both runtimes, and it is the difference between them that carries the continuation claim. The

c{=}40

row is a replay condition only and has no restart measurement. The cross-task restart values are in Table 15 .

Setting	Tool organization	$E_{25}$	Restart $P\!\to\!P$	Restart $P\!\to\!S$	Interpretation
Opaque Knapsack	Long batched inspection	0.53	0.00	0.84	Mismatch-specific failure
Navigation	Broad batched topology	0.04	Too few events	Too few events	Cap rarely exposes mismatch
Navigation $b{=}2$	Narrow topology batches	0.42	0.98	0.94	Both runtimes deteriorate
Rule Diagnosis	Short adaptive probe cycles	0.00	—	—	No exposure

Appendix

Table 15: Cross-task exposure and continuation.

E_{25}

is the fraction of slack-cap action blocks whose replayed call count exceeds the binding cap. Restart is the post-cap re-derivation fraction of Table 14 . Plain Navigation has too few post-cap turns to estimate either restart fraction (

n{=}6

and

11

), and Rule Diagnosis has none at all.

Task	Condition	$Q(c{=}25)$	$Q(c{=}80)$	Tokens @ $25$	Tokens @ $80$
Navigation	$P\!\to\!P$ (matched)	0.55	0.68	142k	155k
	$P\!\to\!S$ (mismatch)	0.41	0.38	352k	366k
	$S\!\to\!P$	0.32	0.44	292k	228k
	$S\!\to\!S$ (matched)	0.53	0.33	200k	256k
Rule Diagnosis	$P\!\to\!P$ (matched)	0.70	0.69	102k	61k
	$P\!\to\!S$ (mismatch)	0.33	0.41	77k	108k

Appendix

Table 16: Control-task outcomes: quality and cost for the full train

\times

runtime factorial at both caps (

n{=}16

paired instances per cell, one seed, one rollout). Quality is each family’s own score, phased shortest-path progress for Navigation and, for Rule Diagnosis, the composite rule score defined in App. E.1 . Tokens are per-episode means of recorded usage for accepted completions.

Task	$G_{P}(25)$	$G_{P}(80)$	$A_{P}$	$G_{S}(25)$	$G_{S}(80)$	$A_{S}$
Navigation	$+0.13$	$+0.30$	$-0.17$ $[-0.50,+0.15]$	$+0.21$	$-0.11$	$+0.32$ $[-0.12,+0.81]$
Rule Diagnosis	$+0.38$	$+0.27$	$+0.11$ $[-0.17,+0.39]$	$-0.16$	$-0.01$	$-0.14$ $[-0.38,+0.10]$

Appendix

Table 17: Cap amplification of each runtime gap on the control tasks, over the same

16

paired instances.

G_{P}(c){=}Q_{PP}(c)-Q_{PS}(c)

is the persistent-trained agent’s runtime gap and

G_{S}(c){=}Q_{SS}(c)-Q_{SP}(c)

is the stateless-trained agent’s. Amplification is the paired difference

G(25)-G(80)

, with a

95\%

bootstrap CI over task instances. Every interval spans zero.

Quantity	$P\!\to\!S$	$+$ Carry	$P\!\to\!P$	Carry (re-run)	$+$ Announced
Quality	0.038	0.132	0.575	0.081	0.080
NameError turns/ep	9.6	1.3	0.1	6.0	5.0
… of them on carried turns	6.2	0.0	0.1	0.0	0.0
Re-derives after a truncation	0.76	0.76	0.00	0.86	0.31
Distinct items	24.2	28.2	34.1	28.0	29.6
Decision achieved/OPT( $\mathcal{Q}$ )	0.08	0.23	0.72	0.11	0.11

Appendix

Table 18: Cap-boundary carryover evaluations, with trace-derived behavioral metrics (

n{=}25

per condition). The first three conditions are one batch and the last two another, so read quality down each block, not across them. “Carried” turns are those following a truncated one. In conditions without carryover those are the turns a checkpoint would have been carried into.

	$T_{\max}{=}40$	$T_{\max}{=}80$	$T_{\max}{=}160$
$P\!\to\!P$ (matched)	$0.66$	$0.59$	$0.66$
$P\!\to\!S$ (mismatch)	$0.076$	$0.058$	$0.088$
$P\!\to\!S$ overflow	$3/25$	$13/20$	$13/20$
$P\!\to\!P$ overflow	$4/25$	—	—

Appendix

Table 19: Episode-horizon sweep at

c{=}25

(Opaque Knapsack, normalized optimality), on the same common set of

20

instances, with the per-turn context behavior that accompanies it. The

80/160

configs raise both the turn limit

T_{\max}

and the episode wall-clock (

3{,}000\!\to\!4{,}000

s). “Overflow” counts episodes with at least one context-window overflow. The runtime hard-errors rather than silently truncating once a turn’s input plus the

8192

-token output reservation would exceed the

40960

window. Overflow denominators are each condition’s own full episode set, not the common set the quality rows use.

Cap	Condition	Attempted	Steps	Failed	Context
$c{=}25$	$P\!\to\!P$ (matched)	13.2	9.8	85	85/85
	$P\!\to\!S$ (mismatch)	37.3	35.2	52	52/52
	$S\!\to\!P$	27.8	18.6	229	227/229
	$S\!\to\!S$ (matched)	21.6	11.2	260	260/260
$c{=}80$	$P\!\to\!P$	4.6	4.6	0	0/0
	$P\!\to\!S$	14.5	13.0	37	37/37

Appendix

Table 20: Horizon accounting for the main evaluation (Opaque Knapsack, seed-0,

n{=}25

episodes per condition,

T_{\max}{=}40

). Attempted and Steps are per-episode means of model calls attempted and steps logged. Failed and Rejected are totals over the

25

episodes, the first counting calls that raised and the second counting completions rejected for not stopping on their own. Context is the share of the failed calls that were serving-window rejections. Termination counts are in Table 10 , and every episode that ended at the horizon satisfies attempted

=

steps

+

failed

+

rejected.

	Qwen3-8B	+2nd rollout	Mistral-7B-v0.3	Llama-3.1-8B
$G_{P}(25)$	+0.59 [+0.46,+0.71]	+0.63 [+0.50,+0.75]	+0.64 [+0.50,+0.78]	+0.58 [+0.43,+0.72]
$G_{P}(80)$	+0.15 [+0.00,+0.32]	+0.08 [-0.06,+0.21]	+0.14 [-0.01,+0.28]	+0.18 [+0.01,+0.35]
Cap amplification	+0.44 [+0.22,+0.65]	+0.55 [+0.38,+0.71]	+0.51 [+0.32,+0.71]	+0.40 [+0.16,+0.63]
$M(80)$	+0.11 [-0.08,+0.29]	-0.02 [-0.15,+0.09]	-0.18 [-0.32,-0.04]	-0.23 [-0.42,-0.04]
$T$	+0.31 [+0.04,+0.57]	+0.40 [+0.24,+0.55]	+0.41 [+0.09,+0.74]	+0.34 [+0.03,+0.65]

Appendix

Table 21: Replications.

G_{P}

is the persistent model’s runtime gap and

G_{P}(25)-G_{P}(80)

its cap amplification.

T

is the change in the train

\times

runtime interaction across the caps, and

M=Q_{PS}-Q_{SS}

is the secondary model-selection difference under stateless deployment. The runtime gap and its amplification are present within every replication.

M(80)

is the quantity whose sign the main text declines to claim: it is never positive with an interval that excludes zero, and is reliably negative in the two base-model settings.

Artifact	Released	Contents
Code	Yes	Task generators, both runtimes, agent harness, training and serving scripts, every condition’s benchmark config, analysis scripts, and the verbatim prompts
Evaluation traces	Yes	Per-task rollouts of every reported condition: main evaluation, dense sweep, horizon control, cross-task controls, both interventions, replications
Opaque Knapsack adapters	Yes	6 Qwen3-8B (persistent/stateless $\times$ 3 seeds) $+$ 4 replication adapters, one card each
Control-task adapters	Yes	The 4 Navigation and Rule Diagnosis adapters, same recipe and card format
Teacher traces	Yes	The filtered, instance-paired SFT data of App. B , as fed to the fine-tuning script. The raw cached teacher responses it was built from are not released

Appendix

Table 22: Released artifacts.

Runtime contracts and conditions
$P$ , $S$	The persistent runtime, under which the variables an agent creates stay available on later turns, and the stateless runtime, under which each turn begins with a fresh set of them. Task and environment state, including the tool objects, survives under both (Sec. 3 )
Runtime contract	The interpreter semantics alone: whether agent-created bindings survive the turn boundary (Sec. 3 )
Runtime condition	The contract together with the prompt package that announces it: a mode instruction, a runtime-state banner and runtime-specific few-shot examples. Cross-runtime comparisons switch the whole package, not the semantics in isolation (Sec. 3 , App. A.3 )
$X\!\to\!Y$	An $X$ -trained agent deployed on a $Y$ runtime. $P\!\to\!P$ and $S\!\to\!S$ are the matched references, $P\!\to\!S$ the main mismatch and $S\!\to\!P$ the reverse transfer (Table 1 )
Condition	One of the four $X\!\to\!Y$ combinations, evaluated at one cap (Table 1 )
Runtime transfer	Whether an agent trained under one runtime still solves the task when deployed under the other (Sec. 1 )

Appendix

Table 23: Notation and terminology: runtimes, conditions and evaluation settings.

Outcome and reported quantities
Normalized optimality	The outcome: achieved value over the exact optimum on allowed-class items, clamped to $0$ on any capacity overflow (Sec. 3 , App. A.2 )
$Q_{XY}(c)$	Mean task quality for an adapter trained under runtime $X$ and evaluated under runtime $Y$ at cap $c$ : normalized optimality for Opaque Knapsack, and each control family’s own score (Sec. 3 , App. E.1 )
$G_{P}(c)$	Runtime gap: $Q_{PP}(c)-Q_{PS}(c)$ (Sec. 3 )
$D(c)$	Train $\times$ runtime interaction: $\bigl(Q_{PP}(c)-Q_{PS}(c)\bigr)-\bigl(Q_{SP}(c)-Q_{SS}(c)\bigr)$ (Sec. 3 )
$T$	Cap moderation: $D(25)-D(80)$ (Sec. 3 )
Cap amplification	The same tightening applied to the runtime gap itself: $G_{P}(25)-G_{P}(80)$ (Sec. 3 , Fig. 3 )

Appendix

Table 24: Notation and terminology: the outcome, the reported quantities and the trace-derived ones.

Tool-Using Agents

Tool-using agents plan and act through external tools, APIs, software interfaces, and environments to complete tasks beyond language generation.

See all 286 papers

Agent Capsules: Quality-Gated Granularity Control for Multi-Agent LLM Pipelines
May 1, 2026Aninda RayLLM Inference Efficiency Multi-Agent LLM Systems
When Does Restricting a Coding Agent to execute_code Help? A Regime $\times$ Agent-Design Ablation
Jul 12, 2026Hong Yang, Qi Yu, Travis DesellCoding Agents Tool-Using Agents
Rochester Institute of Technology Rochester, New York, USA
StateM: Reaching 95.3% Raw Accuracy, or a $15 Frontier Run, on Terminal-Bench 2.1 via Harness Scaling
Aug 15, 2026Ziheng Qin, Yaxin Lu, Zhangyang Atlas Wang +1LLM Agent Harnesses Long-Horizon LLM Agents
Somewhere on the Earth

May 1, 2026J/K move · Enter open · S save

Agent Capsules: Quality-Gated Granularity Control for Multi-Agent LLM Pipelines

Aninda Ray

LLM Inference Efficiency Multi-Agent LLM Systems Multi-Agent System Optimization LLM Agent Orchestration LLM Agent Workflow Optimization LLM Agent Efficiency

Open paper AskPDF arXiv

A multi-agent pipeline with N agents typically issues N LLM calls per run. Merging agents into fewer calls (compound execution) promises token savings, but naively merged calls silently degrade quality through tool loss and prompt compression. We present Agent Capsules, an adaptive execution runtime that treats multi-agent pipeline execution as an optimization problem with empirical quality constraints. The runtime instruments coordination overhead per group, scores composition opportunity, selects among three compound execution strategies, and gates every mode switch on rolling-mean output quality. A controlled negative result confirms that injecting more context into a merged call worsens compression rather than relieving it, so the framework's escalation ladder (standard, then two-phase, then sequential) recovers quality by moving toward per-agent dispatch rather than by rewriting merged prompts. On LLM-judged quality, the controller matches a hand-tuned oracle on every measured (model, group, mode) cell: routing compound whenever the oracle would, and reverting to fine whenever quality would fail the floor, without per-model configuration. Against a hand-crafted LangGraph implementation of a 14-agent competitive intelligence pipeline, Agent Capsules uses 51% fewer fine-mode input tokens and 42% fewer compound-mode input tokens, at +0.020 and +0.017 quality respectively. Against a DSPy implementation of a 5-agent due diligence pipeline, the framework uses 19% fewer tokens than uncompiled DSPy at quality parity, and 68% fewer tokens than MIPROv2 at +0.052 quality. Even before compound mode fires, the runtime delivers efficiency through automatic policy resolution, cache-aligned prompts, and topology-aware context injection, matching both hand-tuned and compile-time baselines without training data or per-pipeline engineering.

Abstract

Tool-Using Agents

Explore similar work

Agent Capsules: Quality-Gated Granularity Control for Multi-Agent LLM Pipelines

When Does Restricting a Coding Agent to execute_code Help? A Regime ×\times× Agent-Design Ablation

StateM: Reaching 95.3% Raw Accuracy, or a $15 Frontier Run, on Terminal-Bench 2.1 via Harness Scaling

When Does Restricting a Coding Agent to execute_code Help? A Regime $\times$ Agent-Design Ablation