Where Does Exactly-Once Live? Model, Harness, and Tool-Contract Effects on Duplicate Side Effects in LLM Agents

Jiapeng Li

cs.LGSep 24, 2026

Where Does Exactly-Once Live? Model, Harness, and Tool-Contract Effects on Duplicate Side Effects in LLM Agents

Authors: Jiapeng Li

Organizations: Microsoft

Abstract

When a tool-using agent's write times out or returns a server error, the action may already have taken effect. Retrying blindly duplicates it -- a second charge, a second announcement, a second deployment -- while giving up skips required work. We ask where exactly-once behaviour should be enforced: in the model, in the agent harness, or in the tool contract. We introduce LIMBO, a deterministic sandbox of six services with realistic contracts (optional idempotency keys, eventually consistent and missing read paths) and twelve fault modes injected at the service boundary, including late commits, redelivery and partial batches; every episode is graded against a ledger of committed effects. Across 25,930 episodes spanning nine recent models, three production agent harnesses, two contract variants and fifteen recovery conditions, the answer depends on the fault. When an immediate read-back can reveal what happened, the model decides: frontier models instructed to act exactly once almost never duplicate a write whose acknowledgement was lost (0.5%), weaker models often do, and the model explains 53% of the explained variance. When it cannot -- the request is still in flight, or the transport delivered it twice -- the same frontier models duplicate in 56% and 74% of episodes, and the contract explains 81%. We prove that no verification-only policy is exactly-once under late commits without a bound on in-flight time. Waiting works when such a bound is short and known, but with heavy-tailed in-flight delays even an hour of waiting per episode falls short of offering an idempotency key on every write, which lowers the duplicate rate from 28% to 4% because agents use keys when they exist. The harness barely matters, a guard that attaches keys transfers across harnesses unchanged, and agents reported success in 90% of the episodes in which they had duplicated an effect.

Figures & tables

Panel of Figure 1 — Figure 1: Why verification is not enough. $W_{\textsf{absent}}$ and $W_{\textsf{late}}$ produce identical observations—a timeout and then an empty read-back—until the in-flight original commits, so verify-then-retry re-issues in both and duplicates in $W_{\textsf{late}}$ (Proposition 1 ). If the re-issue reuses the original’s idempotency key, the late original becomes a no-op in every world (Proposition 2 ).

Service	Writes (idempotency)	Read-back path (consistency)
social	publish (key on mastodon only)	list_posts : weibo lags 180 s; none for x
billing	create_charge (key), refund (naturally idempotent)	list_charges (strong)
tickets	create , add_comment (non-idempotent), update_status (conditional)	list_recent (strong), search (lags 120 s)
mail	send (non-idempotent, irreversible)	search_sent (lags 120 s)
data	insert , insert_many (non-atomic), upsert (idempotent)	query (strong)
deploy	trigger (non-idempotent)	list_runs , get_run (strong)

Table 1: Services and tool contracts. “Key” marks writes that accept an idempotency key in the native contract; lags are documented in the tool descriptions.

Mode	Agent observes	Hidden outcome	Exactly-once recovery
timeout_pre	timeout	not executed	verify, then re-issue
timeout_post	timeout	executed, response lost	verify, then skip
timeout_late	timeout	executed 90 s later (still in flight)	same key, or wait and escalate
timeout_late_tail	timeout	executed 40 s–2 h later (heavy tail)	same key
http500_pre	HTTP 500	not executed	verify, then re-issue
http500_post	HTTP 500	executed	verify, then skip

Table 2: Fault modes. Rows in the same group are observation-equivalent at the time of the response.

Figure 2: Duplicate side-effect rate (%) by model and fault mode (minimal scaffold, vanilla prompt, E1). Columns are grouped as in Table 2 ; rows are ordered roughly by capability. The partial-batch column includes the E1s supplement (ten instances of the batch template per model).

Model	Duplicate rate (%) under committed faults			TS (%) when	EOS (%)
	lost ack / 500 / partial	late commit	redelivery	not executed	all faults
gpt-6-astra	0 [0,9]	25 [13,42]	74 [57,86]	100 [91,100]	79 [66,89]
gpt-6-sol	0 [0,9]	46 [30,63]	74 [57,86]	99 [89,100]	77 [63,87]
gpt-5.6-sol	1 [0,10]	65 [48,79]	74 [57,86]	98 [88,100]	75 [61,85]
claude-opus-5.5	0 [0,9]	68 [51,82]	74 [57,86]	100 [91,100]	72 [58,83]
gemini-3.8-flash	14 [6,29]	59 [42,75]	74 [57,86]	100 [91,100]	69 [54,80]

Table 3: E1 results by model with 95% intervals (Wilson on Kish effective sample sizes). Duplicate rates are conditional on the fault having fired; TS (not executed) is task success in the _pre worlds; EOS is over all faulted episodes. gpt-4.1 covers only 87% of the design (separate low quota).

Factor	pooled (prereg.)	read-back resolves	read-back cannot
contract share (%)	36	30	81
mode share (%)	56	18	10
model share (%)	9	53	8
total pseudo- $R^{2}$	0.66	0.51	0.72
duplicate rate (%)	38	11	66
episodes	3,312	1,696	1,616

Table 4: Shapley decomposition of the explained variance in duplicates. “Pooled” is the preregistered analysis over committed faults and redelivery; the other columns split it into faults that an immediate read-back resolves (lost acknowledgement, misleading 500, partial batch) and faults it cannot resolve (late commit, redelivery). Top: E1 (minimal scaffold, eight models). Bottom: E3 (four harnesses, native condition).

Figure 3: Duplicate rate under committed faults (lost acknowledgement, misleading 500, late commit, partial batch) by tool-contract class and model (E1 and E1s, with 95% intervals).

Contract	Condition	timeout (committed)	500 (committed)	timeout (late commit)	partial batch	redelivery
native	vanilla	9	21	61	25	74
native	guard	0	0	68	0	74
keys-everywhere	vanilla	4	6	9	0	7
keys-everywhere	guard	0	0	7	0	0

Table 5: Duplicate rate (%) by contract variant and condition under committed faults and redelivery (E2, E2k; four models pooled).

	fixed delay (90 s)			heavy-tailed delay (40 s–2 h)
Condition	EOS (%)	dup. (%)	time (min)	EOS (%)	dup. (%)	time (min)	$P(\delta\leq\Delta)$
vanilla	39	61	2.9	34	66	2.8	–
wait 0 s	38	62	2.8	36	64	2.8	0
wait 60 s	43	57	3.2	42	58	2.9	12
wait 120 s	99	1	3.8	–	–	–	–
wait 300 s	99	1	6.2	67	33	6.1	50

Table 6: Waiting versus keys under late commits (E5; four models, all focal writes of instance 0). “time” is the mean agent-visible episode time in simulated minutes;

P(\delta\leq\Delta)

is the fraction of tail-delay worlds whose in-flight delay does not exceed the assumed bound. Fixed-delay rows for vanilla, guard, both oracles and the keys contract are the matched E2/E2k episodes.

Figure 4: Exactly-once success against mean agent-visible episode time for wait-then-verify policies with increasing assumed in-flight bounds, compared with keys (E5). Left: fixed 90 s delay. Right: heavy-tailed delay.

EOS (%)
Model	vanilla	aware	reflect	sdk-retry	rules	vbr	guard	state oracle	outcome oracle
gpt-6-sol	79	78	78	51	80	78	76	82	88
claude-opus-5.5	77	77	77	51	77	77	77	82	88
gemini-3.8-flash	74	77	71	51	74	73	76	81	88
mai-code-1.1-flash	59	67	58	48	59	62	76	74	83
Duplicate rate (%)

Table 7: Recovery conditions (E2). EOS, duplicate rate and TS are over faulted episodes; the bottom rows report mean cost per episode over all four models. The state oracle verifies against the current ground truth and cannot see requests still in flight; the outcome oracle can, and is the upper bound for any client-side policy.

Harness	Model	Duplicate rate (%), native			EOS (%) native	EOS (%) guard	n
		committed	late	redelivery
minimal	gpt-6-sol	0 [0,7]	38 [21,57]	75 [55,88]	77 [69,83]	72 [63,79]	266
minimal	gpt-5.6-sol	0 [0,7]	67 [47,82]	75 [55,88]	71 [62,78]	73 [64,80]	266
minimal	claude-opus-5.5	0 [0,7]	62 [43,79]	75 [55,88]	73 [64,80]	73 [64,80]	266
minimal	gemini-3.8-flash	12 [6,24]	58 [39,76]	75 [55,88]	69 [60,76]	71 [62,78]	266
copilot	gpt-5.6-sol	2 [0,11]	71 [51,85]	75 [55,88]	70 [62,78]	71 [62,78]	266

Table 8: Production harnesses (E3). Duplicate rates are for the native condition by fault family; EOS is over faulted episodes, natively and with the guard deployed in the MCP server.

	native contract		keys everywhere		EOS (%)
Harness	vanilla	guard	vanilla	guard	keys + guard
minimal	35	34	0	0	100
copilot	37	36	0	0	100
hermes	35	31	0	0	100
codex	36	37	0	0	100

Table 9: Duplicate rate (%) under committed faults and redelivery for gpt-5.6-sol in each harness, by contract variant and condition (E3 and E3k); the last column is exactly-once success with keys everywhere and the guard.

Fault family	metric	default (%)	plain (%)	$\Delta$ (pp)	McNemar $p$	pairs
not executed (TS)	TS	98	100	+1.5	0.25	204
read-back resolves	dup.	12	22	+9.7	1.5e-09	414
late commit	dup.	58	71	+12.7	2.6e-06	204
redelivery	dup.	74	75	+1.5	0.25	204
no fault (EOS)	EOS	100	100	+0.0	1	72

Table 10: Cue ablation (E6): the same episodes with the default instruction and with the closing “exactly once” sentence removed (six models, instance 0; exact McNemar tests on paired episodes).

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Tool	Kind	Description shown to the agent
social_publish	write	Publish a post to a social platform (mastodon, weibo, linkedin, x). Returns the new post_id. idempotency_key is honored by mastodon only; other platforms ignore it. On weibo, newly published posts can take up to 3 minutes to appear in social_list_posts.
social_list_posts	read	List the most recent posts on a platform (newest first). Not available for x. Weibo listings are eventually consistent (new posts may take up to 3 minutes to appear).
social_delete_post	write	Delete a post by id.
billing_create_charge	write	Charge a customer’s saved payment method. amount_cents is an integer number of cents. Returns the charge.
billing_list_charges	read	List a customer’s charges (newest first), including refunded ones.
billing_refund_charge	write	Fully refund a charge. Refunding an already-refunded charge returns 409.

Appendix

Table 11: Tools and their agent-facing descriptions (native contract).

	Prediction	Result	Verdict
H1	Non-idempotent writes with an eventual or missing read path are duplicated in $\geq$ 10% of committed-fault episodes, more often than keyable or strongly verifiable writes	mixed-effects OR 6.4 (5.2–7.9); GEE OR 2.9 (1.4–6.0), $p=0.007$	supported
H2	The contract explains more variance than the model; the harness explains $\geq$ 10%	pooled shares: contract 36%, model 9%, harness 0% (E3); stratified: model 53% on resolvable faults, contract 81% on unresolvable ones	contract $>$ model supported only pooled; harness not supported
H3	Given verification, eventual read paths yield more duplicates than strong ones	13.4% vs 0.8% ( $n=947$ ), GEE $p=0.005$	supported
H4	Blind re-issue does not differ between irreversible and reversible writes (TOST, $\pm$ 5 pp)	difference +3.4 pp, 90% interval -1.6 to +8.5 pp	inconclusive
H5	$\geq$ 20% of duplicate-producing episodes end “completed” with no uncertainty listed	80% ( $n=1,279$ ); overclaim 90% (78–96%)	supported
H6	The guard in the MCP server brings duplicates to $\leq$ 2% in every harness with $\leq$ 3 pp TS loss	native contract: 29% (Copilot CLI), 28% (Hermes), 29% (Codex CLI)	not supported

Appendix

Table 12: Preregistered hypotheses and outcomes. Intervals are 95% unless stated otherwise.

Guard vs.	EOS gain (pp)	paired episodes	Holm-adjusted $p$
vanilla	+4.0	820	0.00011
aware	+1.6	820	0.035
reflect	+5.4	820	1e-07
sdk-retry	+26.2	820	9.1e-59
rules	+3.9	820	0.00015
vbr	+3.8	820	5.9e-05

Appendix

Table 13: Paired McNemar tests of the guard against each other condition on exactly-once success (E2; pairs share model, task instance, focal write and fault mode; Holm-adjusted).

Figure 5: Exactly-once success against mean tool calls per episode for each recovery condition and model (E2); marker area is proportional to the duplicate rate.

Figure 6: Recovery behaviour after the focal timeout, by model (minimal scaffold, vanilla, E1). The not-executed and committed panels are observation-equivalent; their behaviour distributions are statistically indistinguishable for every model.

Explore similar work

Aug 3, 2026cs.AI

Beyond Single-Use Tokens: Durable Authorization State for Replay-Resistant LLM Agent Actions

Tool-using large language model agents frequently replan, retry failed operations, delegate tasks, and resume after crashes. These behaviors can cause one user authorization to be requested and executed multiple times under freshly issued token identifiers, even when each individual token is single-use. We call this failure semantic replay: exceeding the execution budget of a token-independent authorization instance rather than merely reusing an old token identifier. We show that identifier-local token consumption cannot prevent fresh reissuance unless the issuer retains monotonic durable state over the authorized action, confirmation event, and remaining execution budget. We introduce CapLease, an authorization-consumption layer that follows proposal- and authority-level defenses, binds an authenticated user confirmation to a canonical action, and enforces transactional Issue-Prepare-Commit transitions. Across LLM-agent replanning, retry, delegation, concurrency, confirmation-replay, and crash-recovery scenarios, identifier-local tokens permit fresh semantic reissuance, whereas CapLease and an equally stateful Server Ledger prevent duplicate admission and, with an idempotent sink, duplicate external effects. Our results identify durable authorization state, rather than token representation alone, as the systems requirement for replay-resistant agent execution.

Jinghan Xu, Longze Fan, Zeyuan Wang +2

1Nankai University, China · 2China University of Petroleum, China · 3Sun Yat-sen University, China +1

Deterministic Replay Commits

Ask PDF

Apr 23, 2026cs.CL

How Consistent Are LLM Agents? Measuring Behavioral Reproducibility in Multi-Step Tool-Calling Pipelines

Large language model (LLM) agents with tool-calling capabilities are increasingly deployed in production systems, yet a fundamental reliability question remains under-explored: does the same agent behave the same way twice? We present a systematic empirical study of behavioral consistency in multi-step tool-calling agents, measuring whether agents select the same tools, in the same order, with the same arguments, across repeated identical invocations. Unlike prior work on consistency in ReAct-style agents(search-only, free-text actions), we study the richer setting of structured tool-calling interfaces with typed parameters and consequential side effects.

Abel Yagubyan

Independent Researcher

Large Language Model Agents Reproducibility

Ask PDF

Jul 13, 2026cs.SE

AgentCheck: A Reproduce-Intervene-Mitigate Workbench for LLM Agents over MCP

Tool-using LLM agents are mostly evaluated assuming all tools work. When a tool times out, returns a week-stale value, or has its description poisoned in deployment, the developer needs a controlled way to reproduce the failure, test a fix, and confirm the fix worked before deployment. We present AgentCheck, an open-source web workbench that turns an MCP server into an intervention surface. AgentCheck runs an agent against its real tools and records every tool response, then re-runs the agent with the response perturbed by a fault (12 types) injector. Matching tool calls are replayed from cache, and later tool calls go live after the agent diverges. This yields a reproduce-intervene-confirm loop: the developer toggles a mitigation, re-runs against the identical fault, and sees if the failure goes away. Scoring has two parts: deterministic pass/fail rules, plus an LLM judge for interpretive labels, validated against human annotations. Across five agents, the best passes 105/120 scenarios and the weakest only 77. The failures are usually silent, confident use of incorrect tool outputs rather than crashes. On the weakest agent, a retry mitigation raises success on timeout error faults from as few as 30% of cases to 100%, whereas stale-data faults remain near 3-4 of 10 regardless of the mitigation. AgentCheck makes these failure modes reproducible, comparable, and verifiable before deployment.

Aritra Mazumder, Nusrat jahan Lia

University of Utah · University of Dhaka

Large Language Model Agents Retrying Model Context Protocol Open-Source Gradient Staleness

Ask PDF