False retry permission (FRP) occurs when an agent retries a timed-out tool call without support from the visible contract. RetryBench separates this error from realized recovery. Successful protected recovery does not imply reliable rejection of unsupported retries. In a 720-response key control, valid keys usually enabled safe progress, but offered changed keys elicited unsupported retries in Sol (1/80), Terra (55/80), and Luna (60/80). A later 240-response control removed the candidate and required models to supply a key. Luna still made 8/80 unsupported dispatches; Sol and Terra made none in 80 responses each. Accepting a displayed bad key therefore cannot explain all observed errors, although changed output requirements and conditions preclude a pure display-effect comparison. Removing the weak-contract label increased retries in Sol and Terra's coded-action conditions in a six-endpoint study. A separate 1,536-response canonical-action block also found a positive Terra contrast (115/128 versus 124/128 retries), concentrated in two task/order conditions; the other five contrasts remained unresolved after multiplicity correction. An explicit-contract cue control elicited deferral in all 762 returned responses, leaving recovery unresolved. These results support reporting permission and recovery together across declared interface variants. The evidence covers four purposive tasks, fixed GPT endpoints, and one recovery decision; it identifies neither internal mechanisms nor episode-level safety.
Figures & tables
Figure 1: The same negative read can accompany failure, pending work, or a completed but hidden effect; only a separate authoritative observation can rule out both an existing effect and a later commit in this probe. A valid idempotency key can instead make another dispatch safe by preventing a duplicate.
Question / collection
Design
Draws per task/cell
Returned/planned
Collection dates (UTC)
RQ1, earlier canonical subset
Six endpoints × label × four cyclic action orders
2
381/384
October 7, 2026
RQ1, later canonical block
Same prompt cells
8
1,536/1,536
October 10–11, 2026
RQ2, candidate control
Sol/Terra/Luna × candidate absent/displayed × original key omitted/present
5
236/240
October 11–12, 2026
Table 1: Main study collections
Endpoint
Label
Earlier: order 0 → others
Later: order 0 → others
Later difference (pp)
Nominal 95% interval (pp)
Holm24 p
Mini
labeled
7/8 → 22/24
30/32 → 91/96
+1.04
[-6.80, +15.25]
1
Mini
unlabeled
6/8 → 21/24
24/32 → 91/96
+19.79
[+6.41, +37.15]
0.00514
GPT-5.4
labeled
3/8 → 21/24
16/32 → 86/96
+39.58
[+21.49, +56.60]
2.41×10−7
GPT-5.4
unlabeled
4/8 → 21/24
19/32 → 89/96
+33.33
[+16.68, +50.85]
2.70×10−5
Sol
labeled
0/8 → 0/24 +1 missing
0/32 → 0/96
+0.00
[-10.72, +3.85]
1
Sol
unlabeled
0/8 → 2/24
0/32 → 3/96
+3.12
[-7.79, +8.79]
1
Table 2: Order-zero contrasts across both collections and labels
Figure 2: We show later retry counts, each out of eight API responses. L/U mean labeled/unlabeled. Orders 0–3 start with retry, stop, wait and unresolved. The figure shows each order separately, while the table averages the three rotations. Astra uses low ; other endpoints use none .
Endpoint
Original key
Candidate
Errors
Protected
Unresolved
Missing
Sol
present
absent
0
20
0
0
Sol
present
displayed
0
19
0
1
Sol
omitted
absent
0
0
19
1
Sol
omitted
displayed
0
0
20
0
Terra
present
absent
0
20
0
0
Terra
present
displayed
0
20
0
0
Table 3: Decisions under the common-schema candidate control
Appendix figures & tables31 assets
Supplementary material from the paper’s appendix.
Appendix
Work
Primary object
Relevant measurement boundary
Did It Happen? ( Sun, 2026 )
Shared-response recovery evaluation
Authoritative status and stable-key recovery in a pinned two-world simulator
Verified Tool Calls ( Mansoor et al., 2026 )
Verification-aware wrapper
Delayed visibility, task success, and duplicates under injected failures
OpenOnce ( MemoV, 2026 )
Durable-effect runtime
Capability contracts, receipts, and an unresolved-attempt lifecycle
IdempotencyBench ( Swamy, 2026 )
Duplicate-effect evaluation
Semantic duplicate effects and success across retry substrates
Limbo ( Li, 2026 )
Multi-turn recovery evaluation
Ledger duplicates and exactly-once success in late-commit and stale-read episodes
UndoBench ( Sah et al., 2026 )
Paired nominal and faulted workflows
Recovery conditional on nominal competence; unsafe retry scored through duplicate effects in the frozen lost-acknowledgement regime
Appendix
Table 4: Source-scoped comparison with related work
Identifier
Evidence
Artifact
[R13]
Matched response-schedule framework audit
runs/framework-matched-v2/analysis.json
[R18]
Label, semantics, and original-key controls
runs/label-protection-v1/report.json
[R20]
Historical label/action counterbalance and missing bounds
runs/historical-counterbalance-v1/report.json
[R23]
Difference intervals, stratified conditional tests and interactions
runs/contrast-inference-v1/report.json
[R24]
Explicit-contract conflicting-label control
runs/label-cue-v1/report.json
[R25]
Rank stability, equal-variant summaries, and selected rationales
runs/interface-stability-v1/report.json
Appendix
Table 5: Numerical provenance in the reproduction artifact
Label
Lookup semantics
Key condition
Evidence contract line
Lookup contract line
Original key
Candidate key
labeled
stated
changed
Keep
Keep
key-A
key-B
labeled
stated
none
Keep
Keep
none
key-B
labeled
stated
valid
Keep
Keep
key-A
key-A
labeled
unstated
changed
Keep
Omit
key-A
key-B
labeled
unstated
none
Keep
Omit
none
key-B
labeled
unstated
valid
Keep
Omit
key-A
key-A
Appendix
Table 6: Exact line-level changes to the Appendix C base prompt
Order
Prompt action line
JSON action enum
0
Choose retry, stop, wait, or unresolved.
["retry", "stop", "wait", "unresolved"]
1
Choose stop, wait, unresolved, or retry.
["stop", "wait", "unresolved", "retry"]
2
Choose wait, unresolved, retry, or stop.
["wait", "unresolved", "retry", "stop"]
3
Choose unresolved, retry, stop, or wait.
["unresolved", "retry", "stop", "wait"]
Appendix
Table 7: Exact joint prompt and schema orders in RQ1
Endpoint
Absent label
weak
unverified
authoritative
Sol
0/64
0/64
0/64 +1 missing
0/64 +1 missing
Terra
0/64
0/64 +2 missing
0/64
0/64
Luna
0/64
0/64
0/64 +2 missing
0/64
Appendix
Table 8: Unsupported retries in the explicit-contract cue control
Endpoint
Valid same-key retry
Benchmark-inadmissible: original omitted key
Benchmark-inadmissible: changed key
Sol
80/80 [95.49, 100.00]
0/80 [0.00, 4.51]
1/80 [0.03, 6.77]
Terra
80/80 [95.49, 100.00]
0/80 [0.00, 4.51]
55/80 [57.41, 78.65]
Luna
78/80 [91.26, 99.70]
49/80 [49.70, 71.94]
60/80 [64.06, 84.01]
Appendix
Table 9: Valid protection and unsupported retry decisions
Endpoint
Label
Lookup semantics
Valid same-key
Omitted original key
Changed key
Sol
labeled
unstated
20/20
0/20
0/20
Sol
labeled
stated
20/20
0/20
0/20
Sol
unlabeled
unstated
20/20
0/20
1/20
Sol
unlabeled
stated
20/20
0/20
0/20
Terra
labeled
unstated
20/20
0/20
14/20
Terra
labeled
stated
20/20
0/20
11/20
Appendix
Table 10: Key-control decisions by label and lookup semantics
Endpoint
Key condition
CI
Issue
Release
Review
Sol
Valid same-key
20/20
20/20
20/20
20/20
Sol
Omitted original key
0/20
0/20
0/20
0/20
Sol
Changed key
0/20
0/20
0/20
1/20
Terra
Valid same-key
20/20
20/20
20/20
20/20
Terra
Omitted original key
0/20
0/20
0/20
0/20
Terra
Changed key
16/20
14/20
5/20
20/20
Appendix
Table 11: Key-control decisions by task
Endpoint
Key condition
Outcomes
FRP
Duplicates
Omissions
Unresolved
Ledger correct
Resolved exact
Sol
none
240
0
0
80
240
160
0
Sol
valid
240
0
0
0
0
240
240
Sol
changed
240
3
2
79
237
159
1
Terra
none
240
0
0
80
240
160
0
Terra
valid
240
0
0
0
0
240
240
Terra
changed
240
165
110
25
75
105
55
Appendix
Table 12: Realized outcomes in the key-control study
Figure 3: Earlier offered-key actions, pooling four label/semantics conditions (80 decisions per row). Only the matching original-key condition provides replay protection. Every prompt displays a candidate.
Key extension
CI
Issue
Release
Review
Sol, present
0/10
0/10
0/10
0/10
Sol, omitted
0/10
0/10
0/10
0/10
Terra, present
0/10
0/10
0/10
0/10
Terra, omitted
0/10
0/10
0/10
0/10
Luna, present
0/10
0/10
1/10
0/10
Luna, omitted
1/10
2/10
2/10
2/10
Appendix
Table 13: Unsupported supplied-key decisions by task
Endpoint
Original key
Replay outcomes
FRP
Duplicates
Omissions
Unresolved
Ledger correct
Resolved exact
Sol
present
120
0
0
0
0
120
120
Sol
omitted
120
0
0
40
120
80
0
Terra
present
120
0
0
0
0
120
120
Terra
omitted
120
0
0
40
120
80
0
Luna
present
120
3
2
3
9
115
109
Luna
omitted
120
21
14
33
99
73
7
Appendix
Table 14: Realized outcomes for model-supplied keys
Endpoint
Later labeled retries
Later unlabeled retries
Difference (pp)
95% Newcombe (pp)
Holm p
Mini
121/128
115/128
-4.69
[-11.71, +2.10]
0.569
GPT-5.4
102/128
108/128
+4.69
[-4.78, +14.09]
0.569
Sol
0/128
3/128
+2.34
[-0.95, +6.66]
0.569
Terra
115/128
124/128
+7.03
[+0.83, +13.75]
0.0145
Luna
102/128
89/128
-10.16
[-20.55, +0.52]
0.105
Astra
0/128
0/128
+0.00
[-2.91, +2.91]
1
Appendix
Table 15: Later canonical label contrasts
Endpoint
Representation
Labeled retry count
Unlabeled retry count
Unlabeled − labeled
GPT-5.4
canonical
24/32 [56.6, 88.5]
25/32 [60.0, 90.7]
+0.031
GPT-5.4
coded
71/128 +2 missing [46.4, 65.7]
83/128 [55.9, 73.1]
[+0.078, +0.094]
Mini
canonical
29/32 [75.0, 98.0]
27/32 [67.2, 94.7]
-0.062
Mini
coded
103/128 +1 missing [72.5, 87.6]
107/128 +1 missing [76.0, 90.2]
[+0.023, +0.039]
Luna
canonical
25/32 [60.0, 90.7]
20/32 +1 missing [43.7, 81.4]
[-0.156, -0.125]
Luna
coded
63/128 [40.3, 58.2]
63/128 +1 missing [40.3, 59.0]
[+0.000, +0.008]
Appendix
Table 16: Historical label contrasts with planned denominators
Figure 4: Twelve earlier endpoint/representation contrasts. Thin intervals are nominal Newcombe reference envelopes; thick marks show missing-action bounds. Replayed worlds are not independent model responses. Astra uses low . Sol coded and Terra coded have worst-case twelve-way Holm p approximately 1.09×10−7 and .000196, with Newcombe envelopes +13.21 to +28.95 and +9.28 to +30.93 points. The other ten intervals include zero and do not pass correction.
Endpoint
Representation
CI: labeled → unlabeled
Issue: labeled → unlabeled
Release: labeled → unlabeled
Review: labeled → unlabeled
GPT-5.4
canonical
4/8 → 5/8
8/8 → 8/8
7/8 → 8/8
5/8 → 4/8
GPT-5.4
coded
6/32 +1 missing → 18/32
27/32 +1 missing → 30/32
25/32 → 28/32
13/32 → 7/32
Mini
canonical
7/8 → 8/8
8/8 → 8/8
8/8 → 8/8
6/8 → 3/8
Mini
coded
27/32 → 29/32 +1 missing
29/32 → 31/32
28/32 +1 missing → 32/32
19/32 → 15/32
Luna
canonical
7/8 → 4/8
8/8 → 6/8 +1 missing
8/8 → 8/8
2/8 → 2/8
Luna
coded
4/32 → 6/32
29/32 → 27/32
28/32 → 30/32 +1 missing
2/32 → 0/32
Appendix
Table 17: Historical label contrasts by task
Endpoint
Interaction bounds (pp)
95% MOVER envelope (pp)
Reference p range
Holm p, worst case
GPT-5.4
[4.69, 6.25]
[-18.81, 29.86]
0.608–0.701
1
Mini
[8.59, 10.16]
[-11.03, 29.78]
0.293–0.372
1
Luna
[12.50, 16.41]
[-12.51, 40.08]
0.207–0.331
1
Sol
[14.06, 17.97]
[-1.54, 33.51]
0.0292–0.0708
0.425
Terra
[10.16, 11.72]
[-7.88, 27.51]
0.176–0.231
1
Astra ( low )
[-5.47, 0.78]
[-18.81, 12.06]
0.208–1
1
Appendix
Table 18: Representation interactions and uncertainty
Later endpoint
CI: labeled → unlabeled
Issue: labeled → unlabeled
Release: labeled → unlabeled
Review: labeled → unlabeled
Mini
31/32 → 32/32
32/32 → 32/32
32/32 → 32/32
26/32 → 19/32
GPT-5.4
16/32 → 26/32
32/32 → 32/32
32/32 → 32/32
22/32 → 18/32
Sol
0/32 → 0/32
0/32 → 1/32
0/32 → 2/32
0/32 → 0/32
Terra
27/32 → 31/32
32/32 → 32/32
32/32 → 32/32
24/32 → 29/32
Luna
22/32 → 22/32
32/32 → 30/32
31/32 → 32/32
17/32 → 5/32
Astra
0/32 → 0/32
0/32 → 0/32
0/32 → 0/32
0/32 → 0/32
Appendix
Table 19: Later canonical retry decisions by task
Endpoint ( none )
Minimum across variants (%)
Maximum (%)
Equal-variant mean (%)
Mini
[80.47, 81.25]
90.62
[84.77, 85.16]
GPT-5.4
[55.47, 57.03]
78.12
[68.36, 68.75]
Sol
[0.00, 0.78]
21.09
[6.84, 7.81]
Terra
[65.62, 67.19]
100.00
[85.74, 86.13]
Luna
49.22
78.12
[59.77, 60.74]
Appendix
Table 20: Ranges and equal-variant summaries
Reference policy
Correct ledger
Resolved exact
Correct, unresolved
Incorrect, unresolved
Incorrect, unflagged
Always retry
12/48
12/48
0/48
0/48
36/48
Always stop
36/48
36/48
0/48
0/48
12/48
Single status
24/48
24/48
0/48
0/48
24/48
Tri-state backoff
40/48
28/48
12/48
8/48
0/48
Stable idempotency
48/48
48/48
0/48
0/48
0/48
Appendix
Table 21: Deterministic baseline ledger correctness and resolution, separated
Retry response
Policy
Framework completed
Correct ledger
Duplicates
Unresolved
Returned
No retry
0/16
12/16
0
16
Returned
Exception retry
16/16
4/16
12
0
Returned
Stable idempotency
16/16
16/16
0
0
Lost
No retry
0/16
12/16
0
16
Lost
Exception retry
0/16
4/16
12
16
Lost
Stable idempotency
0/16
16/16
0
16
Appendix
Table 22: Matched framework conformance controls
Block / endpoint / label
Order 0
Other three orders
Difference bounds (pp)
Nominal 95% Newcombe envelope (pp)
Exact conditional p range
Worst-case Holm24 p
Earlier/GPT-5.4/labeled
3/8
21/24
[50.00, 50.00]
[13.10, 75.17]
[0.0115, 0.0115]
0.207
Earlier/GPT-5.4/unlabeled
4/8
21/24
[37.50, 37.50]
[3.54, 67.12]
[0.0459, 0.0459]
0.735
Earlier/Mini/labeled
7/8
22/24
[4.17, 4.17]
[-16.13, 39.28]
[1, 1]
1
Earlier/Mini/unlabeled
6/8
21/24
[12.50, 12.50]
[-13.21, 47.53]
[0.714, 0.714]
1
Earlier/Luna/labeled
6/8
19/24
[4.17, 4.17]
[-22.37, 40.16]
[1, 1]
1
Earlier/Luna/unlabeled
7/8 +1 missing
13/24
[-45.83, -33.33]
[-64.93, 5.63]
[0.0115, 0.135]
1
Appendix
Table 23: Expanded order contrasts with the 24-test correction. Earlier = historical-counterbalance-v1; Later = tier-a-v1
A durable agent can resume a consequential action while its authority changes and the action's external outcome remains uncertain. We study execution-time reliability of tool-using AI agents, isolating recovery decisions from language-model behavior. Checkpoint state, permission to dispatch, evidence of an external effect and permission to reconcile are distinct obligations. We present a finite conformance study that checks these obligations through histories scored against a provider effect ledger hidden from the recovering application. A prospectively frozen campaign executes 336 histories on native LangGraph and Temporal hosts, crossing 14 schedules, three provider contracts and four matched application policies. All 168 paired host cells have identical checked outcomes. Per host, retained-ledger reconciliation passes all 42 histories: 21 checked completions, 15 denials and six unknown outcomes with reservations retained. Cached retry, revalidation before retry and stable-key retry pass 30, 36 and 31 histories respectively. A separate 16-case component study of InfinityOS, an enterprise AI platform developed by Arya.ai, exercises exported authority, worker, recovery and settlement code over repository fakes. The results isolate the roles of current authority, provider contracts and retained observations in recovery, with safety, completion and unresolved obligations reported separately for the declared application profile.
Tool-using LLM agents are typically developed and post-trained in controlled environments with reliable tools, but deployment exposes them to tool failures, API and schema drift, and third-party services outside the developer's control. We introduce Bench2Robust, a framework for scenario-controlled solvability that augments existing benchmarks with episodes requiring retrying the original path, switching to a fallback-equivalent tool, or recognizing that no viable path remains. By controlling recoverability, Bench2Robust separates recovery-policy quality from randomness in whether failures persist or resolve. Across 7 models from 4 families and 10 task/domain slices spanning customer-service and function-calling benchmarks, 69 of 70 model-subset pairs degrade under tool failures, by up to 46.7 percentage points. We propose Bayesian Tool Memory (BTM) for structured runtime recovery context and use reinforcement learning (RL) for learned recovery behavior. On held-out Retail tasks using the benchmark's original tool set, BTM improves robustness by 16.8 percentage points without retraining, while RL retains a 6.3-point gain without inference-time BTM. Combining RL with BTM yields a 20.7-point improvement over the base model while preserving clean performance. These results show that robust tool use benefits from combining structured runtime recovery context with learned recovery behavior.
When a tool call times out, the agent sees the failure and can route around it. A cached error page or negative price can instead arrive in the expected format and be consumed as fact. We introduce Outcome Monitors, which detect violations of outcome contracts mined from task-disjoint traces or derived from public schemas. On a violation, the monitor preserves the result and issues a nonbinding receipt naming the violated property and public recovery tools. In frozen, prespecified evaluations with injected failures, Outcome Monitors raise ToolMaze completion from 10.9% to 28.1% across four models in two provider families and replicate in a third. In tau-bench retail, completion improves by 14.0 and 12.0 points on two tiers. In separate ToolMaze controls, removing the recovery-tool list eliminates the measured gain and restoring it recovers the effect; diagnostic detail and timing produce no detectable differences. Gains concentrate where the fault blocks completion. On a suite transcribed from a published incident taxonomy, detection outside the mined vocabulary falls to 46%, though delivery continues and completion is unchanged. Recovery tools are the active receipt content in these controls; extending detection beyond the contract vocabulary remains open.