AgentFaultBench: State-Verified Evaluation of Tool-Fault Recovery
Abstract
Does recovery advice help an agent finish a task after a tool fails? AgentFaultBench evaluates LangGraph and CrewAI agents performing specified tasks in an isolated GitLab installation. We inject tool errors and check both the resulting files and issues and the required reads, verification and final response. The prompt supplies the expected issue and report text, so the task tests execution rather than content generation. In the primary comparison of 1,944 planned runs, advice changes success by -8.33, 0.00 and +1.06 percentage points relative to a neutral reminder; missing outcomes leave the latter two comparisons inconclusive. Later experiments test a revised recovery policy. GPT-4o mini improves, while Sol and Claude already complete the recoverable cases. In a separate 624-run study on six related tasks not previously used by that policy, the guided condition records 161 correct final states and one unknown outcome in 192 planned runs, versus 142/192 with neutral feedback. The gain lies between +9.90 and +10.42 pp when missing outcomes are included. Removing issue IDs gives 47/48 successes versus 46/48 with IDs; other execution information remains. A second checker agrees on 1,879 observed primary outcomes and 241 constructed test cases. No human validation was collected. The findings support testing advice against explicit recovery controls rather than assuming it helps.
Figures & tables
| Condition | Injected failure |
|---|---|
| FM0 | Clean control. |
| FM13 | One-second 429 cooldown targeting issue creation. |
| FM19 | Five-second 429 cooldown on eligible mutations. |
| FM18 | Forged verification without genuine execution. |
| FM20 | First three eligible mutations rejected 503 globally, before execution. |
| FM21 | Permission revoked after a commit, blocking full completion. |
| Model | Policy | Full success | Clean full | State completion | After 429 + feedback |
|---|---|---|---|---|---|
| GPT-4o mini | N | 142/216 (65.7%) | 36/36 | 144/216 | 45/47 |
| GPT-4o mini | G | 124/216 (57.4%) | 36/36 | 124/216 | 25/45 |
| GPT-4.1 mini | N | 104/212 (49.1%) | 27/36 | 145/212 | 9/28 |
| GPT-4.1 mini | G | 101/210 (48.1%) | 23/35 | 140/210 | 11/30 |
| GPT-4.1 | N | 67/200 (33.5%) | 17/32 | 134/202 | 13/48 † |
| GPT-4.1 | G | 73/198 (36.9%) | 18/34 | 132/198 | 13/44 |
| Condition | 4o mini | 4.1 mini | 4.1 |
|---|---|---|---|
| FM0 | 0/0/36 | 0/3/32 | 0/0/32 |
| FM13 | 1/9/26 | 1/3/28 | 0/0/34 |
| FM18 | 0/0/36 | 4/0/32 | 0/0/29 |
| FM19 | 1/11/24 | 3/1/32 | 1/1/26 |
| FM20 | 0/0/36 | 0/1/35 | 2/0/31 |
| FM21 | 0/0/36 | 0/0/34 | 0/0/32 |
| GPT-4o mini | GPT-4.1 mini | GPT-4.1 | ||||||||||
| Condition | B | R | N | G | B | R | N | G | B | R | N | G |
| FM0 | ||||||||||||
| FM13 | ||||||||||||
| FM18 | ||||||||||||
| FM19 | ||||||||||||
| FM20 | ||||||||||||
| Model | Endpoint | |||
|---|---|---|---|---|
| GPT-4o mini | State | 67 | 72 | 82 |
| Full | 63 | 66 | 82 | |
| GPT-6.1 Sol | State | 83 u | 83 u | 84 |
| Full | 83 u | 83 u | 84 | |
| Claude Opus 5.5 | State | 84 | 84 | 84 |
| Full | 84 | 84 | 84 |
Appendix figures & tables51 assets
Supplementary material from the paper’s appendix.
Appendix
| Condition | Intervention |
|---|---|
| FM0 | Clean baseline |
| FM1–FM3 | Unavailable tool, bad call schema, malformed result |
| FM4–FM6 | Untrusted instructions, distracting context, stale memory |
| FM7–FM8 | Delegation loop and controlled timeout |
| FM9 | Suppressed verification with an apparent success receipt |
| FM10 | Attempted permission expansion |
| Development arm | Success/ known | Unsafe/ known | Unknown |
|---|---|---|---|
| Raw | 70/108 | 26/108 | 0 |
| Legacy oversight | 49/107 | 18/107 | 1 |
| Safety | 66/108 | 0/108 | 0 |
| Recovery | 89/108 | 0/108 | 0 |
| Condition | ||||||
|---|---|---|---|---|---|---|
| FM0 | 24/24 | 24/24 | 23/24 | 0/24 | 0/24 | 0/24 |
| FM11 | 24/24 | 11/24 | 11/24 | 0/24 | 0/24 | 0/24 |
| FM12 | 18/24 | 23/24 | 24/24 | 6/24 | 0/24 | 0/24 |
| FM13 | 19/23 | 12/24 | 23/24 | 0/23 | 0/24 | 0/24 |
| FM14 | 24/24 | 12/24 | 14/24 | 0/24 | 0/24 | 0/24 |
| FM15 | 0/24 | 0/24 | 0/24 | 0/24 | 0/24 | 0/24 |
| Condition | ||||||
|---|---|---|---|---|---|---|
| FM0 | 24/24 | 18/24 | 19/24 | 0/24 | 0/24 | 0/24 |
| FM11 | 24/24 | 15/24 | 13/24 | 0/24 | 0/24 | 2/24 |
| FM12 | 23/24 | 18/24 | 18/24 | 1/24 | 1/24 | 0/24 |
| FM13 | 23/24 | 12/24 | 19/24 | 0/24 | 0/24 | 0/24 |
| FM14 | 23/24 | 14/24 | 13/24 | 1/24 | 0/24 | 0/24 |
| FM15 | 0/24 | 0/24 | 0/24 | 0/24 | 0/24 | 0/24 |
| Condition | ||||||
|---|---|---|---|---|---|---|
| FM0 | 20/24 | 20/23 | 20/24 | 2/24 | 0/23 | 0/24 |
| FM11 | 17/24 | 11/24 | 15/24 | 6/24 | 1/24 | 1/24 |
| FM12 | 19/24 | 20/24 | 21/24 | 4/24 | 0/24 | 0/24 |
| FM13 | 22/24 | 20/24 | 20/24 | 1/24 | 0/24 | 0/24 |
| FM14 | 18/24 | 17/24 | 16/24 | 5/24 | 1/24 | 0/24 |
| FM15 | 0/24 | 0/24 | 0/24 | 0/24 | 0/24 | 0/24 |
| Configuration | ||||
|---|---|---|---|---|
| GPT-4o mini | 11/24 | 16/24 | 24/24 | 23/24 |
| GPT-4.1 mini | 12/24 | 15/24 | 15/24 | 19/24 |
| GPT-4.1 | 17/24 | 17/23 | 22/24 | 20/24 |
| Guided minus neutral: descriptive subset | GPT-4o mini | GPT-4.1 mini | GPT-4.1 |
|---|---|---|---|
| FM0 | -8.33 (24) | -4.35 (23) | +0.00 (24) |
| FM13 | -4.17 (24) | +0.00 (23) | +4.17 (24) |
| FM18 | +4.17 (24) | +0.00 (24) | -4.17 (24) |
| FM19 | -8.33 (24) | -12.50 (24) | +0.00 (24) |
| FM20 | +0.00 (24) | +0.00 (24) | +0.00 (24) |
| FM21 | +0.00 (24) | +0.00 (24) | +0.00 (24) |
| Unsafe-positive sensitivity: original revised | B | R | N | G |
|---|---|---|---|---|
| GPT-4o mini | 0/144 0/144 | 0/144 0/144 | 0/144 0/144 | 0/144 0/144 |
| GPT-4.1 mini | 0/144 0/144 | 1/143 0/143 | 0/143 0/143 | 2/143 0/143 |
| GPT-4.1 | 1/144 0/144 | 0/144 0/144 | 1/144 0/144 | 0/144 0/144 |
| Configuration | Raw | Safety | Recovery |
|---|---|---|---|
| GPT-4o mini | 139/214 | 129/216 | 164/216 |
| GPT-4.1 mini | 144/216 | 127/216 | 132/216 |
| GPT-4.1 | 118/216 | 135/215 | 142/212 |
| Configuration | B | R | N | G |
|---|---|---|---|---|
| GPT-4o mini | 44/144 | 72/144 | 93/144 | 89/144 |
| GPT-4.1 mini | 65/144 | 73/143 | 73/143 | 69/143 |
| GPT-4.1 | 63/144 | 69/144 | 74/144 | 74/144 |
| Condition | O | N | G |
|---|---|---|---|
| FM0 | 36/36 | 36/36 | 36/36 |
| FM13 | 28/36 | 35/36 | 27/36 |
| FM18 | 35/35 | 36/36 | 36/36 |
| FM19 | 31/36 | 35/36 | 25/36 |
| FM20 | 0/36 | 0/36 | 0/36 |
| FM21 | 0/36 | 0/36 | 0/36 |
| Condition | O: I/H/T | N: I/H/T | G: I/H/T |
|---|---|---|---|
| FM0 | 0/0/36 | 0/0/36 | 0/0/36 |
| FM13 | 36/36/36 | 36/36/36 | 36/36/36 |
| FM18 | 35/0/36 | 36/0/36 | 36/0/36 |
| FM19 | 36/36/36 | 36/36/36 | 36/36/36 |
| FM20 | 36/36/36 | 36/36/36 | 36/36/36 |
| FM21 | 36/36/36 | 36/36/36 | 36/36/36 |
| Condition | O | N | G |
|---|---|---|---|
| FM0 | 23/35 | 27/36 | 23/35 |
| FM13 | 24/34 | 25/34 | 21/32 |
| FM18 | 26/36 | 25/36 | 29/36 |
| FM19 | 20/36 | 26/36 | 28/36 |
| FM20 | 0/36 | 1/36 | 0/36 |
| FM21 | 0/34 | 0/34 | 0/35 |
| Condition | O: I/H/T | N: I/H/T | G: I/H/T |
|---|---|---|---|
| FM0 | 0/0/36 | 0/0/36 | 0/0/36 |
| FM13 | 35/35/36 | 35/34/36 | 34/34/36 |
| FM18 | 36/0/36 | 36/0/36 | 36/0/36 |
| FM19 | 36/36/36 | 36/36/36 | 36/36/36 |
| FM20 | 36/36/36 | 36/36/36 | 36/36/36 |
| FM21 | 35/35/36 | 35/34/36 | 35/35/36 |
| Condition | O | N | G |
|---|---|---|---|
| FM0 | 18/34 | 17/32 | 18/34 |
| FM13 | 16/33 | 17/34 | 18/36 |
| FM18 | 18/35 | 17/33 | 18/30 |
| FM19 | 14/32 | 16/34 | 17/30 |
| FM20 | 1/34 | 0/34 | 2/35 |
| FM21 | 0/33 | 0/33 | 0/33 |
| Condition | O: I/H/T | N: I/H/T | G: I/H/T |
|---|---|---|---|
| FM0 | 0/0/36 | 0/0/36 | 0/0/36 |
| FM13 | 35/34/36 | 36/36/36 | 36/36/36 |
| FM18 | 36/0/36 | 34/0/36 | 34/0/36 |
| FM19 | 32/32/36 | 36/36/36 | 35/35/36 |
| FM20 | 34/34/36 | 35/35/36 | 35/35/36 |
| FM21 | 34/33/36 | 33/33/36 | 33/33/36 |
| Model | N safe/known | G safe/known | G N pp | Task interval | Assigned bounds | Complete pairs |
|---|---|---|---|---|---|---|
| DeepSeek | 69/82 | 68/85 | 81/108 | |||
| Qwen | 32/63 | 30/55 | 39/108 |
| Diagnostic | Arm | Fault safe completion | Returned fault messages | Clean safe completion |
|---|---|---|---|---|
| Rate-limit | Old late guided | 2/2 | 0/2 | 2/2 |
| Earlier neutral | 2/2 | 2/2 | 2/2 | |
| Earlier guided | 2/2 | 2/2 | 2/2 | |
| Harder server-error | Old late guided | 1/2 | 0/2 | 2/2 |
| Earlier neutral | 0/2 | 2/2 | 2/2 | |
| Earlier guided | 1/2 | 2/2 | 2/2 |
| DeepSeek FM20 comparator | Strict successes/18 | Candidate wins | Losses | Ties |
|---|---|---|---|---|
| Simple recurring rule | 8 | 0 | 2 | 16 |
| Recurring neutral | 7 | 1 | 2 | 15 |
| Without advisory memory | 6 | 1 | 1 | 16 |
| Without preferred order | 7 | 0 | 1 | 17 |
| One-shot guidance | 7 | 1 | 2 | 15 |
| One grounded automatic retry | 7 | 2 | 3 | 13 |
| Model | Clean strict/known | FM20 strict/known | FM20 material/known | Known fault pairs/4 | Charge USD |
|---|---|---|---|---|---|
| GPT | 12/12 | 0/12 | 0/12 | 4/4 | 0.50 |
| Claude | 12/12 | 0/12 | 0/12 | 4/4 | 2.00 |
| Gemini | 5/11 | 0/11 | 1/11 | 3/4 | 2.93 |
| Model | Assigned | Known | Unknown |
|---|---|---|---|
| GPT-4o mini | 648 | 647 | 1 |
| GPT-4.1 mini | 648 | 633 | 15 |
| GPT-4.1 | 648 | 599 | 49 |
| Total | 1,944 | 1,879 | 65 |
| Mode | Required route | Contract/path evidence and limit |
|---|---|---|
| FM0 | Genuine artifacts and current proof | 18 original-incident scripted native clean passes, plus eight separate D-pack clean passes; not recovery. |
| FM13/19 | Retry targeted operations after cooldown | Six scripted routes per condition complete. FM13 targets issue creation; FM19 targets eligible mutations more broadly. Untargeted report work can proceed during FM13 cooldown. Dose amendment and limits: Appendix Q ; not a duration-only ablation. |
| FM18 | Genuine verification after forged receipt | One-shot mode permits subsequent genuine verification; permitted route does not validate the grader. |
| FM20 | Genuine effects after three 503 rejections | 18 fresh incident scripted faulted routes complete within 3 attempts per exact operation; 18 clean controls also pass (Appendix Q ). An original model success is separate (Appendix O ). D-v1 intended dependency route needs a blocked fourth attempt. Separate four-attempt D-v2: eight privileged scripted native passes; not LLM efficacy or a claim about all FM20 tasks. |
| FM21 | Two required resources despite later 403 | Valid acknowledged commit arms persistent denial of remaining writes: full completion blocked on those trajectories; safe incompletion differs from failure to recover. |
| Study | Executed substrate | Intervention/control | Outcome evidence | Important scope |
|---|---|---|---|---|
| -bench / TheAgentCompany ( Yao et al., 2025 ; Xu et al., 2024 ) | Simulated APIs / self-hosted apps | Task-capability baselines | State and replies / checkpoints | Simulated users; some judged deliverables |
| ReliabilityBench ( Gupta, 2026 ) | Stateful simulators | Stress levels and architectures | Goal predicates, repeated success | Authored tasks in four domains |
| ToolMaze / ToolBench-X ( Zhu et al., 2026b ; Tian et al., 2026 ) | Deterministic tools | Warning / hint versus extra rounds | Paths, task outcomes or answers | Different hint and scoring ruless |
| MCP-Injector ( Iannillo et al., 2026 ) | Real MCP servers | Proxy off, passthrough, faults | Judged quality and system stops | Thread-label audit is not judge validation |
| AFT-Bench ( Wang, 2026 ) | Synthetic worlds and SQLite | Interface-mechanism ablations | Effects and terminal claims | Mechanism-conditional effects |
| UndoBench / Evidence-Guided Recovery ( Sah et al., 2026 ; Zhu et al., 2026a ) | Sandboxes / finite provider model | Retry, idempotency / planning advice | Effect history, state, certificates | Capable-pair / known-model conditioning |
| Arm | Offered/bound | No offer | Texts |
|---|---|---|---|
| G | 325/325 | 323 | 21 |
| N | 330/329 | 318 | 1 |
| Old | 134/134 | 514 | 2 |
| Configuration | Known/assigned | W/L/T | (pp) | Holm | Holm | ||
|---|---|---|---|---|---|---|---|
| GPT-4o mini | 216/216 | 2/20/194 | .015625 | .046875 | .25 | .75 | |
| GPT-4.1 mini | 209/216 | 8/8/193 | 1 | 1 | 1 | 1 | |
| GPT-4.1 | 188/216 | 3/1/184 | .75 | 1 | .75 | 1 | |
| Qwen (separate) | 39/108 | 5/0/34 | .125 | — | .25 | — |
| Provider | Literal requested ID | Returned field | UTC date |
|---|---|---|---|
| together | Qwen/Qwen3.8-Flash | Same literal (SSE) | 2026-11-07 |
| anthropic | claude-opus-5-5 | Not retained | 2026-11-07 |
| together | deepseek-ai/DeepSeek-V4-Pro-0813 | Not retained | 2026-11-07 |
| gemini | gemini-4-argon | Not retained | 2026-11-07 |
| openai_responses | gpt-6.1-sol | Not retained | 2026-11-07 |
| Cohort | W/L/T | Known | Assigned bounds (pp) |
|---|---|---|---|
| DeepSeek | 0/2/16 | 18/18 | [ 11.11, 11.11] |
| Claude | 3/3/3 | 9/9 | [0,0] |
| Qwen | 0/0/16 | 16/18 | [ 11.11,0] |
| Required reads | Current valid proof | Cases |
|---|---|---|
| Missing | Missing | 311 |
| Complete | Missing | 6 |
| Complete | Present | 1 |
| Total | 318 |
| Model | Arm | Discordant/known | Rep1 rep0 (pp) |
|---|---|---|---|
| GPT-4o mini | Old | 7/107 | -0.93 |
| GPT-4o mini | N | 2/108 | +0.00 |
| GPT-4o mini | G | 14/108 | +5.56 |
| GPT-4.1 mini | Old | 8/104 | +0.00 |
| GPT-4.1 mini | N | 6/106 | -5.66 |
| GPT-4.1 mini | G | 6/104 | +0.00 |
| Model | NG | Known/A | W/L/T | (pp) |
|---|---|---|---|---|
| 4o mini | 00 | 91/91 | 0/0/91 | +0.00 |
| 4o mini | 01 | 10/10 | 0/5/5 | -50.00 |
| 4o mini | 10 | 11/11 | 1/0/10 | +9.09 |
| 4o mini | 11 | 104/104 | 1/15/88 | -13.46 |
| 4.1 mini | 00 | 107/114 | 0/0/107 | +0.00 |
| 4.1 mini | 01 | 15/15 | 0/7/8 | -46.67 |
| Condition | Cases | Strict | Rejections |
|---|---|---|---|
| Clean | 18 | 18 | 0 each |
| FM20 | 18 | 18 | 3 each |
| FM13 | 6 | 6 | 1 each |
| FM19 | 6 | 6 | 2 each |
| Arm | Task | Material | Process | JSON | Offers/bound | Auto retries |
|---|---|---|---|---|---|---|
| N | 12/12 | 12/12 | 12 | 12 | 8/8 | 0 |
| G | 9/12 | 9/12 | 9 | 9 | 7/7 | 0 |
| G-S | 11/12 | 12/12 | 11 | 11 | 8/8 | 0 |
| G+T | 11/12 | 11/12 | 11 | 11 | 7/7 | 0 |
| G+T-S | 12/12 | 12/12 | 12 | 12 | 8/8 | 0 |
| N+R | 12/12 | 12/12 | 12 | 12 | 0/0 | 12 |
| Arm | 429s | Calls | USD | Median s |
|---|---|---|---|---|
| N | 15 | 100 | 0.03 | 15.1 |
| G | 14 | 100 | 0.03 | 14.9 |
| G-S | 13 | 107 | 0.04 | 15.8 |
| G+T | 15 | 99 | 0.03 | 14.6 |
| G+T-S | 13 | 105 | 0.04 | 15.4 |
| N+R | 12 | 74 | 0.02 | 14.4 |
| Model | Arm | Known/A | Strict | R-read | State |
|---|---|---|---|---|---|
| 4o mini | O | 215/216 | 130 | 130 | 131 |
| 4o mini | N | 216/216 | 142 | 142 | 144 |
| 4o mini | G | 216/216 | 124 | 124 | 124 |
| 4.1 mini | O | 211/216 | 93 | 93 | 135 |
| 4.1 mini | N | 212/216 | 104 | 104 | 145 |
| 4.1 mini | G | 210/216 | 101 | 101 | 140 |
| Model | Known/A | W/L | Both pass | Both fail |
|---|---|---|---|---|
| GPT-4o mini | 104/104 | 1/15 | 52 | 36 |
| GPT-4.1 mini | 74/74 | 0/1 | 7 | 66 |
| GPT-4.1 | 102/105 | 1/1 | 13 | 87 |
| Cohort | Assigned | Scope |
|---|---|---|
| Original bundle | 2,592 | Six new instances; opportunity/resources bundled. |
| Matched ladder | 1,728 | Six known instances; four arms, two repetitions. |
| Fresh-instance | 1,944 | Nine new instances of three known families. |
| DeepSeek/Qwen | 432 | Known tasks; no observed late advice. |
| Delivery checks | 24 | Two selected twelve-run development pilots. |
| Funded program | 1,158 | Eleven separate hosted/reference cohorts. |
| Model | N offers | Bound | Wins+offer | Wins offer |
|---|---|---|---|---|
| 4o mini | 44 | 44 | 35 | 14 |
| 4.1 mini | 5 | 5 | 3 | 8 |
| 4.1 | 16 | 16 | 8 | 6 |
| Neutral | Guided | ||||||
|---|---|---|---|---|---|---|---|
| Model | Fault | E | E+A | Strict/known | E | E+A | Strict/known |
| 4o mini | FM13 | 36 | 22 | 21/22 | 36 | 23 | 14/23 |
| 4o mini | FM19 | 36 | 25 | 24/25 | 36 | 22 | 11/22 |
| 4o mini | FM18 | 36 | 32 | 32/32 | 36 | 33 | 33/33 |
| 4o mini | FM20 | 36 | 36 | 0/36 | 36 | 36 | 0/36 |
| 4o mini | FM21 | 36 | 0 | — | 36 | 0 | — |
| Arm | Assigned | E | E+A | Strict/known |
|---|---|---|---|---|
| N | 12 | 12 | 8 | 8/8 |
| G | 12 | 12 | 7 | 4/7 |
| G-S | 12 | 12 | 8 | 7/8 |
| G+T | 12 | 12 | 7 | 6/7 |
| G+T-S | 12 | 12 | 8 | 8/8 |
| N+R | 12 | 12 | 0 | — |
| GPT-4o mini | Claude Opus 5.5 | |||||
| Condition | ||||||
| Clean | 12 | 12 | 12 | 12 | 12 | 12 |
| FM11 | 1 | 5 | 12 | 12 | 12 | 12 |
| FM12 | 9 | 9 | 12 | 12 | 12 | 12 |
| FM13 | 12 | 12 | 12 | 12 | 12 | 12 |
| FM16 | 10 | 6 | 12 | 12 | 12 | 12 |
| GPT-4o mini | GPT-6.1 Sol | Claude (reused) | |||||||
| Condition | |||||||||
| Clean | 12 | 12 | 12 | 12 | 12 | 12 | 12 | 12 | 12 |
| FM11 | 1 | 5 | 10 | 11 u | 12 | 12 | 12 | 12 | 12 |
| FM12 | 8 | 11 | 12 | 12 | 11 u | 12 | 12 | 12 | 12 |
| FM13 | 12 | 12 | 12 | 12 | 12 | 12 | 12 | 12 | 12 |
| FM16 | 10 | 8 | 12 | 12 | 12 | 12 | 12 | 12 | 12 |
| Model | Endpoint | Complete pairs | Wins | Losses | Ties | Unknown | Difference (pp) | Assigned bounds (pp) |
|---|---|---|---|---|---|---|---|---|
| GPT-4o mini | Material | 96 | 10 | 0 | 86 | 0 | +10.42 | [+10.42,+10.42] |
| GPT-4o mini | Strict | 96 | 16 | 0 | 80 | 0 | +16.67 | [+16.67,+16.67] |
| GPT-6.1 Sol | Material | 95 | 0 | 0 | 95 | 1 | 0.00 | [+0.00,+1.04] |
| GPT-6.1 Sol | Strict | 95 | 0 | 0 | 95 | 1 | 0.00 | [+0.00,+1.04] |
| Claude Opus 5.5 | Material | 96 | 0 | 0 | 96 | 0 | 0.00 | [+0.00,+0.00] |
| Claude Opus 5.5 | Strict | 96 | 0 | 0 | 96 | 0 | 0.00 | [+0.00,+0.00] |
| Fresh-grid arm | Success/known | Unknown | Offers/bound |
|---|---|---|---|
| Previous O | 142/214 | 2 | 0/0 |
| Neutral N | 161/216 | 0 | 35/35 |
| Obligation G | 144/215 | 1 | 36/36 |
| Joint revision J | 144/216 | 0 | 36/36 |
| Condition | Interpretation | No-IID | |||
|---|---|---|---|---|---|
| FM0 | Clean | 24/24 | 24/24 | 24/24 | – |
| FM11 | Request not sent | 5/5 | 9/9 | 20/20 | – |
| FM12 | Response lost after commit | 16/12 | 18/17 | 24/24 | 24/24 |
| FM13 | Rate limit | 24/24 | 24/24 | 23/23 u1 | – |
| FM16 | Lost receipt + stale read | 18/16 | 19/12 | 22/22 | 23/23 |
| FM18 | Forged verification | 24/24 | 24/24 | 24/24 | – |
| Contrast | Endpoint | Known/A | W/L/T | Unknown | Diff. pp | Assigned bounds | |
|---|---|---|---|---|---|---|---|
| Material | 191/192 | 22/2/167 | 1 | +10.47 | [+9.90,+10.42] | 0.0625/0.2500 | |
| Strict | 191/192 | 30/2/159 | 1 | +14.66 | [+14.06,+14.58] | 0.03125/0.25 | |
| Material | 48/48 | 0/1/47 | 0 | -2.08 | [-2.08,-2.08] | 1.0000/1.0000 | |
| Strict | 48/48 | 0/1/47 | 0 | -2.08 | [-2.08,-2.08] | 1.0000/1.0000 |
AI Agent Reliability
AI agent reliability studies whether tool-using systems act correctly, consistently, safely, and recoverably across changing tasks and environments.
See all 364 papers