Idempotency Is Not Recovery: A Fault-Injection Study of Language-Model Agent Tool Actions
Organizations: University of Electronic Science and Technology of China, Chengdu, China.
Abstract
When a language-model agent's tool call is interrupted, the action may already have changed an external system without confirmation. Retrying risks duplication; assuming success may conceal an action that never occurred. We study these failure modes in a controlled agent runtime combining authorization, an action ledger, target-side idempotency, reconciliation, and evidence sealing. We inject ten single-host fault classes and compare four configurations and three ablations using a target-side oracle separate from the runtime ledger. On each of Windows and Linux, 7,513 trials used 96 captured model responses from 24 constructed cases. In 200 known original crash and replay pairs, the full configuration recorded 0/200 repeated effects versus 117/200 for each simpler baseline. A ledger-related ablation had 170/170 orphaned effects versus 0/170 and 200/300 recovery successes versus 300/300 for the full configuration. Disabling reconciliation left 100/100 known post-effect crash outcomes ambiguous despite no observed duplicates. A separate supplement designed after the initial results and reusing the same inputs found a similar pattern under response loss or delay. The full runtime had about one-seventh of fail-open throughput, while injected evidence-write faults left 75/100 trials per platform unresolved. These findings distinguish duplicate suppression from recovery and identify their measured reliability and performance tradeoffs within the tested benchmark.
Figures & tables
| Config. | Main capability or change |
|---|---|
| B0 | Volatile, fail-open execution; no durable action reservation or replay suppression. |
| B1 | Fail-closed authorization, without durable action reservation or replay suppression. |
| B2 | Durable authorization and ordinary log; no durable action reservation or replay suppression. |
| B3 | Fail-closed authorization, durable reservation and completion, target idempotency, reconciliation, and sealed evidence. |
| A1 | B3 without reservation, in-flight record, completion transaction, and pre-dispatch replay suppression. |
| A2 | B3 without post-crash reconciliation. |
| ID | Injected condition | Observation focus |
|---|---|---|
| F1 | Invalid or stale authority | Unauthorized effects |
| F2 | Crash after reservation | Lost effect, recovery |
| F3 | Crash in flight, before effect | Lost effect, recovery |
| F4 | Effect followed by response loss or delay | Duplicate, ambiguity, recovery |
| F5 | Effect before ledger completion, then crash | Orphaned effect |
| F6 | Capture write or synchronization error | Missing or corrupt evidence |
| Outcome | Counted when | Scope |
|---|---|---|
| Unauthorized effect | The oracle observes an external effect despite invalid or missing authority | F1 |
| Duplicate effect | The oracle observes more than one external effect with the same logical action ID | F4, F5, F7, F9 |
| Lost effect | The runtime reports completion but the oracle observes no external effect | F2, F3 |
| Orphaned effect | The oracle observes an external effect but the ledger has no terminal record | F5, F7 |
| Ambiguous outcome | The runtime cannot determine whether an external effect occurred | F4, F5 |
| Unresolved state | The runtime remains in flight or unresolved after a recovery attempt | Storage and restart faults |
| Failure mode | Tested mechanism or contrast | Observation and evidence boundary |
|---|---|---|
| Unauthorized effect | Fail-closed authorization | B0 100/100; B1–B3 0/100 in F1. [P] |
| Repeated effect | Identity and target idempotency retained in A1 | Original known F5/F9 pairs: B3 0/200, each B0–B2 117/200 [P]; F4 supplement: 0/100 vs. 66/100 [S]. |
| Orphaned effect | Ledger-related bundle | B3 0/170, A1 170/170; the ablation removes several capabilities together. [P] |
| Orphan or ambiguity | Post-crash reconciliation | F5 known pairs: B3 0/100, A2 100/100 for each outcome [P]. F4 supplement: A2 ambiguous 100/100 (Linux) and 99/99 (Windows), B3 0/100 and 0/99 [S]. |
| Recovery failure | Ledger-related bundle | B3 300/300 and A1 200/300 recovery successes; timing is reported separately. [P] |
| Tamper not detected | Sealing and history verification | B3 detected 89/89 known cases, A3 0/89; localization and unknowns are reported separately. [P] |
| Outcome, comparator | B3 / comparator | Pairs | Difference | Paired 95% CI | Proposal-cluster 95% CI |
|---|---|---|---|---|---|
| Unauthorized effect, B0 | 0/100 / 100/100 | 100 | (13) | ||
| F5 orphan effect, B0 | 0/100 / 100/100 | 100 | (21) | ||
| Orphan effect, A1 | 0/170 / 170/170 | 170 | (21) | ||
| Recovery success, A1 | 300/300 / 200/300 | 300 | (28) |
| Operation | Trials | Each of B0–B2 | B3 |
|---|---|---|---|
| OP1 alert append | 16 | 16 | 0 |
| OP2 ticket upsert | 17 | 17 | 0 |
| OP3 guarded transition | 17 | 0 | 0 |
| Total per subtype | 50 | 33 | 0 |
| Platform | B0 | B1 | B2 | B3 |
|---|---|---|---|---|
| Windows | 20.57 | 21.50 | 11.65 | 3.00 |
| Linux | 318.81 | 319.16 | 163.21 | 41.94 |
| Question | Answer supported by this study |
|---|---|
| RQ1 | Fail-closed configurations blocked the measured unauthorized effects (0/100 versus 100/100); the result is not unique to B3, and the volatile fail-closed configuration did not have lower measured operation-window throughput than fail-open. |
| RQ2 | The full configuration had fewer repeated and orphaned effects in selected known pairs; disabling reconciliation alone left F5 and F4 outcomes ambiguous. Unknown original F4 stimuli and same-team target assumptions limit the claim. |
| RQ3 | The tested sealing/history configuration detected 89/89 known F10 changes versus 0/89 for A3; localization and other observations retain unknowns. |
| RQ4 | B3 recorded 300/300 recovery successes versus 200/300 for A1, and 138/138 in the dedicated recovery family (median 1.09 s on Windows, 0.28 s on Linux); capture failures and measured latency and throughput show availability and cost limitations. |
| RQ5 | All 44 pre-specified contrasts had identical pair-count fields on the two platforms. Both executions used the same captured proposals and fault classes, so this shows consistent behavior of one implementation on two hosts, not independent replication; timing differed, and a cloud host replaced the planned WSL2 environment. |
Tool-Using Language Agents
Tool-using language agents combine language models with APIs, code, and other tools to answer questions and carry out multi-step tasks.
See all 65 papers