Same Error, Different Recovery: How Delivery Changes LLM Agent Behavior
Organizations: Independent Researcher, Bucheon, Republic of Korea · Information Processing and Telecommunications Center, Universidad Politécnica de Madrid, Madrid, Spain
Abstract
A tool-using language model must decide whether to retry a failed call, correct it, or stop and tell the user. The error text alone may not be enough: the same feedback can lead to different actions depending on where it appears in the conversation. We isolate this delivery problem by varying placement, guidance, retained history and turn structure. Two studies with plans fixed before data collection, across 13 open models from 3B to 70B parameters, compare the next action and subsequent task completion, with exact deterministic reruns. A new user turn that retains the failed exchange avoids the severe stopping failure observed in Llama models when that exchange is removed. It is non-inferior to history rewriting in pooled synthetic comparisons, but is not best for every model. We implement this form as an opt-in recovery layer within a deterministic inference engine and evaluate it on public benchmarks and a coding agent. On ToolMaze, it raises success on explicit errors by 14.2 points with 7-8B models and 10.5 points with 32-70B models; Loud/Quiet, 24 multi-turn tasks with visible failures and silent corruption, scored by final-state agreement, shows no measurable gain. A client given the exact fault kind performs better still. The findings support delivery as a distinct factor in recovery, not an engine-exclusive solution or a universal retry policy.
Figures & tables
| Kind | Correct action | Guidance sentence |
|---|---|---|
| Transient | Repeat the same call | This is a temporary failure. Retrying the same call is expected to succeed. |
| Fixable | Call again with the value the error names | This call cannot succeed as written. The error says what to change; call the tool again with the corrected value. |
| Unrecoverable | Stop and tell the user | This cannot be fixed by retrying or by changing the arguments. Tell the user what happened. |
| Ambiguous | Depends on context | (none, the result is left unchanged) |
| Condition | Failed call and result | Note and guidance | History kept |
|---|---|---|---|
| C0 | kept | none | yes |
| C1 | kept | guidance at the end of the tool message | yes |
| C2 (Study 1’s A2B2) | removed | at the end of the last message if it is a user message, else in a new user turn | no |
| C3 | kept | in a new user turn after the failure | yes |
| C4 | removed | in a new user turn | no |
| Small tier (7 models, 3–35B) | Large tier (6 models, 27–70B) | |
|---|---|---|
| H2.1: C3 C2, synthetic | , supported | , supported |
| H2.2: real histories, C3 vs. C2 | 55.6% vs. 51.1%, , not supported | 57.8% vs. 45.6%, , supported |
| C3 C0, real histories | ||
| C3 C2, cluster bootstrap | ||
| S1: C4 C3 | , equivalent | , equivalent |
| S2: C3 C1 | 5 of 7 models, supported | 5 of 6 models, supported |
| Comparison | Main finding | Qualification |
|---|---|---|
| Study 1: note vs. tool message, guidance vs. none (H1.1, H1.2) | Each improves the next action in five of six models | Without guidance, the note reduces correct stopping for two models |
| Study 1: selected form C2 over whole episodes (H1.5) | Completion higher in five models | Not supported: futile calls exceed the margin, all from Llama-3.1-8B |
| Study 2: C3 vs. C2, synthetic errors (H2.1) | and points in the two tiers; non-inferiority passed | Gains driven by Llama stopping; Qwen2.5-7B is worse with C3 |
| Study 2: C3 vs. C2, real histories (H2.2) | Supported in the large tier (57.8% vs. 45.6%) | Not supported in the small tier ( ); 19 clusters limit generalization; significant only under the registered any-call scoring |
| Study 3: G-on vs. G-off, ToolMaze (descriptive) | and points on explicit errors | No detectable change without faults; an oracle client does 17–19 points better |
| Study 3: G-on vs. G-off, Loud/Quiet (descriptive) | No measurable effect ( and points) | The missing tool is ambiguous by design, so the layer gives no guidance |
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Start text | Source |
|---|---|
| Error: , ERROR: , error: | LangChain and LangGraph tool nodes; common CLI form |
| Error executing tool | MCP Python SDK (FastMCP) |
| MCP error - | MCP TypeScript SDK ( McpError ) |
| An error occurred while running the tool | OpenAI Agents SDK default failure function |
| Traceback (most recent call last): | Python exception trace |
| an exception name ending in Error , Exception , Timeout or timeout , followed by : or ( | Python exception string form ( TypeError: … , ConnectionResetError(…) ) |
| Transient (repeat) | Fixable (fix and repeat) | Unrecoverable (stop) | ||||||||||
| Model | A1B1 | A1B2 | A2B1 | A2B2 | A1B1 | A1B2 | A2B1 | A2B2 | A1B1 | A1B2 | A2B1 | A2B2 |
| Qwen2.5-7B | .37 | .43 | 1.00 | 1.00 | .48 | .87 | 1.00 | 1.00 | 1.00 | 1.00 | .30 | 1.00 |
| Qwen3-8B | .00 | .08 | 1.00 | 1.00 | .02 | 1.00 | .95 | 1.00 | 1.00 | 1.00 | .60 | 1.00 |
| Llama-3.1-8B | .98 | .98 | 1.00 | 1.00 | .98 | .93 | .92 | 1.00 | .02 | .10 | .00 | .00 |
| Gemma 4 12B | .12 | .23 | .95 | 1.00 | .80 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| Qwen3.5-9B | .30 | .97 | 1.00 | 1.00 | .13 | .87 | 1.00 | 1.00 | 1.00 | 1.00 | .93 | 1.00 |
| Small tier | Large tier | |||
|---|---|---|---|---|
| Loud/Quiet | ToolMaze | Loud/Quiet | ToolMaze | |
| G-on G-off, explicit errors | ||||
| H3.1: G-on L | * | * | * | |
| H3.1: G-on V | * | * | ( ) | |
| H3.1: G-on S | * | * | * | * |
| H3.2: G-off L | = | |||
| vs. G-off | vs. L | vs. V | vs. S (main) | vs. S (8,192) | |
|---|---|---|---|---|---|
| Flaky: G-on X | |||||
| Forbidden, reports: G-on X | |||||
| Forbidden, requests: G-on X | |||||
| Ok, baseline: G-off X | |||||
| Ok, control: G-on G-off |
LLM Agent Reliability
Large language model (LLM) agent reliability concerns dependable, safe, and verifiable behavior across tools, multi-step tasks, and changing environments.
See all 909 papers