Dependable Long-Horizon LLM Agents: A Trajectory-Centered Empirical Study of Where Errors Enter and Which Controls Reach Them
Organizations: Villanova University, Department of Electrical and Computer Engineering, Villanova, Pennsylvania, USA
Abstract
When an LLM agent fails part-way through a long task, the usual analysis records a symptom: a failure-mode label, or the agent and step judged responsible. It does not say where the error entered, which state carried it to the symptom, whether anything in the trajectory contradicted it, or which control could have caught it. We define an episode record that ties a candidate fault to its origin, propagation, detection, response and consequence, located by operational phase and state domain, and that names the control families whose observation and reach cover it. We turned the record into a written annotation protocol and applied it to 95 failed runs of six multi-agent systems from the public Who&When and MAD corpora, reading each trajectory in full. Faults enter early (median at 0.18 of the run), mostly as wrong decisions or fabrications over task state, and in 37 of 88 Who&When runs before the step the corpus' human labels mark as decisive. In 72% of failed runs the trajectory already contained evidence against the error before the failure; in 88% of those no agent acted on it. Something flagged a deviation in 31% of runs, but no response removed the deviated state. Independent judges reading the unused evidence, task contracts fixed at admission, and re-checking beliefs against the environment reach most episodes; restoring the agent's own state would have sufficed in 35 of 95 runs. A text scan of the raw corpus also shows that 45% of the hand-crafted Who&When failures ended with a provider content-filter crash after which the harness emitted a final answer no agent produced, a failure class that neither symptom taxonomies nor responsible-agent labels represent. Protocol, records and scripts are included.
Figures & tables
| Code | Coordinate | Operational definition |
| P0 | Scope and authorize | Establish goal, constraints, acceptance criteria, permissions, and budget. |
| P1 | Deliberate and decompose | Select or revise a plan, action, or delegation. |
| P2 | Dispatch and execute | Issue and execute a tool, model, agent, or human operation. |
| P3 | Observe and integrate | Interpret a result and update working or persistent state. |
| P4 | Verify and decide | Assess validity, fidelity, progress, or completion. |
| P5 | Recover and escalate | Retry, replan, restore, compensate, or transfer control. |
| Phase | R1 | R2 | R3 | R4 | R5 | R6 | Total |
| P0 | – | 2 | 1 | – | 10 | – | 13 |
| P1 | – | 24 | – | – | – | – | 24 |
| P2 | – | 26 | 6 | 2 | – | – | 34 |
| P3 | – | 9 | 1 | – | – | – | 10 |
| P4 | 1 | 5 | 2 | – | 1 | – | 9 |
| P5 | – | 4 | – | – | – | – | 4 |
| Kind | Runs | Share |
|---|---|---|
| decision | 40 | 42% |
| fabrication | 25 | 26% |
| integration | 10 | 11% |
| coordination | 8 | 8% |
| specification | 4 | 4% |
| interface | 7 | 7% |
| Control | Primary | Any | Any share |
|---|---|---|---|
| TC | 17 | 64 | 67% |
| POL | 3 | 30 | 32% |
| IFS | 1 | 4 | 4% |
| EVS | 17 | 41 | 43% |
| PROV | 20 | 36 | 38% |
| TEST | 8 | 20 | 21% |
| Corpus (system) | P1–P2 | Fabr. | P0 | Evid. | Det. | Dist. | Harness | |
|---|---|---|---|---|---|---|---|---|
| W&W hand-crafted (Magentic-One) | 29 | 52 | 7 | 0 | 90 | 41 | 18 | 55 |
| W&W algorithm-gen. (CaptainAgent) | 59 | 66 | 39 | 20 | 64 | 25 | 5 | 0 |
| MAD (mixed) | 7 | 57 | 0 | 14 | 57 | 29 | – | 0 |
| MAST mode | Runs | Cells | Most frequent origin cells | Origin kinds |
|---|---|---|---|---|
| 1.1 Disobey task specification | 49 | 11 | P1/R2 (14), P2/R2 (12), P0/R5 (6) | decision 23, fabrication 15, coordination 5 |
| 1.2 Disobey role specification | 7 | 4 | P2/R2 (3), P1/R2 (2), P2/R3 (1) | fabrication 3, decision 3, interface 1 |
| 1.3 Step repetition | 33 | 7 | P1/R2 (10), P2/R2 (8), P3/R2 (6) | decision 17, fabrication 7, integration 6 |
| 1.5 Unaware of stopping conditions | 5 | 4 | P1/R2 (2), P2/R3 (1), P2/R2 (1) | decision 3, interface 1, coordination 1 |
| 2.2 Fail to ask for clarification | 12 | 5 | P0/R5 (6), P1/R2 (3), P2/R2 (1) | coordination 5, decision 4, specification 2 |
| 2.3 Task derailment | 15 | 6 | P0/R5 (9), P1/R2 (2), P0/R3 (1) | coordination 6, specification 3, decision 3 |
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
| Code | Dynamic | Distinguishing observation | Useful measurement |
|---|---|---|---|
| H1 | Repeated exposure | More opportunities for an event. | Hazard per opportunity; task failure. |
| H2 | Contamination | A deviated state is retained and reused. | Retention, consumption, retraction. |
| H3 | Drift | Divergence from an explicit reference. | Goal, plan, or policy deviation. |
| H4 | Cascade | One error influences several later events. | Supported dependency edges. |
| H5 | External impact | Restoration omits consequences. | Unrecovered effects and cost. |
| H6 | Observability debt | Diagnosis needs missing dependencies. | Detection delay, attribution error. |
| MAST category | Failure modes with framework coordinates (phase / domain / dynamic; parenthesized dynamics apply only under the stated condition) |
|---|---|
| Specification / system design | 1.1 Disobey task specification (P0–P1 / R2 / H3); 1.2 Disobey role specification (P1 / R5 / H3); 1.3 Step repetition (P1–P2 / R2 / H1; H7 if a budget is exhausted); 1.4 Loss of conversation history (P3 / R2 / –; H3 if later divergence follows); 1.5 Unaware of stopping conditions (P4–P6 / R2 / –; H7 if continuation exhausts a budget) |
| Inter-agent misalignment | 2.1 Conversation reset (P5 / R5 / –; state loss); 2.2 Fail to ask for clarification (P1, P5 / R5–R6 / –; H3 if the unresolved ambiguity causes divergence); 2.3 Task derailment (P1 / R2 / H3); 2.4 Information withholding (P6 / R5 / H6); 2.5 Ignored other agent’s input (P3 / R5 / –; H4 if downstream influence is shown); 2.6 Reasoning–action mismatch (P1–P2 / R2 / H3) |
| Task verification | 3.1 Premature termination (P4–P6 / R2 / –; H7 only if a budget forced termination); 3.2 No or incomplete verification (P4 / R2, R4 / –; H6 if later diagnosis needs the unchecked evidence); 3.3 Incorrect verification (P4 / R2 / –; H2 if the accepted output is reused) |
| System | Recovery boundary | Treatment of released effects | Assurance | Residual dependency |
|---|---|---|---|---|
| ACRFence ( Zheng et al., 2026b ) | Irreversible tool effects across restore | Effect log; replay or fork on restoration | Attack demonstrations; prototype | Effects must be identified as irreversible |
| Cordon ( Chen et al., 2026b ) | Task-level semantic transaction | Shadow state; staged outbox; validation before commit | Experiments | Validation predicates; approval for released effects |
| Execution edits ( Zheng et al., 2026a ) | Checkpoint, fork, restore, merge | Execution record decides which past authorizations and requests an edit must account for | Exact check; mechanized proofs | Complete execution record and policy model |
| AgentRewind ( Zhuang et al., 2026 ) | Agent context and controlled environment | Aligned checkpoints; resume with prior-attempt information | Experiments | Snapshot-capable environment |
| REVISE ( Qi et al., 2026 ) | Partially executed workflow | Dependency-guided invalidation and recomputation | Experiments | Recorded data and control dependencies |
| Planarian ( Guo et al., 2026 ) | Statepoint over local and remote state | Local snapshot; recorded compensating actions replayed on rollback | Experiments | Compensations exist and succeed |
| Family (examples) | Inspection basis | Protects | Enforces or restores | Principal assumptions | Added cost | Residual risk |
| Consistent decisions (Sections C.1 – C.2 ) | ||||||
| Search, decomposition ( Yao et al., 2023a ; Zhou et al., 2024a ; Meyerson et al., 2025 ) | Candidate paths scored by an evaluator or local votes | R2 | Backtrack within the plan | Evaluator discriminates; subtasks separable | Multiples of model calls | Shared evaluator error; effects of explored branches unless sandboxed |
| Reflection, revision ( Shinn et al., 2023 ; Gou et al., 2024 ) | Own outputs, feedback, tool results | R2 | A further attempt, not state | Feedback is informative about the error | Extra trials | Reinforcing a wrong diagnosis |
| Environment-verified task state ( Ma et al., 2026 ; Wang et al., 2024a ) | Observation before a belief or skill is stored | R2 vs R4 | Re-derive the belief | Relevant state is observable and cheap to read | Read and audit calls | Unobservable or lagging state |
| Context compression, KV retention ( Kang et al., 2026 ; Zhang et al., 2023 ; Li et al., 2024 ) | Attention scores or learned compression rules | R1, R2 | Nothing (lossy) | Importance tracks later needs | Saves resources | Silent loss of a constraint |
| Retrieval, memory with provenance ( Asai et al., 2024 ; Yan et al., 2024 ; Chhikara et al., 2025 ) | Relevance, recency, update operations | R2, R5 | Update or delete items | Provenance recorded; invalidation reaches consumers | Storage, retrieval | Stale or poisoned items; dependants not invalidated |
| Case A: shortcut that alters a shared system (TheAgentCompany) | Case B: example accepted as test result (MAST N.8) | |
|---|---|---|
| Symptom-level (source) | Fabricated shortcut (“deceiving oneself”) | 1.2 disobey role specification; 2.4 information withholding |
| State-centered (Ding et al.) | Identity record changed without authority | Stored “tested” claim without provenance |
| Episode: origin | P1/R2: decision to alter the directory instead of searching it | P3/R5: hypothetical output merged with an R4 observation in a handoff |
| Episode: detection, response | Evidence displayed (duplicate name, wrong recipient); no check reads it; no escalation | Executor’s caveat present; dropped at integration; P4 accepts |
| Episode: boundary crossed | P2 write R4/R6 identity R5 message to wrong user R2 belief 81 waiting turns (H7) | R4 observation and example R5 report P4 acceptance P6 termination |
| Indicated control | Scope rule on directory writes; recipient check; stall rule that escalates | Bind “tested” claims to execution records; block termination on unexecuted checks |
| If the agent… | Controls to consider | Evidence to demand |
|---|---|---|
| Releases irreversible effects | Staging; effect logs with replay-or-fork; provider deduplication | Duplicate and residual-impact rates under injected timeouts and restores |
| Reads untrusted content or tools | Data–authority separation; deterministic privilege policies; tool provenance | Attack success and utility under matched permissions |
| Keeps long-lived or shared memory | Provenance and consumers per item; invalidation; decision-based compression | Retraction success and decision fidelity at matched budgets |
| Relies on model judges | Independent evidence; seeded-violation evaluation | False acceptance on the worker’s own errors |
| Coordinates several agents | Enforced ownership; versioned handoffs; bounded termination | Performance under shared-source faults at equal budget |
| Runs for hours or sessions | Task state verified against the environment; reconciliation | and detection latency by horizon component |
Agent Failure Analysis
Agent failure analysis studies how and why AI agents make errors, traces causes through execution, and helps detect or prevent recurrence.
See all 159 papers