Capable but Unreliable: Canonical Path Deviation as a Causal Mechanism of Agent Failure in Long-Horizon Tasks

Wilson Y. Lee

cs.CLFeb 22, 2026

Capable but Unreliable: Canonical Path Deviation as a Causal Mechanism of Agent Failure in Long-Horizon Tasks

Authors: Wilson Y. Lee

Organizations: Independent Researcher United States

Abstract

Why do language agents fail on tasks they are capable of solving? We argue that many such failures are reliability failures caused by stochastic drift from a task's latent solution structure, not capability failures. Every well-defined tool-use task imposes a canonical solution path (i.e., a convergent set of tool invocations shared across successful runs) and agent success depends critically on whether a trajectory stays within this path's operating envelope. We establish this causally using a natural experiment that holds model capability and task difficulty fixed by construction. We analyze trajectories from the Toolathlon benchmark: 22 frontier models each attempt 108 real-world tool-use tasks across 3 independent runs, yielding 515 model $\times$ task units where the same model succeeds on some runs and fails on others due to LLM sampling stochasticity alone. Within these units, successful runs adhere significantly more closely to the canonical solution path than failed runs ( $+$ 0.060 Jaccard, $p<0.0001$ , $n=488$ units, 95% CI [+0.043, +0.077]). This result survives six robustness checks including cross-model-family leave-one-out validation. Critically, the causal mechanism is gradual and self-reinforcing: the adherence gap is statistically indistinguishable from zero through the first 50% of the trajectory, ruling out early-branching selection bias, and each off-canonical tool call raises the probability that the next call is also off-canonical by 22.7 percentage points ( $\hatβ=+0.227$ , $p<0.0001$ ), more than doubling the baseline rate. These findings imply that agent reliability cannot be improved by capability scaling alone, but offer a highly actionable intervention: a simple monitor that restarts the bottom tercile of runs based on mid-trajectory canonical adherence lifts success rates by $+$ 8.8 percentage points among intervened runs.

Figures & tables

Figure 1. Canonical path drift is gradual and self-reinforcing. (A) Within mixed-outcome units (

n=488

), the adherence gap between successful and failed runs is statistically indistinguishable from zero at 10%, 25%, and 50% of trajectory completion, becoming significant only at 75% (

p=0.043

) and strongly significant at full completion (

p<0.0001

), ruling out early branching as the mechanism. (B) Conditional transition probabilities show that off-canonical calls are self-reinforcing: following an off-canonical call, the probability that the next call is also off-canonical rises by

+22.7

pp relative to following a canonical call (

\hat{\beta}=+0.227

p<0.0001

), and this compounding is significantly stronger in runs that ultimately fail. Together, panels A and B establish that agent failure is a gradual compound process , not a single pivotal mistake.

Statistic	Value
Total models	22
Total tasks	108
Runs per model $\times$ task	3
Total trajectories	7,003
Model $\times$ task units (3 runs)	2,286
Always-succeed units	286 (12.5%)

Table 1. Dataset structure and mixed-outcome sample.

Figure 2. Robustness and generalization of the canonical path adherence effect. (A) Within-unit adherence gaps across 70 tasks; 56/70 (80%) show positive effects (mean

=+0.068

, green dashed line). (B) Effect size is larger for shorter tasks (

r=-0.25

p=0.038

), consistent with less opportunity for course-correction after deviation; colour indicates canonical path strength. (C) All 11 model families show positive gaps; 5 reach individual significance (

p<0.05

, dark blue). Error bars show

\pm 1.96

SE.

Specification	Gap	CI	$N$	Sig.
Original (naive)	+0.082	[+0.059, +0.105]	505	∗∗∗
Leave-one-out	+0.067	[+0.050, +0.084]	495	∗∗∗
CF-LOO (headline)	+0.060	[+0.043, +0.077]	488	∗∗∗
Generic tools only	+0.036	[+0.022, +0.051]	473	∗∗∗
Length-residualized	+0.053	[+0.037, +0.071]	481	∗∗∗
Leave-one-task-out	+0.050	[+0.033, +0.066]	471	∗∗∗

Table 2. Main result and robustness checks. The within-unit adherence gap is the difference in canonical path adherence (Jaccard similarity) between successful and failed runs of the same model on the same task , averaged across all mixed-outcome units (model

\times

task pairs where outcomes vary across runs). A positive gap indicates that successful runs adhere more closely to the canonical tool set than failed runs within the same unit, isolating the causal effect of trajectory-level variation on success while holding model capability and task difficulty fixed. All specifications use paired within-unit comparisons; CF-LOO = cross-family leave-one-out canonical path definition.

{}^{***}p<0.001

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

ID	Specification	Estimate	95% CI	$N$	Sig.	What it tests
Canonical path definition
–	Naive (all models)	+0.082	[+0.059, +0.105]	505	∗∗∗	Baseline; circular by construction
R1	Leave-one-out (LOO)	+0.067	[+0.050, +0.084]	495	∗∗∗	Removes target model from path definition
R2	CF-LOO (headline)	+0.060	[+0.043, +0.077]	488	∗∗∗	Removes entire model family
Alternative explanations
R3	Generic tools only	+0.036	[+0.022, +0.051]	473	∗∗∗	Excludes domain-obvious tools

Appendix

Table 4. Full robustness and placebo specification table. For robustness specifications (–/R/R6 rows), the Statistic column reports the within-unit success

-

failure Jaccard adherence gap. For E1, it reports the minority first-tool-choice success rate (binomial test vs.

H_{0}{=}0.50

). For E2, it reports the within-unit adherence gap at that trajectory checkpoint. 95% CIs are bootstrap percentile intervals (5,000 draws, clustered by unit) for Jaccard gaps; exact binomial CI for E1.

{}^{***}p<0.001

{}^{**}p<0.01

{}^{*}p<0.05

, ns

p\geq 0.05

. E2 uses

n=454

units (vs.

n=488

headline) because the partial-trajectory analysis excludes trajectories with fewer than 4 tool calls and those lacking sufficient cross-family successful runs at each checkpoint. E1 uses

n=308

minority-tool observations from 222 units where first tool varies across runs.

Task	Str.	Eff.	$N$	Succ.
interview-report	0.838	+0.067	1	0.11
notion-find-job	0.798	+0.067	5	0.15
landing-task-reminder	0.771	+0.037	5	0.22
imagenet	0.763	-0.009	6	0.21
sla-timeout-monitor	0.752	+0.010	7	0.24
notion-hr	0.739	+0.050	7	0.29

Appendix

Table 6. Per-task statistics for the 67 tasks with mixed-outcome units and valid CF-LOO estimates (3 of the 70 mixed-outcome tasks lack sufficient cross-family successful runs to construct a CF-LOO canonical path and are excluded here). Tasks sorted by canonical path strength (descending). Str. = canonical strength (mean pairwise Jaccard among successful runs); Eff. = within-unit adherence effect (success

-

failure);

N

= mixed-outcome units; Succ. = overall success rate across all models. 54/67 tasks (80.6%) show positive effects.

Agent Failure Analysis

Agent failure analysis studies how and why AI agents make errors, traces causes through execution, and helps detect or prevent recurrence.

See all 153 papers

Mar 31, 2026

Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents

Existing benchmarks measure capability -- whether a model succeeds on a single attempt -- but production deployments require reliability -- consistent success across repeated attempts on tasks of varying duration. We show these properties diverge systematically as task duration grows, and that pass@1 on short tasks is structurally blind to this divergence. We introduce a reliability science framework for long-horizon LLM agents with four metrics: Reliability Decay Curve (RDC), Variance Amplification Factor (VAF), Graceful Degradation Score (GDS), and Meltdown Onset Point (MOP). We evaluate 10 models across 23,392 episodes on a 396-task benchmark spanning four duration buckets and three domains. Key findings: (1) reliability decay is domain-stratified -- SE GDS drops from 0.90 to 0.44 while document processing is nearly flat (0.74 to 0.71); (2) VAF bifurcates by capability tier -- high VAF is a capability signature, not an instability signal; (3) capability and reliability rankings diverge substantially, with multi-rank inversions at long horizons; (4) frontier models have the highest meltdown rates (up to 19%) because they attempt ambitious multi-step strategies that sometimes spiral; and (5) memory scaffolds universally hurt long-horizon performance across all 10 models. These results motivate reliability as a first-class evaluation dimension alongside capability.

Aaditya Khanal, Yangyang Tao, Junxiu Zhou

School of Computing and Analytics, Northern Kentucky University

LLM Reliability LLM Agent Reliability Long-Horizon Agent Evaluation Long-Horizon LLM Agents

Ask PDF

Jun 24, 2026

Beyond Function Calling: Benchmarking Tool-Using Agents under Tool-Environment Unreliability

Large language models are increasingly deployed as agents that solve tasks by interacting with external tool environments. Although recent tool-use benchmarks increasingly cover complex task settings, they still largely assume clean, stable, and trustworthy tool environments, leaving tool-environment unreliability insufficiently examined. We introduce ToolBench-X, a benchmark for evaluating agents under recoverable reliability hazards. ToolBench-X contains executable multi-step tasks across diverse domains and sequential, parallel, and mixed workflows, each paired with deterministic tools and a canonical final answer for automatic evaluation. Starting from clean tool environments, ToolBench-X injects five structured hazard types: Specification Drift, Invocation Error, Execution Failure, Output Drift, and Cross-source Conflict. Crucially, each injected instance remains solvable through at least one valid recovery path, such as retrying, fallback, verification, or cross-checking. Experiments reveal a substantial reliability gap: agents that perform well with reliable tools often fail under recoverable hazards. Further analysis shows that failures are driven less by tool-use volume or inference budget than by limited hazard diagnosis and ineffective recovery. Targeted recovery hints recover many failed tasks, while test-time scaling yields more limited gains. These results suggest that tool-use evaluation should move beyond function-call accuracy toward task completion under unreliable tool environments. The code and data is available at https://github.com/Foreverskyou/ToolBench-X.

Yang Tian, Zhengpeng Shi, Yu Zhou +1

School of AI, Shanghai Jiao Tong University1 · School of Cyber Science and Engineering, Southeast University2

AI Agent Reliability AI Agent Benchmarks Tool-Using Agents Tool Use LLM Agent Reliability

Ask PDF

Jul 7, 2026

Beyond the Leaderboard: A Synthesis of Tool-Use, Planning, and Reasoning Failures in Large Language Model Agents

Large language model (LLM) agents are increasingly evaluated on their ability to use tools, plan multi-step tasks, coordinate with other agents, and operate over extended horizons. Reported benchmark gains often obscure recurring failure modes documented across otherwise unrelated evaluation efforts. This paper synthesizes 27 benchmark, taxonomy, and audit papers (2023-2026), spanning 19 distinct benchmarks, into a cross-cutting taxonomy of agent limitations. To our knowledge, this is the first synthesis that integrates evidence across tool use, planning, long-horizon reasoning, multi-agent coordination, safety, and measurement validity into a single, unified taxonomy of LLM agent limitations. We identify six failure clusters: (1) tool invocation and parameter-level errors, (2) planning and constraint-satisfaction failures, (3) long-horizon degradation from context accumulation, (4) multi-agent coordination failures, (5) safety and security failures under adversarial or underspecified conditions, and (6) measurement validity problems. The taxonomy was derived iteratively by grouping independently reported error categories into themes corresponding to distinct stages of the agent reasoning-to-action pipeline. Across the literature, we find that failures compound nonlinearly with task length, that strong performance on individual sub-tasks does not reliably translate into end-to-end success, and that additional scaffolding does not consistently improve reliability. At the same time, substantial progress has been demonstrated in single-turn tool use, short-horizon web navigation, and narrowly scoped coding tasks.

Wael Albayaydh, Rui Zhao, Ivan Flechais

University of Oxford

AI Agent Reliability LLM Agent Evaluation LLM Agent Safety Long-Horizon LLM Agents Tool-Using Language Agents LLM Agent Planning+1

Ask PDF

Abstract

Agent Failure Analysis

Explore similar work

Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents

Beyond Function Calling: Benchmarking Tool-Using Agents under Tool-Environment Unreliability

Beyond the Leaderboard: A Synthesis of Tool-Use, Planning, and Reasoning Failures in Large Language Model Agents