cs.AIApr 13, 2026

The Long-Horizon Task Mirage? Diagnosing Where and Why Agentic Systems Break

Authors: Xinyu Jessica Wang, Haoyue Bai, Yiyou Sun, Haorui Wang, Shuibai Zhang, Wenjie Hu, Mya Schroder, Bilge Mutlu, 

Organizations: University of Wisconsin–Madison · University of California, Berkeley · Georgia Institute of Technology

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Large language model (LLM) agents perform strongly on short- and mid-horizon tasks, but often break down on long-horizon tasks that require extended, interdependent action sequences. Despite rapid progress in agentic systems, these long-horizon failures remain poorly characterized, hindering principled diagnosis and comparison across domains. To address this gap, we introduce HORIZON, an initial cross-domain diagnostic benchmark for systematically constructing tasks and analyzing long-horizon failure behaviors in LLM-based agents. Using HORIZON, we evaluate state-of-the-art (SOTA) agents from multiple model families (GPT-5 variants and Claude models), collecting 3100+ trajectories across four representative agentic domains to study horizon-dependent degradation patterns. We further propose a trajectory-grounded LLM-as-a-Judge pipeline for scalable and reproducible failure attribution, and validate it with human annotation on trajectories, achieving strong agreement (inter-annotator κ=0.61; human-judge κ=0.84). Our findings offer an initial methodological step toward systematic, cross-domain analysis of long-horizon agent failures and offer practical guidance for building more reliable long-horizon agents. We release our project website at HORIZON Leaderboard and welcome contributions from the community.

Figures & tables

Figure 1: Illustration of general agent execution and failure propagation. Given an instruction 1 , the agent iterates a standard loop: it observes the environment 2 to obtain observations 3 , plans and selects an action 4 drawing on memory-backed context 6 , executes that action to change the environment, and then updates its internal state 5 through memory 6 . Failures can originate at any stage and compound across cycles 7 . The seven legend boxes map directly to the failure categories in Section 3.2 .
Figure 2: The HORIZON diagnostic pipeline for scalable long-horizon failure analysis. The pipeline consists of trace and context collection, taxonomy development with inter-annotator agreement, construction of the HORIZON diagnostic suite (unified horizon measurement and failure attribution), calibration of an LLM-based annotator, and large-scale failure annotation. This figure highlights our systematic approach to approximating the breaking-point problem in long-horizon agents, enabling scalable and reproducible analysis for future cross-domain research. More details are provided in Section 5 .
CategoryWebOSEmbodiedDB
Environment [S] (Environment Disturbance & Unable to Detect Environment Change)After a slow page load with a transient pop-up banner, the agent assumes the page navigation has completed and believes it is on the profile page (“Mark Johnson”), while the UI is actually still on a different profile (“mike_chen”), resulting in actions applied to an incorrect environment state .A shell command fails due to permission denial , but the agent fails to detect the error signal and believes the environment state has changed , continuing to issue follow-up commands that depend on the nonexistent state change.In a simulator with stochastic dynamics , a robot repeatedly drops a block due to random gripper failure, but the agent fails to re-observe the changed environment state , causing repeated execution failures despite correct high-level reasoning.After a database schema change renames columns or adds tables, the agent continues generating SQL based on an outdated schema snapshot because it fails to re-check the current schema , producing invalid queries or silently incorrect results.
Instruction [S] (Ill-defined Instruction & Partially Understand Instruction)An agent is asked to retrieve a specific product field that is not defined on the target website , meaning the instruction requests a non-existent attribute . Instead of detecting the mismatch and asking for clarification, the agent fabricates a plausible value and proceeds.The agent is instructed to change file permissions “except those of mine,” where the instruction contains an ambiguous ownership constraint . The agent misinterprets it and applies a recursive chmod globally.Given the instruction “put the block in the mismatching bowl,” the agent grasps the block but fails to interpret the key constraint “mismatching” and places it into a matching-colored bowl.When asked to “show churned users,” the instruction leaves the definition of churn unspecified . The agent therefore applies a naïve definition (no login in 30 days) instead of the organization’s actual churn definition.
False Assumption [S]When a contact form requires identity fields and the instruction says “do not invent personal info,” the agent still enters a made-up name and email because it assumes placeholder personal information is acceptable .The agent assumes certain files are owned by the user and excludes them from permission changes without checking ownership metadata.In a mismatching-bowl task, the agent assumes the goal is the common “matching” variant and places a red block into a red bowl despite the instruction requiring a mismatching placement.A SQL-generation agent assumes every table contains a created_at column and repeatedly invents similar column names in WHERE clauses.
Planning Error [S] (Sub-plan & Action)While planning the next step to find the highest-upvoted answer, the agent fails to switch sorting from “New” to “Top” , then cites a recently seen low-upvote comment as the “highest-upvoted” evidence.When planning how to update permissions, the agent chooses an incorrect action plan and directly applies a blanket chmod instead of first identifying eligible files.While decomposing the task into steps, the agent generates an incorrect sub-plan and attempts to stack blocks before clearing the base area , producing an invalid task order.During database migration planning, the agent executes an incorrect action sequence and drops the backup before completing data transfer , leaving no recovery path when the migration fails.
Catastrophic Forgetting [L]The agent initially applies the “Condition: New” filter but later adds a “Renewed” item to the cart after a long navigation sequence because the “new-only” constraint is still present in the context but not attended to during later reasoning .Early in a task, the agent is instructed not to modify sensitive system files, but later generates and executes a script that deletes /root/.bashrc even though the original safety instruction is still in the context but ignored during execution .In Minecraft, an agent that has learned to craft wood and stone tools fails on advanced “Iron+” tasks because it does not reuse earlier knowledge that is still present in the context , leading to failure in later crafting stages.After the user specifies “never touch the payments schema,” a later autonomous cleanup step proposes dropping an index on the payments tables because the constraint remains in the interaction history but is overlooked during later planning .
History Error Accumulation [L]After an element click fails, the agent repeats the same failed action across multiple steps , allowing a small initial mistake to accumulate into repeated ineffective actions .After issuing an invalid command that fails silently, the agent treats the failed step as successful and continues executing dependent commands, causing the early incorrect state to propagate through the workflow .After selecting an incorrect initial sub-task, the agent continues executing downstream steps based on that incorrect earlier decision , gradually amplifying the initial mistake .An early intermediate table omits the filter is_deleted = false , and all downstream analytics reuse that table, causing the small upstream error to compound into systematically inflated results .
Table 1: Representative failure cases across domains and failure categories. Each row corresponds to a failure category in our taxonomy, and each column shows a concrete example from a different domain—Web, OS, Embodied, and Database. [L] denotes failure categories that are predominantly long-horizon-specific (rare in short-horizon settings), while [S] denotes categories observed in both short- and long-horizon tasks but often amplified under longer horizons. Details on taxonomy development are provided in Section 3.2 , and additional examples and explanations are provided in Appendix B .
Figure 3: Current model performance as a function of horizon extension. Plots show success rate (accuracy) versus the compositional depth ss , where ss denotes the HORIZON -defined extension level corresponding to the number of high-level subtasks composed within a task. Accuracy is computed over task sets with the same ss . Each point reports the mean over three independent runs with identical task sets; variability across runs is shown as ±\pm one standard deviation (error bars). We evaluate GPT-5-mini and Claude-4-Sonnet across all domains. Across models and domains, performance consistently decreases and exhibits a sharp drop beyond a certain value of ss , empirically supporting the validity of our horizontal horizon definition. More details are provided in Section 4 .
Figure 4: Distribution of failure modes across four task domains on 3100+ failure trajectories. Bars show the proportion of each failure type relative to the total failed traces within each domain. Failures are grouped into Process-level Risks (PFMEA) (72.5%), comprising failures that arise during sequential rollout—environment interaction, instruction following, planning errors, and history error accumulation—and Design-level Risks (DFMEA) (27.5%), reflecting engineered-agent limitations—memory constraints, catastrophic forgetting, and false assumptions.
Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: HORIZON overview with two orthogonal dimensions. Left (Horizontal / Horizon): four domain examples contrasting short- vs. long-horizon tasks under our task-structure definition, where intrinsic horizon H∗H^{*} increases with extension level ss (and may also increase compositional depth CC ). From top-left to bottom-right: Web illustrates theoretical H∗H^{*} growth (e.g., “purchase a flight,” H∗=8H^{*}=8 , vs. “check the weather,” H∗=3H^{*}=3 ); OS shows depth extension by inserting non-skippable intermediate states (e.g., enforce and verify read-only before making a.log writable); Database and Embodied show breadth/length extension by composing multiple short-horizon subtasks into one workflow, increasing number of subtasks and coordination steps. Right (Failure Attribution): 7-category failure taxonomy. Colors indicate a hierarchical risk perspective inspired by FMEA: blue corresponds to design-level risks (DFMEA), originating from limitations in agent architecture, while orange corresponds to process-level risks (PFMEA), emerging during sequential execution. More details are provided in Section 3 .
CategoryRepresentative prior works (3)Typical mechanism (from review notes)
EnvironmentZhou et al. (2024a) ; Chen et al. (2025b) ; Koh et al. (2024)Environment fidelity, transitions, or multimodal observation mismatch.
InstructionTrivedi et al. (2024) ; Chen et al. (2024) ; Wang et al. (2025b)Ambiguous goals, missing constraints, or mis-specified task requirements.
Catastrophic ForgettingSinha et al. (2025) ; Li et al. (2024) ; Feng et al. ()Loss of long-range constraints or competencies as trajectories lengthen.
False AssumptionZhang et al. (2025) ; Yang et al. (2024) ; Wei et al. (2025)Hallucinated facts, unfounded shortcuts, or misaligned beliefs about state.
Planning ErrorErdogan et al. (2025) ; Si et al. (2025) ; Gonzalez-Pumariega et al. (2025)Subgoal decomposition, ordering, or plan repair under dynamics.
History Error AccumulationWang et al. (2025c) ; Liu et al. (2023c) ; Xi et al. (2025b)Multi-step error propagation, unstable rollouts, or interaction loops.
Appendix
Table 2: Representative prior work mapped to each failure category. Entries are drawn from our literature spreadsheet and selected for cross-domain coverage; full paper lists and extracted notes remain in the supplementary spreadsheet.
Figure 6: Failure mode distribution across the four task domains (SR = task success rate; nn = number of failed traces). Embodied (SR 42.2%) and Database (SR 36.9%) are almost entirely dominated by Planning Error (94.9% and 79.3%, respectively), indicating that structured, well-defined action spaces surface planning as the primary bottleneck. Web (SR 24.1%) follows a similar pattern (Planning Error 74.9%) but additionally exhibits Environment (11.3%) and Memory Limitation (6.2%) failures, reflecting dynamic page-state variability and multi-tab navigation demands. OS (SR 40.3%) has by far the most diverse failure profile: Planning Error (36.7%), Instruction (25.9%), Environment (17.3%), and Memory Limitation (15.1%) all contribute substantially, consistent with the open-ended, long-horizon nature of shell tasks. History Error Accumulation appears exclusively in OS (0.1%).
Figure 7: Failure mode distribution for GPT-4o-mini vs. Claude 3.5 Sonnet aggregated across all domains. The two models exhibit qualitatively different failure regimes. GPT-4o-mini (1,145 failed traces, success rate 33.4%) is dominated by Planning Error (64.9%) and Memory Limitation (18.3%), reflecting difficulty maintaining coherent long-horizon plans and retaining intermediate state across extended rollouts. Claude 3.5 Sonnet (850 failed traces, success rate 39.9%) shows markedly higher Environment (32.5%) and Instruction (16.5%) failures while Memory Limitation is near-zero (2.2%), suggesting stronger context retention but greater sensitivity to environment-state changes and task-specification ambiguity. Design-level risks (Memory Limitation, Catastrophic Forgetting, False Assumption) account for 20.8% of GPT failures but only 6.6% of Claude failures.
Figure 8: Failure mode distribution broken down by domain and model. Database and Embodied only have GPT trajectories in the current dataset. Web: Both models are planning-error dominated ( >> 72%), but GPT accumulates more Memory Limitation (10.6% vs. 2.0%) while Claude shows higher False Assumption (5.4% vs. 1.4%), pointing to distinct secondary failure mechanisms. OS: The model divergence is most pronounced here. GPT failures concentrate in Planning Error (57.7%) and Memory Limitation (24.6%), consistent with context-length degradation over long command chains. Claude failures are instead driven by Environment (44.3%) and Instruction (23.4%), indicating sensitivity to shell-environment feedback and command-specification ambiguity. Database: GPT shows Planning Error (79.3%) and False Assumption (7.5%) as the main failure modes, with no Memory Limitation. Embodied: Nearly all failures are Planning Errors (94.9%), reflecting the precise sequential action ordering demands of embodied tasks.
More in this topic

Long-Horizon Agent Tasks

Long-Horizon Agent Tasks study how AI agents plan, remember, and act reliably across extended, multi-step interactions.

See all 192 papers

Explore similar work

CardsList
Aug 7, 2026

The Horizon Gap: Planning, Memory, Execution, Training, and Evaluation for Long-Horizon LLM Agents

Frontier language models solve reasoning problems in a single forward pass that would have been research contributions years ago, yet fail at multi-hour tasks: losing track of earlier decisions, declaring half-finished work done, or drifting from goals. We call this the horizon gap and survey 1,547 arXiv papers (2024-2026) collected via systematic seed harvest with a disclosed 26.8% bleed filter, extended by targeted supplementation. We disambiguate three routinely conflated properties: long-horizon (task property: required steps), long-context (model property: token capacity), and long-term memory (system property: persistence across steps/sessions). We organize the corpus into six categories tracking a long-horizon task's lifecycle -- planning, memory, execution, training, evaluation, and foundations/safety -- crossed with an axis capturing where horizons are carried (within-context, within-task-beyond-context, or cross-task-persistent). Across all categories, we find the same pattern: outcome-only signals grow uninformative as horizons lengthen, and the field's response -- whether process reward models, credit assignment, or trajectory-level diagnostics -- manufactures denser step-level signals. We treat critical and diagnostic literature as first-class threads throughout, arguing that segregating critique from method would routinely split single papers across chapters. We close by naming open measurement problems: decomposing model versus harness capability, managing correlated bias in process-level signals used for both training and evaluation, and whether long-horizon reliability admits general predictive theory.

Mingguang Chen, Licheng Wang, Bo Qu

1DeepGrounding · 2AlphaAvatar

Memory-Augmented Language Model AgentsLong-Horizon PlanningLong-Horizon Agent EvaluationLong-Horizon Agent TasksLong-Horizon LLM AgentsLong-Horizon LLM Reasoning+1
AskPDF
Aug 7, 2026

Long-Horizon Agent Trajectory Attribution: A Unified Benchmark and Fine-Grained Annotation Framework

Large language model (LLM) agents increasingly operate through long-horizon trajectories involving user instructions, tool use, external observations, and memory. Existing benchmarks primarily evaluate behavioral outcomes but provide limited support for fine-grained attribution analysis. We introduce trajectory attribution and develop a benchmark and annotation framework for this task. The benchmark organizes heterogeneous trajectories under a unified component schema and provides annotations of the primary attribution component, together with attack and execution chains where applicable. Instantiating the benchmark with trajectories from AgentDojo and the Stage and Canary settings of Agent3Sigma yields more than 1,300 annotated trajectories covering task-aligned actions, unsafe actions, and safety refusals. The benchmark defines two evaluation tasks, primary attribution localization and attribution-chain recovery, and provides reference baselines based on incremental trajectory contribution and component-level leave-one-out perturbation. It captures diverse attribution settings, including local and long-range attribution as well as structured attribution chains. Reference baseline results exhibit substantial performance differences across these settings, providing an initial characterization of the benchmark's attribution challenges. Beyond this initial instantiation, we release a reusable annotation skill that enables trajectories generated by new agent models to be standardized, annotated, and evaluated under the same framework. Project resources and future releases are available at https://github.com/chenjing-2024/agent-trajectory-attribution.

Jing Chen, Yang Sun, Li Zhang +2

Huawei Technologies Ltd.

Long-Horizon Agent EvaluationAI Agent BenchmarksAgent Trajectory Analysis
AskPDF
Aug 31, 2026

How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making

Production deployments of large language model (LLM) agents remain unreliable on long, multi-step workflows even as benchmark success rates climb steadily. We argue this gap is largely an artifact of task horizon: benchmarks are dominated by short-to-medium horizons where success remains high, while production workloads demand an order of magnitude more dependent steps. We measure the effect directly, characterizing the shape of agent degradation and disentangling its cause across a large controlled study spanning nine models, six open models from 1.2B to 671B parameters, and three deployed proprietary systems; four task families, including a genuinely agentic tool-use loop; five horizons; and three context regimes. Task success follows a geometric law governed by a single per-step reliability parameter, which rises with model scale but saturates well below 1 even for the strongest models, guaranteeing eventual collapse at sufficiently long horizons. The effect is sharpest on the agentic task, where every model tested, including widely deployed systems, falls from near-perfect success to near zero within sixteen steps of (n=10,664 analyzed trajectories. Degradation is driven by step count rather than context length: bounding the context window steepens decay rather than easing it (logit slope -0.69 vs. -0.44), p=3x10-6), contradicting a lost-in-the-middle explanation and warning against a common production shortcut. Projecting measured reliability onto representative benchmark horizons quantifies a substantial gap between benchmark and production conditions, from 0.42 at GAIA-length horizons to 0.24 at hundred-step production horizons. For teams responsible for agent orchestration and reliability at scale, these results argue for horizon-aware evaluation and reliability budgeting in place of aggregate pass-rate metrics. Code, prompts, seeds, and raw trajectories are released.

Shubhra Mittal

Independent Researcher; Principal Software Engineer, Microsoft AI

Long-Horizon Agent TasksLLM Agent ReliabilityLong-Horizon Agent Evaluation
AskPDF