Long-Horizon Agent Evaluation

Overview

This field studies whether agents can complete sustained, multi-step work—not just isolated prompts—in settings such as browser and GUI use, software maintenance, healthcare, business simulations, and robotics. Researchers build interactive environments with executable state checks, then assess progress through checkpoints, rubrics, and trajectory-level analysis of errors, safety, memory, and recovery. A central finding is that success on short tasks often fails to predict performance over longer workflows, where forgotten constraints, cascading errors, changing conditions, and weak verification can undermine results. A continuing challenge is making evaluations realistic and verifiable while separating model capability from the effects of the agent harness, tools, and evaluation protocol.

Latest papers

CardsList

382 papers37 papers in the last four weeks, down 51% on the four weeks before. 0.3% of all new papers.

  1. Shopping Companion: Benchmarking and Training LLM Agents for Long-Horizon Preference-Grounded E-Commerce Tasks

    Mar 16, 2026Zijian Yu, Kejun Xiao, Huaipeng Zhao +2LLM Agent TrainingAI Agent Benchmarks

  2. AgentProcessBench: Diagnosing Step-Level Process Quality in Tool-Using Agents

    Mar 15, 2026Shengda Fan, Xuyan Ye, Yupeng Huo +9LLM Agent EvaluationProcess-Based Agent Evaluation

  3. EnterpriseOps-Gym: Environments and Evaluations for Stateful Agentic Planning and Tool Use in Enterprise Settings

    Mar 13, 2026Shiva Krishna Reddy Malay, Shravan Nayak, Jishnu Sethumadhavan Nair +6AI Agent BenchmarksLLM Agent Planning

  4. SWE-Milestone: Evaluating AI Agents on Continuous Software Evolution

    Mar 13, 2026Gangda Deng, Zhaoling Chen, Zhongming Yu +11Software MaintenanceSoftware Engineering

  5. LifeSim: Long-Horizon User Life Simulator for Personalized Assistant Evaluation

    Mar 12, 2026Feiyu Duan, Xuanjing Huang, Zhongyu WeiUser SimulationLong-Horizon Agent Evaluation

  6. Beyond Scaling: Assessing Strategic Reasoning and Rapid Decision-Making Capability of LLMs in Zero-sum Environments

    Mar 10, 2026Yang Li, Xing Chen, Yutao Liu +5AI Agent EvaluationAI Agent Benchmarks

  7. ττ-Knowledge: Evaluating Conversational Agents over Unstructured Knowledge

    Mar 4, 2026Quan Shi, Alexandra Zytek, Pedram Razavi +2LLM Agent EvaluationAgent Evaluation

  8. SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via Continuous Integration

    Mar 4, 2026Jialong Chen, Xander Xu, Hu Wei +2Software MaintenanceSoftware Engineering

  9. LifeBench: A Benchmark for Long-Horizon Multi-Source Memory

    Mar 4, 2026Zihao Cheng, Weixin Wang, Yu Zhao +15Agent MemoryMemory Agent Benchmarks

  10. EMPA: Evaluating Persona-Aligned Empathy as a Process

    Feb 28, 2026Shiya Zhang, Yuhan Zhan, Ruixi Su +4LLM Agent EvaluationMulti-Turn Dialogue Evaluation

  11. AgentLAB: Benchmarking LLM Agents against Long-Horizon Attacks

    Feb 18, 2026Tanqiu Jiang, Yuhui Wang, Jiacheng Liang +1LLM Agent SecurityAI Agent Security Benchmarks

  12. MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks

    Feb 18, 2026Zexue He, Yu Wang, Churan Zhi +11Agent MemoryLLM Agent Memory

  13. OdysseyArena: Benchmarking Large Language Models For Long-Horizon, Active and Inductive Interactions

    Feb 5, 2026Fangzhi Xu, Hang Yan, Qiushi Sun +14AI Agent BenchmarksLong-Horizon Agent Evaluation

  14. EMemBench: Interactive Benchmarking of Episodic Memory for VLM Agents

    Jan 23, 2026Xinze Li, Ziyue Zhu, Siyuan Liu +4Episodic MemoryLong-Horizon Agent Evaluation

  15. LifeAgentBench: Benchmarking LLMs for Long-Horizon, Cross-Dimensional Lifestyle Health Reasoning

    Jan 20, 2026Ye Tian, Zihao Wang, Onat Gungor +2HealthcareLong-Horizon Agent Evaluation

  16. SWITCH: Benchmarking Modeling and Handling of Tangible Interfaces in Long-horizon Embodied Scenarios

    Nov 20, 2025Juntao Cheng, Wanyue Zhang, Zhiwei Yu +7Long-Horizon Agent EvaluationEgocentric Video Understanding

  17. Lost in the Maze: Overcoming Context Limitations in Long-Horizon Agentic Search

    Oct 21, 2025Howard Yen, Yoonsang Lee, Ashwin Paranjape +5Agentic SearchLong-Horizon Agent Evaluation

  18. BuilderBench: The Building Blocks of Intelligent Agents

    Oct 7, 2025Raj Ghugare, Roger Creus Castanyer, Catherine Ji +4Long-Horizon Agent EvaluationRobot Manipulation Benchmarks

  19. HiMA-Ecom: A Long-Horizon Memory Benchmark for Hierarchical Multi-Agent E-commerce Assistants

    Jun 24, 2025Junxing Hu, Ai Han, Haolan Zhan +8Hierarchical MARLLong-Horizon Agent Evaluation