Long-Horizon Agent Evaluation

Overview

This field studies whether agents can complete sustained, multi-step work—not just isolated prompts—in settings such as browser and GUI use, software maintenance, healthcare, business simulations, and robotics. Researchers build interactive environments with executable state checks, then assess progress through checkpoints, rubrics, and trajectory-level analysis of errors, safety, memory, and recovery. A central finding is that success on short tasks often fails to predict performance over longer workflows, where forgotten constraints, cascading errors, changing conditions, and weak verification can undermine results. A continuing challenge is making evaluations realistic and verifiable while separating model capability from the effects of the agent harness, tools, and evaluation protocol.

Latest papers

CardsList

382 papers37 papers in the last four weeks, down 51% on the four weeks before. 0.3% of all new papers.

  1. 3D GameDev Arena: Benchmarking Coding Agents That Plan, Build, and Playtest extreme Long-Horizon Browser Games

    Nov 18, 2026Bizhe Bai, Jiliang Jian, Yuqing Yan +2Coding Agent BenchmarksLong-Horizon Agent Evaluation

  2. Metrics-in-the-Loop: Complexity Feedback Reshapes Agent-Written Code

    Nov 18, 2026Nicolas Chapados, Jean-François Gagné, Nouha DziriSoftware EngineeringSoftware Engineering Agents

  3. Stopping Is Not the Limit: Understanding and Eliciting Agent Capability in Long-Horizon Vulnerability Exploitation

    Nov 17, 2026Deokhyung Kang, Minjae Gwon, Sangdon ParkLLM AgentsAutomated Exploit Generation

  4. HospitalBench: Diagnosing Where and Why Embodied Agents Fail on Long Hospital Task Chains

    Nov 14, 2026Gaytri Jena, Kapil Wanaskar, Aman Chadha +3Long-Horizon Agent EvaluationEmbodied Agent Benchmarks

  5. OutfitterBench: Evaluating LLM Agents on Long-Horizon Resource Allocation and Commitment Coordination

    Nov 13, 2026Jie Huang, Yuecheng Hong, Yao Chen +3Long-Horizon Agent EvaluationLLM Agents

  6. Actor, Judge, or Teacher? Understanding Decision Models in Long-Horizon Agents

    Nov 12, 2026Haochen Liu, Qianhao Zhu, Zixuan Wang +7LLM Decision-MakingLLM-Guided RL

  7. Singularity-Bench: Can Agents Build Agents?

    Nov 12, 2026Yang Zhou, Krish Agarwal, Zhuoming Chen +16AI Agent BenchmarksLong-Horizon Agent Evaluation

  8. RiSE-Drive: Risk-Sensitive Long-Horizon Candidate Evaluation for End-to-End Autonomous Driving

    Nov 10, 2026Yu Zhao, Hongyi Duan, Ming Yin +4End-to-End Autonomous DrivingLong-Horizon Agent Evaluation

  9. Infrathon: Benchmarking Language Agents for Real-World, Long-Horizon AI Infra Optimization

    Nov 9, 2026Zhengyu Hu, Zheyuan Xiao, Yike Wang +18Long-Horizon Agent EvaluationAI Agent Benchmarks

  10. THEATRON: A Benchmark for Evaluating Persona Persistence in Long-Horizon LLM Decision-Making

    Nov 8, 2026Yuyan Wang, Yuechen Jiang, Zhiwei Liu +2LLM Decision-MakingBehavioral Consistency of LLM Agents

  11. DeepFinance: Can AI Diligence a $100MM Deal?

    Nov 7, 2026Drew Hahn, Tyler Shibata, Jackson Grove +1Financial Language Model EvaluationFinancial Reasoning Benchmarks

  12. Aligning, Fast and Slow: A Control-Theoretic Oversight Framework for Long-Horizon Trajectories and Maintained Deployments

    Nov 5, 2026Stephen Peng, Vineet Jagadeesan Nair, Noah Broestl +3AI Agent SafetyAI Agent Governance

  13. On Turn-Level Best-of-N Sampling for Scaling LLM Agents

    Nov 4, 2026Jalaj Bhandari, Dhawal Gupta, Hongli Zhan +5Inference-Time ScalingBest-of-N Sampling

  14. Towards Robust Long-Lived Agents: Benchmarking Experience Lifecycles in Evolving Environments

    Nov 3, 2026Shijue Huang, Hangyu Guo, Hengheng Zhang +4Experience Learning for LLM AgentsAI Agent Benchmarks

  15. Is Your Long-Horizon Benchmark Really Deep? Measuring and Synthesizing Depth in Terminal Tasks

    Nov 1, 2026Yu Jiang, Yue Huang, Hang Hua +6Terminal-Agent BenchmarksLong-Horizon Agent Evaluation

  16. FeatureArc: Benchmarking Coding Agents on Evolving Features

    Oct 27, 2026Yunlong Wang, Alyssa Wu, Cuiyun Gao +3Coding Agent BenchmarksSoftware Engineering

  17. LongSearchEval: Evaluating Long-Horizon Web Search Capabilities of Large Language Models via a Trajectory-Aware Agent Framework with Failure-Node Localization

    Oct 26, 2026Jialiang Liu, Shusheng Li, Wenjun TanLong-Horizon Agent EvaluationAutomated Agent Evaluation

  18. SlowLab: Benchmarking Autonomous AI Agents on a Year of Experiments They Cannot Undo

    Oct 26, 2026Ningpei Ding, Zai Wen FengScientific Agent BenchmarksAutonomous Experimentation

  19. SkillShift-Bench: Benchmarking Skill Learning under Environment Evolution for Web Agents

    Oct 24, 2026Bo-Yu Chen, Zhi Zhou, Yu-Feng LiAgent Skill LearningWeb Agent Benchmarks

  20. 4DMemory: Benchmarking Long-Horizon Spatiotemporal Memory with Harness-Augmented Agents

    Oct 23, 2026Yanhao Jia, Yi Zhu, Yiran Zhong +7Long-Horizon Agent EvaluationTemporal Reasoning in VLMs

  21. Compact-Bench: Measuring What Context Compaction Costs Long-Horizon Coding Agents

    Oct 22, 2026Ruopei Sun, Jianfeng Cai, Kangwen Zhao +2Context Compaction for LLM AgentsLLM Context Compression

  22. Why Bother Me Again? Benchmarking Personal Agent Memory for Interaction Preferences

    Oct 21, 2026Jialin Li, Zhenhao Chen, Hanjun Luo +1Memory Agent BenchmarksPersonal AI Agents