Long-Horizon Agent Evaluation

Follow this topic Ask about this topic

This field studies whether agents can complete sustained, multi-step work—not just isolated prompts—in settings such as browser and GUI use, software maintenance, healthcare, business simulations, and robotics. Researchers build interactive environments with executable state checks, then assess progress through checkpoints, rubrics, and trajectory-level analysis of errors, safety, memory, and recovery. A central finding is that success on short tasks often fails to predict performance over longer workflows, where forgotten constraints, cascading errors, changing conditions, and weak verification can undermine results. A continuing challenge is making evaluations realistic and verifiable while separating model capability from the effects of the agent harness, tools, and evaluation protocol.

Latest papers

359 papers70 papers in the last four weeks, up 37% on the four weeks before. 0.4% of all new papers.

Efficient Agent Evaluation via Diversity-Guided User Simulation
Apr 23, 2026Itay Nakash, George Kour, Ateret Anaby-TavorAutomated Software Testing LLM Agent Evaluation
IBM Research
From Recall to Forgetting: Benchmarking Long-Term Memory for Personalized Agents
Apr 21, 2026Md Nayem Uddin, Kumar Shubham, Eduardo Blanco +2Long-Horizon Agent Evaluation AI Agent Benchmarks
Arizona State University · Genies · University of Arizona
Four-Axis Decision Alignment for Long-Horizon Enterprise AI Agents
Apr 21, 2026Vasundra SrininvasanLong-Horizon Agent Evaluation AI Agent Benchmarks
Stanford School of Engineering
SkillFlow:Benchmarking Lifelong Skill Discovery and Evolution for Autonomous Agents
Apr 19, 2026Ziao Zhang, Kou Shi, Shiting Huang +13Lifelong Learning Agents Long-Horizon Agent Evaluation
University of Science and Technology of China · University of Toronto · University of Sydney
MemEvoBench: Benchmarking Safety Risks from Memory Misevolution in LLM Agents
Apr 17, 2026Weiwei Xie, Shaoxiong Guo, Fan Zhang +5Long-Horizon Agent Evaluation AI Agent Security Benchmarks
Shanghai Jiao Tong University · East China Normal University · Shandong University +1
GTA-2: Benchmarking General Tool Agents from Atomic Tool-Use to Open-Ended Workflows
Apr 17, 2026Jize Wang, Xuanxuan Liu, Yining Li +7Tool-Use Evaluation Long-Horizon Agent Evaluation
1Shanghai Jiao Tong University · 2Shanghai AI Laboratory · 3Tencent +1
DR $^{3}$ -Eval: Towards Realistic and Reproducible Deep Research Evaluation
Apr 16, 2026Qianqian Xie, Qingheng Xiong, He Zhu +16LLM Agent Evaluation Long-Horizon Agent Evaluation
Nanjing University · M-A-P · Nanjing University of Science and Technology +2
HINTBench: Horizon-agent Intrinsic Non-attack Trajectory Benchmark
Apr 15, 2026Jiacheng Wang, Jinchang Hou, Fabian Wang +3AI Agent Auditing Long-Horizon Agent Evaluation
Baidu Inc., Beijing, China · Beijing Institute of Technology, Beijing, China
Evaluation of Agents under Simulated AI Marketplace Dynamics
Apr 15, 2026To Eun Kim, Alireza Salemi, Hamed Zamani +1AI Agent Evaluation IR Evaluation
Carnegie Mellon University, Pittsburgh, PA, United States · University of Massachusetts Amherst, Amherst, MA, United States
The Long-Horizon Task Mirage? Diagnosing Where and Why Agentic Systems Break
Apr 13, 2026Xinyu Jessica Wang, Haoyue Bai, Yiyou Sun +7Long-Horizon Agent Tasks Agent Failure Analysis
University of Wisconsin–Madison · University of California, Berkeley · Georgia Institute of Technology
CocoaBench: Evaluating Unified Digital Agents in the Wild
Apr 13, 2026CocoaBench Team, Shibo Hao, Zhining Zhang +29AI Agent Benchmarks Computer-Use Agent Evaluation
SEA-Eval: A Benchmark for Evaluating Self-Evolving Agents Beyond Episodic Assessment
Apr 10, 2026Sihang Jiang, Lipeng Ma, Zhonghua Hong +9Self-Evolving Agents AI Agent Benchmarks
Fudan University
Mastering PokeGym: Graph-Guided Multimodal Evolution at Test Time
Apr 9, 2026Ruizhi Zhang, Ye Huang, Yuangang Pan +5Game-Playing Agents Multimodal Agents
Shenzhen Institute for Advanced Study, University of Electronic Science and Technology of China · A*STAR · College of Computer Science and Software Engineering, Shenzhen University
Aligning Agents via Planning: A Benchmark for Trajectory-Level Reward Modeling
Apr 9, 2026Jiaxuan Wang, Yulan Hu, Wenjin Yang +3Reward Model Evaluation AI Agent Benchmarks
State Key Laboratory of Novel Software Technology, Nanjing University · School of Intelligence Science and Technology, Nanjing University · AMAP, Alibaba Group
Sustained Impact of Agentic Personalisation in Marketing: A Longitudinal Case Study
Apr 9, 2026Olivier Jeunen, Eleanor Hanna, Schaun WheelerHuman-in-the-Loop Evaluation Long-Horizon Agent Evaluation
aampe Antwerp, Belgium · aampe Raleigh, NC, USA · aampe Cary, NC, USA
CivBench: Progress-Based Evaluation for LLMs' Strategic Decision-Making in Civilization V
Apr 9, 2026John Chen, Sihan Cheng, Can Gurkan +1LLM Agent Evaluation AI Agent Benchmarks
Northwestern University
Gym-Anything: Turn any Software into an Agent Environment
Apr 7, 2026Pranjal Aggarwal, Graham Neubig, Sean WelleckComputer-Use Agents Computer-Use Agent Benchmarks
CMU
AgentCE-Bench: Agent Configurable Evaluation with Scalable Horizons and Controllable Difficulty under Lightweight Environments
Apr 7, 2026Wang Yang, Chaoda Song, Xinpeng Li +7AI Agent Benchmarks Agent Benchmarks
Case Western Reserve University · NII LLMC (Japan) · University of Zurich
GUIDE: Interpretable GUI Agent Evaluation via Hierarchical Diagnosis
Apr 6, 2026Yuwen Zhai, Runze Li, Liang Wang +6AI Agent Evaluation GUI Agents
Alibaba Group · School of Computer Science and Technology, East China Normal University · Shanghai Innovation Institute
Beyond Isolated Tasks: A Framework for Evaluating Coding Agents on Sequential Software Evolution
Apr 3, 2026KN Ajay Shastry, Ganesh Senrayan, Shrey Satapara +2Code Generation Evaluation Software Engineering
Fujitsu Research
ESL-Bench: An Event-Driven Synthetic Longitudinal Benchmark for Health Agents
Apr 3, 2026Chao Li, Cailiang Liu, Ang Gao +7Synthetic Clinical Data Generation AI Agent Benchmarks
Shanda Group
ATBench: A Diverse and Realistic Agent Trajectory Benchmark for Safety Evaluation and Diagnosis
Apr 2, 2026Yu Li, Haoyu Luo, Yuejin Xie +10AI Agent Safety Benchmarks LLM Agent Safety
Shanghai AI Lab · Fudan Univ. · Shanghai Jiao Tong Univ. +2
HippoCamp: Benchmarking Contextual Agents on Personal Computers
Apr 1, 2026Zhe Yang, Shulin Tian, Kairui Hu +9AI Agent Benchmarks Multimodal Search Agents
S-Lab, Nanyang Technological University, Singapore · Synvo AI
$\texttt{YC-Bench}$ : Benchmarking AI Agents for Long-Term Planning and Consistent Execution
Apr 1, 2026Muyu He, Adit Jain, Anand Kumar +4Long-Horizon Agent Evaluation AI Agent Benchmarks
When Users Change Their Mind: Evaluating Interruptible Agents in Long-Horizon Web Navigation
Apr 1, 2026Henry Peng Zou, Chunyu Miao, Wei-Chieh Huang +16LLM Agents LLM Agent Evaluation
University of Illinois Chicago · McGill University · MBZUAI +2
Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents
Mar 31, 2026Aaditya Khanal, Yangyang Tao, Junxiu ZhouLLM Reliability LLM Agent Reliability
School of Computing and Analytics, Northern Kentucky University
Multi-Layered Memory Architectures for LLM Agents: An Experimental Evaluation of Long-Term Context Retention
Mar 31, 2026Sunil Tiwari, Payal FofadiyaPersistent Memory for Language Models Long-Term Conversational Memory
Fulloop
From Intent to Evidence: A Categorical Approach for Structural Evaluation of Deep Research Agents
Mar 26, 2026Shuoling Liu, Zhiquan Tan, Kun Yi +6Category Theory Deep Research Agents
The Hong Kong University of Science and Technology · E Fund Management Co., Ltd. · The Hong Kong Polytechnic University
SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks
Mar 25, 2026Gabriel Orlanski, Devjeet Roy, Alexander Yun +7AI Coding Agents Coding Agent Benchmarks
University of Wisconsin–Madison · Washington State University · MIT
CarePilot: A Multi-Agent Framework for Long-Horizon Computer Task Automation in Healthcare
Mar 25, 2026Akash Ghosh, Tajamul Ashraf, Rishu Kumar Singh +4Computer-Use Agents Long-Horizon Agent Tasks
Indian Institute of Technology Patna · Mohamed bin Zayed University of AI (MBZUAI)

Apr 23, 2026J/K move · Enter open · S save

Efficient Agent Evaluation via Diversity-Guided User Simulation

Itay Nakash, George Kour, Ateret Anaby-Tavor

IBM Research

Automated Software Testing LLM Agent Evaluation Long-Horizon Agent Evaluation User Simulation LLM Agent Reliability

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly deployed as customer-facing agents, yet evaluating their reliability remains challenging due to stochastic, multi-turn interactions. Current evaluation protocols rely on linear Monte Carlo rollouts of complete agent-user conversations to estimate success. However, this approach is computationally inefficient, repeatedly regenerating identical early prefixes, and often fails to uncover deep failure modes that arise from rare user behaviors. We introduce DIVERT (Diversity-Induced Evaluation via Branching of Trajectories), an efficient, snapshot-based, coverage-guided user simulation framework for systematic exploration of agent-user interactions. DIVERT captures the full agent-environment state at critical decision points and resumes execution from these snapshots, enabling reuse of shared conversation prefixes and reducing redundant computation. From each junction, the framework branches using targeted, diversity-inducing user responses, allowing directed exploration of alternative interaction paths. By focusing evaluation on semantically diverse and underexplored trajectories, DIVERT improves both efficiency and coverage. Empirical results show that it discovers more failures per token compared to standard linear rollout protocols, while expanding the set of tasks on which failures are identified.