Long-Horizon Agent Evaluation

Follow this topic Ask about this topic

This field studies whether agents can complete sustained, multi-step work—not just isolated prompts—in settings such as browser and GUI use, software maintenance, healthcare, business simulations, and robotics. Researchers build interactive environments with executable state checks, then assess progress through checkpoints, rubrics, and trajectory-level analysis of errors, safety, memory, and recovery. A central finding is that success on short tasks often fails to predict performance over longer workflows, where forgotten constraints, cascading errors, changing conditions, and weak verification can undermine results. A continuing challenge is making evaluations realistic and verifiable while separating model capability from the effects of the agent harness, tools, and evaluation protocol.

Latest papers

359 papers70 papers in the last four weeks, up 37% on the four weeks before. 0.4% of all new papers.

Emergence World: A Platform for Evaluating Long-Horizon Multi-Agent Autonomy
Jun 6, 2026Deepak Akkil, Ravi Kokku, Karthik Vikram +3Multi-Agent LLM Systems Long-Horizon Agent Evaluation
Emergence AI
TRACE: Trajectory Reasoning through Adaptive Cross-Step Evidence Aggregation for LLM Agents
Jun 5, 2026Vijitha Mittapalli, Shreyaa Jayant Dani, Satya Srujana Pilli +7Long-Horizon Agent Evaluation AI Agent Security
University of Massachusetts at Amherst · Adobe Research · Dolby Labs +2
SWE-Marathon: Can Agents Autonomously Complete Ultra-Long-Horizon Software Work?
Jun 5, 2026Rishi Desai, Jesse Hu, Joan Cabezas +23AI Coding Agents Software Engineering Benchmarks
Abundant · Zenity · Harvard University +14
SubtleMemory: A Benchmark for Fine-Grained Relational Memory Discrimination in Long-Horizon AI Agents
Jun 4, 2026Wenxuan Wang, Haoyu Sun, Fukuan Hou +4Persistent Memory for Language Models Long-Horizon Agent Evaluation
Harbin Institute of Technology · Shanghai AI Laboratory · Tongji University +4
Agents' Last Exam
Jun 3, 2026Yiyou Sun, Xinyang Han, Weichen Zhang +376Computer-Use Agent Benchmarks Long-Horizon Agent Evaluation
SentinelBench: A Benchmark for Long-Running Monitoring Agents
Jun 3, 2026Matheus Kunzler Maldaner, Adam Fourney, Amanda Swearngin +5Computer-Use Agent Benchmarks Web Agent Benchmarks
University of Florida · Microsoft Research, AI Frontiers
AutoLab: Can Frontier Models Solve Long-Horizon Auto Research and Engineering Tasks?
Jun 3, 2026Zhangchen Xu, Junda Chen, Yue Huang +16Long-Horizon Agent Evaluation AI Agent Benchmarks
University of Washington · Bake AI · UCSD +11
Policy-Conditioned Counterfactual Credit for Verifiable Reinforcement Learning of Long-Horizon Language Agents
Jun 3, 2026Renwei MengCredit Assignment in RL Long-Horizon Agent Evaluation
Ahu
LifeSide: Benchmarking Agents as Lifelong Digital Companions
Jun 3, 2026Yuqian Wu, Zhijie Deng, Wei Chen +8Long-Horizon Agent Evaluation AI Companions
Hong Kong University of Science and Technology (Guangzhou) · Tencent · Hong Kong University of Science and Technology
ClinEnv: An Interactive Multi-Stage Long Horizon EHR Environment for Agents
Jun 1, 2026Yuxing Lu, Yushuhong Lin, Wenqi Shi +4Long-Horizon Agent Evaluation Clinical Decision-Making
1Georgia Institute of Technology · 2Peking University · University of Texas Southwestern Medical Center +1
RescueBench: Can Embodied Agents Save Lives in the Wild ?
Jun 1, 2026Kui Wu, Beiyu Guo, Hao Chen +6Agent Memory Embodied Navigation
1Beihang University · 5ATEC2025 Challenge Committee · 4City University of Macau +2
Momento: Evaluating Persistent Memory and Reasoning with Multi-Session Agentic Conversations
May 30, 2026Adril Putra Merin, David Anugraha, Ayu Purwarianti +1Agent Memory Temporal Reasoning
Institut Teknologi Bandung · Stanford University · Capital One
AgentOdyssey: Open-Ended Long-Horizon Text Game Generation for Test-Time Continual Learning Agents
May 29, 2026Zheyuan Zhang, Zehao Wen, Alvin Zhang +4Continual Test-Time Adaptation Long-Horizon Agent Evaluation
Johns Hopkins University
MineExplorer: Evaluating Open-World Exploration of MLLM Agents in Minecraft
May 29, 2026Tianjie Ju, Yueqing Sun, Zheng Wu +7Long-Horizon Agent Evaluation AI Agent Benchmarks
School of Computer Science, Shanghai Jiao Tong University · 2Meituan
LongDS-Bench: On the Failure of Long-Horizon Agentic Data Analysis
May 28, 2026Kewei Xu, Xiaoben Lu, Shuofei Qiao +4Long-Horizon Agent Evaluation Long-Horizon Agent Tasks
Zhejiang University · Zhejiang University - Ant Group Joint Laboratory of Knowledge Graph · Ant Group
Recovering Policy-Induced Errors: Benchmarking and Trajectory Synthesis for Robust GUI Agents
May 28, 2026Tianpeng Bu, Xin Liu, Qihua Chen +7GUI Agents Long-Horizon Agent Evaluation
Alibaba Cloud Computing.
WorldMemArena: Evaluating Multimodal Agent Memory Through Action-World Interaction
May 28, 2026Chengzhi Liu, Yuzhe Yang, Sophia Xiao Pu +14Multimodal Agent Evaluation Multimodal Memory
University of California, Santa Barbara · Johns Hopkins University · Carnegie Mellon University +3
DynSess: Dynamic Session-Level Evaluation and Optimization Framework for Role-Playing Agents
May 28, 2026Rongsheng Zhang, Jiji Tang, Junnan Ren +6Large Language Model-Based Role-Play Simulation Long-Horizon Agent Evaluation
1Zhejiang University · 2Fuxi AI Lab, NetEase Inc. · 3Xiamen University
Verifiable Benchmarking of Long-Horizon Spatial Biology
May 27, 2026Ian Diks, Harihara Muralidharan, Tim Proctor +1Spatial Transcriptomics Benchmark Design
LatchBio, San Francisco, CA, USA
VibeSearchBench: Benchmarking Long-horizon Proactive Search in the Wild
May 27, 2026Xiaohongshu IncAgentic Search Long-Horizon Agent Evaluation
Xiaohongshu Dots Studio & Unipat AI
AndroidDaily: A Verifiable Benchmark for Mobile GUI Agents on Real-World Closed-Source Applications
May 26, 2026Yifan Sui, Xin Huang, Hongbing Li +14Automated Evaluation Computer-Use Agent Benchmarks
1Beijing University of Posts and Telecommunications · 2StepFun · 3Waseda University
Benchmarks are Not Enough: RAMP for Runtime Assessing of Agentic Models in Production Systems
May 26, 2026Yipeng Ouyang, Xin Huang, Bingjie Liu +3Software Engineering Agents LLM Agent Evaluation
School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, China
VitaBench 2.0: Evaluating Personalized and Proactive Agents in Long-Term User Interactions
May 26, 2026Yuxin Chen, Yi Zhang, Zhengzhou Cai +11LLM Agent Memory AI Agent Evaluation
1National University of Singapore · 2Meituan · University of Science and Technology of China +2
Persistent AI Agents in Academic Research: A Single-Investigator Implementation Case Study
May 26, 2026Anas H. AlzahraniLong-Horizon Agent Evaluation AI Agent Governance
Department of Preventive Medicine and Public Health, Faculty of Medicine, King Abdulaziz University, Jeddah, Saudi Arabia
Anchor: Mitigating Artifact Drift in Agent Benchmark Generation
May 25, 2026Maksim Ivanov, Abhijay RanaComputer-Use Agent Benchmarks Long-Horizon Agent Evaluation
Agentic Labs New York, United States
EvoCode-Bench: Evaluating Coding Agents in Multi-Turn Iterative Interactions
May 22, 2026Haiyang Shen, Xuanzhong Chen, Wendong Xu +3AI Coding Agents Software Engineering Benchmarks
1UniPat AI · 2Peking University · 3Tsinghua University +1
Push Your Agent: Measuring and Enforcing Quantitative Goal Persistence in Long-Horizon LLM Agents
May 22, 2026Yuandao Cai, Yuzhang Zhu, Liyou Gao +2Long-Horizon Planning Long-Horizon Agent Evaluation
Independent Researcher · Xidian University
TerminalWorld: Benchmarking Agents on Real-World Terminal Tasks
May 21, 2026Zhaoyang Chu, Jiarui Hu, Xingyu Jiang +8Computer-Use Agent Benchmarks Terminal-Agent Benchmarks
University College London · Nanjing University · Tencent
SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agents
May 20, 2026Bingchen Zhao, Dhruv Srikanth, Yuxiang Wu +1Reward Hacking AI Coding Agents
Weco AI
Open-World Evaluations for Measuring Frontier AI Capabilities
May 19, 2026Sayash Kapoor, Peter Kirgis, Andrew Schwartz +15AI Agent Evaluation Long-Horizon Agent Evaluation
Princeton University · Cornflower Labs · Meridian Labs +10

Jun 6, 2026J/K move · Enter open · S save

Emergence World: A Platform for Evaluating Long-Horizon Multi-Agent Autonomy

Deepak Akkil, Ravi Kokku, Karthik Vikram, Tamer Abuelsaad+2

Emergence AI

Multi-Agent LLM Systems Long-Horizon Agent Evaluation Agent-Based Modeling AI Agent Governance Emergent Behavior in Multi-Agent Systems

Open paper AskPDF arXiv

Most evaluations of LLM agents look like exams: a discrete task, a clean environment, a score in minutes or hours. We argue that this approach is mismatched with the deployment conditions of autonomous systems, where the relevant timescale can be weeks to months, and where the dynamics that matter most, such as behavioral drift, governance in diverse environmental contexts, and cross-influence between agents from different model families, only emerge over time. We introduce Emergence World, a continuously running multi-agent simulation platform designed to make those dynamics measurable. The platform hosts populations of LLM-driven agents in a shared spatial world grounded in live external data (e.g. real-time weather, news APIs, internet access), equips each agent with 120+ specialized tools and three persistent memory systems, and lets them govern themselves through democratic mechanisms with consequential outcomes. The platform is model-agnostic at the reasoning layer and supports heterogeneous populations in which agents from different vendors share the same world. To illustrate the kinds of questions the platform makes tractable, we present a 15-day cross-vendor study with five parallel worlds powered by Claude Sonnet 4.6, Grok 4.1 Fast, Gemini 3 Flash, GPT-5-mini, and a mixed population. Identical roles and starting conditions produced radically different outcomes, ranging from stable deliberative governance to total population collapse. We release the prompts, log data and configurations to support further research on long-horizon multi-agent autonomy.