Long-Horizon Agent Evaluation

Follow this topic Ask about this topic

This field studies whether agents can complete sustained, multi-step work—not just isolated prompts—in settings such as browser and GUI use, software maintenance, healthcare, business simulations, and robotics. Researchers build interactive environments with executable state checks, then assess progress through checkpoints, rubrics, and trajectory-level analysis of errors, safety, memory, and recovery. A central finding is that success on short tasks often fails to predict performance over longer workflows, where forgotten constraints, cascading errors, changing conditions, and weak verification can undermine results. A continuing challenge is making evaluations realistic and verifiable while separating model capability from the effects of the agent harness, tools, and evaluation protocol.

Latest papers

359 papers70 papers in the last four weeks, up 37% on the four weeks before. 0.4% of all new papers.

CutVerse: A Compositional GUI Agents Benchmark for Media Post-Production Editing
May 19, 2026Haobo Hu, Xiangwu Guo, Zhiheng Chen +4GUI Agents Video Editing
MIPG, Communication University of China · National University of Singapore · USEIT AI
DecisionBench: A Benchmark for Emergent Delegation in Long-Horizon Agentic Workflows
May 18, 2026Yuxuan Gao, Megan Wang, Yi Ling Yu +2LLM Agent Orchestration Long-Horizon Agent Evaluation
1OpenMesh AI · University of Pennsylvania · 3Columbia University +2
MementoGUI: Learning Agentic Multimodal Memory Control for Long-Horizon GUI Agents
May 18, 2026Ziyun Zeng, Hang Hua, Bocheng Zou +3Multimodal Memory GUI Agents
University of Rochester · MIT-IBM Watson AI Lab · University of Wisconsin-Madison
MINTEval: Evaluating Memory under Multi-Target Interference in Long-Horizon Agent Systems
May 18, 2026Hyunji Lee, Justin Chih-Yao Chen, Joykirat Singh +3Agent Memory Long-Horizon Agent Evaluation
UNC Chapel Hill · The University of Texas at Austin
Remembering More, Risking More: Longitudinal Safety Risks in Memory-Equipped LLM Agents
May 18, 2026Ahmad Al-Tawaha, Shangding Gu, Peizhi Niu +2Long-Horizon Agent Evaluation LLM Safety Evaluation
Virginia Tech · University of California, Berkeley · University of Illinois Urbana-Champaign
SaaSBench: Exploring the Boundaries of Coding Agents in Long-Horizon Enterprise SaaS Engineering
May 17, 2026Qingnan Ren, Shun Zou, Shiting Huang +11Software Engineering AI Coding Agents
University of Science and Technology of China · 2AMAP, Alibaba Group
CHI-Bench: Can AI Agents Automate End-to-End, Long-Horizon, Policy-Rich Healthcare Workflows?
May 15, 2026Haolin Chen, Deon Metelski, Leon Qi +30Healthcare Long-Horizon Agent Evaluation
Johns Hopkins Medicine · Wellstar Health System · Stanford University +16
RoadmapBench: Evaluating Long-Horizon Agentic Software Development Across Version Upgrades
May 15, 2026Xinbo Xu, Ruihan Yang, Haiyang Shen +13AI Coding Agents Software Engineering Benchmarks
UniPat AI · Peking University · Fudan University +4
FutureSim: Replaying World Events to Evaluate Adaptive Agents
May 14, 2026Shashwat Goel, Nikhil Chandak, Arvindh Arun +5Continual Test-Time Adaptation Forecasting Benchmarks
1ELLIS Institute Tübingen · 2Max Planck Institute for Intelligent Systems · 4Tübingen AI Center +3
MemEye: A Visual-Centric Evaluation Framework for Multimodal Agent Memory
May 14, 2026Minghao Guo, Qingyue Jiao, Zeru Shi +14Multimodal Agent Evaluation Multimodal Memory
Rutgers · Notre Dame · Princeton +1
Holistic Evaluation and Failure Diagnosis of AI Agents
May 14, 2026Netta Madvil, Gilad Dym, Alon Mecilati +12Agent Failure Analysis AI Agent Evaluation
Deepchecks, Ramat Gan, Israel
$π$ -Bench: Evaluating Proactive Personal Assistant Agents in Long-Horizon Workflows
May 14, 2026Haoran Zhang, Luxin Xu, Zhilin Wang +11Long-Horizon Agent Evaluation AI Agent Benchmarks
Shanghai Jiao Tong University · Shanghai AI Laboratory · Fudan University +7
TwinRouterBench: Fast Static and Live Dynamic Evaluation for Realistic Agentic LLM Routing
May 14, 2026Pei Yang, Wanyi Chen, Tongyun Yang +15Benchmark Design LLM Agent Evaluation
Gradient · Soochow University · Independent Researcher +7
Cattle Trade: A Multi-Agent Benchmark for LLM Bluffing, Bidding, and Bargaining
May 14, 2026Robert Müller, Clemens MüllerMulti-Agent Negotiation Game-Playing Agents
When Robots Do the Chores: A Benchmark and Agent for Long-Horizon Household Task Execution
May 14, 2026Zilin Zhu, Longteng Guo, Yanghong Mei +5Long-Horizon Planning Long-Horizon Agent Evaluation
Institute of Automation, Chinese Academy of Sciences, Beijing, China · Zhongguancun Academy, Beijing, China · University of Chinese Academy of Sciences, Beijing, China
Herculean: An Agentic Benchmark for Financial Intelligence
May 14, 2026Xueqing Peng, Zhuohan Xie, Yupeng Cao +60AI Agent Benchmarks LLM Agent Evaluation
Why We Need World Models for AGI: Where LLMs Fail and How World Models May Outperform
May 13, 2026Feisal Alaswad, Batoul Aljaddouh, Maher Alrahhal +2Latent Dynamics Modeling Long-Horizon Agent Evaluation
Department of Computing Technologies SRM Institute of Science and Technology Kattankulathur 603203, Tamil Nadu, India · Bio-Sensing and Bio-Sensors Group Smart Automation and Communication Technologies Research Institute of Sciences and Engineering University of Sharjah, UAE · Department of Computer Engineering College of Computing and Informatics University of Sharjah, UAE
Ego2World: Compiling Egocentric Cooking Videos into Executable Worlds for Belief-State Planning
May 13, 2026Qinchuan Cheng, Zhantao Gong, Pengzhan Sun +3Belief-Space Planning Long-Horizon Agent Evaluation
Xi’an Jiaotong University · Nankai University · National University of Singapore +1
MedMemoryBench: Benchmarking Agent Memory in Personalized Healthcare
May 12, 2026Yihao Wang, Haoran Xu, Renjie Gu +10Agent Memory Healthcare
Zhejiang University · Ant Group · Alibaba Group +1
PRISM: : Planning and Reasoning with Intent in Simulated Embodied Environments
May 12, 2026Yunn Kang Lim, Pengzhan Sun, Ziyi Bai +4Long-Horizon Agent Evaluation AI Agent Benchmarks
1A*STAR · 2National University of Singapore · 3BAAI
WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation
May 11, 2026Shuangrui Ding, Xuanlang Dai, Long Xing +14LLM Agent Harnesses AI Agent Benchmarks
1Shanghai AI Laboratory, 2The Chinese University of Hong Kong · 1Shanghai AI Laboratory, 3Fudan University · 1Shanghai AI Laboratory, 4University of Science and Technology of China +8
Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?
May 8, 2026Anmol Gulati, Hariom Gupta, Elias Lumer +2LLM Agent Evaluation Long-Horizon Agent Evaluation
PricewaterhouseCoopers U.S.
AgentEscapeBench: Evaluating Out-of-Domain Tool-Grounded Reasoning in LLM Agents
May 8, 2026Zhengkang Guo, Yiyang Li, Lin Qiu +7LLM Agent Evaluation Long-Horizon Agent Evaluation
1Fudan University · 2Meituan Longcat Team
Stop Comparing LLM Agents Without Disclosing the Harness
May 7, 2026Yunbei Zhang, Janet Wang, Yingqiang Ge +3LLM Agent Evaluation Long-Horizon Agent Evaluation
Tulane University · Rutgers University · Independent Researcher +1
Agentic Coding Needs Proactivity, Not Just Autonomy
May 7, 2026Nghi D. Q. Bui, Georgios EvangelopoulosProactive Agents AI Coding Agents
Google Labs
PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments
May 4, 2026Ruoqi Liu, Imran Q. Mohiuddin, Austin J. Schoeffler +10AI Agent Benchmarks Electronic Health Records
Stanford University
AgentFloor: How Far Up the tool use Ladder Can Small Open-Weight Models Go?
May 1, 2026Ranit Karmakar, Jayita ChatterjeeComputer-Use Agent Benchmarks LLM Agent Evaluation
Harvard University, Boston, MA 02115 · Boston, MA 02115
KellyBench: A Benchmark for Long-Horizon Sequential Decision Making
Apr 30, 2026Thomas Grady, Kip Parker, Iliyan Zarov +3Sequential Decision Making Long-Horizon Agent Evaluation
General Reasoning, Inc.
Odysseys: Benchmarking Web Agents on Realistic Long Horizon Tasks
Apr 27, 2026Lawrence Keunho Jang, Jing Yu Koh, Daniel Fried +1Computer-Use Agent Benchmarks Web Agent Benchmarks
Carnegie Mellon University
ClawMark: A Living-World Benchmark for Multi-Turn, Multi-Day, Multimodal Coworker Agents
Apr 26, 2026Fanqing Meng, Lingxiao Du, Zijian Wu +46Computer-Use Agent Benchmarks LLM Agent Evaluation

May 19, 2026J/K move · Enter open · S save

CutVerse: A Compositional GUI Agents Benchmark for Media Post-Production Editing

Haobo Hu, Xiangwu Guo, Zhiheng Chen, Difei Gao+3

MIPG, Communication University of China · National University of Singapore · USEIT AI

GUI Agents Video Editing Long-Horizon Agent Evaluation AI Agent Benchmarks

Open paper AskPDF arXiv

While GUI agents have made significant progress in web navigation and basic operating system tasks, their capabilities in professional creative workflows remain largely underexplored. To bridge this gap, we introduce Cutverse, a benchmark designed to systematically evaluate autonomous GUI agents in realistic media post-production environments. We curate expert demonstrations across 7 professional applications (e.g., Premiere Pro, Photoshop), covering 186 complex, long-horizon tasks grounded in authentic editing workflows, involving dense multimodal interfaces and tightly coupled interaction sequences. To support scalable evaluation, we develop a lightweight parser that transforms raw screen recordings and low-level interaction logs into structured, compositional GUI action trajectories with precise grounding. Extensive evaluations reveal that existing agents achieve only 36.0% task success on realistic media editing tasks, underscoring the challenges posed by complex, long-horizon media post-production workflows in our benchmark.While current models demonstrate promising spatial grounding, multimodal alignment, and coordinated action execution, they remain limited in long-horizon reliability and domain-specific planning.