Long-Horizon Agent Evaluation

Follow this topic Ask about this topic

This field studies whether agents can complete sustained, multi-step work—not just isolated prompts—in settings such as browser and GUI use, software maintenance, healthcare, business simulations, and robotics. Researchers build interactive environments with executable state checks, then assess progress through checkpoints, rubrics, and trajectory-level analysis of errors, safety, memory, and recovery. A central finding is that success on short tasks often fails to predict performance over longer workflows, where forgotten constraints, cascading errors, changing conditions, and weak verification can undermine results. A continuing challenge is making evaluations realistic and verifiable while separating model capability from the effects of the agent harness, tools, and evaluation protocol.

Latest papers

359 papers70 papers in the last four weeks, up 37% on the four weeks before. 0.4% of all new papers.

FeatureArc: Benchmarking Coding Agents on Evolving Features
Oct 27, 2026Yunlong Wang, Alyssa Wu, Cuiyun Gao +3Coding Agent Benchmarks Software Engineering
Hunyuan Team, Tencent · The Chinese University of Hong Kong
LongSearchEval: Evaluating Long-Horizon Web Search Capabilities of Large Language Models via a Trajectory-Aware Agent Framework with Failure-Node Localization
Oct 26, 2026Jialiang Liu, Shusheng Li, Wenjun TanLong-Horizon Agent Evaluation Automated Agent Evaluation
College of Information Science and Engineering, Northeastern University, China · Key Laboratory of Intelligent Computing of Medical Images, Ministry of Education, Northeastern University, China. · School of Computer Science and Engineering, Northeastern University, Shenyang 110819, China +1
SlowLab: Benchmarking Autonomous AI Agents on a Year of Experiments They Cannot Undo
Oct 26, 2026Ningpei Ding, Zai Wen FengScientific Agent Benchmarks Autonomous Experimentation
College of Informatics, Huazhong Agricultural University
Continual Learning Exam: Benchmarking Long-Horizon RSI and Experiential Learning in LLM Agents
Oct 25, 2026Haoyu Zhao, Sitian Tu, Yuxuan Huang +16Continual Learning for LLM Agents Long-Horizon Agent Evaluation
SkillShift-Bench: Benchmarking Skill Learning under Environment Evolution for Web Agents
Oct 24, 2026Bo-Yu Chen, Zhi Zhou, Yu-Feng LiAgent Skill Learning Web Agent Benchmarks
School of Artificial Intelligence, Nanjing University · National Key Laboratory for Novel Software Technology, Nanjing University
4DMemory: Benchmarking Long-Horizon Spatiotemporal Memory with Harness-Augmented Agents
Oct 23, 2026Yanhao Jia, Yi Zhu, Yiran Zhong +7Long-Horizon Agent Evaluation Temporal Reasoning in VLMs
Nanyang Technological University · Token Foundry, Alibaba · The Chinese University of Hong Kong +2
Compact-Bench: Measuring What Context Compaction Costs Long-Horizon Coding Agents
Oct 22, 2026Ruopei Sun, Jianfeng Cai, Kangwen Zhao +2Context Compaction for LLM Agents LLM Context Compression
University of Science and Technology of China
Why Bother Me Again? Benchmarking Personal Agent Memory for Interaction Preferences
Oct 21, 2026Jialin Li, Zhenhao Chen, Hanjun Luo +1Memory Agent Benchmarks Personal AI Agents
New York University Abu Dhabi, Abu Dhabi, UAE
TARSphere: A Robot-Oriented Full-Stack Framework for Long-Horizon Embodied Agents in Evolving Physical and Social Environments
Oct 21, 2026Yicheng Xu, Zhikai Xu, Ziyuan Chen +5Long-Horizon Agent Evaluation Embodied Agent Benchmarks
Zhejiang University · Fudan University
Benchmarking and Error Analysis of Deep Search and Deep Research LLM Agents
Oct 21, 2026Zixuan Yu, Zhenheng Tang, Xiaowen Chu +1LLM Agent Reliability Process-Based Agent Evaluation
School of Computer science and engineering, Sun Yat-sen University · CSE, The Hong Kong University of Science and Technology · DSA Thrust, The Hong Kong University of Science and Technology (GuangZhou) +1
Automating ML for Science: Can Frontier Agents Climb Scientific Hills in the Wild?
Oct 20, 2026Ming Zhong, Stacy Li, Nicholas Carlini +1AI Agents for Scientific Discovery Scientific Agent Benchmarks
Anthropic Fellows Program · Anthropic
Lost in Recovery: Demystifying Recovery Behavior of Computer-Use Agents via Goal-Oriented Action Replay
Oct 20, 2026Byeonghun Bang, Sangwoo Park, Jinheon Baek +3Computer-Use Agents Computer-Use Agent Evaluation
KAIST · ETRI
CUA-Recovery: Benchmarking Error Recovery in Long-Horizon Computer-Use Agents
Oct 20, 2026Xu Hu, Haoming Li, Yuxiang Wang +11Computer-Use Agents Computer-Use Agent Benchmarks
University of Texas at Dallas · Independent Researcher · University of Manchester +3
ProMemAgent: Long-Horizon Proactive Assistance with Context-Aware Prospective Memory from Sensory Perceptions
Oct 19, 2026Lilin Xu, Linyang He, Kaiyuan Hou +4Proactive Agents Long-Horizon Agent Tasks
Columbia University
Evolving Self-tuning Benchmarks for Long-Horizon Tool Use
Oct 19, 2026Hyunji Alex Nam, Michael Dennis, Natasha Jaques +1AI Agent Benchmarks Tool-Use Evaluation
Stanford University · Google Deepmind · University of Washington
Embodied Playground: A Long-Horizon Agentic Robotics Benchmark for Physical, Strategic, and Spatial Reasoning
Oct 19, 2026Feng Chen, Pei Zhou, Lixuan He +7Embodied Agent Benchmarks Long-Horizon Agent Evaluation
HKU · Sharpa · Tsinghua University +3
From Code to Playable Systems: CEGame-Bench for Long-Horizon Game Creation Agents
Oct 19, 2026Zixuan Dong, Yanghai Wang, Hanyan Bian +8Automated Game Testing Software Engineering Agents
National University of Defense Technology · Nanjing University · The University of Tokyo +2
From Trace to Fix: Residual Task State in Long-Horizon Agents
Oct 19, 2026Zile Huang, Ser-nam LimLLM Agent Reliability LLM Agent Planning
University of Central Florida
SWE-Bench Pro V2: A Cleaner, Harder-to-Game Coding Benchmark
Oct 16, 2026Soham Dan, Jeff Da, Sanyam Bhutani +8Coding Agent Benchmarks Software Engineering Benchmarks
Scale AI · Reflection AI
When to Forget: Risk-Controlled Context Adaptation for Long-Horizon Agents
Oct 16, 2026Chenle Chen, Yangbo Wei, Zhijie Zheng +6LLM Context Management Long-Horizon Agent Tasks
University of California, Los Angeles · Shanghai Jiao Tong University · Arizona State University +1
RealCLI: Toward a Unified Evaluation of Human-Agent Collaboration for CLI Agents
Oct 15, 2026Yunqi Liu, Peiyu Wang, Bingli Wang +21Terminal-Agent Benchmarks LLM Agent Evaluation
IQuest Research · Beihang University · Renmin University
Kolab: Benchmarking Human-Agent Collaboration in Long-Horizon Workflows
Oct 15, 2026Shirley Anugrah Hayati, Patricia Anugrah Setiani, Ruizi Wang +4Human-AI Collaboration Long-Horizon Agent Evaluation
Department of Computer Science, University of Minnesota · Department of Economics, University of Minnesota
OmicsBench: a long-horizon multi-turn benchmark of AI agents on research-study-scale bioinformatics
Oct 15, 2026Gideon Shaked, Siddharth NagisettyBioinformatics AI Agent Benchmarks
Department of Computational Medicine and Bioinformatics, University of Michigan · AfterQuery
Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agents
Oct 14, 2026Yuyao Ge, Shenghua Liu, Peizhong Gao +8LLM Agent Harnesses AI Agent Benchmarks
Institute of Computing Technology, Chinese Academy of Sciences · Tsinghua University · Peking University +1
Can an Agent Learn to Predict Its Own Failure? Outcome-Grounded Confidence Calibration in LLM Agents
Oct 13, 2026Ashutosh HathidaraConfidence Estimation in Language Models LLM Agent Reliability
SAP
Benchmarking Interaction-Oriented Aerial Mobility across Heterogeneous Environments
Oct 13, 2026Zhibo Duan, Hanlin Kong, Tianran Ma +8UAV Navigation Aerial Manipulation
Zhejiang University · Differential Robotics
I2O2I-Nav: A Benchmark with a Modular Agent for Long-Horizon Indoor-to-Outdoor-to-Indoor Navigation
Oct 13, 2026Zhengjie Shu, Wanyu Li, Di Xiao +11Indoor Robot Navigation Robot Navigation Benchmarks
University of Hong Kong · Shanghai Jiao Tong University · Independent Researcher +2
LifeSearch: Benchmarking Deep Research Agents for Real-World Life and Productivity
Oct 13, 2026Yibo Wang, Senjie Jin, Yansong Ning +6Deep Research Agents LLM Agent Evaluation
LLM Department, Tencent
Empowering Edge Agents: A Systematic Analysis of Edge-Cloud Collaboration in LLM Agent Execution
Oct 13, 2026Pengcheng Huang, Zhenghao Liu, Shixin Ren +5Edge-Cloud Collaborative Inference Edge-Cloud Computing
School of Computer Science and Engineering, Northeastern University, Shenyang, China · Department of Computer Science and Technology, Tsinghua University, Beijing, China · China Mobile Online Services Co., Ltd. Beijing, China
Towards a Systematic Study of Embodied System 2 over Full Task Execution Cycles
Oct 13, 2026Yuxi Wei, Ye Fang, Shuai Yang +3Embodied Reasoning AI Agent Evaluation
The University of Hong Kong · Morphi Robot · Fudan University +1

Oct 27, 2026J/K move · Enter open · S save

FeatureArc: Benchmarking Coding Agents on Evolving Features

Yunlong Wang, Alyssa Wu, Cuiyun Gao, Jia Feng+2

Hunyuan Team, Tencent · The Chinese University of Hong Kong

Coding Agent Benchmarks Software Engineering Software Engineering Agents Long-Horizon Agent Evaluation

Open paper AskPDF arXiv

A software feature usually evolves over time: developers extend its first version as new use cases and requirements arrive, and each extension builds on code written earlier. Most feature benchmarks evaluate a single request on a fixed codebase. Feature evolution requires evaluating successive extensions on an agent's own earlier code and checking whether the feature remains correct. We introduce FeatureArc to evaluate coding agents on evolving features, together with a pipeline that constructs its tasks from repository histories. The benchmark combines dependencies between successive stages of the same feature, explicit requirements and interface contracts, and executable checks across four programming languages. The pipeline validates fail-to-pass (F2P) tests for new behavior and pass-to-pass (P2P) tests for existing behavior on the reference code before and after each stage. We evaluated 10 coding agents on 73 manually reviewed tasks of 2 to 10 stages. Each agent works in one persistent session and workspace without external test feedback or rollback. The evaluation reveals a gap between first-stage and whole-feature success: pooled success falls from 62.7% at the first stage to 28.5% when the whole feature is tested on the final code. Trajectory analysis identifies inherited defects as a common failure mechanism, present in 72.9% of failed later stages. These earlier errors often remain undetected: in stages following a failure, agents leave the earlier error unnoticed about 95% of the time, and their self-tests rarely revisit earlier stages. Evolving a feature autonomously, without human checks between stages, therefore remains a substantial challenge for current coding agents. At Tencent, the pipeline extends internal evaluations to this setting and guides the development of Hunyuan, Tencent's family of large language models.