Long-Horizon Agent Evaluation

Follow this topic Ask about this topic

This field studies whether agents can complete sustained, multi-step work—not just isolated prompts—in settings such as browser and GUI use, software maintenance, healthcare, business simulations, and robotics. Researchers build interactive environments with executable state checks, then assess progress through checkpoints, rubrics, and trajectory-level analysis of errors, safety, memory, and recovery. A central finding is that success on short tasks often fails to predict performance over longer workflows, where forgotten constraints, cascading errors, changing conditions, and weak verification can undermine results. A continuing challenge is making evaluations realistic and verifiable while separating model capability from the effects of the agent harness, tools, and evaluation protocol.

Latest papers

359 papers70 papers in the last four weeks, up 37% on the four weeks before. 0.4% of all new papers.

VehicleMemBench: An Executable Benchmark for Multi-User Long-Term Memory in In-Vehicle Agents
Mar 25, 2026Yuhao Chen, Yi Xu, Xinyun Ding +7Memory Agent Benchmarks Memory-Augmented Agents
University of Science and Technology of China · iFLYTEK Research · Xiamen University +1
Can LLM Agents Be CFOs? Benchmarking Long-Horizon Resource Allocation in an Uncertain Enterprise Environment
Mar 24, 2026Yi Han, Yan Wang, Lingfei Qian +12Long-Horizon Agent Evaluation LLM Agents
Georgia Institute of Technology · The Fin AI · Stevens Institute of Technology +7
LH-Bench: Skill-Grounded Evaluation of Long-Horizon Agents on Subjective Enterprise Tasks
Mar 24, 2026Abhishek Chandwani, Ishan GuptaLong-Horizon Agent Evaluation AI Agent Benchmarks
Metaphi Inc. USA
AndroTMem: From Interaction Trajectories to Anchored Memory in Long-Horizon GUI Agents
Mar 19, 2026Yibo Shi, Jungang Li, Linghao Zhang +25Agent Memory Memory-Augmented Agents
XJTU · HKUST(GZ) · HKUST +8
Enactor: From Traffic Simulators to Surrogate World Models
Mar 18, 2026Yash Ranjan, Rahul Sengupta, Anand Rangarajan +1Interactive World Models Neural Surrogate Modeling
Department of CISE, University of Florida
When the Specification Emerges: Benchmarking Faithfulness Loss in Long-Horizon Coding Agents
Mar 17, 2026Lu Yan, Xuan Chen, Xiangyu ZhangAI Coding Agents Coding Agents
Purdue University
RetailBench: Evaluating Long-Horizon Autonomous Decision-Making and Strategy Stability of LLM Agents in Realistic Retail Environments
Mar 17, 2026Linghua Zhang, Jun Wang, Jingtong Wu +1Long-Horizon Agent Evaluation AI Agent Benchmarks
Ant Group · City University of Hong Kong
The PokeAgent Challenge: Competitive and Long-Context Learning at Scale
Mar 16, 2026Seth Karten, Jake Grigsby, Tersoo Upaa +28AI Agent Benchmarks Game-Playing Agents
Princeton · UT-Austin · Team Heatz +12
Shopping Companion: Benchmarking and Training LLM Agents for Long-Horizon Preference-Grounded E-Commerce Tasks
Mar 16, 2026Zijian Yu, Kejun Xiao, Huaipeng Zhao +2LLM Agent Training AI Agent Benchmarks
Alibaba International Digital Commercial Group
AgentProcessBench: Diagnosing Step-Level Process Quality in Tool-Using Agents
Mar 15, 2026Shengda Fan, Xuyan Ye, Yupeng Huo +9LLM Agent Evaluation Process-Based Agent Evaluation
Renmin University of China Beijing, China · Beijing Jiaotong University Beijing, China · Shanghai Jiao Tong University Shanghai, China +1
EnterpriseOps-Gym: Environments and Evaluations for Stateful Agentic Planning and Tool Use in Enterprise Settings
Mar 13, 2026Shiva Krishna Reddy Malay, Shravan Nayak, Jishnu Sethumadhavan Nair +6AI Agent Benchmarks LLM Agent Planning
ServiceNow Research · Mila - Quebec AI Institute · Université de Montréal
SWE-Milestone: Evaluating AI Agents on Continuous Software Evolution
Mar 13, 2026Gangda Deng, Zhaoling Chen, Zhongming Yu +11Software Engineering Software Engineering Benchmarks
USC · UCR · UCSD +5
LifeSim: Long-Horizon User Life Simulator for Personalized Assistant Evaluation
Mar 12, 2026Feiyu Duan, Xuanjing Huang, Zhongyu WeiUser Simulation Long-Horizon Agent Evaluation
School of Data Science, Fudan University · College of Computer Science and Artificial Intelligence, Fudan University · Shanghai Innovation Institute +2
Beyond Scaling: Assessing Strategic Reasoning and Rapid Decision-Making Capability of LLMs in Zero-sum Environments
Mar 10, 2026Yang Li, Xing Chen, Yutao Liu +5AI Agent Evaluation AI Agent Benchmarks
Tsinghua University · Ocean University of China (OUC) · CAICT +4
$τ$ -Knowledge: Evaluating Conversational Agents over Unstructured Knowledge
Mar 4, 2026Quan Shi, Alexandra Zytek, Pedram Razavi +2LLM Agent Evaluation Agent Evaluation
Sierra · Princeton University
SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via Continuous Integration
Mar 4, 2026Jialong Chen, Xander Xu, Hu Wei +2Software Engineering Software Engineering Benchmarks
Sun Yat-sen University · Skylenage · Alibaba Group
LifeBench: A Benchmark for Long-Horizon Multi-Source Memory
Mar 4, 2026Zihao Cheng, Weixin Wang, Yu Zhao +15Agent Memory Memory Agent Benchmarks
State Key Laboratory for Novel Software Technology, Nanjing University · School of Artificial Intelligence, Nanjing University · Huawei Technologies Co., Ltd.
EMPA: Evaluating Persona-Aligned Empathy as a Process
Feb 28, 2026Shiya Zhang, Yuhan Zhan, Ruixi Su +4LLM Agent Evaluation Multi-Turn Dialogue Evaluation
Team Echo, Nature Select · Sun Yat-sen University
AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applications
Feb 26, 2026Yujie Zhao, Boqin Yuan, Junbo Huang +9Agent Memory Long-Horizon Agent Evaluation
UCSD · Recursive.
Search More, Think Less: Rethinking Long-Horizon Agentic Search for Efficiency and Generalization
Feb 26, 2026Qianben Chen, Tianrui Qin, King Zhu +21Agentic Search Long-Horizon LLM Agents
AgentLAB: Benchmarking LLM Agents against Long-Horizon Attacks
Feb 18, 2026Tanqiu Jiang, Yuhui Wang, Jiacheng Liang +1LLM Agent Security AI Agent Security Benchmarks
Department of Computer Science, Stony Brook University, Stony Brook, USA.
MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks
Feb 18, 2026Zexue He, Yu Wang, Churan Zhi +11Agent Memory LLM Agent Memory
Stanford University · UCSD · UIUC +3
OdysseyArena: Benchmarking Large Language Models For Long-Horizon, Active and Inductive Interactions
Feb 5, 2026Fangzhi Xu, Hang Yan, Qiushi Sun +14AI Agent Benchmarks Long-Horizon Agent Evaluation
Xi’an Jiaotong University · The University of Hong Kong · Tsinghua University +5
EMemBench: Interactive Benchmarking of Episodic Memory for VLM Agents
Jan 23, 2026Xinze Li, Ziyue Zhu, Siyuan Liu +4Episodic Memory Long-Horizon Agent Evaluation
Nanyang Technological University · Shanghai AI Laboratory · Fudan University
LifeAgentBench: Benchmarking LLMs for Long-Horizon, Cross-Dimensional Lifestyle Health Reasoning
Jan 20, 2026Ye Tian, Zihao Wang, Onat Gungor +2Healthcare Long-Horizon Agent Evaluation
University of California San Diego, La Jolla, CA, USA · Google Research, Mountain View, CA, USA
SWITCH: Benchmarking Modeling and Handling of Tangible Interfaces in Long-horizon Embodied Scenarios
Nov 20, 2025Juntao Cheng, Wanyue Zhang, Zhiwei Yu +7Long-Horizon Agent Evaluation Egocentric Video Understanding
Beijing Academy of Artificial Intelligence (BAAI) · Shanghai Jiao Tong University · School of Artificial Intelligence, University of Chinese Academy of Sciences +3
Lost in the Maze: Overcoming Context Limitations in Long-Horizon Agentic Search
Oct 21, 2025Howard Yen, Yoonsang Lee, Ashwin Paranjape +5Agentic Search Long-Horizon Agent Evaluation
Princeton Language and Intelligence, Princeton University · Work done as an intern at Samaya AI. · Samaya AI
BuilderBench: The Building Blocks of Intelligent Agents
Oct 7, 2025Raj Ghugare, Roger Creus Castanyer, Catherine Ji +4Long-Horizon Agent Evaluation Robot Manipulation Benchmarks
Princeton University · Mila – Quebec AI Institute · Université de Montréal
HiMA-Ecom: A Long-Horizon Memory Benchmark for Hierarchical Multi-Agent E-commerce Assistants
Jun 24, 2025Junxing Hu, Ai Han, Haolan Zhan +8Long-Horizon Agent Evaluation AI Agent Benchmarks
JD Retail, JD.com, Inc., Beijing, 101111, China · Faculty of Engineering and Information Technology, The University of Melbourne, Parkville, Victoria, 3082, Australia · School of Mathematical Sciences, University of Chinese Academy of Sciences, Beijing, 100049, China

Mar 25, 2026J/K move · Enter open · S save

VehicleMemBench: An Executable Benchmark for Multi-User Long-Term Memory in In-Vehicle Agents

Yuhao Chen, Yi Xu, Xinyun Ding, Xiang Fang+6

University of Science and Technology of China · iFLYTEK Research · Xiamen University+1

Memory Agent Benchmarks Memory-Augmented Agents Long-Horizon Agent Evaluation

Open paper AskPDF arXiv

With the growing demand for intelligent in-vehicle experiences, vehicle-based agents are evolving from simple assistants to long-term companions. This evolution requires agents to continuously model multi-user preferences and make reliable decisions in the face of inter-user preference conflicts and changing habits over time. However, existing benchmarks are largely limited to single-user, static question-answer settings, failing to capture the temporal evolution of preferences and the multi-user, tool-interactive nature of real vehicle environments. To address this gap, we introduce VehicleMemBench, a multi-user long-context memory benchmark built on an executable in-vehicle simulation environment. The benchmark evaluates tool use and memory by comparing the post-action environment state with a predefined target state, enabling objective and reproducible evaluation without LLM-based or human scoring. VehicleMemBench includes 23 tool modules, and each sample contains over 80 historical memory events. Experiments show that powerful models perform well on direct instruction tasks but struggle in scenarios involving memory evolution, particularly when user preferences change dynamically. Even advanced memory systems struggle to handle domain-specific memory requirements in this environment. These findings highlight the need for more robust and specialized memory management mechanisms to support long-term adaptive decision-making in real-world in-vehicle systems. To facilitate future research, we release the data and code.