Long-Horizon Agent Evaluation

Follow this topic Ask about this topic

Momentum

47 papers in the last four weeks, up 176% on the four weeks before. 0.5% of all new papers.

Jul 13Week of Sep 28

Latest papers 232

All topics

AgenticSTS: A Bounded-Memory Testbed for Long-Horizon LLM Agents
Jul 2, 2026Xiangchen Cheng, Yunwei Jiang, Jianwen Sun +7LLM Agent Memory Long-Horizon Agent Evaluation
Alaya Lab · Shanghai Jiao Tong University · Shanghai Innovation Institute +2
QVal: Cheaply Evaluating Dense Supervision Signals for Long-Horizon LLM Agents
Jun 30, 2026Sergio Hernández-Gutiérrez, Matteo Merler, Ilze Amanda Auzina +3Long-Horizon Agent Evaluation AI Agent Benchmarks
Tübingen AI Center, University of Tübingen · Fondazione Bruno Kessler
FinPersona-Bench: A Benchmark for Longitudinal Psychometric Stability of Autonomous Financial Agents
Jun 30, 2026Muhammad Usman Safder, Ayesha Gull, Rania Elbadry +7Personality Modeling in Language Models Long-Horizon Agent Evaluation
Steve · MBZUAI · The University of Tokyo +2
World-Model Collapse as a Phase Transition
Jun 30, 2026Xinyuan Song, Zekun CaiPhase Transitions Model Collapse
Emory University, Atlanta, GA, USA · The University of Tokyo, Tokyo, Japan · LocationMind, Tokyo, Japan
HealthAgentBench: A Unified Benchmark Suite of Realistic Agentic Healthcare Environments for Challenging Frontier AI Agents
Jun 30, 2026Qianchu Liu, Sheng Zhang, Guanghui Qin +16Long-Horizon Agent Evaluation AI Agent Benchmarks
Microsoft Research
AhaBench: Do Agents Turn Experience into Reusable Insights? A Long-Horizon Benchmark for Continual Learning
Jun 30, 2026Zerui Cheng, Jiawei Xu, Huacan Chai +3Continual Learning for LLM Agents Lifelong Learning Agents
Princeton University · Tencent Hy · Tsinghua University +2
OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks
Jun 28, 2026Mengqi Yuan, Zilong Zhou, Xinzhuang Xiong +33Computer-Use Agent Benchmarks Computer-Use Agents
XLANG Lab
MedEvoEval: Evaluating Continual Evolution of Doctor Agents through Simulated Clinical Episodes
Jun 27, 2026Hui ZhangContinual Learning for LLM Agents Long-Horizon Agent Evaluation
Beijing Institute of Technology (BIT), China
DMV-Bench: Diagnosing Long-Horizon Multimodal Agents' Visual Memory with Incidental Cue Injection
Jun 25, 2026Yujin Tang, Chenming Shang, Ruize Xu +1Multimodal Memory Long-Horizon Agent Evaluation
Dartmouth College
scBench-Long: Verifiable Benchmarking of Long-Horizon Single-Cell Biology
Jun 25, 2026Ian Diks, Zhen Yang, Arjun Banerjee +2AI Agents for Scientific Discovery Long-Horizon Agent Evaluation
LatchBio, San Francisco, CA, USA
SidConArena: An Environment Evaluating Agents in Open-Ended,Positive-Sum Bargaining Game
Jun 24, 2026Yeqi Feng, Yuxin Chen, Tianxing HeMulti-Agent Negotiation Game Theory
Institute for Interdisciplinary Information Sciences, Tsinghua University
Long-Term Simulation Exposes Cognitive-Developmental Risks in AI Companions
Jun 24, 2026Kaicheng Shen, Lingyu Li, Wen Wu +3Language Model Safety Evaluation Long-Horizon Agent Evaluation
Shanghai Institute of Artificial Intelligence for Education, East China Normal University, Shanghai, China · Shanghai Artificial Intelligence Laboratory, Shanghai, China · School of Computer Science and Technology, East China Normal University, Shanghai, China
World Models in Pieces: Structural Certification for General Agents
Jun 23, 2026Yikai Lu, Yifei Wu, Xinyu Lu +1Long-Horizon Agent Evaluation Model-Based Planning
School of Data Science, The Chinese University of Hong Kong, Shenzhen, Shenzhen, China.
SAFARI: Scaling Long Horizon Agentic Fault Attribution via Active Investigation
Jun 23, 2026Chenyang Zhu, Jiayu Yao, Kushal Chawla +10Agent Failure Analysis Long-Horizon Agent Evaluation
1AI Foundations, Capital One · 2General Intelligence Com · Department of Engineering Sciences and Ap
When Retrieval Metrics Mislead: Measuring Policy Signal in Long-Horizon Tool-Use Agents
Jun 22, 2026Tianyu Ding, Juan Pablo De la Cruz WeinsteinTool-Use Evaluation Retrieval-Augmented Classification
Amazon Web Services
ENVS: Environment-Native Verified Search for Long-Horizon GUI Agents
Jun 22, 2026Yincheng Zhou, Athena Zhuoming Zhong, Shijie Zhang +3Distribution Shift Robustness Computer-Use Agent Benchmarks
University of Pennsylvania · 2Peking University
DynamicMem: A Long-Horizon Memory Benchmark in Real-World Settings
Jun 22, 2026Wenya Xie, Shengming Zhou, Zelin Li +9Agent Memory LLM Agent Memory
University of Minnesota · University of North Carolina at Charlotte · Johns Hopkins University +2
MacAgentBench: Benchmarking AI Agents on Real-World macOS Desktop
Jun 21, 2026Yikun Fu, Bowen Fu, Zhenyu Wu +10Computer-Use Agent Benchmarks Computer-Use Agents
1Shanghai Jiao Tong University · 2Shanghai AI Laboratory · 3Xi’an Jiaotong University +5
PlanBench-XL: Evaluating Long-Horizon Planning of LLM Tool-Use Agents in Large-Scale Tool Ecosystems
Jun 21, 2026Jiayu Liu, Qihan Lin, Cheng Qian +8Long-Horizon Agent Evaluation AI Agent Benchmarks
University of Illinois Urbana-Champaign
ChainWorld: Composing Long-Horizon Desktop Workloads from Atomic OSWorld Tasks
Jun 19, 2026Vincent Siu, Manasi Sharma, Dawn Song +2Computer-Use Agent Benchmarks Computer-Use Agents
1Scale AI · University of California, Santa Cruz · *Work done during an internship at Scale AI. +1
StaminaBench: Stress-Testing Coding Agents over 100 Interaction Turns
Jun 17, 2026Vlad Sobal, Shuo Yang, Yuting Zhang +2AI Coding Agents Benchmark Design
AWS Agentic AI
WorldLines: Benchmarking and Modeling Long-Horizon Stateful Embodied Agents
Jun 17, 2026Yehang Zhang, Jianchong Su, Haojian Huang +7Embodied QA Long-Horizon Agent Evaluation
HKUST(GZ) · Knowin · HKUST
CEO-Bench: Can Agents Play the Long Game?
Jun 16, 2026Haozhe Chen, Karthik Narasimhan, Zhuang LiuLong-Horizon Agent Evaluation AI Agent Benchmarks
Princeton University
When Web Agents Finish but Still Fail: Reproducible Triggers and Trace Diagnostics for Parallel Web Exploration
Jun 16, 2026Aagam Sogani, Botao Rui, Swetha Vaidyanathan +3Agent Failure Analysis Web Agent Benchmarks
University of Wisconsin–Madison, Madison, WI, USA.
EComAgentBench: Benchmarking Shopping Agents on Long-Horizon Tasks with Distributed Hidden Intent
Jun 16, 2026Zeyao Du, Tong Li, Haibo ZhangTool-Use Evaluation Long-Horizon Agent Evaluation
Shopee
SidewalkBench: Benchmarking Visual Navigation on Urban Sidewalks
Jun 15, 2026Zhizheng Liu, Honglin He, Vivek Alumootil +4Visual Navigation Long-Horizon Agent Evaluation
University of California, Los Angeles · Coco Robotics
LabOSBench: Benchmarking Computer Use Agents for Scientific Instrument Control
Jun 15, 2026Anqi Zou, Han Deng, Chengyu Zhang +9Computer-Use Agent Benchmarks Computer-Use Agents
Shenzhen Loop Area Institute · Dalian University of Technology · The Chinese University of Hong Kong +1
CoffeeBench: Benchmarking Long-Horizon LLM Agents in Heterogeneous Multi-Agent Economies
Jun 15, 2026Issa Sugiura, Daichi Hattori, Kazuo Araragi +5Long-Horizon Agent Evaluation AI Agent Benchmarks
Sakana AI · KPMG AZSA LLC
RetailBench: Benchmarking long horizon reasoning and coherent decision making of LLM agents in realistic retail environments
Jun 14, 2026Linghua Zhang, Jun Wang, Jingtong Wu +1Long-Horizon Agent Evaluation AI Agent Benchmarks
Ant Group · City University of Hong Kong
StreamMemBench: Streaming Evaluation of Agent Memory for Future-Oriented Assistance
Jun 12, 2026Guanming Liu, Yuqi Ren, Hansu Gu +5Agent Memory Long-Horizon Agent Evaluation
1Fudan University · 2Amazon

Jul 2, 2026, cs.AIJ/K move · Enter open · S save

AgenticSTS: A Bounded-Memory Testbed for Long-Horizon LLM Agents

Xiangchen Cheng, Yunwei Jiang, Jianwen Sun, Zizhen Li+6

Alaya Lab · Shanghai Jiao Tong University · Shanghai Innovation Institute+2

LLM Agent Memory Long-Horizon Agent Evaluation AI Agent Benchmarks LLM Agents Long-Horizon LLM Agents

Open paper AskPDF arXiv

Memory for a long-horizon LLM agent is a contract about what each future decision is allowed to see. The simplest contract appends past observations, tool calls, and reflections to every prompt, which makes prior context easy to access but also turns it into a jumbled mixture in which the effect of any single memory component is hard to isolate. We introduce and instrument an alternative bounded contract: every decision is made from a fresh user message assembled by typed retrieval, with no raw cross-decision transcript appended. The prompt thus stays bounded across runs of any length, and any single layer can be ablated in isolation. We instantiate the contract in Slay the Spire 2, a closed-rule stochastic deck-building game whose runs require hundreds of tactical and strategic decisions. A public online benchmark of frontier LLMs on the same game reports zero wins at the lowest difficulty across five configurations, and the developer-reported human win rate at the same difficulty is 16%; the task is hard but not saturated. Within our harness, a fixed-A0 ablation shows the largest observed difference when triggered strategic skills are enabled: the no-store baseline wins 3/10 games and adding the skill layer 6/10. At this sample size the comparison is directional rather than statistically decisive (Fisher exact p\approx0.37); a cross-backbone probe and public accumulating-context baselines are reported as operational comparisons rather than controlled tests of the contract variable itself. We release a reproducible testbed: 298 completed trajectories with condition tags, frozen memory/skill snapshots, prompt records, and analysis scripts -- an agent design and a validated, reusable methodology for studying how explicit memory layers shape long-horizon LLM-agent decisions.

Momentum

Related topics

Latest papers 232