Long-Horizon Agent Evaluation

Follow this topic Ask about this topic

Momentum

47 papers in the last four weeks, up 176% on the four weeks before. 0.5% of all new papers.

Jul 13Week of Sep 28

Latest papers 232

All topics

Self-Designed Evaluators and Warm Memory for Long-Horizon Agents
Sep 27, 2026Saeid Asgari, Emre Kiciman, Leonardo de Oliveira Nunes +1Language Model Generation Evaluation Long-Horizon Agent Evaluation
Microsoft
EverMine: Dissecting the Self-Evolution of Research Capabilities in Long-Horizon Alpha Research
Sep 27, 2026Siyuan Li, Jiangfeng Zhang, Rui Yao +3Quantitative Finance Long-Horizon Agent Evaluation
The Hong Kong University of Science and Technology (Guangzhou) · The Hong Kong Polytechnic University · Singapore Management University
Long-Horizon Analog Design Bench: Benchmarking Agents on Hours-Long Analog and Mixed-Signal Circuit Design Tasks
Sep 27, 2026Analog Design Bench TeamBenchmark Design Electronic Design Automation
Adaptive Consistency Graph for Long-Horizon Agents
Sep 26, 2026Jiecong Wang, Hao Peng, Zhanyi WangLong-Horizon Agent Evaluation Graph-Based Agent Memory
Beihang University · QI-ANXIN GROUP
Dude, Where's My State? Execution Information Requirements for Stateful Agents
Sep 26, 2026Nikita Mehrotra, Ashish Tiwari, Priyanshu Gupta +1Software Engineering Agents Long-Horizon Agent Evaluation
SCLATE: a Substrate for Continual-Learning Agent Training and Evaluation
Sep 26, 2026Youngmok Jung, Sirajul Salekin, Henry Tran +3Continual Learning for LLM Agents Long-Horizon Agent Evaluation
Apple
Advancing Model Research in AgentX: Long-Horizon Autonomy for Industrial Recommender Systems
Sep 24, 2026Shuang Yang, Zijie Zhuang, Changxin Lao +20AI Agents for Scientific Discovery Long-Horizon Agent Evaluation
The Tasteful Agent: Measuring and Improving Taste in Long-Horizon Tasks
Sep 22, 2026Wenbo Pan, Zhichao Liu, Shujie Liu +6Software Engineering Benchmarks Long-Horizon Agent Evaluation
City University of Hong Kong · Microsoft · Independent Researcher
GameHorizon Suite: Multi-Horizon Data and Evaluation in Gameplay
Sep 21, 2026Yiran Wang, Xingyilang Yin, Junfu Pu +12Game-Playing Agents Long-Horizon Agent Evaluation
ARC Lab, Tencent · GVC Lab, Great Bay University · University of Macau +3
DolphinBench: Mapping the Pareto Frontier of Agent Memory
Sep 21, 2026Soumil Rathi, Deshraj Yadav, Taranjeet SinghAgent Memory Persistent Memory for Language Models
Mem0
DeliveryGym: An RL Environment for Long-Horizon Embodied Agent Planning with Adaptive Curriculum
Sep 17, 2026Haoqiang Kang, Yiming Zhang, Yiyang Guo +5Long-Horizon Planning Long-Horizon Agent Evaluation
UC San Diego · Independent · Johns Hopkins University +1
Reach or Solve? Attributing Agentic RL Gains with Checkpoint Handoffs
Sep 17, 2026Xuan Liu, Jingbin QianAgentic RL Long-Horizon Agent Evaluation
Shanghai Jiao Tong University · Rice University
SIMLIFE: Pattern Understanding for Long-Horizon Human-Agent Partnership
Sep 17, 2026Run Peng, Zinnia Nie, Jing Ding +7Human Behavior Simulation Long-Horizon Agent Evaluation
University of Michigan · ByteDance · Osaka University +1
Locating Hidden Failures Makes Long-Horizon Agents More Reliable
Sep 15, 2026Salman Rahman, Yubin Kim, Mihir Parmar +15Agent Failure Analysis AI Agent Reliability
University of California, Los Angeles · New York University · Google Research +1
Emergence World: Adversarial Stress-Testing of Long-Horizon Multi-Agent Systems
Sep 15, 2026Deepak Akkil, Tamer Abuelsaad, Karthik Vikram +5Multi-Agent LLM Systems Long-Horizon Agent Evaluation
Emergence AI
BLINDSPOT: A Benchmark for Safety and Refusal Calibration in Long-Horizon Tool-Using Agents
Sep 14, 2026Sadia Asif, Mohammad Mohammadi Amiri, Momin Abbas +2Long-Horizon Agent Evaluation AI Agent Safety Benchmarks
Rensselaer Polytechnic Institute · IBM Research
When Agents Slow Down: Understanding LLM Agents' Test-Time Strategies via Elo-per-token Analysis
Sep 14, 2026Kaiyuan Liu, Qiuyang Mang, Bo Peng +6LLM Agent Evaluation Test-Time Scaling
UC Berkeley · University of Washington · Princeton University +1
MemRiskBench: Trace-Aware Risk-Preserving Evaluation for Long-Horizon LLM Agents
Sep 14, 2026Jianhua Jiang, Dongbo Yuan, Weihua LiLong-Horizon Agent Evaluation AI Agent Safety Benchmarks
School of Artificial Intelligence and Computer Science, Jilin University of Finance and Economics, 130117 Changchun, China · Jilin Province Key Laboratory of Fintech, Jilin University of Finance and Economics, 130117 Changchun, China · School of Engineering, Computer and Mathematical Sciences, Auckland University of Technology, Auckland, New Zealand
DynSTEER: Dynamic Stage-wise Trajectory Evaluation and Execution-time Review for Agents
Sep 13, 2026Zhichao Shi, Xuhui Jiang, Wenjie Zhang +5LLM Agent Evaluation AI Agent Evaluation
School of Advanced Interdisciplinary Sciences, UCAS · State Key Lab of AI Safety, Institute of Computing Technology, CAS · DataArc Tech Ltd. +3
Safety Signals to Verify NetOps Agents with Action-Level Granularity
Sep 13, 2026Tobias Labarta, Frederik Pahde, Novak Boškov +5Long-Horizon Agent Evaluation AI Agent Safety
Fraunhofer Heinrich-Hertz-Institut, Berlin, Germany · Nokia Bell Labs, Stuttgart, Germany · Nokia Bell Labs, Murray Hill, NJ 07974, USA +2
T1: Terminal Agent Reinforcement Learning for Long-Horizon Tasks
Sep 12, 2026Junyao Yang, Yucheng Shi, Zhongzhi Li +4Terminal Agents Long-Horizon Agent Evaluation
Tencent Hy Foundation Model · National University of Singapore · University of Georgia +2
Root-Cause Attribution Is a Search Problem: Continual Search for Long-Horizon Agent Failures
Sep 11, 2026Harsh Raj, David Lee, Anas Mahmoud +7Agent Failure Analysis Agentic Search
Scale AI · Scale AI Research
JarvisGUI: Towards Cross-Device GUI Agents with Dynamic Task Composition
Sep 9, 2026Zixiang Chen, Yuheng Lu, Zihao Cheng +8Computer-Use Agent Benchmarks GUI Agents
School of Computer Science and Engineering, Beihang University, Beijing, China · Beijing Institute of Technology, Beijing, China · Baidu Inc., Beijing, China
EvoNav-Bench: Benchmarking Lifelong Navigation in Evolving Environments
Sep 8, 2026Xilin Wang, Guoxi Zhang, Hongming Xu +3Embodied Navigation Robot Navigation
State Key Laboratory of General Artificial Intelligence, BIGAI · 2Tsinghua University
Linguistic Trajectory Encoding for Efficient Long-Horizon Spatial Memory in Embodied Agents
Sep 4, 2026Tianyidan Xie, Shenyi Wang, Qiang Tang +7Long-Horizon Agent Evaluation Trajectory Representation Learning
Nanjing University · University of British Columbia · Zhejiang Sci-Tech University +2
Parsing the Stream: A Live Trace Model for Long-Horizon Agents and Their Observers
Sep 1, 2026Egor Pakhomov, Erik NijkampLong-Horizon Agent Evaluation AI Agent Monitoring
Salesforce AI Research
Measuring the Behavioral Fidelity of Long-Horizon Human Activity Simulations
Sep 1, 2026Yi Fei Cheng, Fan Yang, Iremsu Bas +3Human Behavior Simulation Long-Horizon Agent Evaluation
Human-Computer Interaction Institute, School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA · Fujitsu Ltd., Kawasaki, Japan · Fujitsu Research of America, Pittsburgh, PA, USA
Deploying and Evaluating a Smart-Agriculture Agentic Engine for Full-Season Soybean Farm Operations
Aug 31, 2026Ao Qu, Panagiotis Michelakis, Linyuan Han +8Software Engineering Multi-Agent Orchestration
Faculty of Computing Harbin Institute of Technology State Key Laboratory of Smart Farm Technologies and Systems Harbin, China · School of Electrical and Computer Engineering National Technical University of Athens Athens, Greece
E-Commerce Bench: Evaluating LLM Agents on Long-Horizon Autonomous Business Operation
Aug 31, 2026Wei Fan, Xinjie Shen, Xudong Guo +8Long-Horizon Agent Evaluation AI Agent Benchmarks
1Qwen Team, Alibaba Group · Department of Computer Science and Engineering, HKUST · 2Taobao & Tmall Group, Alibaba Group
ATLAS: Dual-Horizon Diagnostic Evaluation for Industrial Tool-Use Agents
Aug 31, 2026Wei Chen, Peilun Zhou, Zhaoyu Hu +8Tool-Use Evaluation LLM Agent Evaluation
University of Science and Technology of China, Meituan · Meituan

Sep 27, 2026, cs.AIJ/K move · Enter open · S save

Self-Designed Evaluators and Warm Memory for Long-Horizon Agents

Saeid Asgari, Emre Kiciman, Leonardo de Oliveira Nunes, Ranveer Chandra

Microsoft

Language Model Generation Evaluation Long-Horizon Agent Evaluation Language Model Self-Assessment LLM Agent Self-Improvement AI Agent Benchmarks Memory-Augmented Language Model Agents

Open paper AskPDF arXiv

A tool-using language-model agent deployed over a long stream of tasks receives no reward, so it cannot tell whether it succeeded, cannot safely retry, and cannot label the experience it needs to improve. We present SelfSuite, in which the agent's own base model, given only the world's public materials, designs a small evaluation suite of weighted judges and grounded per-task briefs, freezes it, and uses it to gate a keep-best retry and to label a typed, outcome-tracked memory. On matched five-repeat benchmarks over tau2-bench and AppWorld, SelfSuite scores above the plain agent without any labels, matches methods given ten expert labels on tau2-bench, and trails Agentic Context Engineering (ACE) on AppWorld, where code execution gives a direct success signal. In an ablation campaign run on the same tasks, it is above label-free ACE in every repeat, and the gated second attempt is the only component whose removal hurts in every repeat. We also simulate a subject-matter expert who grades ten onboarding tasks per world. Using those labels to calibrate SelfSuite's evaluator gives a small, consistent gain, and using them to warm up ACE's memory lifts ACE to tie calibrated SelfSuite. A single-run study on a second model family shows the same ordering.

Momentum

Related topics

Latest papers 232