Agent Benchmarks

Follow this topic Ask about this topic

Momentum

9 papers in the last four weeks, up 50% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 87

All topics

Ask Now, Use Later: Benchmarking the Proactivity Gap in Long-Lived LLM Agents
May 27, 2026Bin Wu, Guanyun Zou, Bingbing Wang +2LLM Agent Memory LLM Agent Evaluation
Beijing University of Posts and Telecommunications · Nanjing University of Aeronautics and Astronautics · Harbin Institute of Technology (Shenzhen) +1
FastKernels: Benchmarking GPU Kernel Generation in Production
May 22, 2026Gabriele Oliaro, Yichao Fu, May Jiang +5GPU Kernel Optimization GPU Acceleration
1Snowflake AI Research · 2CMU · 3UCSD +1
MemGym: a Long-Horizon Memory Environment for LLM Agents
May 20, 2026Wujiang Xu, Yu Wang, Kai Mei +8LLM Agent Memory LLM Agent Evaluation
1Rutgers University · 2Capital One · 3Princeton University +1
EvoMemBench: Benchmarking Agent Memory from a Self-Evolving Perspective
May 18, 2026Yuyao Wang, Zhongjian Zhang, Mo Chi +7LLM Agent Memory LLM Agent Evaluation
Hong Kong University of Science and Technology (Guangzhou) · Beijing University of Posts and Telecommunications · Beijing Institute of Technology +1
SMMBench: A Benchmark for Source-Distributed Multimodal Agent Memory
May 15, 2026Huacan Chai, Yukai Wang, Yingxuan Yang +7Multimodal Memory Multimodal Reasoning
Shanghai Jiao Tong University, China · OPPO, China
ClawForge: Generating Executable Interactive Benchmarks for Command-Line Agents
May 13, 2026Yuxiang Lai, Peng Xia, Haonian Ji +8Computer-Use Agent Benchmarks LLM Agent Evaluation
University of North Carolina at Chapel Hill · Stanford University · University of Southern California +2
ChromaFlow: A Negative Ablation Study of Orchestration Overhead in Tool-Augmented Agent Evaluation
May 13, 2026Tarun MittalAI Agent Reliability LLM Agent Orchestration
Octave-X
FIKA-Bench: From Fine-grained Recognition to Fine-Grained Knowledge Acquisition
May 13, 2026Geng Li, Yuxin PengVLM Evaluation Fine-Grained Visual Perception
1Wangxuan Institute of Computer Technology, Peking University, China
LongMemEval-V2: Evaluating Long-Term Agent Memory Toward Experienced Colleagues
May 12, 2026Di Wu, Zixiang Ji, Asmi Kawatkar +4Computer-Use Agent Benchmarks Retrieval-Augmented Generation
University of California, Los Angeles
MedMemoryBench: Benchmarking Agent Memory in Personalized Healthcare
May 12, 2026Yihao Wang, Haoran Xu, Renjie Gu +10Agent Memory Healthcare
Zhejiang University · Ant Group · Alibaba Group +1
CodeClinic: Evaluating Automation of Coding Skills for Clinical Reasoning Agents
May 10, 2026Timothy Ossowski, Xinchi Liu, Danyal Maqbool +6Healthcare LLM Agent Skill Learning
University of Wisconsin–Madison · 2Microsoft Research
ProactBench: Beyond What The User Asked For
May 9, 2026Sepehr Harfi, Ahmad Salimi, Dongming Shen +1LLM Agent Evaluation Proactive Assistance
Boson AI Toronto, ON, Canada · Boson AI Santa Clara, CA, USA
DataClawBench: An Agent Benchmark for Exploratory Real-World Financial Data Analysis
May 4, 2026Qiaohong Zhang, Weihao Ye, Jialong Chen +7LLM Agent Reliability Agent Benchmarks
School of Computer Science and Engineering, Sun Yat-sen University · School of Software Engineering, Sun Yat-sen University · Lingnan College, Sun Yat-sen University
ESARBench: A Benchmark for Agentic UAV Embodied Search and Rescue
May 2, 2026Daoxuan Zhang, Ping Chen, Jianyi Zhou +1Aerial Robotics Robotics Simulation
Harbin Institute of Technology, Shenzhen
EO-Gym: A Multimodal, Interactive Environment for Earth Observation Agents
May 2, 2026Sai Ma, Zhuang Li, Sichao Li +4Spatiotemporal Reasoning Remote Sensing
The Australian National University · Royal Melbourne Institute of Technology · University of Sydney +1
Healthcare AI GYM for Medical Agents
May 1, 2026Minbyul JeongHealthcare On-Policy Distillation
Upstage AI
LATTICE: Evaluating Decision Support Utility of Crypto Agents
Apr 29, 2026Aaron Chan, Tengfei Li, Tianyi Xiao +3AI-Assisted Decision Making AI Agent Benchmarks
Sahara AI · University of Southern California
DV-World: Benchmarking Data Visualization Agents in Real-World Scenarios
Apr 28, 2026Jinxiang Meng, Shaoping Huang, Fangyu Lei +17Data Visualization AI Agent Benchmarks
Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences · National University of Singapore +1
PersonalHomeBench: Evaluating Agents in Personalized Smart Homes
Apr 18, 2026Manasa Bharadwaj, Yolanda Liu, InJung Yang +5AI Agent Evaluation Multimodal Agents
LG Toronto AI Lab · LG Electronics, Korea.
GTA-2: Benchmarking General Tool Agents from Atomic Tool-Use to Open-Ended Workflows
Apr 17, 2026Jize Wang, Xuanxuan Liu, Yining Li +7Tool-Use Evaluation Long-Horizon Agent Evaluation
1Shanghai Jiao Tong University · 2Shanghai AI Laboratory · 3Tencent +1
Spatial Atlas: Compute-Grounded Reasoning for Spatial-Aware Research Agent Benchmarks
Apr 13, 2026Arun SharmaSpatial Reasoning Benchmarks Question Answering
University of Minnesota, Twin Cities
AMIGO: Agentic Multi-Image Grounding Oracle Benchmark
Mar 30, 2026Min Wang, Ata MahjoubfarVLM Evaluation Visual Question Answering
Target Corporation
Two-Bridge: Exclusive Objectives and Extended Horizon StarCraft II Benchmark
Feb 19, 2026Sourav Panda, Tanmay Ambadkar, Shreyash Kale +2RL Benchmarks Game-Playing Agents
Pennsylvania State University, PA, USA.
When Actions Go Off-Task: Detecting and Correcting Misaligned Actions in Computer-Use Agents
Feb 9, 2026Yuting Ning, Jaylen Jones, Zhehao Zhang +5LLM Self-Correction Computer-Use Agents
The Ohio State University · Amazon AGI
ProAct: A Benchmark and Multimodal Framework for Structure-Aware Proactive Response
Feb 3, 2026Xiaomeng Zhu, Fengming Zhu, Weijie Zhou +8AI Agent Benchmarks Proactive Assistance
Department of Computer Science and Engineering, The Hong Kong University of Science and Technology (HKUST), Hong Kong SAR, China · Tencent, Shenzhen, China · Futian Laboratory, Shenzhen, China +1
EvoHarnessBench: Can Your Agents Keep Pace with an Evolving Harness?
Date pendingZixuan Ke, Vaidehi Patil, Haizhou Shi +9Continual Learning for LLM Agents LLM Agent Harnesses
Salesforce Research · University of North Carolina at Chapel Hill, conducted this work during an internship at Salesforce Research · University of Wisconsin–Madison +1
Harbor Adapters and Harbor-Index: Infrastructure and a Curated Meta-Dataset for Large-Scale Agentic Evaluation
Date pendingLin Shi (Audrey), Haowei Lin (Audrey), Zixuan Zhu (Audrey) +123LLM Agent Evaluation AI Agent Evaluation

May 27, 2026, cs.CLJ/K move · Enter open · S save

Ask Now, Use Later: Benchmarking the Proactivity Gap in Long-Lived LLM Agents

Bin Wu, Guanyun Zou, Bingbing Wang, Huan Zhao+1

Beijing University of Posts and Telecommunications · Nanjing University of Aeronautics and Astronautics · Harbin Institute of Technology (Shenzhen)+1

LLM Agent Memory LLM Agent Evaluation Proactive Assistance Long-Horizon LLM Agents Preference Elicitation Agent Benchmarks

Open paper AskPDF arXiv

A long-lived LLM agent, such as OpenClaw, earns its value by acting on a user's preferences and constraints across sessions, not just the current request. Yet today's agents keep what a user volunteers but rarely ask for what stays unspoken, leaving a proactivity gap in long-lived LLM agents: an agent cannot act on a preference it never obtained. As users delegate more of their affairs to agents, the impact of this gap grows. We isolate one concrete, controllable slice of this gap as Ask-to-Remember (ATR): the agent decides whether to ask now for a reusable user preference that the current task does not need but a later session with the same user will. ATR is hard even to evaluate: the right question is underdetermined and its payoff deferred to tasks that may never arise. ATRBench, to the best of our knowledge the first ATR benchmark, makes it measurable by fixing each user's preferences as hidden ground truth, so success demands asking, not recall. Across eight frontier LLM agents, defaults fall at least 62 points below an oracle handed the relevant preference, and prompting closes little of it. Diagnostics identify acquisition as the bottleneck. ATRBench surfaces this proactivity gap in current agents and offers a diagnostic testbed for closing it.

Momentum

Related topics

Latest papers 87