LLM Agent Evaluation

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

115 papers in the last four weeks, up 140% on the four weeks before. 1.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 793

All topics

Same Pieces, Different Servers: A Tetris Benchmark for AI Agents as Served
Mar 2, 2026Haochuan WangLLM Inference Efficiency LLM Agent Evaluation
Massachusetts Institute of Technology
AgenticRAGTracer: A Hop-Aware Benchmark for Diagnosing Multi-Step Retrieval Reasoning in Agentic RAG
Feb 22, 2026Qijie You, Wenkai Yu, Wentao ZhangMulti-Hop QA LLM Agent Evaluation
University of Science and Technology Beijing · 2Peking University · 3Zhongguancun Academy +1
Evaluating Test-Time Scaling of General LLM Agents
Feb 22, 2026Xiaochuan Li, Ryan Ming, Pranav Setlur +6LLM Agent Evaluation Test-Time Scaling
Language Technologies Institute, School of Computer Science, Carnegie Mellon University · Meta
Evaluating Collective Behaviour of Hundreds of LLM Agents
Feb 18, 2026Richard Willis, Jianing Zhao, Yali Du +1Social Dilemmas Multi-Agent LLM Systems
King’s College London London, United Kingdom · King’s College London · Turing Institute +1
AD-Bench: A Real-World, Trajectory-Aware Advertising Analytics Benchmark for LLM Agents
Feb 15, 2026Lingxiang Hu, Yiding Sun, Tianle Xia +6Online Advertising LLM Agent Evaluation
Tencent
Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents?
Feb 12, 2026Thibaud Gloaguen, Niels Mündler-Sasahara, Mark Niklas Müller +2AI Coding Agents Coding Agents
Department of Computer Science ETH Zurich
When Agents Disagree With Themselves: Behavioral Consistency as an Uncertainty Signal for LLM Agents
Feb 12, 2026Aman MehtaAI Agent Reliability Selective Prediction
Snowflake AI Research
ReplicatorBench: Benchmarking LLM Agents for Replicability in Social and Behavioral Sciences
Feb 11, 2026Bang Nguyen, Dominik Soós, Qian Ma +8LLM Agent Evaluation AI Agent Benchmarks
University of Notre Dame · Computer Science and Engineering · Notre Dame, IN, United States +8
Evaluating Memory Structure in LLM Agents
Feb 11, 2026Alina Shutova, Alexandra Olenina, Ivan Vinogradov +1LLM Agent Memory LLM Agent Evaluation
HSE University · Yandex · YSDA +1
Do Web Agents Investigate Before They Decide?
Feb 5, 2026Syed Nazmus Sakib, Nafiul Haque, Tapodhir Karmakar Taton +2Web Agent Benchmarks Hallucination in Language Models
Department of Robotics and Mechatronics Engineering, University of Dhaka · Department of Computer Science and Engineering, University of Dhaka
Architectural Design, Not Only Model Intelligence, Governs Multi-Agent LLM Performance
Feb 3, 2026Abdelghny Orogat, Ana Rostam, Essam MansourMulti-Agent LLM Systems Multi-Agent Orchestration
Concordia University
MAS-ProVe: Understanding the Process Verification of Multi-Agent Systems
Feb 3, 2026Vishal Venkataramani, Haizhou Shi, Zixuan Ke +6Multi-Agent LLM Systems LLM-as-a-Judge
1Rutgers University · 2Salesforce AI Research
AgentRx: Diagnosing AI Agent Failures from Execution Trajectories
Feb 2, 2026Shraddha Barke, Arnav Goyal, Alind Khare +3Agent Failure Analysis AI Agent Reliability
1Microsoft Research · 2Microsoft · 3UIUC
Why Your Deep Research Agent Fails? On Hallucination Evaluation in Full Research Trajectory
Jan 30, 2026Yuhao Zhan, Tianyu Fan, Linxuan Huang +2Deep Research Agents Hallucination Detection
Zhejiang University · The University of Hong Kong
DSAEval: Evaluating Data Science Agents on a Wide Range of Real-World Data Science Problems
Jan 20, 2026Maojun Sun, Yifei Xie, Yue Wu +5LLM Agent Evaluation Agent Evaluation
Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University · Department of Applied Mathematics, The Hong Kong Polytechnic University
Real-Time Deadlines Reveal Fragile Temporal Adaptation in LLM Strategic Dialogues
Jan 19, 2026Neil K. R. Sehgal, Sharath Chandra Guntuku, Lyle UngarLLM Agent Evaluation LLM Decision-Making
University of Pennsylvania
Replayable Financial Agents: A Determinism-Faithfulness Assurance Harness for Tool-Using LLM Agents
Jan 17, 2026Raffi KhatchadourianAI Agent Reliability LLM Agent Evaluation
IBM Financial Services Market, New York, NY, USA
AstroAgentBench: Evaluating Agentic Planning on Space Mission Planning Tasks
Jan 16, 2026Weiyi Wang, Xinchi Chen, Jingjing Gong +2LLM Agent Evaluation AI Agent Benchmarks
Fudan University · OpenMOSS Team · Shanghai Innovation Institute
IDRBench: Benchmarking the Interactive Capabilities of Deep Research Agents
Jan 10, 2026Yingchaojie Feng, Qiang Huang, Xiaoya Xie +4Deep Research Agents LLM Agent Evaluation
School of Computing, National University of Singapore · School of Intelligence Science and Engineering, Harbin Institute of Technology (Shenzhen) · Zhejiang University +1
Identifying and Mitigating Bottlenecks in Role-Playing Agents: A Systematic Study of Disentangling Character Profile Axes
Jan 8, 2026Yonghyun Jun, Junhyuk Choi, Jeonghyun Park +3LLM Alignment Personality Modeling in Language Models
Chung-Ang University, Seoul, Korea
MCPAgentBench: A Real-world Task Benchmark for Evaluating LLM Agent MCP Tool Use
Dec 31, 2025Zixiang Liu, Wenrui Liu, Elsie Dai +5LLM Agent Evaluation AI Agent Benchmarks
Peking University · ZTE
DEER: A Benchmark for Evaluating Deep Research Agents on Expert Report Generation
Dec 19, 2025Janghoon Han, Heegyu Kim, Changho Lee +6Deep Research Agents LLM Agent Evaluation
LG AI Research · University of Illinois Chicago · University of Michigan, Ann Arbor
AGAPI-Agents: An Open-Access Agentic AI Platform for Accelerated Materials Design on AtomGPT.org
Dec 12, 2025Jaehyung Lee, Justin Ely, Kent Zhang +3AI Agents for Scientific Discovery LLM Agent Evaluation
Department of Materials Science and Engineering, Johns Hopkins University, Baltimore, MD 21218, USA · Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD 21218, USA · Department of Physics and Astronomy, West Virginia University, Morgantown, WV 26506, USA
PPTArena: A Benchmark for PowerPoint Editing
Dec 2, 2025Michael Ofengenden, Yunze Man, Ziqi Pang +2LLM Agent Evaluation AI Agent Benchmarks
University of Illinois Urbana-Champaign · University of California, Berkeley
Multi-Agent LLM Orchestration Achieves Deterministic, High-Quality Decision Support for Incident Response
Nov 19, 2025Philip DrammehMulti-Agent LLM Systems LLM Agent Orchestration
Independent Researcher
CostBench: Evaluating Multi-Turn Cost-Optimal Planning and Adaptation in Dynamic Environments for LLM Tool-Use Agents
Nov 4, 2025Jiayu Liu, Cheng Qian, Zhaochen Su +4LLM Planning LLM Agent Evaluation
The Hong Kong University of Science and Technology · University of Illinois Urbana-Champaign · Tsinghua University
Beyond Reactivity: Measuring Proactive Problem Solving in LLM Agents
Oct 22, 2025Gil Pasternak, Dheeraj Rajagopal, Julia White +4LLM Agent Evaluation AI Agent Benchmarks
Check Yourself Before You Wreck Yourself: Selectively Quitting Improves LLM Agent Safety
Oct 18, 2025Vamshi Krishna Bonagiri, Ponnurangam Kumaragurum, Khanh Nguyen +1LLM Agent Evaluation Tool-Using Agents
University of California, Berkeley · Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · International Institute of Information Technology, Hyderabad (IIIT Hyderabad)
BuildBench: Benchmarking LLM Agents on Compiling Real-World Open-Source Software
Sep 27, 2025Zehua Zhang, Ati Priya Bajaj, Divij Handa +14Software Engineering Agents Software Engineering Benchmarks
School of Computing and Augmented Intelligence, Arizona State University
HumanAgencyBench: Scalable Evaluation of Human Agency Support in AI Assistants
Sep 10, 2025Benjamin Sturgeon, Daniel Samuelson, Jacob Haimes +1LLM Agent Evaluation Human-AI Interaction
Apart Research · AI Safety Cape Town · University of Chicago +2

Mar 2, 2026, cs.LGJ/K move · Enter open · S save

Same Pieces, Different Servers: A Tetris Benchmark for AI Agents as Served

Haochuan Wang

Massachusetts Institute of Technology

LLM Inference Efficiency LLM Agent Evaluation LLM Inference Serving AI Agent Benchmarks LLM Agent Serving

Open paper AskPDF arXiv

An agent meets a model as served: through an endpoint with a price card, a shared cache and other tenants, or on whatever hardware a self-hosted model runs. Benchmarks rank the weights. We introduce a Tetris benchmark that measures what agents get from models as served: every move is scored against an oracle, and every agent receives the same pieces. In five pre-specified experiments with nine open-weight models on one serverless provider, plus an open decision model self-hosted on a CPU, we find that price and size do not predict decision quality; that resending history costs almost nothing when cached input is free, would cost eleven to twelve times more if it were not, and makes play worse; that deployments keep between 4 and more than 64 agent contexts warm, in line with their throughput rather than the model's KV-cache size; that an agent's own long requests slow its slowest short decisions more than tenfold, which a simple admission rule cuts by 59% without hurting play; and that the decision model plays mid-pack but takes 27 s per move on a CPU, about 650 times longer than reported on a GPU. Architecture predicts some of what an agent sees; the deployment sets the rest.

Momentum

Related topics

Latest papers 793