LLM Agent Evaluation

LLM: Large Language Model

The field studies whether language-model agents can use tools, manage state, follow instructions, and achieve users’ goals across multi-step interactions. Benchmarks place agents in executable settings such as coding repositories, websites, terminals, business workflows, and medical or scientific tasks, while methods use hidden tests, state checks, trajectory analysis, and simulated users to assess performance. Evaluation increasingly includes cost, latency, privacy, safety, consistency, and process quality—not just whether the final answer is correct. Results remain difficult to compare because harnesses and scoring choices can change rankings, and strong performance on short or isolated tasks often fails to transfer to long-horizon real-world work.

Latest papers

1,234 papers218 papers in the last four weeks, up 31% on the four weeks before. 1.4% of all new papers.

Adversarial Feeds Steer LLM Agent Decisions Against Their Defaults
May 30, 2026Rana Muhammad UsmanAdversarial Attacks on LLMs LLM Agent Evaluation
Independent Researcher
Not All Flips Are Conformity: Decomposing Stance Convergence in Multi-Agent LLM Debate
May 30, 2026Xiqi Hao, Zengqing Wu, Yu-Xuan Qiu +4LLM Agent Evaluation Multi-Agent Debate
Beijing Institute of Technology, Zhuhai · University of Osaka · Shenzhen University
ForeSci: Evaluating LLM Agents for Forward-Looking AI Research Judgment
May 30, 2026Qiuyu Tian, Haojie Yin, Yingce Xia +2LLM Agent Evaluation AI Agent Benchmarks
1Southeast University, Nanjing, China · 3Duke Kunshan University, Kunshan, China · 2Beijing Zhongguancun Academy, Beijing, China
Doing What They Say, Not What They Reason: Locating the Faithfulness Gap in LLM Agents
May 30, 2026Yufeng WangAI Agent Reliability LLM Agent Evaluation
BAGEN: Are LLM Agents Budget-Aware?
May 29, 2026Yuxiang Lin, Zihan Wang, Mengyang Liu +9LLM Agent Evaluation AI Agent Benchmarks
1Northwestern University · 2O2 Lab · 3Independent +5
Skill Availability and Presentation Granularity in Large-Language-Model Agents: A Controlled SkillsBench Study
May 29, 2026Xiaonan Xu, Wenjing WuLLM Agent Evaluation LLM Agent Skill Learning
Computer Information Technology, Northern Arizona University, Flagstaff, AZ, USA · Computer Science, University of Colorado Boulder, Boulder, CO, USA
Industrializing Prediction-Powered Inference: The GLIDE Library for Reliable GenAI and Agentic Systems Evaluation
May 29, 2026Grégoire Martinon, Ibrahim Merad, Mohammed RakiLLM-Assisted Annotation LLM Agent Evaluation
Emerton Data, 16 Avenue Hoche, 75008, Paris, France.
TUX: Measuring Human--AI Tacit Understanding
May 29, 2026Yueshen Li, Hanyi Min, Vedant Das Swain +1LLM Alignment LLM Agent Evaluation
University of Illinois Urbana-Champaign · New York University
BlueFin: Benchmarking LLM Agents on Financial Spreadsheets
May 29, 2026Srivatsa Kundurthy, Clara Na, Colton Moraine +6LLM Agent Evaluation AI Agent Benchmarks
1Longitude Labs Inc. · 2Cornell University · 3Carnegie Mellon University
MAVEN: Improving Generalization in Agentic Tool Calling
May 29, 2026Omkar Ghugarkar, Vishvesh Bhat, Muhammad Ahmed Mohsin +1Logical Reasoning LLM Agent Evaluation
CoreThink AI, USA · Stanford University, Stanford, CA, USA
Depth-Dependent Indirect Prompt Injection in Tool-Calling ReAct Agents: Injection Depth, Payload Framing, and Turn-Budget Sensitivity
May 29, 2026Mohammadreza RashidiLLM Agent Evaluation AI Agent Security
Department of Computer Science AI and Media Analysis Lab Berlin, Germany
On Effectiveness and Efficiency of Agentic Tool-calling and RL Training
May 28, 2026Tong Liu, Cheng Qian, Matej Cief +4Tool-Use Evaluation LLM Agent Evaluation
1LMU Munich & Munich Center for Machine Learning · 2UIUC · 3Amazon +1
Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents
May 28, 2026Minhua Lin, Juncheng Wu, Zijun Wang +14LLM Agent Evaluation Agent Harness Optimization
1The Pennsylvania State University · 2UC Santa Cruz · 3Amazon +3
Counterfactual Evaluation Reveals Hidden Capability Profiles in Clinical LLMs and Agents
May 28, 2026Matt TurkCounterfactual Evaluation Healthcare
Protege Data Lab New York City, USA
Can LLM Teams Play What? Where? When?
May 28, 2026Anastasia Kotelnikova, Viktor Byzov, Maria Dolzhenkova +1Multi-Agent LLM Systems LLM Agent Evaluation
Vyatka State University Kirov, Russia · European University at St. Petersburg St. Petersburg, Russia
SpecBench: Evaluating Specification-Level Reasoning for Software Engineering LLM Agents
May 28, 2026Grant Hamblin, Kevin Song, Zhanda Zhu +4Requirements Engineering Software Engineering Agents
University of Toronto · Toronto, Ontario, Canada · University of Waterloo +3
Specialize Roles, Mix Deployments: Pushing the Cost-Accuracy Frontier of LLM Agent Teams
May 28, 2026Yinsicheng Jiang, Liang Cheng, Yeqi Huang +6Multi-Agent LLM Systems LLM Agent Evaluation
University of Edinburgh · 2Microsoft Research
ResearchClawBench: A Benchmark for End-to-End Autonomous Scientific Research
May 28, 2026Wanghan Xu, Shuo Li, Tianlin Ye +48LLM Agent Evaluation AI Agent Benchmarks
HEART-Bench: Do LLMs Make Persona-Consistent Decisions?
May 28, 2026Weihan Peng, Chenxu Zhang, Qianao Wang +7Personality Modeling in Language Models Human Behavior Simulation
Shanghai Jiao Tong University · Imperial College London · Quwan Group +2
Does The Way You Plan Matter? An Empirical Study of Planning Representations for LLM Web Agents
May 28, 2026Alejandra Zambrano, Sara Vera Marjanovic, Imene Kerboua +2LLM Agent Evaluation LLM Agent Planning
Concordia University · Mila - Quebec AI Institute · University of Copenhagen +2
Redundant or Necessary? A Benchmark for Detecting Redundant Steps in Agent Trajectories
May 28, 2026Liang Zhu, Bo Yang, Zhinuo Zhou +5LLM Agent Evaluation Agent Trajectory Analysis
Huawei Technologies, Noah Arks’ Lab · Institute of Computing Technology, Chinese Academy of Sciences
Evolutionary Dynamics of Cooperation in Next-Generation LLM Agent Systems: A Cross-Provider Empirical Extension
May 28, 2026Francisco León Zúñiga BolívarMulti-Agent LLM Systems LLM Agent Evaluation
Institución Universitaria Colegio Mayor del Cauca · Institución Universitaria Colegio Mayor del Cauca Popayán, Colombia
Realistic honeypot evaluations for scheming propensity
May 28, 2026Victoria Krakovna, David Lindner, Lewis Ho +2LLM Agent Evaluation AI Agent Safety
1Google DeepMind
PTCG-Bench: Can LLM Agents Master Pokémon Trading Card Game?
May 28, 2026Dongdong Hua, Yifei Sun, Renhong Huang +3Game-Playing Agents LLM Agent Evaluation
Zhejiang University · FinVolution Group
MINDGAMES: A Live Arena for Evaluating Social and Strategic Reasoning in Multi-Agent LLMs
May 28, 2026Kevin Wang, Anna Thöni, Benjamin Kempinski +50LLM Agent Evaluation Strategic Reasoning Risks in Language Models
Revisiting Observation Reduction for Web Agents: Comprehensive Evaluation with a Lightweight Framework
May 28, 2026Masafumi Enomoto, Ryoma Obara, Haochen Zhang +1Web Agent Benchmarks LLM Agent Evaluation
NEC Corporation
BenchTrace: A Benchmark for Testing Reflection Ability and Controlled Evolution in LLM Agents
May 28, 2026Jiahao Huang, Fei Cheng, Junfeng Jiang +2LLM Agent Evaluation Self-Reflection in Language Model Agents
University of Tokyo · Kyoto University · National Institute of Informatics
Hallucination Mitigation with Agentic AI, Nested Learning, and AI Sustainability via Semantic Caching
May 27, 2026Diego Gosmar, Deborah A. DahlAI Agent Reliability Multi-Agent LLM Systems
Head of AI, Tesisquare · Member, Open Voice Interoperability Initiative · Linux Foundation AI & Data +4
Frontier LLM-based agents can overcome the ontology curation bottleneck for natural phenotypes
May 27, 2026James P. Balhoff, Hilmar LappLLM-Assisted Annotation LLM Agent Evaluation
Renaissance Computing Institute, University of North Carolina, Chapel Hill, North Carolina, USA · Neuromatch, USA
Do Agents Need Semantic Metadata? A Comparative Study in Agentic Data Retrieval
May 27, 2026Shiyu Chen, Tarfah Alrashed, Alon Halevy +1LLM Agent Evaluation Agentic Retrieval
Google, Mountain View, CA, USA

May 30, 2026J/K move · Enter open · S save

Adversarial Feeds Steer LLM Agent Decisions Against Their Defaults

Rana Muhammad Usman

Independent Researcher

Adversarial Attacks on LLMs LLM Agent Evaluation LLM Decision-Making LLM Agent Safety

Open paper AskPDF arXiv

LLM agents increasingly act after consuming ranked external information streams such as social feeds, search results, retrieval contexts, and email queues, yet safety evaluations almost always test the model or the user prompt in isolation, never the upstream ranker that decides what the agent reads just before it acts. We introduce a controlled protocol that holds the model, persona, topic, and final decision prompt fixed and varies only the composition and ordering of the posts an agent encounters during a preceding ten-turn "scrolling" phase, isolating the causal effect of feed curation on a downstream decision. Across 2,785 decision rollouts on four modern open instruct LLMs from three independent labs, we identify three response regimes: adversarial capitulation, default saturation, and a default-direction asymmetry in which a one-sided feed tips a decision the model was genuinely uncertain about (in the clearest cases from 5% to 100%; Fisher p as low as 3 x 10^-10) but cannot dislodge one it already favors or holds firmly. The effect follows a dose-response curve, survives a generator swap that rules out a writing-style artifact, generalizes across several decision domains including security-relevant choices such as removing a deployment approval gate or relaxing access controls, and is partly mitigated by two simple feed-level defenses; a frontier model retains its default. We characterize the recommender as a practical, default-bounded control surface for LLM agents, and argue that agent evaluations must audit the feed layer rather than the final prompt alone.