LLM Agents

LLM: Large Language Model

LLM agents combine language models with tools, memory, and execution environments to pursue goals through sequences of decisions rather than producing a single response. Common approaches include ReAct-style planning and tool use, retrieval and reusable skill libraries, reflection and verification, multi-agent coordination, and training with supervised trajectories or reinforcement learning; runtime harnesses also manage state, permissions, and model routing. They are tested on interactive web and coding tasks, tool-use and long-horizon benchmarks, domain workflows, and simulations of social or physical settings. A central challenge is making agents reliable and safe when tasks, instructions, and environments change while controlling cost and evaluating performance beyond narrow benchmark success.

Latest papers

711 papers129 papers in the last four weeks, up 59% on the four weeks before. 0.7% of all new papers.

Ambig-DS: A Benchmark for Task-Framing Ambiguity in Data-Science Agents
May 10, 2026Josefa Lia Stoisser, Marc Boubnovski Martell, Sidsel Boldsen +2LLM Agent Evaluation AI Agent Benchmarks
Novo Nordisk London, UK
SkillGen: Verified Inference-Time Agent Skill Synthesis
May 9, 2026Yuchen Ma, Yue Huang, Han Bao +5LLM Agent Self-Improvement LLM Agent Skill Learning
Munich Center for Machine Learning, LMU Munich · University of Notre Dame · Microsoft Research
When Agents Overtrust Environmental Evidence: An Extensible Agentic Framework for Benchmarking Evidence-Grounding Defects in LLM Agents
May 9, 2026Strick Sheng, Ziyue Wang, Liyi ZhouAI Agent Reliability LLM Agent Evaluation
The University of Sydney · Nanjing University
PAAC: Privacy-Aware Agentic Device-Cloud Collaboration
May 9, 2026Liangqi Yuan, Wenzhi Fang, Shiqiang Wang +1Privacy-Preserving Language Models AI Agent Security
Purdue University · University of Exeter
Behavioral Determinants of Deployed AI Agents in Social Networks: A Multi-Factor Study of Personality, Model, and Guardrail Specification
May 8, 2026Sarah Wilson, Diem Linh Dang, Usman Ali Moazzam +2AI Agent Evaluation LLM Agents
Columbia University
Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?
May 8, 2026Anmol Gulati, Hariom Gupta, Elias Lumer +2LLM Agent Evaluation Long-Horizon Agent Evaluation
PricewaterhouseCoopers U.S.
RelAgent: LLM Agents as Data Scientists for Relational Learning
May 8, 2026Xingyue Huang, Louis Tichelman, Jinwoo Kim +2Interpretable ML LLM Agents
University of Oxford · TU Wien · AITHYRA +1
SkillSelect-Serve: QoS-Aware Budgeted Skill Service Recommendation for LLM Agents
May 8, 2026Jingyuan Zheng, Dongjing Wang, Xin Zhang +7LLM Agent Skill Retrieval LLM Agents
School of Computer Science and Technology, Hangzhou Dianzi University, Hangzhou, 310018, China · Sanya Traditional Chinese Medicine Hospital, Sanya, 572000, China · HDU-ITMO Joint Institute, Hangzhou Dianzi University +3
The Context Gathering Decision Process: A POMDP Framework for Agentic Search
May 7, 2026Chinmaya Kausik, Adith Swaminathan, Nathan KallusAgentic Search LLM Agents
University of Michigan · Netflix
Cognitive Agent Compilation for Explicit Problem Solver Modeling
May 7, 2026Hyeongdon Moon, Carolyn Rosé, John StamperCognitive Architectures for AI Agents Cognitive Modeling
Carnegie Mellon University, Pittsburgh PA 15213, USA
Designing Intelligent Enterprise Agents: A Capability-Aligned Multi-Agent Architecture
May 7, 2026John deVadossEnterprise AI Agents AI Agent Architecture
InterWork Alliance Washington DC
Self-Programmed Execution for Language-Model Agents
May 7, 2026Luke J. O'ConnorLLM Agent Orchestration LLM Agents
Harvard Medical School Boston, MA 02115
Strat-LLM: Stratified Strategy Alignment for LLM-based Stock Trading with Real-time Multi-Source Signals
May 7, 2026Wenliang Huang, Zengyi YuLLM Evaluation LLM Alignment
School of Economics Zhejiang University of Technology Hangzhou, China · Faculty of Education East China Normal University Shanghai, China
SkillRet: A Large-Scale Benchmark for Skill Retrieval in LLM Agents
May 7, 2026Ryangkyung Kang, Hongcheol Cho, Youngeun KimAI Agent Benchmarks LLM Agent Skill Retrieval
ThakiCloud
Inference-Time Budget Control for LLM Search Agents
May 7, 2026Zhengru Fang, Senkang Forest Hu, Zhonghao Chang +6Multi-Hop QA Inference-Time Search
City University of Hong Kong · Tsinghua University · Alibaba Ant Group
SADE: Symptom-Aware Diagnostic Escalation for LLM-Based Network Troubleshooting
May 6, 2026Kuan-Hao Tseng, Niruth Bogahawatta, Yasod Ginige +3LLM Agent Orchestration LLM Agents
University of Sydney, Sydney, Australia · University of New South Wales, Sydney, Australia
OpenSeeker-v2: Pushing the Limits of Search Agents with Informative and High-Difficulty Trajectories
May 5, 2026Yuwen Du, Rui Ye, Shuo Tang +4Supervised Fine-Tuning Tool-Augmented Language Model Agents
1Shanghai Jiao Tong University
SymptomAI: Toward a Conversational AI Agent for Everyday Symptom Assessment
May 5, 2026Joseph Breda, Fadi Yousif, Beszel Hawkins +30Healthcare Medical Diagnosis
Google Research · Work done while at Google Research · Google DeepMind
An Ethical eValuation Agent (EeVA): Results of a Proof-of-Concept Test on a Prototype Agentic-like Workflow to Assist Ethical Deliberations
May 5, 2026Stephen Milford, B. Zara Malgir, Miguel VazquezAI-Assisted Decision Making LLM Agents
Institute for Biomedical Ethics, Basel University, Switzerland · Barcelona Supercomputing Center
A Skill-Based AI Agentic Pipeline for Library of Congress Subject Indexing
May 5, 2026Eric H. C. ChowLibrary and Information Science LLM Agents
School of Humanities, The University of Hong Kong
SkCC: Portable and Secure Skill Compilation for Cross-Framework LLM Agents
May 5, 2026Yipeng Ouyang, Yi Xiao, Yuhao Gu +1LLM Agent Security LLM Agent Skill Retrieval
Sun Yat-sen University
FlexSQL: Flexible Exploration and Execution Make Better Text-to-SQL Agents
May 4, 2026Quang Hieu Pham, Yang He, Ping Nie +5Execution-Guided Text-to-SQL Text-to-SQL
University of Alberta · Simon Fraser University · University of Waterloo +1
AIs and Humans with Agency
May 4, 2026David MumfordHuman-in-the-Loop AI LLM Agents
LLM-enabled Social Agents
May 4, 2026Önder Gürcan, Moharram ChallengerLLM Agents
NORCE Research AS, Universitetsveien 19, Kristiansand, Norway. · University of Antwerp and Flanders Make, Belgium.
PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments
May 4, 2026Ruoqi Liu, Imran Q. Mohiuddin, Austin J. Schoeffler +10AI Agent Benchmarks Electronic Health Records
Stanford University
A Language for Describing Agentic LLM Contexts
May 3, 2026Noga Peleg Pelc, Gal A. Kaminka, Yoav GoldbergLLM Prompting LLM Agents
Dept. of Computer Science and AI Bar-Ilan University
BIM Information Extraction Through LLM-based Adaptive Exploration
May 3, 2026Sylvain Hellin, Suhyung Jang, Stefan Fuchs +2Building Information Modeling LLM Information Extraction
Chair of Computing in Civil and Building Engineering, Georg Nemetschek Institute, Technical University of Munich, Munich, Germany
AgentStop: Terminating Local AI Agents Early to Save Energy in Consumer Devices
May 1, 2026Dzung Pham, Kleomenis Katevas, Ali Shahin Shamsabadi +1LLM Inference Efficiency On-Device Language Model Inference
University of Massachusetts Amherst Amherst, Massachusetts, USA · Brave Software London, UK · Brave Software, Imperial College London London, UK
Collaborative Agent Reasoning Engineering (CARE): A Three-Party Design Methodology for Systematically Engineering AI Agents with Subject Matter Experts, Developers, and Helper Agents
Apr 30, 2026Rahul Ramachandran, Nidhi Jha, Muthukumaran RamasubramanianLLM Agents Human-AI Collaboration
Marshall Space Flight Center, Huntsville, Alabama · The University of Alabama in Huntsville, Huntsville, Alabama
Exploring LLM Agent Designs and Interaction Modalities for Scientific Visualization
Apr 30, 2026Jackson Vonderhorst, Kuangshi Ai, Haichao Miao +2Scientific Visualization Data Visualization
Univ. Notre Dame · LLNL

May 10, 2026J/K move · Enter open · S save

Ambig-DS: A Benchmark for Task-Framing Ambiguity in Data-Science Agents

Josefa Lia Stoisser, Marc Boubnovski Martell, Sidsel Boldsen, Kaspar Märtens+1

Novo Nordisk London, UK

LLM Agent Evaluation AI Agent Benchmarks LLM Agents Data Analysis Agents

Open paper AskPDF arXiv

As data-science agents shift from co-pilots to auto-pilots, silent misframing becomes a critical failure mode. Agents quietly commit to plausible but unintended task framings, producing clean, executable artifacts that hide their incorrect assessment of the task. Existing benchmarks score whether the pipeline runs, ignoring whether the agent recognized the task was underspecified. We introduce Ambig-DS, two diagnostic suites: one for prediction-target ambiguity (Ambig-DS-Target, 51 tasks built on DSBench, a tabular modeling benchmark) and one for evaluation-objective ambiguity (Ambig-DS-Objective, 61 tasks built on MLE-bench, a Kaggle-style ML competition benchmark), constructed so that scoring uses each source benchmark's original evaluator. For every task we pair the original, fully specified version with an ambiguous variant produced by controlled edits; a human-and-LLM verification pipeline confirms each variant admits multiple plausible interpretations with decision-relevant consequences. The suites are analyzed independently and ambiguity lowers performance in both. Across five agents spanning efficient to frontier-class models, we find in our controlled diagnostic setting: (i) failures are silent commitments: wrong-target submissions on Target, wrong-metric or non-committal baseline submissions on Objective, rather than execution errors; (ii) allowing the agent to ask one clarifying question recovers much of the loss under idealized conditions, suggesting missing framing information drives a substantial part of the observed degradation; but (iii) agents cannot reliably tell when to use it: permissive prompts induce over-asking on clear tasks, while conservative prompts induce silent defaulting on ambiguous ones. Recognizing target and objective underspecification, not pipeline execution, is the bottleneck missing from standard DS-agent evaluations.