LLM Agent Evaluation

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

115 papers in the last four weeks, up 140% on the four weeks before. 1.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 793

All topics

Do Agents Dream of Root Shells? Partial-Credit Evaluation of LLM Agents in Capture the Flag Challenges
Apr 21, 2026Ali Al-Kaswan, Maksim Plotnikov, Maxim Hájek +3LLM Agent Evaluation AI Agent Security Benchmarks
Delft University of Technology Delft, The Netherlands
Forage V2: Knowledge Evolution and Transfer in Autonomous Agent Organizations
Apr 21, 2026Huaqing XieLLM Agent Evaluation Memory-Augmented Agents
Independent Researcher
AutomationBench
Apr 21, 2026Daniel Shepard, Robin SalimansLLM Agent Evaluation Business Process Automation
AI scientists produce results without reasoning scientifically
Apr 20, 2026Martiño Ríos-García, Nawaf Alampara, Chandan Gupta +5AI Agent Reliability Scientific Reasoning
Laboratory of Organic and Macromolecular Chemistry (IOMC), Friedrich Schiller University Jena, Humboldstr. 10, 07743 Jena, Germany · Department of Civil Engineering, Indian Institute of Technology Delhi, Hauz Khas, New Delhi 110016, India · School of Interdisciplinary Research, Indian Institute of Technology Delhi, Hauz Khas, New Delhi 110016, India +3
AJ-Bench: Benchmarking Agent-as-a-Judge for Environment-Aware Evaluation
Apr 20, 2026Wentao Shi, Yu Wang, Yuyang Zhao +8LLM-as-a-Judge LLM Agent Evaluation
University of Science and Technology of China · National University of Singapore · Meituan
WebCompass: Towards Multimodal Web Coding Evaluation for Code Language Models
Apr 20, 2026Xinping Lei, Xinyu Che, Junqi Xiong +16LLM Agent Evaluation Code Language Models
Nanjing University · Kuaishou Technology
Do LLMs Need to See Everything? A Benchmark and Study of Failures in LLM-driven Smartphone Automation using Screentext vs. Screenshots
Apr 20, 2026Shiquan Zhang, Tianyi Zhang, Le Fang +3Mobile GUI Automation LLM Agent Evaluation
School of Computing and Information Systems, University of Melbourne, Melbourne, Australia · School of Computer Science, University of Auckland, Auckland, New Zealand
Agents Explore but Agents Ignore: LLMs Lack Environmental Curiosity
Apr 19, 2026Leon Engländer, Sophia Althammer, Ahmet Üstün +2LLM Agent Evaluation Tool-Using Agents
Cohere · Poolside
HalluClear: Diagnosing, Evaluating and Mitigating Hallucinations in GUI Agents
Apr 19, 2026Chao Jin, Wenkui Yang, Hao Sun +6GUI Agents Hallucination in Language Models
MAIS&NLPR, Institute of Automation, Chinese Academy of Sciences
Systematic Capability Benchmarking of Frontier Large Language Models for Offensive Cyber Tasks
Apr 18, 2026Tyler H. Merves, Michael H. Conaway, Joseph M. Escobar +2LLM Agent Evaluation Cybersecurity
Department of Cybersecurity · Department of Information Sciences and Technology
If Only My CGM Could Speak: A Privacy-Preserving Agent for Question Answering over Continuous Glucose Data
Apr 18, 2026Yanjun Cui, Ali Emami, Temiloluwa Prioleau +1Privacy-Preserving Language Models Healthcare
Dartmouth College · Emory University
Don't Start What You Can't Finish: A Counterfactual Audit of Support-State Triage in LLM Agents
Apr 17, 2026Eren UnluLLM Agent Evaluation LLM Refusal Behavior
Globeholder Paris, France
Evaluating Tool-Using Language Agents: Judge Reliability, Propagation Cascades, and Runtime Mitigation in AgentProp-Bench
Apr 17, 2026Bhaskar GurramAI Agent Reliability Tool-Augmented Language Model Agents
Zasti Inc., Ashburn, VA, USA
SocialGrid: A Benchmark for Planning and Social Reasoning in Embodied Multi-Agent Systems
Apr 17, 2026Hikaru Shindo, Hanzhao Lin, Lukas Helff +2LLM Agent Evaluation AI Agent Benchmarks
Technical University of Darmstadt · German Research Center for Artificial Intelligence · CERTAIN +2
Understanding Tool-Augmented Agents for Lean Formalization: A Factorial Analysis
Apr 16, 2026Ke Zhang, Patricio Gallardo, Maziar Raissi +1Code Translation LLM Agent Evaluation
University of California, Riverside.
CoopEval: Benchmarking Cooperation-Sustaining Mechanisms and LLM Agents in Social Dilemmas
Apr 16, 2026Emanuel Tewolde, Xiao Zhang, David Guzman Piedrahita +2Social Dilemmas Cooperative Game Theory
Carnegie Mellon University · Foundations of Cooperative AI Lab (FOCAL) · Jinesis Lab, University of Toronto & Vector Institute +3
From Reactive to Proactive: Assessing the Proactivity of Voice Agents via ProVoice-Bench
Apr 16, 2026Ke Xu, Yuhao Wang, Yu WangLLM Agent Evaluation Agent Evaluation
Shanghai Jiao Tong University
Task-Level AI Readiness Assessment for Business Process Management:The T-IPO Model and LARA Matrix in Financial-Services IT Operations
Apr 16, 2026Mingjun Li, Xiaojun YeLLM Agent Evaluation Multi-Criteria Decision Analysis
Tsinghua University, Beijing, China
HWE-Bench: Benchmarking LLM Agents on Real-World Hardware Bug Repair Tasks
Apr 16, 2026Fan Cui, Hongyuan Hou, Zizhang Luo +2Automated Program Repair Electronic Design Automation
Peking University · Beijing, China
DR $^{3}$ -Eval: Towards Realistic and Reproducible Deep Research Evaluation
Apr 16, 2026Qianqian Xie, Qingheng Xiong, He Zhu +16LLM Agent Evaluation Long-Horizon Agent Evaluation
Nanjing University · M-A-P · Nanjing University of Science and Technology +2
Grounded autonomous scrutiny at scale: emergent critique from reproduction of published computational physics papers
Apr 14, 2026Haonan HuangLLM Agent Evaluation Scientific Reproducibility
Department of Physics, Princeton University
UniToolCall: Unifying Tool-Use Representation, Data, and Evaluation for LLM Agents
Apr 13, 2026Yijuan Liang, Xinghao Chen, Yifan Ge +5LLM Agent Evaluation AI Agent Benchmarks
University of Science and Technology of China · Zhejiang Key Laboratory of Industrial Intelligence and Digital Twin, Eastern Institute of Technology, Ningbo · Department of Computing, The Hong Kong Polytechnic University
Diversity Without Fidelity: A Solver-Sampler Mismatch in Multi-Agent LLM Negotiation Simulation
Apr 12, 2026Sandro AndricMulti-Agent LLM Systems Multi-Agent Negotiation
Agentic Tool Use in Large Language Models
Apr 1, 2026Jinchao Hu, Meizhi Zhong, Kehai Chen +2LLM Agent Evaluation Tool-Using Agents
School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen, China · Independent Researcher, China
Helping Customers in Distress: An LLM-powered Agent that Converses, Probes, and Routes
Mar 31, 2026Alankar Atreya, Stefan Sylvius Wagner, Devesh Batra +7Financial Services Customer Support Automation
NatWest AI Research
SciVisAgentBench: A Benchmark for Evaluating Scientific Data Analysis and Visualization Agents
Mar 31, 2026Kuangshi Ai, Haichao Miao, Kaiyuan Tang +13Scientific Visualization Data Visualization
University of Notre Dame · Lawrence Livermore National Laboratory · University of Utah +4
Confident and Wrong: Silent Semantic Failures in Coding Agents
Mar 26, 2026Aman MehtaAI Agent Reliability AI Coding Agents
Snowflake AI Research
LiveClawBench: Benchmarking LLM Agents on Complex, Real-World Assistant Tasks
Mar 20, 2026Xiang Long, Li Du, Yilong Xu +11Computer-Use Agent Benchmarks Tool-Augmented Language Model Agents
Samsung Research, Beijing, China · HKUST (Guangzhou) · City University of Hong Kong +1
$OneMillion-Bench: How Far are Language Agents from Human Experts?
Mar 9, 2026Yang Liu, Jiaqi Li, Jun Bai +20LLM Agent Evaluation AI Agent Benchmarks
BIGAI · Humanlaya · xbench +1
Safety Under Scaffolding: How Evaluation Conditions Shape Measured Safety
Mar 8, 2026David GringrasLLM Safety Benchmarks LLM Agent Evaluation
Harvard University

Apr 21, 2026, cs.AIJ/K move · Enter open · S save

Do Agents Dream of Root Shells? Partial-Credit Evaluation of LLM Agents in Capture the Flag Challenges

Ali Al-Kaswan, Maksim Plotnikov, Maxim Hájek, Roland Vízner+2

Delft University of Technology Delft, The Netherlands

LLM Agent Evaluation AI Agent Security Benchmarks LLMs for Cybersecurity Automated Grading

Open paper AskPDF arXiv

Large Language Model (LLM) agents are increasingly proposed for autonomous cybersecurity tasks, but their capabilities in realistic offensive settings remain poorly understood. We present DeepRed, an open-source benchmark for evaluating LLM-based agents on realistic Capture The Flag (CTF) challenges in isolated virtualized environments. DeepRed places an agent in a Kali attacker environment with terminal tools and optional web search, connected over a private network to a target challenge, and records full execution traces for analysis. To move beyond binary solved/unsolved outcomes, we introduce a partial-credit scoring method based on challenge-specific checkpoints derived from public writeups, together with an automated summarise-then-judge labelling pipeline for assigning checkpoint completion from logs. Using DeepRed, we benchmark ten commercially accessible LLMs on ten VM-based CTF challenges spanning different challenge categories. The results indicate that current agents remain limited: the best model achieves only 35% average checkpoint completion, performing strongest on common challenge types and weakest on tasks requiring non-standard discovery and longer-horizon adaptation.

Momentum

Related topics

Latest papers 793