LLM Agent Evaluation

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

115 papers in the last four weeks, up 140% on the four weeks before. 1.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 793

All topics

PowerBench: A Benchmark for Agentic Retrieval and Reasoning in Power Systems
Sep 28, 2026Xijing Wang, Yinsheng Yao, Jinru Ding +5LLM Agent Evaluation AI Agent Benchmarks
Tongji University · Johns Hopkins University · Shanghai Artificial Intelligence Laboratory +3
AgentHop: A Diagnostic Benchmark for Agentic Multi-Hop Scientific Question Answering
Sep 28, 2026Chanhee Park, Jeongho Yoon, Sungbin Han +2Multi-Hop QA Tool-Augmented Language Model Agents
Korea University · Sookmyung Women’s University
Certified Selective Automation of LLM Agent Evaluation
Sep 28, 2026Chengguang Gan, Yunhao Liang, Qinghao Zhang +1LLM-as-a-Judge Selective Prediction
Independent Researcher · University of Chinese Academy of Sciences · Pusan National University +1
ControlScope: Workflow Revision and Reliability in LLM Agents
Sep 28, 2026Jingjie Ning, Xueqi Li, Yibo Kong +1LLM Agent Evaluation LLM Agent Reliability
Carnegie Mellon University · Tsinghua University
Same Winners, Different Success Rates: Evaluating How LLM Agents Recover from Failures
Sep 28, 2026Dong Xu, Zhangfan Yang, Jiantao Wu +5LLM Agent Evaluation AI Agent Benchmarks
School of Artificial Intelligence, Shenzhen University · EasternDawn · School of Computer Science, University of Nottingham Ningbo
Frozen Judges, Moving Agents: Version-Dependent LLM-Judge Error and the Limits of Judge-Assisted Agent Evaluation
Sep 28, 2026Jiapeng LiLLM-as-a-Judge AI Agent Auditing
Designing Reliable LLM-as-a-Judge Measurement Systems for Multi-Turn Business Agents
Sep 27, 2026Kaiwen Luo, Ming GaoLLM-as-a-Judge LLM Agent Evaluation
Meta
Identical Runs, Different Results: Benchmarking AI Coding Agents on Open-Weight Models
Sep 27, 2026Eduardo Ariño de la Rubia, Szilard PafkaAI Coding Agents LLM Agent Evaluation
Central European University · Epoch
When Do Agents Help? Embedding, LLM and Agentic Alignment of Classical Texts and Their Translations
Sep 27, 2026Máté MetzgerLLM Alignment LLM Agent Evaluation
Independent Researcher
SWE-Game: Can Coding Agents Build the Games We Want?
Sep 27, 2026Xiaoyu Chen, Lai Wei, Jin Wang +8Software Engineering Agents AI Coding Agents
Shanghai Jiao Tong University · Zhongguancun Academy · Shenzhen University +3
AgentBoundary: Counterfactual Evaluation of Safety in Tool-Using LLM Agents
Sep 27, 2026Tianzhuo Yang, Zirui Mi, Yantao Huang +4LLM Agent Evaluation LLM Agents
Peking University · Beijing Academy of Artificial Intelligence
LiveOption: Evaluating LLM Agents in Structured Option Trading with Nonlinear Payoffs
Sep 27, 2026Haochen Luo, Yifan Li, Binh Minh An +4Quantitative Finance LLM Agent Evaluation
City University of Hong Kong · The Chinese University of Hong Kong (Shenzhen) · Shanghai University of Finance and Economics
TraceDance: An Automated System for Building Agent Behavior Benchmarks from Real-World Agent Deployment Traces
Sep 27, 2026Dehai Min, Daoan Zhang, Yiming Zeng +13Benchmark Construction LLM Agent Evaluation
ByteDance Inc., USA · University of Illinois at Chicago
What Does a Skill Actually Do? Estimands and Evaluation Validity for Tool and Skill Use in LLM Agents: A Critical Review
Sep 27, 2026Shuyang ZhangTool-Use Evaluation LLM Agent Evaluation
The Hong Kong Polytechnic University, Hong Kong, China
Evaluating Budgeted Context Projection with Unexecuted Companion Runs
Sep 25, 2026Guangzhe ZhangLLM Context Management LLM Agent Evaluation
Independent AI Researcher
Cheap, open agents make LLM pollution harder to mitigate
Sep 25, 2026Raluca Rilla, Anne-Marie Nussberger, Rui Mata +1AI-Generated Text Detection LLM Agent Evaluation
Center for Humans and Machines, Max Planck Institute for Human Development, Berlin, Germany · International Max Planck Research School on Learning, Institutions, and Future Evolution (LIFE), Berlin, Germany · University of Basel, Basel, Switzerland +2
Screen Before You Serve: Simulation for Production Customer Experience AI Agents at 140M Scale
Sep 24, 2026Edesio Alcoba, Kevin Rossell, Aman Gupta +13Large Language Model-Based Role-Play Simulation Customer Support Automation
Nubank · Guardrails AI
Era by Eon: Benchmarking Enterprise Agents on Hidden Knowledge
Sep 24, 2026Benjamin Gruenbaum, Doron Porat, Assaf Natanzon +3Tool-Augmented Language Model Agents LLM Agent Evaluation
Eon
SkinAgent AI: A Safety-Grounded Multimodal Agentic Framework for Non-Diagnostic Skincare Support
Sep 24, 2026Muhammad Muhtasim Shahriar, Abdullah Mohammad Sayem, Tze Hui Liew +2LLM Agent Orchestration LLM Agent Evaluation
Department of Computer Science, International Islamic University Chittagong, Chittagong, Bangladesh · Centre for Intelligent Cloud Computing, Centre of Excellence for Advanced Cloud, Faculty of Information Science and Technology, Multimedia University, Melaka, Malaysia · Multimedia University, Jalan Ayer Keroh Lama, 75450 Bukit Beruang, Melaka, Malaysia +1
IndicBankBench: Evaluating Safety and Reliability of Language Model Assistants in Indian Retail Banking
Sep 24, 2026Suvradip Paul, Chandra Bhushan, Harsh Sharma +4LLM Safety Benchmarks Financial Services
National Payments Corporation of India Mumbai, India
Forecast-Dojo: Replayable Environments for Benchmarking and Training LLM Forecasting Agents
Sep 24, 2026Liqin Ye, Haorui Wang, Fardin Ahmed +8LLM Agent Evaluation Forecasting Benchmarks
Georgia Institute of Technology · Amazon · University of Florida
RECLAIM: Can Agents Reproduce the Claims of Machine Learning Papers?
Sep 23, 2026Mithil Salunkhe, Haochen Ding, Samridhi Verma +1LLM Agent Evaluation AI Agent Benchmarks
University of Illinois Urbana-Champaign · National Center for Supercomputing Applications
Where Cyber Agents Struggle: Bottleneck Analysis of Multi-Stage LLM Agents
Sep 23, 2026Saeedeh Lohrasbi, Mohammad Mamun, Ahmed Yehia +2Agent Failure Analysis LLM Agent Evaluation
Cybersecurity, DTRC, National Research Council, Ottawa, Canada · University of Windsor, Windsor, Ontario, Canada
MolDesignBench: Evaluating LLM-based Agent for Scenario-grounded Molecular Design
Sep 23, 2026Yongjun Jeong, Hanbum Ko, Ye Rin Kim +6Molecular Optimization LLM Agent Evaluation
Department of Artificial Intelligence, Korea University · LG AI Research · Department of Statistics, Korea University
REFLEX with Jev for Efficient Selective Control in LLM Agents
Sep 22, 2026Tiantong Wu, Wei Yang Bryan LimLLM Inference Efficiency LLM Agent Evaluation
Nanyang Technological University Singapore
How Strongly Should Task State Influence an LLM Agent?
Sep 22, 2026Chenyu Zhang, Wonbin Kweon, Jiawei HanLLM Agent Evaluation Runtime Enforcement for AI Agents
University of Waterloo · Sungkyunkwan University · University of Illinois Urbana-Champaign
Testing-Driven Reliability Audit of Trajectory-Based Early Outcome Prediction for LLM Agents: Target-Specific Calibration Transfer Persists Within a Single Benchmark
Sep 22, 2026YanZe CaoLLM Agent Evaluation LLM Agent Reliability
Lizhi College, Shaanxi University of Technology, No. 1 East 1st Ring Road, Hantai District, Hanzhong, Shaanxi 723001, China
Passes Alone, Fails Together: Benchmarking Semantic Coordination in Parallel LLM-Agent Development
Sep 21, 2026Haocheng Xia, Eugene Wu, Yongjoo ParkLLM Agent Evaluation AI Agent Benchmarks
University of Illinois Urbana-Champaign Champaign, USA · Columbia University New York, USA
When LLM Agents Fail to Read the Room: ReAdapt for Relational Social Reasoning
Sep 21, 2026Jianzhe Lin, Xiaolin Li, Yunda Liu +2LLM Agent Evaluation Relational Reasoning
MetaAI · Work done at Meta
The AI Neuroscientist: An Interactive Agentic Interface for Neuroimaging Analysis
Sep 21, 2026Aakash Patel, Panos Ketonis, Shreya Saxena +2AI Agents for Scientific Discovery LLM Agent Evaluation
Department of Computer Science, Yale University · Wu Tsai Institute, Yale University · Department of Biomedical Engineering, Yale University +2

Sep 28, 2026, cs.AIJ/K move · Enter open · S save

PowerBench: A Benchmark for Agentic Retrieval and Reasoning in Power Systems

Xijing Wang, Yinsheng Yao, Jinru Ding, Yidong Jiang+4

Tongji University · Johns Hopkins University · Shanghai Artificial Intelligence Laboratory+3

LLM Agent Evaluation AI Agent Benchmarks Agentic Retrieval Power Systems

Open paper AskPDF arXiv

Large language model (LLM) agents offer new opportunities for automated analysis in industry. However, rigorous evaluation of such agents-for example, within power system scenarios-remains hindered: real operational data are confidential, and existing public resources fail to fully capture the chained dependencies and heterogeneous evidence. To address this gap, we propose PowerBench, comprising (1) a generation framework that derives interconnected heterogeneous operational data through a common dependency chain, and (2) a synthetic dataset generated by this framework. The dataset covers 761 devices across 100 device types, with 13.35 million hourly telemetry records spanning two years and 24,939 operational documents. Building on this dataset, we construct 300 questions across three task families that evaluate frontier LLMs' ability to complete analysis tasks that require autonomous evidence retrieval and reasoning across interconnected and heterogeneous data under restricted tool calls and time budgets. Results demonstrate that the evaluated frontier LLMs remain challenged on these tasks: the best model reaches only 74.2% joint accuracy. Our trace analysis further reveals that model performance varies across evidence discovery, content retrieval, tool use, reasoning over evidence, and answer submission. These findings provide detailed insights for evaluating LLM agents and guiding their reliable deployment in industry. The framework, dataset, and benchmark tasks are available at https://github.com/open-compass/PowerBench.

Momentum

Related topics

Latest papers 793