LLM Agent Evaluation

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

109 papers in the last four weeks, up 88% on the four weeks before. 1.3% of all new papers.

Jul 6Week of Sep 21

Latest papers 793

All topics

ReproRepo: Scaling Reproducibility Audits with GitHub Repository Issues
Jun 16, 2026Shanda Li, Qiuhong Anna Wei, Jingwu Tang +5LLM Agent Evaluation AI Agent Benchmarks
School of Computer Science, Carnegie Mellon University · Datadog
A Framework for Evaluating Agentic Skills at Scale
Jun 16, 2026Maksim Shaposhnikov, Nicolas Fortuin, Simon Stipcich +3LLM Agent Evaluation Instruction Following
Tessl London, United Kingdom
SEAGym: An Evaluation Environment for Self-Evolving LLM Agents
Jun 16, 2026Congjie Zheng, Chuanyi Xue, Bin Liang +2LLM Agent Evaluation AI Agent Benchmarks
Department of Automation, Tsinghua University, Beijing, 100084, China · Beijing National Research Center for Information Science and Technology (BNRist), Tsinghua University, Beijing, 100084, China
Can LLMs Be CEOs? Benchmarking Strategic Resource Reallocation with Multi-Role Agent Simulation
Jun 16, 2026Yuyang Dai, Xueqing Peng, Lingfei Qian +1LLM Agent Evaluation AI Agent Benchmarks
1MBZUAI · 2Yale University
Dissecting model behavior through agent trajectories
Jun 16, 2026Gaurav Gupta, Vatshank Chaturvedi, Jun Huan +1Software Engineering Agents LLM Agent Evaluation
AWS AI Labs
MapSatisfyBench: Benchmarking Satisfaction-Aware Map Agents through Behavior-Grounded Implicit Decision Factors
Jun 16, 2026Lubin Bai, Mengyu Cao, Sixue Wang +5LLM Agent Evaluation User Preference Modeling
AMAP, Alibaba Group, Beijing, China · College of Urban and Environmental Sciences, Peking University, Beijing, China
Benchmarking LLM Agents on Meta-Analysis Articles from Nature Portfolio
Jun 15, 2026Anzhe Xie, Weihang Su, Yujia Zhou +2AI-Assisted Scientific Research LLM Agent Evaluation
Tsinghua University Beijing, China
How Much Can We Trust LLM Search Agents? Measuring Endorsement Vulnerability to Web Content Manipulation
Jun 15, 2026Yimeng Chen, Zhe Ren, Firas Laakom +3LLM Agent Evaluation AI Agent Security
Center of Excellence for Generative AI, KAUST · 2Jilin University · 3Zhejiang University +2
AgentFairBench: Do LLM Agents Discriminate When They Act?
Jun 15, 2026Triveni Morla, Rohith Reddy Bellibaltu, Manpreet Singh +1Algorithmic Fairness LLM Agent Evaluation
Florida International University, Miami, FL 33199 USA · Boston University, Boston, MA 02215 USA · Department of Computer Science and Engineering, Indian Institute of Technology Patna, Patna, Bihar, India
FraudSMSWalker: Benchmarking Agentic Large Language Models for SMS-to-Webpage Fraud Detection
Jun 15, 2026Y. H. Zhou, Z. M. Ma, Y. J. Zhou +12Web Agent Benchmarks LLM Agent Evaluation
Can Agents Infer Environment from Interaction? Evidence from Agentic Automata Learning
Jun 15, 2026Reef Menaged, Gili Lior, Shauli Ravfogel +2LLM Agent Evaluation Active Learning
The Hebrew University of Jerusalem · New York University · Google Research
Can LLM Coding Agents Reason About Time Series?
Jun 15, 2026Filip Rechtorík, Ondřej Dušek, Zdeněk KasnerTime Series Reasoning Time Series QA
Institute of Formal and Applied Linguistics Faculty of Mathematics and Physics, Charles University
An Evaluation of Data Leakage Risks in Tool-Using LLM Agents in Realistic Scenarios
Jun 15, 2026Hankyul Baek, Jaewon Noh, Sang Seo +5Data Leakage LLM Agent Evaluation
Korea AI Safety Institute · Singapore AI Safety Institute
UXBench: Measuring the Actionability of LLM-Generated UX Critiques
Jun 15, 2026Wenjie Wang, Yue Huang, Zipeng Ling +11LLM-as-a-Judge LLM Agent Evaluation
University of Notre Dame · University of Pennsylvania · University of Rochester +4
From Question Answering to Task Completion: A Survey on Agent System and Harness Design
Jun 14, 2026Jianyuan Guo, Zhiwei Hao, Chengcheng Wang +14LLM Agent Evaluation LLM Agents
Department of Computer Science, City University of Hong Kong, HKSAR, China
ToolMenuBench: Benchmarking Tool-Menu Filtering Strategies for Reliable and Efficient LLM Agents
Jun 13, 2026Rahul Suresh Babu, Laxmipriya Ganesh IyerAI Agent Reliability Tool-Use Evaluation
Independent Researcher United States of America
Can Agents Read the Room? Benchmarking Visual Social Intelligence in Multimodal Simulation
Jun 13, 2026Shijun Wan, Xuehai Wu, Jiwen Zhang +2LLM Agent Evaluation AI Agent Benchmarks
Fudan University · The Chinese University of Hong Kong · Shanghai Innovation Institute
Are Online Skill and Memory Modules Always Worth Their Tokens? A Budget-Constrained Study of Web Agents
Jun 12, 2026Sina Hajimiri, Masih Aminbeidokhti, Jose Dolz +4Web Agent Benchmarks LLM Agent Evaluation
ServiceNow AI Research · ÉTS Montreal · University of British Columbia +1
When the Tool Decides: LLM Agents Defer Blindly to Graph Neural Network Tools, and Stronger Backbones Defer More
Jun 12, 2026Zhongyuan Wang, Pratyusha VemuriGraph Neural Networks LLM Reasoning with Graphs
Communication Policy Evolution for Proactive LLM Agents
Jun 12, 2026Xinbei Ma, Jiyang Qiu, Yao Yao +10LLM Agent Evaluation LLM Agent Self-Improvement
1Shanghai Jiao Tong University · 2OPPO Research Institute
Closing the Reflection Gap: A Free Calibration Bonus for Agentic RL
Jun 12, 2026Yinglun ZhuLLM Agent Evaluation Language Model Calibration
University of California, Riverside
Formalizing Numerical Analysis: An Agent Pipeline and Quality Audit Beyond Kernel Acceptance
Jun 12, 2026Theodore Meek, Siyuan Ge, Di Qiu Xiang +2LLM Agent Evaluation Autoformalization
Math AI Lab, University of Washington, Seattle, United States · Department of Mathematics, University of Washington, University of Washington, Seattle, United States · Department of Computer Science & Engineering, University of Washington, Seattle, United States +1
Dialogue SWE-Bench: A Benchmark for Dialogue-Driven Coding Agents
Jun 12, 2026Brendan King, Jeffrey FlaniganCoding Agents LLM Agent Evaluation
University of California, Santa Cruz
SANA: What Matters for QA Agents over Massive Data Lakes?
Jun 11, 2026Austin Senna Wijaya, Jiaxiang Liu, Haonan Wang +1LLM Agent Evaluation Question Answering
Columbia University New York, USA
SpheriCity: Designing Trustworthy Conversational AI for Sustainability Decision Support
Jun 11, 2026Ahmed Qayyum, Madison Werner, Kathryn Youngblood +2AI-Assisted Decision Making Data Provenance
Department of Computer Science, Colby College Waterville, Maine, USA · Circularity Informatics Lab, University of Georgia Athens, Georgia, USA
When Plausible Is Not Realistic: Evaluating Human Mobility in LLM-Based Urban Simulation
Jun 11, 2026Gustavo H. Santos, Aline Carneiro Viana, Thiago H. SilvaHuman Behavior Simulation LLM Agent Evaluation
UTFPR, Brazil · Inria, France · U. of Toronto, Canada
LLM-as-an-Investigator: Evidence-First Reasoning for Robust Interactive Problem Diagnosis
Jun 11, 2026Fabrizio Marozzo, Pietro LiòLLM Sycophancy LLM Agent Evaluation
University of Calabria, Italy · University of Cambridge, United Kingdom
TerraBench: Can Agents Reason Over Heterogeneous Earth-System Data?
Jun 11, 2026Dat Tien Nguyen, Thao Nguyen, Fadillah Adamsyah Maani +5AI Agents for Scientific Discovery Scientific Reasoning
Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE · ADIA Lab, Abu Dhabi, UAE
Nous: An Attempt to Extract and Inject the Cognition Behind Prediction-Market Behavior
Jun 11, 2026Haowei QianCognitive Modeling LLM Agent Evaluation
Independent Researcher
Iterating Toward Better Search: A Two-Agent Simulation Framework for Evaluating Agentic Search Architectures in E-Commerce
Jun 11, 2026Jetlir Duraj, Jayanth Yetukuri, Shuang Zhou +4LLM Agent Memory E-Commerce Search
eBay Inc.

Jun 16, 2026, cs.CLJ/K move · Enter open · S save

ReproRepo: Scaling Reproducibility Audits with GitHub Repository Issues

Shanda Li, Qiuhong Anna Wei, Jingwu Tang, Valerie Chen+4

School of Computer Science, Carnegie Mellon University · Datadog

LLM Agent Evaluation AI Agent Benchmarks ML Reproducibility Scientific Reproducibility

Open paper AskPDF arXiv

Reproducing research results from papers and released code is central to scientific progress. Existing works have introduced benchmarks to evaluate whether LLM agents can assist with reproducibility, but they are difficult to scale due to their reliance on substantial manual effort for data curation and evaluation. We introduce ReproRepo, a scalable framework for reproducibility evaluation that leverages human-raised GitHub issues as naturally occurring supervision on realistic reproduction blockers. We instantiate ReproRepo on 1,149 recent machine learning papers from major conferences and evaluate four frontier model-agent configurations. Our results show that LLM agents, even without executing code, can identify many real-world reproducibility problems from paper-repository pairs: the best agent in our study, namely Codex with GPT-5.5, surfaces at least one semantically related human-reported blocker for

\sim

90% of papers in the study. Further analysis shows that agents are particularly effective for surfacing visible failures and identifying the right semantic region, but may still be insufficient in exact localization. ReproRepo can serve as a reusable, scalable framework for future evaluations of LLM agents on real-world reproducibility auditing. Our code is released at https://github.com/LithiumDA/ReproRepo.

Momentum

Related topics

Latest papers 793