LLM Agent Evaluation

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

115 papers in the last four weeks, up 140% on the four weeks before. 1.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 793

All topics

Analogical Deep Research: Retrieving and Integrating Historical Analogies for Foresight Analysis
Jul 15, 2026Yongqiang Chen, Guangyi Chen, Yuewen Sun +1Analogical Reasoning LLM Agent Evaluation
1Mohamed bin Zayed University of Artificial Intelligence · 2Carnegie Mellon University
Structured Feedback Improves Repair in an LLM Agent Loop
Jul 15, 2026Jaideep Ray, Ankit GoyalLLM Self-Correction LLM Agent Evaluation
Independent Researcher
DevicesWorld: Benchmarking Cross-Device Agents in Heterogeneous Environments
Jul 15, 2026Huatao Li, Xinwei Geng, Yuheng Wang +9Computer-Use Agent Benchmarks LLM Agent Evaluation
School of Artificial Intelligence, Shanghai Jiao Tong University · 2Honor Device Co., Ltd
Set-shifting Behavioral Test for Harnessed Agents
Jul 15, 2026Ziwei YeLLM Agent Evaluation AI Agent Benchmarks
Learning Engagement Assistant (LEA): Cross-Course Scalability and Classroom Evaluation of an Agentic AI Tutoring System
Jul 15, 2026Teri Rumble, Javad Zarrin, P. George Lovell +1Retrieval-Augmented Generation Intelligent Tutoring Systems
Faculty of Design, Informatics and Business, Abertay University, Dundee, United Kingdom.
Do AI Agents Know When a Task Is Simple? Toward Complexity-Aware Reasoning and Execution
Jul 14, 2026Junjie Yin, Xinyu FengSoftware Engineering Agents LLM Agent Evaluation
University of Tennessee, Knoxville, TN 37996, USA · Microsoft, Redmond, WA 98052, USA
Win by Silence: Deletion Non-Monotonicity, Autonomous Exploitation, and Typed-State Gating in LLM Plan Evaluation
Jul 14, 2026Aleh ManchuliantsauReward Hacking LLM Agent Evaluation
Independent Researcher
Who Grades the Grader? Co-Evolving Evaluation Metrics and Skills for Self-Improving LLM Agents
Jul 14, 2026Xing Zhang, Guanghui Wang, Yanwei Cui +4LLM Agent Evaluation LLM Agent Self-Improvement
1AWS Generative AI Innovation Center · 2HSBC Holdings Plc., HSBC Technology Center, China
Critic Experience Bank: Self-Evolving Step-Level Confidence Estimation for LLM Agents
Jul 14, 2026Yaopei Zeng, Congchao Wang, JianHang Chen +3AI Agent Reliability LLM Agent Evaluation
Pennsylvania State University · Virginia Tech · Purdue University +1
PM-Bench: Evaluating Prospective Memory in LLM Agents
Jul 14, 2026Genglin Liu, Saadia GabrielLLM Agent Memory LLM Agent Evaluation
University of California, Los Angeles
How Many Tasks Are Enough for Agent Benchmark Decisions? A Replay Analysis of Public LLM Agent Benchmarks
Jul 14, 2026Wei-Jung HuangPairwise Comparison Benchmark Design
Independent Researcher · United States
Fin-Analyst at FinMMEval 2026 Task 3: A Live Hybrid Trading Agent with LLM Specialists and Rule-Based Signals
Jul 14, 2026Mohotarema Rashid, Lingzi Hong, Junhua Ding +1Multi-Agent LLM Systems LLM Agent Evaluation
Department of Information Science, University of North Texas, Denton, TX, United States · Department of Data Science, University of North Texas, Denton, TX, United States
Rethinking the Evaluation of Harness Evolution for Agents
Jul 14, 2026Yike Wang, Huaisheng Zhu, Zhengyu Hu +8LLM Agent Evaluation Agent Harness Optimization
Allen Institute for AI · University of Washington · Independent
The Hidden Footprint: Making Storage a First-Class Metric for LLM Agent Evaluation
Jul 13, 2026Chenglin Yu, Hongquan Gui, Ying Yu +3LLM Agent Evaluation AI Agent Benchmarks
Department of Industrial and Systems Engineering, The Hong Kong Polytechnic University · College of Economics and Management, Zhejiang Normal University · Department of Computing, The Hong Kong Polytechnic University +2
NextFund: A Unified Performance Tracking Platform for Agentic Portfolio Management
Jul 13, 2026Changlun Li, Peixian Ma, Qiqi Duan +2Quantitative Finance LLM Agent Evaluation
Paradoox AI Research
AgentCheck: A Reproduce-Intervene-Mitigate Workbench for LLM Agents over MCP
Jul 13, 2026Aritra Mazumder, Nusrat jahan LiaAI Agent Reliability LLM Agent Evaluation
University of Utah · University of Dhaka
How Do Practitioners Build SE Agents? Insights from a Mixed-Methods Study
Jul 12, 2026Yunbo Lyu, David Williams, Jieke Shi +5Software Engineering Agents LLM Agent Evaluation
Singapore Management University, Singapore · University College London, UK · Tencent, China +2
Auditing Belief-Conditioned LLM Agents in Hidden-Information Social Deduction Games
Jul 12, 2026Yuan Gao, Jiangyi Yang, Yao Zhao +1Imperfect-Information Games Social Deduction Games
University of Melbourne
Opti-Agent-Bench: Benchmarking End-to-End Optimization R&D Agents on Real-World Business Problems
Jul 12, 2026Yongchang Fu, Xinjie Huang, Chengjun Dai +3Large Language Model-Guided Optimization LLM Agent Evaluation
Ding Talk, Alibaba Group, Hangzhou, China · Zhejiang University, Hangzhou, China
MafiaScope: Non-Invasive, Time-Resolved Belief Probing for LLM Agents in Social Deduction Games
Jul 12, 2026Ilia KarpovSocial Deduction Games Theory of Mind
HSE University
Agents Don't Just Agree, They Remember: Benchmarking Persistent Sycophancy in Self-Improving Personal Agents
Jul 12, 2026Xutao Mao, Liangjie Zhao, Leyao Wang +6LLM Sycophancy LLM Agent Memory
City University of Hong Kong · Adelaide University · Yale University +3
Can Agentic Trading Systems Pay for Their Own Intelligence?
Jul 11, 2026Qiqi Duan, Changlun Li, Chen Wang +10LLM Agent Evaluation Agent Evaluation
HKUST(GZ) · Paradoox AI Research · MBZUAI +1
Dynamic Agent Skills: A Lifecycle Survey and Taxonomy of Evolving Skill Libraries
Jul 11, 2026Yubo LiLLM Agent Evaluation LLM Agent Skill Learning
Carnegie Mellon University
AgentAbstain: Do LLM Agents Know When Not to Act?
Jul 11, 2026Xun Liu, Yi Evie Zhang, Vira Kasprova +5AI Agent Reliability LLM Agent Evaluation
University of Illinois Urbana-Champaign
Who&When Pro: Can LLMs Really Attribute Failures in AI Agents?
Jul 10, 2026Jiale Liu, Huajun Xi, Shaokun Zhang +6Agent Failure Analysis LLM Agent Evaluation
Penn State University · AG2ai, Inc. · Mohamed bin Zayed University of Artificial Intelligence +2
DynamicMCPBench: A Trace-Grounded, Effect-Scored Benchmark for LLM Agents over Live MCP Servers
Jul 10, 2026Jerzy Kamiński, Ilya Galyukshev, Artem Kuznetsov +4Computer-Use Agent Benchmarks LLM Agent Evaluation
ITMO University · Central University
Do LLM-Generated Skills Make Better AI Data Scientists? A Component Ablation Across Data-Science Workflows
Jul 8, 2026Wei-Jung HuangLLM Prompting LLM Agent Evaluation
Independent Researcher United States
Evaluating SageMath-Augmented LLM Agents for Computational and Experimental Mathematics
Jul 7, 2026Pavel Snopov, German MagaiMathematical Reasoning Benchmarks Knowledge Augmentation for Language Models
School of Mathematical and Statistical Sciences, The University of Texas Rio Grande Valley, USA · Noeon Research, Tokyo, Japan.
Reliable and Developer-Aligned Evaluation of Agents for Software Engineering
Jul 7, 2026Razvan Mihai PopescuSoftware Engineering Agents LLM Agent Evaluation
Delft University of Technology Delft, The Netherlands
RuBench: A Repository-Level Agentic Coding Benchmark with Natively Authored Russian Task Specifications
Jul 7, 2026Evgeny ShilovComputer-Use Agent Benchmarks Tool-Augmented Language Model Agents
Independent Researcher

Jul 15, 2026, cs.CLJ/K move · Enter open · S save

Analogical Deep Research: Retrieving and Integrating Historical Analogies for Foresight Analysis

Yongqiang Chen, Guangyi Chen, Yuewen Sun, Kun Zhang

1Mohamed bin Zayed University of Artificial Intelligence · 2Carnegie Mellon University

Analogical Reasoning LLM Agent Evaluation Causal Reasoning in Language Models Autonomous Research Agents

Open paper AskPDF arXiv

Systematic comparisons between current situations and structurally similar past events in the historical, i.e., historical analogies, is among the most powerful tools for foresight analysis. In this work, we present a new task called Analogical Deep Research (ADR) to Large Language Model (LLM) agents and construct the first ADR benchmark ADR-bench to study whether LLM agents are able to find and leverage historical analogies when doing foresight analysis. Our investigation reveals a key obstacle: LLM agents are poor at finding analogies because they match on surface features rather than underlying mechanisms. We argue that ADR is inherently a causal question as it requires understanding why the event occurred. Based on our theoretical analysis, we propose two principles required for ADR, including the mechanism alignment and cross-analogy confirmation. Built upon our theoretical results, we propose a new agentic framework called Causal Analogical Researcher (CANA) that guides LLMs to find and integrate historical analogies. CANA incorporates a simple yet effective structural decomposition representation, and integrates structural feedback for reflective improvements of historical analogy identification and integration. We show that CANA brings up to 10% improvements in historical analogy generation, and surpasses the state-of-the-art deep research agents in the ADR-bench. Case studies with the ongoing events confirm the effectiveness of CANA in leveraging historical analogies.

Momentum

Related topics

Latest papers 793