LLM Agent Evaluation

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

115 papers in the last four weeks, up 140% on the four weeks before. 1.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 793

All topics

What Do CAE Simulation Agents Really Need Beyond a Generic Harness?
Sep 3, 2026Jiasheng Shi, Tianhan ZhangMulti-Agent LLM Systems LLM Agent Evaluation
DP Technology · Beijing, China · School of Astronautics, Beihang University +1
KC-Bench: A Dynamic Interactive Benchmark for Evaluating Knowledge Conflicts in LLM Agents
Sep 3, 2026Yaxing Lyu, Shengjie Zhou, Binbin Toh +2Knowledge Conflicts in Language Models LLM Agent Evaluation
1Shanghai Artificial Intelligence Laboratory · 2The University of Hong Kong · 3Xiamen University Malaysia +1
MasterControl Seventeen Every Time
Sep 2, 2026MasterControl AI LabLLM Agent Evaluation Text-to-SQL
MasterControl AI Lab
EarlyEval: Cheaper Agent Evaluation via Early Outcome Prediction
Sep 2, 2026Yuling Shi, Zhensu Sun, Junsen Dong +3LLM Agent Evaluation
Shanghai Jiao Tong University, Shanghai, China · Singapore Management University, Singapore · East China Normal University, Shanghai, China +1
CivBench: A Long-Horizon Benchmark for Tool-Mediated Agents in Civilization VI
Sep 2, 2026Austin Tudor David Andrews, Liam Wilkinson, Jamie Heagerty +3LLM Agent Evaluation LLM Tool Use
University of Oxford · Tony Blair Institute for Global Change · Google DeepMind +1
A Tri-Agent Framework for Evaluating and Aligning Question Clarification Capabilities of Large Language Models
Sep 2, 2026Yikai Zhao, Saurabh Pandey, Pradeep Kumar MisraMulti-Agent LLM Systems Clarification Question Generation
Amazon Dallas, Texas, USA · Amazon Seattle, Washington, USA
When Agents Implement Systems: A Case Study in Defects, Detection, and Evaluation Rigor
Sep 2, 2026Phanindra Reddy MadduruSoftware Engineering Agents AI Coding Agents
Efficient SWE Agent Benchmarking via Trajectory-Aware Evaluation
Sep 1, 2026Kefeng Duan, Dewu Zheng, Yanlin Wang +7Software Engineering Benchmarks LLM Agent Evaluation
School of Software Engineering, Sun Yat-sen University, Zhuhai, China · Huawei Cloud Computing Technologies Co., Ltd., China · College of Computer Science and Technology, Zhejiang University, Hangzhou, China
When Guardrails Look Effective: Construct Validity Failures in LLM Agent Commerce Evaluation
Sep 1, 2026Peiying Zhu, Sidi ChangLLM Guardrails LLM Agent Evaluation
Blossom AI · San Francisco, CA, USA · Blossom AI Labs +1
HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?
Sep 1, 2026Yuhao Wu, Jingyuan Zhang, Jiajun Shi +16LLM Agent Evaluation Agent Harness Optimization
What Does an Agentic Software Engineering Benchmark Measure? Profiling Task Demands and Agent Behaviour Beyond What Category Labels Reveal
Sep 1, 2026Radin Shayanfar, Keheliya Gallaba, Ahmed E. HassanSoftware Engineering Benchmark Design
Queen’s University
WorldBench: Culturally Grounded Benchmark for Multilingual Agents
Sep 1, 2026Leonardo Ranaldi, Sherrie Shen, Jushi Kai +1Cross-Cultural Language Model Evaluation LLM Agent Evaluation
(•) ILCC, School of Informatics, University of Edinburgh · (•,◦) ILCC, School of Informatics, University of Edinburgh · (◦) School of Artificial Intelligence, Shanghai Jiao Tong University
Calibration is the Bottleneck: An Action-Class Diagnostic of Multi-Turn Tool-Calling
Sep 1, 2026Kangjia Zhao, Jiajun Li, Haozhan Shen +8Tool-Use Evaluation LLM Agent Evaluation
Zhejiang University · Shanghai Jiao Tong University · National University of Singapore +3
Skill Following: Evaluating Actual Skill Use in Retrieval-Enabled LLM Agents
Sep 1, 2026Seonghyeon Cho, Chanjun ParkTool-Use Evaluation Tool-Augmented Language Model Agents
Soongsil University
Exploring Collaboration between a language and a non-language agent
Aug 31, 2026Harini S, Somesh Singh, Yaman K Singla +3Multi-Agent LLM Systems LLM Agent Evaluation
Adobe Media and Data Science Research (MDSR) · IIIT-Delhi · IIT Kanpur +1
mimeo: Compiling Public Expert Corpora into Agent Skills and Testing What Transfers
Aug 31, 2026Timothy KassisTool-Augmented Language Model Agents Personality Modeling in Language Models
K-Dense, Inc.
RestoreBench: Can AI Agents Restore Power Flow Convergence?
Aug 31, 2026Riccardo Mansutti, Andrea Pomarico, Robert Jakob +3Multi-Agent LLM Systems LLM Agent Evaluation
1ETH Zürich · 2Politecnico di Milano · 3Harvard University
Toward Workflow-Aware Benchmarking for Healthcare NLP Agents
Aug 31, 2026Junyi Yao, Baichuan Li, Zihao Zheng +1Benchmark Design LLM Agent Evaluation
Washington University in St. Louis, USA · Southern Methodist University, USA
S3Gym: Can LLMs Turn Self-Testing and Self-Judging into Self-Improvement?
Aug 31, 2026Jiajun Shi, Siyuan Tao, Yuhao Wu +18LLM Agent Evaluation Language Model Self-Assessment
ByteDance Seed
Measure Before You Manage: Evaluating Agent Working Memory in Coding Agents
Aug 31, 2026Le Chen, Zishen Wan, Baixi Sun +6Agent Memory AI Coding Agents
Argonne National Laboratory · Columbia University · University of Houston
ATLAS: Dual-Horizon Diagnostic Evaluation for Industrial Tool-Use Agents
Aug 31, 2026Wei Chen, Peilun Zhou, Zhaoyu Hu +8Tool-Use Evaluation LLM Agent Evaluation
University of Science and Technology of China, Meituan · Meituan
Ignorance or Incompetence? Constructing Knowledge-Gated, Verifiable Tasks for LLM Agents
Aug 31, 2026Hanlin Tian, Minhao Li, Yu Mi +5LLM Agent Evaluation AI Agent Benchmarks
1DataGrids · 2Shanghai University · 3Peking University +3
Can LLMs Take the Pulse of the Economy? A Real-Time Evaluation of LLM Nowcasts on Macroeconomic Indicators
Aug 31, 2026Xinyue Zhao, Ruiyi Zhang, Liqin Ye +3Time Series Forecasting LLM Agent Evaluation
Georgia Institute of Technology · University of California San Diego
Can LLM Agents Discover? Evaluating Creativity on ML Engineering Tasks
Aug 30, 2026Shitanshu Bhushan, Yunxiang Zhang, Lu WangCreativity Assessment LLM Agent Evaluation
Computer Science and Engineering University of Michigan Ann Arbor, MI, USA
EDGE: Engine for Deterministic Graph Evaluation through Conversation Simulation from Graph Structured DSL Configuration
Aug 30, 2026Ram Kulathumani, Pushkar Nagar, Regunathan Radhakrishnan +6AI Agent Reliability LLM Agent Evaluation
Salesforce, Palo Alto, USA.
Benchmarking LLM Judges for Voice-Agent Evaluation: Reliability, Calibration, and Human Oversight
Aug 25, 2026Anupam Purwar, Shashank Singh, Kritika SrivastavaVoice Agent Evaluation LLM-as-a-Judge
Sprinklr AI Gurugram, India · Sprinklr AI Bengaluru, India
K-Bench: measuring model performance on real scientific agent requests
Aug 21, 2026Aubrey M. Brueckner, Darshil Patel, Yuhuan He +1AI for Science LLM Agent Evaluation
1K-Dense, Inc.
Designing a Robust LLM-Based Evaluation System for Agentic AI in Drug Discovery Through Human Alignment
Aug 21, 2026Emma Granqvist, Rocío Mercado, Samuel GenhedenAI Agents for Scientific Discovery LLM-as-a-Judge
Molecular AI, Discovery Sciences, R&D, AstraZeneca, Gothenburg, Sweden · Department of Computer Science and Engineering, Section for Data Science and AI, Chalmers University of Technology and University of Gothenburg, Gothenburg, Sweden · Science for Life Laboratory (SciLifeLab), Gothenburg, Sweden
QuoteBench: How Matched Scores Can Hide Command-Path Failures
Aug 13, 2026Shangao Li, Yao Zhang, Volker Tresp +1LLM Agent Evaluation Agent Evaluation
1Stony Brook University · 2LMU Munich · 3Munich Center for Machine Learning
SteerBench-Work: A Benchmark for Agent Steering at Action Boundaries
Aug 12, 2026Oguz Serdar, Cuneyt MertayakLLM Agent Evaluation Tool-Using Agents
AgentDock

Sep 3, 2026, cs.CEJ/K move · Enter open · S save

What Do CAE Simulation Agents Really Need Beyond a Generic Harness?

Jiasheng Shi, Tianhan Zhang

DP Technology · Beijing, China · School of Astronautics, Beihang University+1

Multi-Agent LLM Systems LLM Agent Evaluation LLM Agent Harnesses

Open paper AskPDF arXiv

Computer-aided engineering (CAE) simulation is among the largest and most demanding areas of engineering, where setting up a solver such as OpenFOAM, FEniCS, or COMSOL takes real expertise. Large language model (LLM) agents promise to turn a natural-language request into a working simulation, and recent CAE agents add simulation-specific machinery: multi-agent decomposition, domain retrieval, and scripted reflection. That machinery suited weak base models; modern harnesses already supply multi-turn reasoning, tool use, and execution feedback. We ask what a CAE simulation agent still needs beyond a generic harness. With information access and repair budget held fixed, a single-agent harness matches or beats multi-agent specialized systems (FoamBench 96.4% vs.\ 88.2%). Ablations trace this to capabilities the harness already provides: execution-feedback repair lifts FoamBench from 71.8% with no repair round to 96.4%, while scripted reflection adds nothing. The one input that still helps is domain knowledge supplied as solver tutorials, our largest measured gain (80.9% to 96.4%).

Momentum

Related topics

Latest papers 793