Automated Evaluation

Follow this topic Ask about this topic

Momentum

48 papers in the last four weeks, up 12% on the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28

Latest papers 321

All topics

SkillAudit: From Fixed-Suite Benchmarking to Skill-Centered Assessment
Jun 21, 2026Dexu Yu, Youhua Li, Zhaoyang Guan +12AI Agent Auditing Automated Evaluation
Northeastern University · City University of Hong Kong · DeciLix Lab +8
MacAgentBench: Benchmarking AI Agents on Real-World macOS Desktop
Jun 21, 2026Yikun Fu, Bowen Fu, Zhenyu Wu +10Computer-Use Agent Benchmarks Computer-Use Agents
1Shanghai Jiao Tong University · 2Shanghai AI Laboratory · 3Xi’an Jiaotong University +5
Can LLMs Control Readability? A Multi-Dimensional Evaluation Framework for CEFR-Controlled Arabic Generation
Jun 20, 2026Nour Rabih, Chatrine Qwaider, Ted BriscoeLLM Evaluation Controllable Text Generation
Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)
LLM-Based Multi-Reference Evaluation for Efficient and Robust Assessment of Phrase Break Annotations
Jun 19, 2026Younghan Park, Hoyeon Lee, Hawon Jeong +1Speech Prosody Automated Evaluation
NAVER Cloud, South Korea · Yonsei University, South Korea · KAIST AI, South Korea
Power Systems Agent Benchmark: Executable Evaluation of AI Agents in Electric Power Engineering
Jun 18, 2026Sergei TrashchenkovLLM Agent Evaluation AI Agent Benchmarks
Blame is easier than praise: Measuring off-ball defensive performance in football
Jun 18, 2026Jonas Bischofberger, Runqing Ma, Pascal Bauer +2Automated Evaluation Fine-Grained Credit Assignment
Centre for Sport Science and University Sports, University of Vienna, Vienna, Austria · Vienna Doctoral School of Pharmaceutical, Nutritional and Sport Sciences (VDS-PhaNuSpo), University of Vienna, Vienna, Austria · VfB Stuttgart 1893 AG, Stuttgart, Germany +2
RECOM: A Validity Discrimination Tradeoff in Automatic Metrics for Open Ended Reddit Question Answering
Jun 17, 2026Pushwitha Krishnappa, Amit Das, Vinija Jain +2Open-Ended Generation LLM Evaluation
University of Alabama Huntsville · University of North Alabama · Stanford University +2
A Clinician-Centered Pipeline for Annotation and Evaluation in Ultrasound AI Studies
Jun 17, 2026Fangyijie Wang, Jianjun Yu, Wentao Shi +4Human-in-the-Loop Annotation Human-in-the-Loop Evaluation
Research Ireland Centre for Research Training in Machine Learning · School of Medicine, University College Dublin, Dublin, Ireland · The Third People’s Hospital of Zhenjiang City, Zhenjiang, China +3
Mitigating Scoring Errors and Compensating for Nonverbal Subtests in Speech-Based Dementia Assessment
Jun 17, 2026Franziska Braun, Christopher Witzl, Andreas Erzigkeit +4Speech-Based Dementia Detection Automated Evaluation
Technische Hochschule Nürnberg · Geromed GmbH · PMU Klinikum Nürnberg
ELSA: Acoustic Event-Level Semantic Alignment for Fine-Grained Reference-Free Text-to-Audio Evaluation
Jun 16, 2026Shuntaro Suzuki, Kento Tokura, Daichi Yashima +3Text-to-Audio Generation Automated Evaluation
Keio University, Japan
UXBench: Measuring the Actionability of LLM-Generated UX Critiques
Jun 15, 2026Wenjie Wang, Yue Huang, Zipeng Ling +11LLM-as-a-Judge LLM Agent Evaluation
University of Notre Dame · University of Pennsylvania · University of Rochester +4
RecourseBench: A Modular Framework for Reproducible Algorithmic Recourse Evaluation
Jun 15, 2026Zahra Khotanlou, Hashir Ahmed, Chenghao Tan +2Benchmark Design Automated Evaluation
University of Waterloo
AIChilles: Automatically Uncovering Hidden Weaknesses in AI-Evolved Systems
Jun 14, 2026Yajie Zhou, Ao Li, Ashwin Silla +2Automated Software Testing Differential Testing
University of Maryland, College Park · Carnegie Mellon University
ReportQA: QA-Based Radiology Report Evaluation
Jun 13, 2026Yiming Shi, Shaoshuai Yang, Xi Chen +10Radiology Report Generation Medical VLMs
Department of Electronic Engineering, Tsinghua University · Beijing Electronic Digital & Intelligence · College of AI, Tsinghua University +1
A Benchmark and Framework for Evaluating Next Action Predictions in Spreadsheets
Jun 11, 2026Tejas Agrawal, Vu Le, Sumit Gulwani +1Next-Action Prediction Automated Evaluation
1Microsoft, Bangalore, India · 2Microsoft, Redmond, USA · 3Microsoft, Keerbergen, Belgium.
AgentBeats: Agentifying Agent Assessment for Openness, Standardization, and Reproducibility
Jun 11, 2026Xiaoyuan Liu, Jianhong Tu, Yuqi Chen +26Software Engineering Agents AI Agent Evaluation
UC Berkeley · University of California, Santa Cruz · EPFL +12
DailyReport: An Open-ended Benchmark for Evaluating Search Agents on Daily Search Tasks
Jun 11, 2026Jingxuan Han, Wei Liu, Mingyang Zhu +8LLM Agent Evaluation Web Search Agents
University of Science and Technology of China · Meituan
Measuring Semantic Progress in Multi-turn Dialogue via Information Gain
Jun 10, 2026Paul He, Shiva Kasiviswanathan, Dominik JanzingLLM Evaluation Multi-Turn Dialogue Evaluation
NTU Singapore · Amazon Research, Tübingen, Germany · Amazon
Automated Creativity Evaluation of Language Models Across Open-Ended Tasks
Jun 10, 2026Min Sen Tan, Zachary Kit Chun Choy, Syed Ali Redha Alsagoff +4Creativity Assessment LLM Evaluation
1Raffles Institution · College of Computing and Data Science, Nanyang Technological University · 3Lee Kong Chian School of Medicine, Nanyang Technological University +1
Sensitivity Analysis of Generative Spatial Audio Metrics: A Study on Responsiveness, Smoothness, and Symmetry
Jun 10, 2026Purnima Kamath, Adrian S. Roman, Koichi Saito +2Automated Evaluation Spatial Audio
New York University, New York, USA · Sony AI, New York, USA · Sony Group Corporation, Tokyo, Japan
Are We Evaluating Knowledge or Phrasing? Mitigating MCQA Sensitivity with ParaEval
Jun 9, 2026João Maria Janeiro, Mathurin Videau, Andrea Caciolai +3LLM Evaluation Multiple-Choice Question Answering
FAIR at Meta · Sorbonne Université, CNRS, ISIR, F-75005 Paris, France · Criteo AI Lab, Paris, France
DeRA-MOS: Optimizing Text-to-Music Evaluation via Decoupled Listwise Ranking and Modality Alignment
Jun 8, 2026Chien-Chun Wang, Hung-Shin Lee, Hsin-Min Wang +1Text-to-Music Generation Music Generation Evaluation
E.SUN Financial Holding Co., Ltd., Taipei 10546, Taiwan · United Link Co., Ltd., Taipei 11493, Taiwan · Institute of Information Science, Academia Sinica, Taipei 11529, Taiwan +1
A Finetuned SpeechLLM for Joint Multi-Granular L2 Assessment and Natural-Language Rationales
Jun 8, 2026Aditya Kamlesh Parikh, Cristian Tejedor-Garcia, Catia Cucchiarini +1Audio-Language Model Evaluation Speech Processing
Centre for Language Studies, Radboud University, Nijmegen, The Netherlands
Reasoning without Gold Standards: A Proxy-Judge Theory of Autoformalization
Jun 8, 2026Lei Xu, Xin Quan, André FreitasLLM Self-Refinement LLM-as-a-Judge
1Idiap Research Institute, Switzerland · 2École Polytechnique Fédérale de Lausanne (EPFL), Switzerland · Department of Computer Science, University of Manchester, United Kingdom +1
Correct Looks Better: Pairwise Comparisons Reveal Accuracy Rankings
Jun 8, 2026Mina Remeli, Moritz HardtPairwise Comparison LLM-as-a-Judge
Max Planck Institute for Intelligent Systems, Tübingen, Germany, Tübingen AI Center
RadOT-Eval: Auditable Structured-Evidence Transport for Radiology Report Evaluation
Jun 7, 2026Weixin Liu, Juming Xiong, Yang Li +5Radiology Report Generation Automated Evaluation
Vanderbilt University, Nashville, TN, USA · Vanderbilt University Medical Center, Nashville, TN, USA · Virginia Tech, Blacksburg, VA, USA
SLMJury: Can Small Language Models Judge as Well as Large Ones?
Jun 5, 2026Anish Laddha, Nitesh Pradhan, Gaurav SrivastavaLLM-as-a-Judge Language Model Generation Evaluation
Department of Computer Science and Engineering, LNMIIT, Jaipur, India · Department of Computer Science, Virginia Tech, Blacksburg, VA, USA
A Framework for Evaluating and Benchmarking Concept Drift Detection Methods
Jun 5, 2026Vitor Cerqueira, Heitor Murilo Gomes, Marco Heyden +2Concept Drift Synthetic Benchmark Generation
University of Coimbra · Victoria University of Wellington · Commerzbank +2
Meaning in Order, Order in Meaning: Semantic R-precision for Keyphrase Evaluation
Jun 5, 2026Shamira Venturini, Steffen KinkelAutomated Evaluation IR Evaluation
ILIN - Institute for Learning and Innovation in Networks, Karlsruhe University of Applied Sciences, Moltkestraße 30, 76133 Karlsruhe, Germany · Karlsruhe Institute of Technology, Kaiserstraße 12, 76131 Karlsruhe, Germany
RadSEM: A Finding-by-Finding Metric for Clinical Consistency in Radiology Reports
Jun 3, 2026Zhenhong Yang, Zhuoyun Liu, Jintao Fei +4Factual Consistency Evaluation Automated Evaluation
JDH Algo, JD Health International Inc., China · Department of Big Data in Health Science, The First Affiliated Hospital of Wenzhou Medical University, China · Zhejiang Engineering Research Center for Hospital Emergency and Process Digitization, Wenzhou, China +1

Jun 21, 2026, cs.AIJ/K move · Enter open · S save

SkillAudit: From Fixed-Suite Benchmarking to Skill-Centered Assessment

Dexu Yu, Youhua Li, Zhaoyang Guan, Xianhao Lin+11

Northeastern University · City University of Hong Kong · DeciLix Lab+8

AI Agent Auditing Automated Evaluation Agent Skill Security Auditing

Open paper AskPDF arXiv

Agent skills have become a practical way to extend large language model agents, but the growing skill ecosystem still lacks a reliable way to judge whether a skill is worth deploying. Existing evaluation methods remain largely anchored to fixed task suites, assessing skills through performance on predefined tasks and environments. As skill marketplaces expand, this paradigm becomes inadequate: fixed suites can conflate a skill's marginal contribution with backbone strength and miss its value when tasks fall outside the skill's intended scope. We introduce SkillAudit, an end-to-end framework for skill-centered assessment that takes an arbitrary agent skill as input and automatically generates a comprehensive, multi-dimensional evaluation report spanning utility, efficiency/cost, and safety. SkillAudit focuses on the skill artifact itself and constructs capability-aligned evaluation tasks directly from the skill package. The generated tasks are conducted in isolated sandbox environments to collect execution evidence, followed by automated checks with LLM-based judging to produce auditable results. To dissect the agent skills, we propose the baseline comparison principle to measure utility and efficiency/cost, and introduce a two-stage detection paradigm combining static semantic analysis with dynamic runtime verification to assess safety risks. After scanning top-ranked real-world skill packages spanning 23 occupational categories, we found that over 7% of skills are at risky status.

Momentum

Related topics

Latest papers 321