Automated Evaluation

Ask about this topic

Momentum

48 papers in the last four weeks, up 12% on the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 321

All topics
CardsList

  1. How Benchmarks Mis-Score Computer-Use Agents

    Jul 30, 2026Zihan Dong, Zhiyuan Ma, Zekun Wang +5Agent Failure AnalysisComputer-Use Agent Benchmarks

    Georgia Institute of Technology · North Carolina State University · Lenovo AI Technology Center +3

  2. (Towards) Scalable Reliable Automated Evaluation with Large Language Models

    Jul 30, 2026Bertil Braun, Martin ForellLLM EvaluationPairwise Comparison

    KIT

  3. DataClawEval: A Benchmark for Data Engineering Agents in Real Industrial Harness

    Jul 30, 2026Debin Meng, Jiaming Yang, Zefang Zong +4LLM Agent EvaluationAutomated Evaluation

    Tencent Shenzhen, China · Xidian University Xi’an, China · Tencent Beijing, China

  4. Evaluating and Pricing Advertisements in AI-Generated Responses

    Jul 30, 2026John L. Turner-Smith, Zimeng Huang, Yuhan Fu +2Mechanism DesignOnline Advertising

    Carnegie Mellon University, Pittsburgh, PA, USA · College of AI, Tsinghua University, Beijing, China

  5. Automated Transcript Analysis for Detecting Flaws in Agentic Benchmarks

    Jul 29, 2026Jeff Mohl, Nelson Gardner-Challis, Magda Dubois +6Benchmark AuditingAI Agent Benchmarks

    Independent · UK AI Security Institute · MARS +4

  6. SciFigQual-Bench: A Benchmark for Scientific Figure Quality Assessment with Full-Manuscript Context

    Jul 29, 2026Zihan Deng, Chuanzhi Xu, Huiqi Liang +3Automated EvaluationImage Quality Assessment

    The University of Hong Kong · The University of Sydney · University of Electronic Science and Technology of China

  7. AnnoBench: A Benchmark for Visualization Annotation Generation

    Jul 28, 2026Md Rahat-uz-Zaman, Md Dilshadur Rahman, Andrew McNutt +1Data VisualizationAutomated Evaluation

    University of Utah

  8. Interactive Reward Agent: GUI Task Evaluation via Environment-State Verification

    Jul 28, 2026Chenrui Shi, Yuwei Wu, Yang Liu +5RL for GUI AgentsAutomated Evaluation

    Beijing Key Laboratory of Intelligent Information Technology, School of Computer Science & Technology, Beijing Institute of Technology, Beijing, China · State Key Laboratory of General Artificial Intelligence, BIGAI, Beijing, China · Guangdong Laboratory of Machine Perception and Intelligent Computing, Shenzhen MSU-BIT University, Shenzhen, China +1

  9. Efficiency Matters in Autonomous Research

    Jul 27, 2026Haiqian Yang, Yuan CaoAlgorithm SelectionAutomated Evaluation

  10. GAUGE: Grading Agent-Built Financial Models Without a Golden Answer

    Jul 27, 2026Jiacheng Lu, Sinuo Wang, Wentao Zhao +12Quantitative FinanceFinancial Forecasting

    Shanghai Jiao Tong University Shanghai, China · StepFun Shanghai, China · University of Adelaide Adelaide, Australia +5

  11. The Cost of Knowing: A Resource-Aware Protocol for Benchmarking Hallucination Beyond Static Leaderboards

    Jul 27, 2026Keyu Li, Jin Gao, Dequan WangFactual Consistency EvaluationAI Agent Benchmarks

    Shanghai Jiao Tong University Shanghai, China

  12. A Reconstruction-Based Framework for Caption Evaluation Beyond Reference Captions

    Jul 25, 2026Zhijiang Tang, Jiaxin Qi, Kaihua Tang +2Image CaptioningAutomated Evaluation

    2Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences, China · 1Computer Network Information Center, Chinese Academy of Sciences, China · 3Tongji University, China +1

  13. grapheme-kit: Grapheme-Level Metrics and Text Processing for Multilingual NLP

    Jul 24, 2026Izzath Nisfer, Ashini Kavindya, Ovindu Atukorala +2Automated Evaluation

    Department of Computer Science & Engineering, University of Moratuwa, Sri Lanka · Computer Science and Engineering, University of Nevada, Nevada, USA · Department of Information & Computing Sciences, Utrecht University, Utrecht, The Netherlands

  14. An Evaluation Framework for Structured Audio Captions Validated by Controlled Perturbations

    Jul 23, 2026Liang-Yuan Wu, Sripathi Sridhar, Mark Cartwright +1Audio UnderstandingAutomated Evaluation

    New York University, New York, USA · New Jersey Institute of Technology, New Jersey, USA

  15. Reexamining zero-shot summarization: Empirical investigation of trustworthiness of LLM-summarizers

    Jul 23, 2026Vasudha Bhatnagar, Purnima Bindal, Vikas Kumar +1LLM EvaluationText Summarization

    Department of Computer Science, University of Delhi, Delhi, 110007, Delhi, India · Department of Statistics, Ramjas College, University of Delhi, Delhi, 110007, Delhi, India

  16. Learning to Detect UI Principle Violations via Reinforcement Learning

    Jul 22, 2026Nishi Mehta, Swathi Alse, Himani Kumawat +2Automated EvaluationLanguage Model Auditing

    University of California, Santa Cruz · Carnegie Mellon University

  17. A Diagnostic Evaluation Framework for AI-Generated Cover Songs Using Music-Theoretic and Acoustic Features

    Jul 22, 2026Yingxin LiangMusic Generation EvaluationMusic Generation

    Hangzhou Xiaoying Innovation Technology Co., Ltd. (Rythmix AI)

  18. Two-Level Meta-Rubrics for Evaluating Open-Ended Generation: GAMUT, a Benchmark for Factual Completeness

    Jul 21, 2026Xilun Chen, Zhaleh Feizollahi, Ross Goodwin +5Open-Ended GenerationLLM-as-a-Judge

    Meta AI

  19. Assessment in Team Problem-Solving Exercises in Computing Education

    Jul 21, 2026Valdemar Švábenský, Jan Vykopal, Sukrit Leelaluk +3Educational AssessmentRubric-Based Evaluation

    Faculty of Informatics Masaryk University Brno, Czech Republic · Center for ICT Infrastructure Yamaguchi University Yamaguchi, Japan · Graduate School of Information Science and Electrical Engineering, Kyushu University Fukuoka, Japan

  20. Translation as Augmentation: Effect of Translated Data on Assessment of Difficulty

    Jul 21, 2026Yiheng Wu, Jue Hou, Roman YangarberSynthetic Data AugmentationAutomated Evaluation

    University of Helsinki, Finland

  21. Human Grounded Evaluation of Large Language Models for Optical Network Automation

    Jul 20, 2026Kiarash Rezaei, Omran Ayoub, Paolo Monti +1LLM Inference EfficiencyLLM Evaluation

    Department of Electrical Engineering, Chalmers University of Technology, 412 96 Gothenburg, Sweden · University of Applied Sciences and Arts of Southern Switzerland, 6928 Lugano, Switzerland

  22. Auto Research for Materials: Auditable AI-Scientist Workflows with Held-Out Transfer

    Jul 19, 2026Jingjie Ning, Xiaochuan Li, Shanshan Zhong +2Automated Algorithm DiscoveryAutomated Evaluation

    School of Computer Science, Carnegie Mellon University · 2DP Technology

  23. Controlling Implicit Shortcut Reliance in L2 Spoken English Auto-markers

    Jul 17, 2026Shilin Gao, Mark J. F. Gales, Kate M. KnillEducational AssessmentAutomated Evaluation

    ALTA Institute, Machine Intelligence Laboratory, Cambridge University Engineering Department Cambridge, UK

  24. Project Kaleidoscope: Contextual, Human-Aligned Evaluation for Real-World AI Applications

    Jul 16, 2026Leanne Tan, Rohan Jaggi, Shaun Khoo +1Human-in-the-Loop EvaluationRubric-Based Evaluation

    GovTech, Singapore · National University of Singapore · University of British Columbia

  25. ChartGenEval: Corruption-Tested Multi-Dimensional Feedback for Rhythm-Game Chart Generation

    Jul 14, 2026Jhen-Ke LinSymbolic Music GenerationAutomated Evaluation

    National Yang Ming Chiao Tung University

  26. Operationalising Multi-Dimensional Evaluation for Conversational Agents: A Scalable, Governed Pipeline with Selective Re-evaluation and Model Benchmarking

    Jul 13, 2026Niranjan Kumar M, Balaji Nagarajan, Karthik Nair +2Language Model Generation EvaluationCustomer Support Automation

    Lowe's

  27. Interaction Scaling: Grounding the Third Axis of Test-Time Compute

    Jul 13, 2026Bojie Li, Noah ShiTest-Time ScalingAutomated Evaluation

    Pine AI · University of Washington

  28. DynEval: Holistic Evaluations of T2I Generative Models in the Wild

    Jul 13, 2026Shyam Marjit, Dheeraj Baiju, Anuj Shikarkhane +3VLM EvaluationAutomated Evaluation

    Indian Institute of Science · VCL, IISc · Hugging Face

  29. From Checker to Forecaster: Code-Owned Evaluation of Model-Generated Strategic Routes Under Delayed Ground Truth

    Jul 13, 2026Aleh ManchuliantsauLLM EvaluationProbabilistic Forecasting

    Independent Researcher

  30. Did We Actually Fix It? An Independent Adversarial Stress-Test of Post-Point-Adjustment Evaluation Metrics for Time-Series Anomaly Detection

    Jul 12, 2026Zongye LyuAdversarial RobustnessAutomated Evaluation

    Faculty of Information Technology, Monash University, Melbourne, VIC 3800, Australia