Automated Evaluation

Ask about this topic

Momentum

36 papers in the last four weeks, up 57% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 289

All topics
CardsList

  1. Can LLMs Fix It Without Code? Toward Automated Verification of No-Code Bug Fixes

    Oct 8, 2026Utku Boran Torun, Veli Karakaya, Eray TüzünAutomated EvaluationLLM Agent Reliability

    Bilkent University, Ankara, Turkey

  2. Who Verifies the Verifier? Co-Evolving Inspectable Graders with Self-Improving Agents

    Oct 8, 2026Xing Zhang, Guanghui Wang, Yanwei Cui +4Automated EvaluationLLM Agent Self-Improvement

    AWS Forward Deployed Engineering · HSBC Holdings Plc., HSBC Technology Center, China

  3. LLM Persuasion Is in the Eye of the Evaluation

    Oct 7, 2026Kamile Dementaviciute, Julija Vaitonyte, Tijl De BiePersuasion in Language ModelsLLM Evaluation

    Ghent University · Tilburg University, Department of Computational Cognitive Science · ISM University of Management and Economics

  4. Perceptually Aligned Evaluation of Style Transfer

    Oct 7, 2026Yang Deng, Eleftherios Ioannou, David Mould +3Image Style TransferAutomated Evaluation

    Cardiff University, UK · The Cyprus Institute, Cyprus · Carleton University, Canada +1

  5. A Tale of Two Error Categories: Exploring Concealed Trade-Offs in the Errors of Automated Judges in Evaluation of Uncertainty Quantifiers

    Oct 7, 2026Evgenia Ilia, Wilker AzizUncertainty QuantificationAutomated Evaluation

    University of Amsterdam

  6. From Plausible Hierarchies to Useful Taxonomies: Evaluating Agentic Harnesses on Customer Feedback

    Oct 7, 2026Prabhath Chellingi, Raviraja G, Viraj BagalAutomated EvaluationLLM Agent Evaluation

    Enterpret

  7. Language Carries the Expert's Impression: Instrument-Anchored LLM Judges Transfer Counseling-Quality Assessment and Beat In-Domain Training

    Oct 6, 2026Tobias Hallmen, Elisabeth AndréMental Health CounselingCross-Domain Transfer Learning

    Chair for Human-Centered Artificial Intelligence University of Augsburg

  8. CroissantMiner: Automated Extraction and Validation of Croissant Metadata for ML Datasets

    Oct 5, 2026Berke Arda, Ahmetcan Yavuz, Paul Gerry +7Document Information ExtractionLLM Information Extraction

    ETH Zurich · CSAIL, MIT · Helmholtz Zentrum München +5

  9. Old Ideas, Novel Problems: The Instability of LLM-Based Novelty Evaluation

    Oct 1, 2026Noy Sternlicht, Simra Shahid, Peter Jansen +3Prompt SensitivityLLM-as-a-Judge

    Hebrew University of Jerusalem · Allen Institute for AI · Microsoft +2

  10. Judgement in the Age of Jev: From Evaluation Scarcity to Evaluation Abundance

    Oct 1, 2026Richard HillAutomated EvaluationHuman-AI Decision Making

    Department of Computer Science, University of Huddersfield, Huddersfield, UK.

  11. When Does Exercise-Specific Joint Selection Help? An Audit of Evaluation and Control Design

    Oct 1, 2026Haotian Chen, Jingkun Yu, Yuning Zhang +1Skeleton-Based Action RecognitionClassification

    School of Cyber Science and Technology, University of Science and Technology of China · SWJTU-Leeds Joint School, Southwest Jiaotong University · School of Education, Shanghai Jiao Tong University

  12. Sentence Specificity Scores for Collaborative Technical Documentation: A Domain-Transfer Study

    Oct 1, 2026Rocker D'Antonio, Thomas Benton Townsend, Dimitrios Michael ManiasDocument UnderstandingDomain Generalization

    Mississippi State University, USA

  13. Ontology-Based Contextual AI Evaluations (OB-CAIE) Methodology

    Sep 30, 2026Julie Krugler Hollek, Michael Zargham, Mala KumarHuman-in-the-Loop EvaluationAutomated Evaluation

    Humane Intelligence · Dynamical Systems Group

  14. LLM-as-a-Judge for Low-Resource Languages: Adapting Ragas and Comparative Ranking for Romanian

    Sep 30, 2026Claudiu Creanga, Liviu P. DinuLLM EvaluationLLM-as-a-Judge

    Interdisciplinary School of Doctoral Studies, University of Bucharest, Romania · HLT Research Center, University of Bucharest, Romania · Faculty of Mathematics and Computer Science, University of Bucharest, Romania

  15. Which papyrus HTR is good enough? Character-error-rate tolerance of four papyrological tasks on Greek texts

    Sep 29, 2026Anton Repushko, Elena ChepelAutomated EvaluationOptical Character Recognition

    Independent Researcher, Berlin, Germany. · University of Vienna, Vienna, Austria.

  16. Do Music Generative Models Understand Musical Qualities? Automatic Music Evaluation with Model-Intrinsic Signals

    Sep 29, 2026Xiaosha Li, Chun Liu, Ziyu WangMusic Generation EvaluationAutomated Evaluation

    Georgia Institute of Technology · ByteDance Inc. · New York University · MBZUAI

  17. Mubric: Mutation Testing-Guided Rubric Generation for LLM Evaluation

    Sep 29, 2026Jiayuxuan Yang, Jie M. Zhang, Yiling Lou +1Language Model Generation EvaluationRubric-Based Evaluation

    Tsinghua University · King’s College London · University of Illinois Urbana-Champaign

  18. SymbolicArena: A Unified Infrastructure for Benchmark Distillation and Dynamic Evaluation in Symbolic Regression

    Sep 28, 2026Ziwen Zhang, Xiju Wu, Yuheng Jing +9Benchmark DesignAutomated Evaluation

    School of Artificial Intelligence, University of Chinese Academy of Sciences · C2DL, Institute of Automation, Chinese Academy of Sciences · Nanjing University of Science and Technology +1

  19. TermJudge: A Document-Level Metric Judging, Not Counting, Terminology in Machine Translation Evaluation

    Sep 28, 2026Nicolas Dahan, Fran{\cc}ois Yvon, Rachel BawdenAutomated EvaluationDocument-Level MT

    ISIR, ALMAnaCH · MLIA, ISIR · ALMAnaCH

  20. SIVIA-RSI: Source-Grounded Adaptation of Diagramming Skills

    Sep 27, 2026Feng Yuan, Yifan Gao, Haoyue Li +1Cross-Domain Transfer LearningAgent Skill Learning

    School of Biomedical Engineering (Suzhou), Division of Life Science and Medicine, University of Science and Technology of China, Hefei, China · Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Sciences, Suzhou, China · Shanghai Innovation Institute, Shanghai, China

  21. CHI: A Composite Hallucination Index Unifying Entity, Relation, and Quantity Dimensions for Summarization Evaluation

    Sep 27, 2026Praveenkumar Katwe, Rakesh Chandra Balabantaray, Kali Prasad VittalaHallucination in Language ModelsFactual Consistency Evaluation

    International Institute of Information Technology Bhubaneswar, India · Salesforce India Pvt Ltd, Bengaluru, India

  22. Which Self-Improvements Should We Trust? Reliable Self-Improvement When Agents Reuse Their Benchmarks

    Sep 27, 2026Xiaojing Sun, Yuhan Zeng, Zihua She +1Benchmark ContaminationAutomated Evaluation

    Purdue University

  23. SEEK: Skill-Routed Evaluation with Evolvable Knowledge for Industrial Search

    Sep 24, 2026Zhongxin Huang, Songyang Li, Renzhe Zhou +5LLM EvaluationAutomated Evaluation

    Peking University Beijing, China · Kuaishou Technology Beijing, China · Kuaishou Technology Hangzhou, China +2

  24. Does Graph Structure Earn Its Place in Microservice Root-Cause Analysis? A Controlled Study on RCAEval, and What the Benchmark Was Really Measuring

    Sep 22, 2026Imad BuljićGraph Neural NetworksBenchmark Auditing

    Polytechnic Faculty, University of Zenica

  25. JEV-as-a-Judge: Accept When Confident, Escalate When Unsure

    Sep 22, 2026Yubo Li, Yidi Miao, Ramayya Krishnan +1LLM-as-a-JudgeLanguage Model Generation Evaluation

    Carnegie Mellon University

  26. Auditing Proxy-Based Validation Across Text Spans

    Sep 22, 2026Daein Weon, Dong Ho KangBenchmark ValidityAutomated Evaluation

    Kookmin University · UStechlab

  27. LLJ Cards: Best practices for the Use of LLMs as Judges

    Sep 21, 2026Khaoula Chehbouni, Melina Medjdoub, Florian Carichon +2LLM EvaluationLLM-as-a-Judge

    McGill University · Mila - Quebec AI Institute · Concordia University

  28. Preserving What Matters: Semantic Scaffolds Beyond Saturation in Summarization Evaluation

    Sep 18, 2026Nikhil Reddy Pottanigari, Ramin Fahimi, Noah Bolger +2LLM EvaluationText Summarization

    ServiceNow Canada

  29. greCAPTCHA: Assessing Understanding as Evidence of Research Authorship Under Generative AI

    Sep 17, 2026Justin Payan, Bálint Gyevnár, Atoosa Kasirzadeh +1Authorship AttributionAutomated Evaluation

    Machine Learning Department, Carnegie Mellon University · Institute for Complex Social Dynamics, Carnegie Mellon University · Departments of Philosophy & Software and Societal Systems, Carnegie Mellon University +1

  30. E-AVI: Evidence-Grounded Multimodal Assessment for Automated Video Interviews

    Sep 17, 2026Haoshen Wang, Dongbo Che, Zeyi Xie +3Multimodal GroundingAudio-Visual Understanding

    The Hong Kong Polytechnic University · Independent Researcher