Automated Evaluation

Ask about this topic

Momentum

48 papers in the last four weeks, up 12% on the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 321

All topics
CardsList

  1. JEV-as-a-Judge: Accept When Confident, Escalate When Unsure

    Sep 22, 2026Yubo Li, Yidi Miao, Ramayya Krishnan +1LLM-as-a-JudgeLanguage Model Generation Evaluation

    Carnegie Mellon University

  2. Auditing Proxy-Based Validation Across Text Spans

    Sep 22, 2026Daein Weon, Dong Ho KangBenchmark ValidityAutomated Evaluation

    Kookmin University · UStechlab

  3. LLJ Cards: Best practices for the Use of LLMs as Judges

    Sep 21, 2026Khaoula Chehbouni, Melina Medjdoub, Florian Carichon +2LLM EvaluationLLM-as-a-Judge

    McGill University · Mila - Quebec AI Institute · Concordia University

  4. Automated Evaluation of Multi-Turn Dialogues in In-Car Conversational Assistants

    Sep 20, 2026Vaishnav Negi, Lev Sorokin, Soroosh Tayebi Arasteh +1Multi-Turn Dialogue EvaluationVoice Agent Evaluation

    Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany · BMW Group, Germany · Technical University of Munich, Munich, Germany +2

  5. Preserving What Matters: Semantic Scaffolds Beyond Saturation in Summarization Evaluation

    Sep 18, 2026Nikhil Reddy Pottanigari, Ramin Fahimi, Noah Bolger +2LLM EvaluationText Summarization

    ServiceNow Canada

  6. Beyond Reference-Based Evaluation: Reward Models for Meta-Evaluation of Grammatical Error Correction

    Sep 18, 2026Ruotian Wu, Bill E. Johnson, Gene Saunders +3Grammatical Error CorrectionAutomated Evaluation

    University of Waterloo · Vector Institute · Scribendi Inc.

  7. greCAPTCHA: Assessing Understanding as Evidence of Research Authorship Under Generative AI

    Sep 17, 2026Justin Payan, Bálint Gyevnár, Atoosa Kasirzadeh +1Authorship AttributionAutomated Evaluation

    Machine Learning Department, Carnegie Mellon University · Institute for Complex Social Dynamics, Carnegie Mellon University · Departments of Philosophy & Software and Societal Systems, Carnegie Mellon University +1

  8. E-AVI: Evidence-Grounded Multimodal Assessment for Automated Video Interviews

    Sep 17, 2026Haoshen Wang, Dongbo Che, Zeyi Xie +3Multimodal GroundingAudio-Visual Understanding

    The Hong Kong Polytechnic University · Independent Researcher

  9. A Benchmark Framework for Screening Automation in Systematic Reviews

    Sep 16, 2026Gauransh Kumar, Luciano Marchezan, Guillaume Genois +2LLM EvaluationEvidence Selection

    Université de Montréal Montreal, Canada

  10. I code or AI code: A comparative evaluation of AI-rated scores in classroom observations

    Sep 16, 2026Y. Fong, J. Xiang, T. Y. D. Chan +2LLM-as-a-JudgeEducational Assessment

    Department of Early Childhood Education, The Education University of Hong Kong, 10 Lo Ping Road, Tai Po, Hong Kong SAR, China · Yew Chung College of Early Childhood Education, 2 Tin Wan Hill Road, Tin Wan, Hong Kong SAR, China

  11. The record is part of the task: matched-record evaluation of text classifiers across maintenance, safety and recall reporting

    Sep 14, 2026Hisham Ihshaish, Peter Mayhew, Tasnim M. A. Zayet +1Text ClassificationAutomated Evaluation

    School of Computing and Creative Technologies, University of the West of England, Bristol, UK · Department of Computer Science, Birzeit University, Birzeit, Palestine · GE Aerospace, Cheltenham, UK +1

  12. Don't Count the Edits, Judge by the Outcome Alone: Reward-Based Evaluation for Grammatical Error Correction

    Sep 14, 2026Hayeong Ryu, Sunhee Jo, Seunguk Yu +1Reward ModelingReference-Free Evaluation

    Department of Artificial Intelligence, Chung-Ang University · Graduate School of Advanced Imaging Sciences, Multimedia and Film, Chung-Ang University

  13. WebCraftBench: Evaluating Web Application Generation from a Software Testing Perspective

    Sep 14, 2026Chenxu Liu, Zilu Zou, Peizhong Gao +9Automated Software TestingWeb Application Generation

    Hunyuan Team, Tencent · Tsinghua University · Peking University

  14. ProIQA: A Process-Based Framework for Fine-Grained Math Item Quality Assessment

    Sep 14, 2026Junkai Tong, Mingjia Li, Haoran Chen +4Educational AssessmentAutomated Evaluation

    East China Normal University · Shanghai, China · East China Normal University, Shanghai, China +1

  15. AppliedScientist: Automated Scientific Revision Through Iterative AI Reviewing

    Sep 13, 2026Vidushee Vats, Karun Sharma, Shengzhi Li +1AI-Assisted Scientific ResearchAutomated Peer Review

    Department of Computer Science University of Massachusetts Boston

  16. Inter-Rater Reliability of LLM and Rule-Based Annotation for Inferential Narrative Features: Three Studies on a Turkish Corpus

    Sep 12, 2026Levent BulutLLM-Assisted AnnotationInter-Rater Reliability

    Independent Researcher

  17. Certifying Model Upgrades with Slice-Wise Non-Regression and Incumbent Fallback

    Sep 12, 2026Shengwei Zhang, Tao Wu, Fei QianModel SelectionAutomated Evaluation

  18. PA-CDM: Position-Aware Character Detection Matching for Evaluating Handwritten Mathematical Expression Recognition

    Sep 11, 2026Shiliang LuoAutomated EvaluationOptical Character Recognition

    East China Normal University

  19. PRISMA-LLM: An Empirical Reporting Framework for AI-Assisted Systematic Reviews

    Sep 10, 2026Miguel Zabaleta, Baihan LinLLM AuditingAI-Assisted Scientific Research

    Department of AI and Human Health, Icahn School of Medicine at Mount Sinai, New York, NY, USA · Department of Psychiatry, Icahn School of Medicine at Mount Sinai, New York, NY, USA · Department of Neuroscience, Icahn School of Medicine at Mount Sinai, New York, NY, USA

  20. Rubric-Aligned Disentangled Evaluation of Human Simultaneous Interpreting

    Sep 10, 2026Ziyu Zhang, Satoshi NakamuraRubric-Based EvaluationAutomated Evaluation

    School of Data Science, The Chinese University of Hong Kong, Shenzhen, China · School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen, China · Shenzhen Loop Area Institute, China

  21. TimeCues Studio: A Workspace for Music Annotation and Algorithm Prototyping

    Sep 9, 2026Sapir Caduri, Yoav GoldbergHuman-in-the-Loop AnnotationMusic Information Retrieval

    Bar-Ilan University Ramat Gan, Israel

  22. SQLMorph: Query Mutation and Fine-Grained Metrics for Text-to-SQL Evaluation

    Sep 8, 2026Mohammadhossein Malekpour, Mohamed Riahi, Maxime Lamothe +1Mutation TestingText-to-SQL

    Polytechnique Montréal

  23. Limitations of Automated Simulatability: LLM Simulators Can Bypass Explanations

    Sep 8, 2026Antonin Poché, Fanny Jourdan, Nils Feldhus +6LLM EvaluationAutomated Evaluation

    IRT Saint Exupéry · IRIT, Université de Toulouse · University of Groningen +7

  24. EviSI: An Evidence-Based Evaluation Agent for Simultaneous Interpreting

    Sep 8, 2026Ben Yan, Zongyao Li, Xiaoyu Chen +6LLM-as-a-JudgeSimultaneous Speech Translation

    1Huawei Translation Service Center, Beijing, China · School of Informatics, Xiamen University, China

  25. What Does an LLM-Agent Leaderboard Rank Actually Compare?

    Sep 7, 2026Wei-Jung HuangLLM Agent EvaluationAI Agent Benchmarks

    Independent Researcher United States

  26. An Empirical Measurement of Jailbreaking Evaluators

    Sep 7, 2026Yujie MuAutomated EvaluationJailbreak Detection

    Independent Researcher

  27. Used, Mentioned, or Condemned? A Controlled Contrast-Set Diagnostic for the Use-Mention Distinction in Code-Mixed Hinglish Misogyny Detection

    Sep 6, 2026Ashanvi Yadav, Shubham BhardwajHate Speech DetectionContent Moderation

    Manipal University Jaipur Rajasthan, India · BITS Pilani, Hyderabad Campus Telangana, India

  28. Beyond Aggregate Scores: Behavioral Correctness Assumptions for Assessing Reference-Based Automatic Evaluation Methods

    Sep 4, 2026Maria Mahbub, Ashley Rice, Michael R. Munroe +2Automated EvaluationLanguage Model Generation Evaluation

    Oak Ridge National Laboratory, Oak Ridge, TN 37830, USA · DHS Science and Technology Directorate, Washington, DC 20528, USA

  29. TruthInsightBench: An Evidence-Grounded Benchmark for Automated Evaluation of Open-Ended Scientific Discovery Agents

    Sep 4, 2026Zhibo Yang, Chen Zhang, Yuewei Zhang +1AI Agents for Scientific DiscoveryAI Agent Benchmarks

    TruthInsight-AI

  30. Decoupled Analysis-Judging: An Automated Creativity Evaluator Using LLMs in Complex Multi-step Creativity Tasks

    Sep 3, 2026Xiangyu Wang, Jin Wu, Xiaoyu Li +2Creativity AssessmentLLM-as-a-Judge

    Department of Educational Psychology, East China Normal University · 2Shanghai Institute of Artificial Intelligence for Education, East China Normal University · School of Computer Science and Technology, East China Normal University +2