Automated Evaluation

Ask about this topic

Momentum

48 papers in the last four weeks, up 12% on the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 321

All topics
CardsList

  1. LongSumEval: Question-Answering Based Evaluation and Feedback-Driven Refinement for Long Document Summarization

    Apr 28, 2026Huyen Nguyen, Haoxuan Zhang, Yang Zhang +2Language Model Generation EvaluationFactual Consistency Evaluation

    Cigna Group Evernorth Health Services Austin, Texas, USA · dept. of Information Science University of North Texas Denton, Texas, USA · dept. of Data Science University of North Texas Denton, Texas, USA

  2. STELLAR-E: a Synthetic, Tailored, End-to-end LLM Application Rigorous Evaluator

    Apr 27, 2026Alessio Sordo, Lingxiao Du, Meeka-Hanna Lenisa +2LLM EvaluationSynthetic Data Generation

    1Deutsche Bank, Berlin Technology Center, Berlin, Germany

  3. Understanding the Limits of Automated Evaluation for Code Review Bots in Practice

    Apr 27, 2026Veli Karakaya, Utku Boran Torun, Baykal Mehmet Uçar +1LLM-as-a-JudgeHuman-in-the-Loop Evaluation

    Bilkent University Ankara, Turkey · Beko İstanbul, Turkey

  4. RAS: a Reliability Oriented Metric for Automatic Speech Recognition

    Apr 27, 2026Wenbin Huang, Yuhang Qiu, Bohan Li +5ASR EvaluationSelective Prediction

    X-LANCE Lab, School of Computer Science, Shanghai Jiao Tong University, China · MoE Key Lab of Artificial Intelligence; Jiangsu Key Lab of Language Computing, China

  5. MetaGAI: A Large-Scale and High-Quality Benchmark for Generative AI Model and Data Card Generation

    Apr 26, 2026Haoxuan Zhang, Ruochi Li, Yang Zhang +4Automated EvaluationGenerative AI Evaluation

    University of North Texas, Denton, TX, USA · North Carolina State University, Raleigh, NC, USA

  6. Your Students Don't Use LLMs Like You Wish They Did

    Apr 26, 2026Sebastian Kobler, Matthew Clemson, Angela Sun +1Educational AssessmentAI in Education

    The University of Sydney

  7. GSAR: Typed Grounding for Hallucination Detection and Recovery in Multi-Agent LLMs

    Apr 25, 2026Federico A. KamelharAI Agent ReliabilityMulti-Agent LLM Systems

    Senior Principal Architect, Agentic AI Oracle Corporation

  8. AmaraSpatial-10K: A Spatially and Semantically Aligned 3D Dataset for Spatial Computing and Embodied AI

    Apr 24, 2026Mohammad Sadegh Salehi, Alex Perkins, Igor Maurell +2Synthetic Data Generation for RoboticsAutomated Evaluation

    Zero One Creative

  9. Quantifying and Mitigating Self-Preference Bias of LLM Judges

    Apr 24, 2026Jinming Yang, Zheng Hu, Chuxian Qiu +3LLM-as-a-JudgeLanguage Model Bias Evaluation

    CompleX Lab, School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, China

  10. Deep FinResearch Bench: Evaluating AI's Ability to Conduct Professional Financial Investment Research

    Apr 22, 2026Mirazul Haque, Antony Papadimitriou, Samuel Mensah +6Quantitative FinanceDeep Research Agents

    AI Research, JPMorganChase

  11. Escaping the Agreement Trap: Defensibility Signals for Evaluating Rule-Governed AI

    Apr 22, 2026Michael O'Herlihy, Rosa CatalàContent ModerationAutomated Evaluation

    Reddit, Inc.

  12. LAF-Based Evaluation and UTTL-Based Learning Strategies with MIATTs

    Apr 22, 2026Yongquan YangWeakly Supervised LearningAutomated Evaluation

    Institute of Sciences for AI, Chengdu, Sichuan, China

  13. Centering Ecological Goals in Automated Identification of Individual Animals

    Apr 22, 2026Lukas Picek, Timm Haucke, Lukáš Adam +16Biometric IdentificationAnimal Re-Identification

    University of West Bohemia in Pilsen, Pilsen, Czechia. · Massachusetts Institute of Technology, Cambridge, Massachusetts, USA. · LUT University, Lappeenranta, Finland. +13

  14. Embedding-Based Intrusive Evaluation Metrics for Musical Source Separation Using MERT Representations

    Apr 22, 2026Paul A. Bereuter, Alois SontacchiMusic Information RetrievalAutomated Evaluation

    University of Music and Performing Arts Graz, Institute of Electronic Music and Acoustics (IEM) A-8010 Graz, Austria

  15. In-Situ Behavioral Evaluation for LLM Fairness, Not Standardized-Test Scores

    Apr 21, 2026Zeyu Tang, Sang T. Truong, Deonna Owens +4LLM EvaluationSocial Bias in Language Models

    Stanford University

  16. Towards a Linguistic Evaluation of Narratives: A Quantitative Stylistic Framework

    Apr 21, 2026Alessandro MaistoStylometryAutomated Evaluation

    Università degli Studi di Salerno, Via Giovanni Paolo II, 132, 84084 Fisciano (SA), Italia

  17. SCURank: Ranking Multiple Candidate Summaries with Summary Content Units for Enhanced Summarization

    Apr 21, 2026Bo-Jyun Wang, Ying-Jia Lin, Hung-Yu KaoLearning to RankText Summarization

    Department of Computer Science and Information Engineering, National Cheng Kung University · Department of Artificial Intelligence, Chang Gung University · Artificial Intelligence Research Center, Chang Gung University +1

  18. TactileEval: A Step Towards Automated Fine-Grained Evaluation and Editing of Tactile Graphics

    Apr 20, 2026Adnan Khan, Abbas Akkasi, Majid KomeiliRubric-Based EvaluationAutomated Evaluation

    School of Computer Science, Carleton University, Ottawa, Canada

  19. ClawEnvKit: Automatic Environment Generation for Claw-Like Agents

    Apr 20, 2026Xirui Li, Ming Li, Ion Stoica +2Agent EvaluationAI Agent Benchmarks

    University of Maryland · 5Mohamed bin Zayed University of Artificial Intelligence · 2Arena +2

  20. FSEVAL: Feature Selection Evaluation Toolbox and Dashboard

    Apr 20, 2026Muhammad Rajabinasab, Arthur ZimekFeature SelectionAutomated Evaluation

    Department of Mathematics and Computer Science University of Southern Denmark Odense, Denmark

  21. Who Watches the Watchmen? Humans Disagree With Translation Metrics on Unseen Domains

    Apr 19, 2026Finn Schmidt, Jan Philip Wahle, Terry Ruas +1Annotator DisagreementAutomated Evaluation

    University of Göttingen, Germany

  22. Rethinking Meeting Effectiveness: A Benchmark and Framework for Temporal Fine-grained Automatic Meeting Effectiveness Evaluation

    Apr 19, 2026Yihang Li, Chenhui ChuAutomated Evaluation

    Kyoto University

  23. Calibrating Model-Based Evaluation Metrics for Summarization

    Apr 19, 2026Hongye Liu, Dhanajit Brahma, Ricardo HenaoLanguage Model CalibrationSummarization Evaluation

    Duke University

  24. Diagnosing LLM Judge Reliability: Conformal Prediction Sets and Transitivity Violations

    Apr 16, 2026Manan Gupta, Dhruv KumarLLM EvaluationLLM-as-a-Judge

    1BITS Pilani, Pilani Campus, India

  25. AI-Assisted Requirements Engineering: An Empirical Evaluation Relative to Expert Judgment

    Apr 16, 2026Oz Levy, Ilya Dikman, Natan Levy +1Requirements EngineeringHuman-in-the-Loop Evaluation

    Faculty of Industrial Engineering and Technology Management, Holon Institute of Technology (HIT) Holon, Israel · School of Computer Science and Engineering Hebrew University Jerusalem (HUJI) Jerusalem, Israel

  26. Exploring and Testing Skill-Based Behavioral Profile Annotation: Human Operability and LLM Feasibility under Schema-Guided Execution

    Apr 16, 2026Yufeng WuLLM-Assisted AnnotationHuman-in-the-Loop Annotation

    City University of Hong Kong

  27. Learning in Blocks: A Multi Agent Debate Assisted Personalized Adaptive Learning Framework for Language Learning

    Mar 29, 2026Nicy Scaria, Silvester John Joseph Kennedy, Deepak SubramaniMulti-Agent DebateAutomated Evaluation

    Indian Institute of Science, India · Talking Yak, India · Indian Institute of Technology Patna, India

  28. ECoLAD: Selecting Anomaly Detectors for Automotive Deployment via Compute-Reduction Evaluation

    Mar 11, 2026Kadir-Kaan Özer, René Ebeling, Markus EnzweilerEfficient Neural Network InferenceAutomated Evaluation

    Mercedes-Benz AG, Germany. · Esslingen University of Applied Sciences, Germany.

  29. Toward Robust LLM-Based Judges: Taxonomic Bias Evaluation and Debiasing Optimization

    Mar 9, 2026Hongli Zhou, Hui Huang, Rui Zhang +5LLM-as-a-JudgeSocial Bias in Language Models

    Faculty of Computing, Harbin Institute of Technology, Harbin, 150001, China · School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen, 518055, China

  30. Human or Machine? A Preliminary Turing Test for Speech-to-Speech Interaction

    Feb 27, 2026Xiang Li, Jiabao Gao, Sipei Lin +5Voice Agent EvaluationSpoken Dialogue Systems

    State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications · Shenzhen Research Institute of Big Data · The Chinese University of Hong Kong, Shenzhen +2