Automated Evaluation

Ask about this topic

Momentum

48 papers in the last four weeks, up 12% on the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 321

All topics
CardsList

  1. Abstractiveness Metrics for Evaluating Text Summarization: A Refined Formulation with Empirical Validation

    Jul 12, 2026Praveenkumar Katwe, Rakesh Chandra Balabantaray, Kali Prasad VittalaText SummarizationSummarization Evaluation

    Department of Computer Science and Engineering, International Institute of Information Technology, Bhubaneswar, India · Salesforce India Pvt Ltd, Bengaluru, India

  2. Knowledge Distillation for Automated AI Tutor Evaluation

    Jul 12, 2026Tahmid Al Hannan, Diego Garcia, Alex Njoroge +2LLM-as-a-JudgeAI in Education

    Folsom Lake College Folsom, CA, USA

  3. CAFE: A Compound-AI Factorial Evaluation Framework

    Jul 11, 2026Fabian Lukassen, Christoph Weisser, Thomas Kneib +1RAG EvaluationAutomated Evaluation

    University of Göttingen · Bielefeld University of Applied Sciences and Arts (HSBI)

  4. KGCQual: An Interpretable Framework for Evaluating the Knowledge Graph Construction Quality from Text

    Jul 11, 2026Nipun Misra, Vikranth Udandarao, Aanchal Gupta +3KG ConstructionAutomated Evaluation

    VIT Vellore, Vellore, India · IIIT-Delhi, Delhi, India · Indian Institute of Technology Palakkad, Kerala, India.

  5. Stop Guessing When to Stop Testing: Efficient Model Evaluation with Just Enough Data

    Jul 9, 2026Ofir Arviv, Kristjan Greenewald, Yotam Perlitz +3Early StoppingOptimal Stopping

    IBM Research

  6. False Confidence: Automated Labels Confound Fairness Audits in Cervical Spine Segmentation

    Jul 8, 2026Linus Juni, Aasa Feragen, Aditya ParikhAlgorithmic FairnessMedical Image Segmentation

    Section for Visual Computing, DTU Compute, Technical University of Denmark, Kongens Lyngby, Denmark

  7. Evaluating RAG Metrics in Applied Contexts: An Experiment, Its Findings and Its Limitations

    Jul 8, 2026Quentin BrabantRAG EvaluationAutomated Evaluation

    Orange Research, Lannion, France

  8. Auto-DSM Under the Lens: A Black-Box Evaluation Framework for LLM-Based DSM Generation

    Jul 7, 2026Niels Potters, Theo HofmanLLM EvaluationLLM Auditing

    Eindhoven University of Technology (TU/e), Dept. of Mechanical Engineering, Control Systems Technology section, Engineering Systems Design group, P.O.Box 513, 5600 MB Eindhoven, The Netherlands.

  9. FormalRx: Rectify and eXamine Semantic Failures in Autoformalization

    Jul 6, 2026Haocheng Wang, Baiyu Huang, Yingjia Wan +4Automated EvaluationAutoformalization

    LARK Lab, HKUST(GZ) · ETH Zurich · UCLA +3

  10. Prompt-to-Paper: Agentic AI System for Bioinformatics

    Jul 5, 2026Ramsha Kamran, Maheera Amjad, Zartasha Mustansar +3Long-Form Document GenerationAI-Assisted Scientific Research

    School of Interdisciplinary Engineering and Sciences (SINES), National University of Sciences and Technology (NUST), Islamabad, Pakistan · College of Electrical and Mechanical Engineering (CEME), NUST, Rawalpindi, Pakistan

  11. Flow-A11y: Flow-Aware Accessibility Testing

    Jul 3, 2026Nasr Eddine Fliti, Leisan Kokorina, Florian Tambon +1Automated Software TestingAutomated Evaluation

    University of Luxembourg Luxembourg, Luxembourg

  12. Object Aligner: A Configurable JSON Schema Similarity Score for Graphs, Applied to LLM Prompt Optimization

    Jul 2, 2026Jan DrchalLLM EvaluationLanguage Model Generation Evaluation

    Artificial Intelligence Center, Faculty of Electrical Engineering, Czech Technical University in Prague, Czech Republic

  13. TurnNat: Automatic Evaluation of Turn-Taking Naturalness in Dyadic Spoken Dialogue

    Jul 1, 2026Hao Zhang, Thomas Thebaud, Georgi Tinchev +2Automated EvaluationFull-Duplex Spoken Dialogue Systems

    Center for Language and Speech Processing, Johns Hopkins University, USA · Amazon Research, UK · Amazon, USA

  14. PPT-Eval: A Benchmark for Computer-Use Agents on PowerPoint Tasks

    Jun 30, 2026Apurva Gandhi, Vishwas Suryanarayanan, Raja Hasnain Anwar +6Computer-Use AgentsRubric-Based Evaluation

    Carnegie Mellon University · Microsoft · UMass Amherst +3

  15. Rigel: Self-Distilled Score Adaptation for Image and Video Captioning Evaluation

    Jun 29, 2026Shuitsu Koyama, Kazuki Matsuda, Yuiga Wada +3Image CaptioningVideo Captioning

    Keio University

  16. SABER-Math: Automated Benchmark for Information Retrieval Evaluation in Mathematics

    Jun 29, 2026Nikolay Georgiev, Maria Drencheva, Kseniia Ibragimova +3Text Embedding EvaluationLLM Reranking

    INSAIT, Sofia University “St. Kliment Ohridski” · ETH Zurich

  17. Faults in Our Formal Benchmarking: Dataset Defects and Evaluation Failures in Lean Theorem Proving

    Jun 28, 2026Pawan Sasanka Ammanamanchi, Siddharth Bhat, Stella BidermanBenchmark DesignBenchmark Validity

    Eleuther AI · Cambridge

  18. MedEvoEval: Evaluating Continual Evolution of Doctor Agents through Simulated Clinical Episodes

    Jun 27, 2026Hui ZhangContinual Learning for LLM AgentsLong-Horizon Agent Evaluation

    Beijing Institute of Technology (BIT), China

  19. BackTranslation2.0 -- A Linguistically Motivated Metric to Assess Sign Language Production

    Jun 27, 2026Oliver Cory, Maksym Ivashechkin, Karahan Sahin +6Sign Language TranslationAutomated Evaluation

    Centre for Vision, Speech and Signal Processing, University of Surrey, Guildford, UK

  20. PerceptionRubrics: Calibrating Multimodal Evaluation to Human Perception

    Jun 26, 2026Yana Wei, Hongbo Peng, Yanlin Lai +14Rubric-Based EvaluationAutomated Evaluation

    Johns Hopkins University · StepFun · Tsinghua University +1

  21. The Signal-Coverage Matrix: Stratifying Type and Semantic Errors in Statement Autoformalization

    Jun 26, 2026Chengxiao Dai, Zhaokun Yan, Zhanhui LinLLM Self-CorrectionLean Theorem Proving

    School of Computer Science, University of Sydney, Sydney, Australia · China Academy of Information and Communications Technology, China

  22. Ask, Don't Judge: Binary Questions for Interpretable LLM Evaluation and Self-Improvement

    Jun 25, 2026Sangwoo Cho, Kushal Chawla, Pengshan Cai +4LLM EvaluationLLM-as-a-Judge

    Capital One, AI Foundations, McLean, VA 22102, USA.

  23. Empirical Software Engineering TerraProbe: A Layered-Oracle Framework for Detecting Deceptive Fixes in LLM-Assisted Terraform

    Jun 25, 2026Manar Alsaid, Chimdumebi Nebolisa, Faris AbbasLLM EvaluationAutomated Program Repair

    Department of Computer Science and Information Systems, East Texas A&M University, 2600 W. Neal Street, Commerce, 75428, Texas, United States · Department of Dual Enrollment and P-16 Programs, Texas Woman’s University, 3042026 Administration Drive, Denton, 76204, Texas, United States

  24. Theory-Scale Auto-Formalization of Logics for Computer Science

    Jun 25, 2026Yuming Feng, Frederick Pu, One An +5Automated Theorem ProvingLean Theorem Proving

    Johns Hopkins University · University of Toronto · University of Pennsylvania +1

  25. Estimating Uncertainty in Classifier Performance with Applications to Large Language Models and Nested Data

    Jun 24, 2026Kylie AnglinLLM EvaluationConfidence Region Estimation

    Department of Educational Psychology, Neag School of Education, University of Connecticut

  26. Agentic System as Compressor: Quantifying System Intelligence in Bits

    Jun 24, 2026Zihan Qin, Hongrui ZhangAI Agent EvaluationAutomated Evaluation

    Peking University

  27. Automated Residual Plot Assessment With the R Package autovi and the Shiny Application autovi.web

    Jun 23, 2026Weihao Li, Dianne Cook, Emi Tanaka +2Automated Evaluation

    Department of Econometrics and Business Statistics, Monash University, Wellington Road, VIC 3800, Australia · Research School of Finance, Actuarial Studies and Statistics, The Australian National University, CBE Building 26C, Kingsley Street, ACT 2600, Australia · Department of Statistics, University of Nebraska, Hardin Hall, 3310 Holdrege St Suite 340, Lincoln, NE 68583, United States

  28. Do LLM Attribution Metrics Transfer? Auditing Retrieval-Augmented Generation Evaluation Across Datasets and Constructs

    Jun 22, 2026Tianyu Ding, Aditya Nannapaneni, Juan Pablo De la Cruz WeinsteinLanguage Model Generation EvaluationLLM Auditing

    Amazon Web Services

  29. Litmus: Zero-Label, Code-Driven Metric Specification for Evaluating AI Systems

    Jun 22, 2026Prajjwal Gupta, Prasang Gupta, Vishal Bhutani +4AI Agent EvaluationAI Agent Monitoring

    PricewaterhouseCoopers, U.S.

  30. Expert Consensus on Criteria for the Automated Assessment of Laparoscopic Camera Navigation

    Jun 22, 2026Amir Ebrahimzadeh, Nazila Esmaeili, Michael Ghadimi +1Surgical Video UnderstandingAutomated Evaluation

    Center for Digital Surgery, Department of General, Visceral and Pediatric Surgery, University Medical Center Göttingen, Robert-Koch-Straße 40, Göttingen, 37075, Germany · Fraunhofer Research Institution for Individualized and Cell-based Medical Engineering (IMTE), Mönkhofer Weg 239a, Lübeck, 23562, Germany