Human-in-the-Loop Evaluation

Ask about this topic

Overview

This field examines how people evaluate AI outputs and collaborate with AI during testing and deployment, including whether systems are accurate, useful, safe, and appropriately trusted. Common approaches include expert rubrics and pairwise preferences, user studies and interaction traces, human-validated benchmarks, and LLM judges calibrated against human ratings, with uncertain or disputed cases escalated for review. Researchers also study how to allocate limited review effort, model annotator disagreement, and measure effects on human decisions and learning over time. The central tension is scaling evaluation with automated judges and simulations without losing expert context or mistaking agreement, confidence, or benchmark scores for real-world quality.

Latest papers

CardsList

160 papers19 papers in the last four weeks, level with the four weeks before. 0.1% of all new papers.

  1. End-to-End Evaluation and Governance of an EHR-Embedded AI Agent for Clinicians

    Apr 30, 2026Aaryan Shah, Andrew Hines, Alexia Downs +6HealthcareAI Agent Evaluation

    1Canvas Medical, San Francisco, CA, USA · Department of Biomedical Data Science, Stanford University, Stanford, CA, USA · 3XPC (X Primary Care) +2

  2. Human-in-the-Loop Benchmarking of Heterogeneous LLMs for Automated Competency Assessment in Secondary Level Mathematics

    Apr 29, 2026Jatin Bhusal, Nancy Mahatha, Aayush Acharya +1LLM EvaluationEducational Assessment

    Research and Incubation Center (RAIN), Sunway College Kathmandu

  3. Understanding the Limits of Automated Evaluation for Code Review Bots in Practice

    Apr 27, 2026Veli Karakaya, Utku Boran Torun, Baykal Mehmet Uçar +1LLM-as-a-JudgeHuman-in-the-Loop Evaluation

    Bilkent University Ankara, Turkey · Beko İstanbul, Turkey

  4. Multi-Dimensional Evaluation of Sustainable City Trips with LLM-as-a-Judge and Human-in-the-Loop

    Apr 27, 2026Ashmi Banerjee, Adithi Satish, Wolfgang Wörndl +1LLM EvaluationLLM-as-a-Judge

    Technical University of Munich Munich, Germany · Polytechnic University of Bari Bari, Italy

  5. Rethinking XAI Evaluation: A Human-Centered Audit of Shapley Benchmarks in High-Stakes Settings

    Apr 24, 2026Inês Oliveira e Silva, Sérgio Jesus, Iker Perez +4Human-in-the-Loop EvaluationShapley Value Attribution

    University of Porto, Feedzai · Feedzai · University of Porto

  6. Who Defines "Best"? Towards Interactive, User-Defined Evaluation of LLM Leaderboards

    Apr 23, 2026Minji Jung, Minjae Lee, Yejin Kim +2LLM EvaluationHuman-in-the-Loop Evaluation

    Yonsei University, South Korea

  7. CollabSkill: Evaluating Human-Agent Collaboration On Real-World Tasks

    Apr 20, 2026Yijia Shao, Zora Zhiruo Wang, Neel Ahuja +3AI Agent EvaluationHuman-in-the-Loop Evaluation

    Stanford University · Carnegie Mellon University · University of Rochester +1

  8. AI-Assisted Requirements Engineering: An Empirical Evaluation Relative to Expert Judgment

    Apr 16, 2026Oz Levy, Ilya Dikman, Natan Levy +1Requirements EngineeringHuman-in-the-Loop Evaluation

    Faculty of Industrial Engineering and Technology Management, Holon Institute of Technology (HIT) Holon, Israel · School of Computer Science and Engineering Hebrew University Jerusalem (HUJI) Jerusalem, Israel

  9. RuleEdit: Failure-Guided Human-AI Model Editing with Prospective Impact Preview

    Apr 12, 2026Min Hun Lee, Justin Yu Feng TeoModel EditingHuman-AI Decision Making

    Singapore Management University, Singapore

  10. HiL-Bench (Human-in-Loop Benchmark): Do Agents Know When to Ask for Help?

    Apr 10, 2026Tu Trinh, Mohamed Elfeki, Guangze Luo +9LLM AgentsAI Agent Benchmarks

  11. MuTSE: A Human-in-the-Loop Multi-use Text Simplification Evaluator

    Apr 10, 2026Rares-Alexandru Roscan, Gabriel Petre1, Adrian-Marius Dumitran +1Semantic AlignmentText Simplification

    University of Bucharest, Faculty of Mathematics and Computer Science, Academiei 14, 010014, Bucharest, Romania · Cu Drag si Sport SRL, Bucharest, Romania · Softbinator Technologies, Bucharest, Romania +1

  12. Sustained Impact of Agentic Personalisation in Marketing: A Longitudinal Case Study

    Apr 9, 2026Olivier Jeunen, Eleanor Hanna, Schaun WheelerHuman-in-the-Loop EvaluationLong-Horizon Agent Evaluation

    aampe Antwerp, Belgium · aampe Raleigh, NC, USA · aampe Cary, NC, USA

  13. "Why This Avoidance Maneuver?" Contrastive Explanations in Human-Supervised Maritime Autonomous Navigation

    Apr 9, 2026Joel Jose, Andreas Madsen, Andreas Brandsæter +2Explainable Motion PlanningCounterfactual Explanations

    Department of ICT and Natural Sciences, Norwegian University of Science and Technology (NTNU), Ålesund, Norway. · Department of Ocean Operations and Civil Engineering, Norwegian University of Science and Technology (NTNU), Ålesund, Norway. · Department of Science and Mathematics, Volda University College (HVO), Volda, Norway. +1

  14. Let's Have a Conversation: Designing and Evaluating LLM Agents for Interactive Optimization

    Apr 3, 2026Joshua Drossman, Alexandre Jacquillat, Sébastien MartinLLM Agent EvaluationHuman-in-the-Loop Evaluation

    Operations Research Center and Sloan School of Management, Massachusetts Institute of Technology, Cambridge, MA. · Kellogg School of Management, Northwestern University, Evanston, IL

  15. Semantic Interaction for Narrative Map Sensemaking: An Insight-based Evaluation

    Mar 31, 2026Brian Felipe Keith-Norambuena, Fausto German, Eric Krokos +2Visual AnalyticsHuman-in-the-Loop Evaluation

    Universidad Católica del Norte, Av. Angamos 0610, Antofagasta, 1270709, Chile · Virginia Tech, 620 Drillfield Drive, Blacksburg, VA, 24061, USA · U.S. Government, Washington, D.C. 20500, USA

  16. Culturally Adaptive Explainable LLM Assessment for Multilingual Information Disorder: A Human-in-the-Loop Approach

    Mar 28, 2026Maziar Kianimoghadam JouneghaniCross-Cultural Language Model EvaluationFaithfulness of Language Model Explanations

    University of Turin

  17. User Involvement in Robotic Wheelchair Development: A Decade of Limited Progress

    Mar 27, 2026Mario Andres Chavarria, Santiago Price Torrendell, Aude Billard +7Participatory DesignAssistive Robotics

    Geneva School of Health Science, University of Applied Sciences and Arts Western Switzerland, Geneva, Switzerland · Institute Of Systems and Information Engineering, University of Tsukuba, Tsukuba, Japan · Learning Algorithms and Systems Laboratory, Ecole Polytechnique Fédérale de Lausanne, Lausanne, Switzerland +7

  18. Human-in-the-Loop Pareto Optimization: Trade-off Characterization for Assist-as-Needed Training and Performance Evaluation

    Mar 24, 2026Harun Tolasa, Volkan PatogluHuman-in-the-Loop EvaluationPareto Optimization

    the Faculty of Engineering and Natural Sciences at Sabanci University, Istanbul, Turkiye.

  19. LH-Bench: Skill-Grounded Evaluation of Long-Horizon Agents on Subjective Enterprise Tasks

    Mar 24, 2026Abhishek Chandwani, Ishan GuptaLong-Horizon Agent EvaluationAI Agent Benchmarks

    Metaphi Inc. USA

  20. From Accuracy to Readiness: Metrics and Benchmarks for Human-AI Decision-Making

    Mar 19, 2026Min Hun LeeHuman-AI Decision MakingHuman-AI Collaboration

    Singapore Management University, Singapore

  21. A Human-in/on-the-Loop Framework for Accessible Text Generation

    Mar 19, 2026Lourdes Moreno, Paloma MartínezText SimplificationHuman-in-the-Loop AI

    Universidad Carlos III de Madrid Av. Universidad, 30, Leganés, 28911, Spain

  22. Multisource human-in-the-loop digital twin testbed for connected and autonomous vehicles in mixed traffic flow

    Mar 18, 2026Jianghong Dong, Chunying Yang, Mengchi Cai +5Intelligent Transportation SystemsAutonomous Driving Simulation

    School of Vehicle and Mobility, Tsinghua University, Beijing 100084, China. · Department of Civil and Environmental Engineering, University of Michigan, Ann Arbor, MI 48109, USA.

  23. From Optimizable to Interactable: Mixed Digital Twin-Empowered Testing of Vehicle-Infrastructure Cooperation Systems

    Mar 18, 2026Jianghong Dong, Chunying Yang, Mengchi Cai +4Intelligent Transportation SystemsDigital Twins

    School of Vehicle and Mobility, Tsinghua University, 100084 Beijing, China

  24. When Should Humans Step In? Optimal Human Dispatching in AI-Assisted Decisions

    Mar 14, 2026Lezhi Tan, Naomi Sagan, Lihua Lei +1AI-Assisted Decision MakingHuman-AI Collaboration

    Stanford University, USA

  25. Human-in-the-Loop LLM Grading for Handwritten Mathematics Assessments

    Mar 13, 2026Arne Vanhoyweghen, Vincent Holst, Melika Mobini +9Automated GradingAI in Education

    Data Analytics Lab, Vrije Universiteit Brussel, 1050 Brussel, Belgium · School of Engineering and Applied Sciences, Harvard University, Cambridge, Massachusetts 02138, USA

  26. Reference-Free Image Quality Assessment for Virtual Try-On via Human Feedback

    Mar 13, 2026Yuki Hirakawa, Takashi Wada, Ryotaro Shimizu +6Virtual Try-OnHuman-in-the-Loop Evaluation

    ZOZO Research · ZOZO Inc. · Keio University

  27. Human-Centered Evaluation of an LLM-Based Process Modeling Copilot: A Mixed-Methods Study with Domain Experts

    Mar 13, 2026Chantale Lauer, Peter Pfeiffer, Nijat MehdiyevBusiness Process ModelingLLM Agent Evaluation

    Saarland University, Saarbrücken, Saarland, Germany · German Research Institute for Artificial Intelligence, Saarbrücken, Saarland, Germany · 5Plus GmbH, Würzburg, Bavaria, Germany

  28. Agreement Between Large Language Models, Human Reviewers, and Authors in Evaluating STROBE Checklists for Observational Studies in Rheumatology

    Mar 12, 2026Emre Bilgin, Ebru Ozturk, Meera Shah +6LLM EvaluationHealthcare

    Sakarya University, Faculty of Medicine, Department of Internal Medicine, Division of Rheumatology, Sakarya, Turkiye · Topiwala NaBonal Medical College and B.Y.L. Nair Hospital, Department of Rheumatology, , Mumbai, India · St Luke’s Medical Center (Global City), Philippines +9

  29. CHiL(L)Grader: Calibrated Human-in-the-Loop Short-Answer Grading

    Mar 12, 2026Pranav Raikote, Korbinian Randl, Ioanna Miliou +2Automated GradingAI in Education

    Department of Computer and Systems Sciences, Stockholm University, 164 25 Kista, Sweden

  30. Mind the Sim2Real Gap in User Simulation for Agentic Tasks

    Mar 11, 2026Xuhui Zhou, Weiwei Sun, Qianou Ma +8User SimulationSim-to-Real Transfer

    Carnegie Mellon University, Language Technologies Institute