Human-in-the-Loop Evaluation

Overview

This field examines how people evaluate AI outputs and collaborate with AI during testing and deployment, including whether systems are accurate, useful, safe, and appropriately trusted. Common approaches include expert rubrics and pairwise preferences, user studies and interaction traces, human-validated benchmarks, and LLM judges calibrated against human ratings, with uncertain or disputed cases escalated for review. Researchers also study how to allocate limited review effort, model annotator disagreement, and measure effects on human decisions and learning over time. The central tension is scaling evaluation with automated judges and simulations without losing expert context or mistaking agreement, confidence, or benchmark scores for real-world quality.

Latest papers

CardsList

174 papers17 papers in the last four weeks, down 15% on the four weeks before. 0.1% of all new papers.

  1. Remote Teleoperation of Endovascular Robots: A Network Emulation Framework for Systematic Performance Evaluation

    Nov 18, 2026Xingyu Chen, Nikola Fischer, Harry Robertshaw +3Robot TeleoperationRobotic Endovascular Intervention

  2. VeriMath: Human-in-the-Loop Generation and Verification of Curriculum-Aligned Mathematics Assessment Items

    Nov 17, 2026Quy P. Nguyen, Duong Q. Nguyen, Dinh P. T. Nguyen +9Mathematical Problem GenerationEducational Assessment

  3. Prioritize, Don't Decide: What We Learned Auditing a Deployed Human-in-the-Loop CV Sorter

    Nov 17, 2026Hadi Mohammadi, Jeroen Sparla, Alexandros Paliouras +2Human-in-the-Loop EvaluationLearning to Rank

  4. When Lower Harm Misleads: Auditing Concept Corrections

    Nov 17, 2026Tianyou Lai, Wentao Yue, Hongji Li +2Human-in-the-Loop Evaluation

  5. Prediction-Powered LLM Ranking with Adaptively Collected Human Feedback

    Nov 9, 2026Gabriel Sargent, Will Wei Sun, Zhengwu Zhang +1LLM EvaluationHuman-in-the-Loop Evaluation

  6. Multi-Agent Screening: What Agreement and Acceptance Cannot Tell You

    Nov 9, 2026Benjamin Chang, Dell ZhangAutomated Agent EvaluationHuman-in-the-Loop Evaluation

  7. Q-OIL: Optimistic Exploration in Learning from Human Interventions

    Nov 5, 2026Pranav Teegavarapu, Sriyash Poddar, Rico Qi +5RL for RoboticsHuman-in-the-Loop AI

  8. Interactive Visual Gym: Evaluating Visual Interaction Models with Simulated Users in Interactive Worlds

    Nov 5, 2026Chenhao Zheng, Zixian Ma, Kai Li +11VLM EvaluationMultimodal Agent Evaluation

  9. Clinician-AI Collaboration in Endovascular Neurointerventions: In-Vitro User Study of Copilot Assistance Modes for Robotic Navigation Across the Autonomy Spectrum

    Nov 2, 2026Nikola Fischer, Harry Robertshaw, S. M. Hadi Sadati +16Human-AI CollaborationRobotic Endovascular Intervention

  10. Recursive Self-Improvement of Exoskeleton Assistance Policies through Human-in-the-Loop Feedback

    Nov 1, 2026Muyuan Ma, Shuo Jiang, Lijun Han +5Assistive RoboticsRobot Policy Adaptation

  11. Principled Human-AI Collaboration in Agentic Workflows

    Nov 1, 2026Young Hyun Cho, Will Wei SunHuman-AI CollaborationAgentic Workflows

  12. HALO: Human-Grounded Granular Evaluation of Proactive Agents

    Oct 29, 2026Wenqi Pei, Xinyu Wang, Jinrui Liu +5Proactive AgentsProactive Assistance

  13. Human Evaluation with Automatic Scores: Separating Sampling and Estimation Gains

    Oct 28, 2026Yang Liu, Ran Zou, Zhengtao Gui +1Human-in-the-Loop EvaluationLLM Evaluation

  14. From Self-Confidence to Statistical Accuracy: Scaling Human-in-the-Loop Evaluation with Direct and LLM-Assisted Auditing

    Oct 26, 2026Harine Choi, Chenchen Kuai, Zihao Li +5LLM EvaluationLLM Auditing

  15. DealsFlow: An Evidence-Gated Agentic Harness for Venture Capital Deal Triage

    Oct 23, 2026Ming Ze Tang, David Cutting, Boyd Carson +1Financial Language Model EvaluationLLM Agent Evaluation

  16. SciDiagBench: A Type-Specific Benchmark for Human-Aligned Evaluation of Scientific Diagrams

    Oct 20, 2026Hao Wang, Zixuan Li, Long Bai +4VLM EvaluationImage Generation Evaluation

  17. VEILED-SH: Vulnerability Expression through Indirect Language Evaluation Dataset (Self-Harm)

    Oct 19, 2026Leanne Tan, Keith Chua, Shaun Khoo +7LLM Safety BenchmarksLanguage Model Safety Evaluation

  18. Towards Human-AI Complementarity for AI Safety

    Oct 18, 2026Sander Lanbo Shi, Miranda Anna Christ, Eric Modesitt +20Human-AI ComplementarityHuman Oversight of AI

  19. Edit Discovery in Human-AI Interactions

    Oct 15, 2026Alireza Mirrokni, Shayan Kiyani, George Pappas +1Human-in-the-Loop EvaluationHuman-AI Interaction

  20. Author-in-the-loop Agentic Reproducibility Auditing for a Machine Learning Conference

    Oct 15, 2026Yukun Dong, Linjun Zhang, Tian ZhengML ReproducibilityScientific Reproducibility

  21. Kolab: Benchmarking Human-Agent Collaboration in Long-Horizon Workflows

    Oct 15, 2026Shirley Anugrah Hayati, Patricia Anugrah Setiani, Ruizi Wang +4Human-AI CollaborationLong-Horizon Agent Evaluation

  22. One-Sided Verification Under Limited Expert Review

    Oct 13, 2026Fan Nie, Haotian Ye, Shurui Liu +3Reasoning VerificationHuman-in-the-Loop Evaluation

  23. A Scoping Review and Experimental Study on Reinforcement Learning from Human Feedback for Human-Robot Collaboration

    Oct 7, 2026Alexandra Coroiu, Andrea Vogt, Viktor Werbilo +3RL from Human FeedbackHuman-Robot Collaboration

  24. Evaluating human-AI workflows for field research in viticulture

    Oct 6, 2026Niko Carvajal Janke, Daoyuan Jin, Shivranjani Baruah +4AI-Assisted Scientific ResearchHuman-in-the-Loop Evaluation