Human-in-the-Loop Evaluation

Ask about this topic

Momentum

11 papers in the last four weeks, up 38% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 102

All topics
CardsList

  1. The Checking Problem: What must be true before AI ships in a regulated firm

    Jul 25, 2026Prerit AhujaHuman-in-the-Loop EvaluationHuman Oversight of AI

    Independent Researcher

  2. HARP: The Human--AI Research Platform

    Jul 22, 2026Zeshu Zhu, Natalie Friedman, Kevin Weatherwax +1Human-in-the-Loop EvaluationHuman-AI Interaction

    BTPX Innovation Lab · BTPX User Assistance

  3. Mitigating Fabrication in Multi-Stage LLM Pipelines for Hiring: An Empirical Evaluation of Prompt Guardrails and Human-in-the-Loop Checkpoints

    Jul 22, 2026Hiroko TakanoAI-Assisted Decision MakingLLM Hallucination Mitigation

    Lemmanode LLC

  4. PhaseAware: Interpretable Human-in-the-Loop Rehabilitation Scoring with Boundary Monitoring

    Jul 22, 2026Yankai Zheng, Yuhe Liu, Yuxin Ma +8Interpretable MLHuman-in-the-Loop Evaluation

    Department of Medical Imaging, The First Affiliated Hospital of Xi'an Jiaotong University, Xi'an, 710061, China · Faculty of Electronic and Information Engineering, Xi'an Jiaotong University, Xi'an, 710049, China · School of Future Technology, Xi'an Jiaotong University, Xi'an, 710049, China +2

  5. EduPanel: A Three-Agent LLM Judge for Teaching Videos -- Reliability, Complementarity, and Human Trust Calibration

    Jul 20, 2026Jia-Kai Dong, Yi-Cheng Lin, Hung-yi LeeLLM-as-a-JudgeEducational Assessment

    National Taiwan University · NTU Artificial Intelligence Center of Research Excellence

  6. Human Grounded Evaluation of Large Language Models for Optical Network Automation

    Jul 20, 2026Kiarash Rezaei, Omran Ayoub, Paolo Monti +1LLM Inference EfficiencyLLM Evaluation

    Department of Electrical Engineering, Chalmers University of Technology, 412 96 Gothenburg, Sweden · University of Applied Sciences and Arts of Southern Switzerland, 6928 Lugano, Switzerland

  7. Human-in-the-Loop User Feedback Affects Perceived Accuracy and Trust, but Task Subjectivity Matters

    Jul 20, 2026Donald R. Honeycutt, Mahsan Nourani, Eric D. RaganHuman Preference EvaluationHuman-in-the-Loop Evaluation

    University of Florida, USA · Northeastern University, USA

  8. Project Kaleidoscope: Contextual, Human-Aligned Evaluation for Real-World AI Applications

    Jul 16, 2026Leanne Tan, Rohan Jaggi, Shaun Khoo +1Human-in-the-Loop EvaluationRubric-Based Evaluation

    GovTech, Singapore · National University of Singapore · University of British Columbia

  9. Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops

    Jul 8, 2026Mingguang Chen, Licheng Wang, Bo QuLLM Self-RefinementHuman-in-the-Loop Evaluation

    University of California, Riverside (UCR) · AlphaAvatar · Illinois Institute of Technology (IIT)

  10. Manual, Joystick, or Haptic Control? An In Vitro Comparison of Navigation Strategies for Robotic Interventional Neuroradiology Procedures

    Jul 8, 2026Benjamin Jackson, Nikola Fischer, Harry Robershaw +16Robot TeleoperationHuman-in-the-Loop Evaluation

    School of Biomedical Engineering & Imaging Sciences, King’s College London, UK

  11. HAS-Bench: Evaluating LLM-Based Human-Agent Systems under Configurable Human Participation

    Jul 5, 2026Yaozu Wu, Wei-Chieh Huang, Jizhou Guo +11LLM Agent EvaluationHuman-in-the-Loop Evaluation

    The University of Tokyo · University of Illinois Chicago · MBZUAI +2

  12. LOTUSim: Multi-Domain Simulator for Marine Robotics

    Jul 3, 2026Cédric Buche, Juliette Grosset, Hélène Lechêne +4Multi-Robot SystemsRobotics Simulation

    CROSSING, IMT Atlantique · CROSSING IRL 2010, CNRS · IMT Atlantique +2

  13. CommonRoad-Game: A Human-in-the-Loop Simulation Framework for Autonomous Driving

    Jul 1, 2026Yunfei Bi, Youran WangHuman-in-the-Loop EvaluationAutonomous Driving Planning

    Technical University of Munich Munich, Germany

  14. From Technical Metrics to User Perception: A User Study of a Multimodal Human-Robot Interaction System for Object Detection and Grasping

    Jul 1, 2026Jian Song, Tian Zi, Shen GuantingRobotic GraspingHuman-in-the-Loop Evaluation

    The authors are with the Dalian University of Technology, Dalian, China.

  15. CLExEval: A Human-in-the-Loop Framework for Qualitative Evaluation of LLM Clinical Reasoning

    Jun 30, 2026Ajmal M., Abin Roy, Afthab Salam Kanniyan +4LLM-as-a-JudgeHuman-in-the-Loop Evaluation

    1MBZUAI · 2IIT Madras · 3Calicut Medical College

  16. HERO: Improving the Reliability and Sensitivity of Generative Model Evaluation Using Historical Data

    Jun 29, 2026Xinrui Ruan, Zhenyu Zhao, Waverly Wei +4Human-in-the-Loop EvaluationGenerative AI Evaluation

    Division of Biostatistics, University of California, Berkeley · Roblox Corporation · Department of Data Sciences and Operations, University of Southern California +2

  17. Helpful or Harmful? Evaluating LLM-Assisted Vulnerability Patching via a Human Study

    Jun 24, 2026Giulian Biolo, Michael Tezza, Yuanjun Gong +1Automated Program RepairHuman-in-the-Loop Evaluation

    University of Trento, IT · Vrije Universiteit Amsterdam, NL

  18. Grading the Grader: Lessons from Evaluating an Agentic Data Analysis System

    Jun 23, 2026Tian Zheng, Kai-Tai HsuLLM-as-a-JudgeHuman-in-the-Loop Evaluation

    Department of Statistics Columbia University New York, NY

  19. When CQs Go Wrong: Challenges in CQ Verification with OE-Assist

    Jun 23, 2026Anna Sofia Lippolis, Mohammad Javad Saeedizade, Robin Keskisärkkä +3Human-in-the-Loop Evaluation

    University of Bologna, Italy · Linköping University, Sweden · ISTC-CNR, Italy

  20. Judgment-Grounded Expansion for Peer Review Generation

    Jun 22, 2026Sheng Lu, Lizhen Qu, Iryna GurevychHuman-in-the-Loop EvaluationAutomated Peer Review

    1Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science, Technical University of Darmstadt and National Research Center for Applied Cybersecurity ATHENE · Department of Data Science and Artificial Intelligence (DSAI), Monash University

  21. Counsel: A Meta-Evaluation Dataset for Agentic Tasks

    Jun 19, 2026Sashank Pisupati, Henry Broomfield, Eujeong Choi +5LLM-as-a-JudgeLLM Agent Evaluation

    1Atla AI · 2Cohere AI · 3Mistral AI +1

  22. AURA: Adaptive Uncertainty-aware Refinement for LLM-as-a-Judge Auditing

    Jun 18, 2026Zilong Zhang, Yi-Ting Hung, Weiyi He +3LLM-as-a-JudgeLLM Auditing

    Department of Mathematics and Statistics Georgia State University · Department of Probability and Statistics Department of Computer Science and Engineering Michigan State University · Department of Mathematics and Statistics Concordia University +1

  23. A Clinician-Centered Pipeline for Annotation and Evaluation in Ultrasound AI Studies

    Jun 17, 2026Fangyijie Wang, Jianjun Yu, Wentao Shi +4Human-in-the-Loop AnnotationHuman-in-the-Loop Evaluation

    Research Ireland Centre for Research Training in Machine Learning · School of Medicine, University College Dublin, Dublin, Ireland · The Third People’s Hospital of Zhenjiang City, Zhenjiang, China +3

  24. RubricsTree: Scalable and Evolving Open-Ended Evaluation of Personal Health Agents across Health Memory and Medical Skills

    Jun 16, 2026Weizhi Zhang, Zechen Li, Hamid Palangi +16AI Agent EvaluationHuman-in-the-Loop Evaluation

    1Google Research, 2University of Illinois Chicago, *Work done during an internship at Google, †Corresponding Author

  25. Human-on-the-Bridge: Scalable Evaluation for AI Agents

    Jun 15, 2026Fouad BousetouaneAI Agent ReliabilityAI Agent Evaluation

    The University of Chicago, USA

  26. JoyAI-Sim: A Simulation-Enabled Interconversion Toolchain for the Embodied Data Pyramid

    Jun 15, 2026Peidong Liu, Yongce Liu, Songyan Guo +34Robotic Data CollectionRobotics Simulation

    Joy Future Academy, JD Group

  27. Is Your Trajectory Displacement Safe in Long-tail?

    Jun 15, 2026Qiao Sun, Weicheng Zheng, Yixin Huang +1Autonomous Driving Safety EvaluationAutonomous Driving

    Shanghai Qi Zhi Institute · Tongji University · Tsinghua University

  28. Ride, Track, and Recover: Pilot Randomized Trial of a Wearable Digital Self-Management Intervention During a Veteran Endurance-Cycling Program

    Jun 11, 2026Alan Ta, Nilsu Salgin, Caleb Armstrong +2Mental HealthHuman-in-the-Loop Evaluation

    Department of Industrial and Systems Engineering, Texas A&M University, 3127 TAMU, College Station, TX 77843-3127 · Texas A&M Health Telehealth Institute, 1700 Research Pkwy, Suite 250, College Station, TX 77845

  29. RogueAI: A Reverse Turing Test for Detecting Licensed AI Deception in Dialogue

    Jun 11, 2026Sara Candussio, Emanuele Ballarin, Lorenzo Bonin +2AI Agent ReliabilityLanguage Model Safety Evaluation

    1AILab, MIGe, University of Trieste, 34127 Trieste, IT · 2Computational Statistics and Machine Learning, Istituto Italiano di Tecnologia, Genova, IT · 3DIA, University of Trieste, 34127 Trieste, IT

  30. IntElicit: Eliciting and Assessing Contextualized Creativity via Dialogue Policy Optimization

    Jun 10, 2026Mingjia Li, Jin Wu, Hong Qian +7Creativity AssessmentPolicy Optimization

    East China Normal University · Shanghai Innovation Institute