Human-in-the-Loop Evaluation

Follow this topic Ask about this topic

Momentum

11 papers in the last four weeks, up 38% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 102

All topics

Catching One in Five: LLM-as-Judge Blind Spots in Production Multi-Turn Transaction Agents
Jun 9, 2026Sawyer Zhang, Alexander Wang, Sophie LeiLLM-as-a-Judge LLM Agent Evaluation
Lumivate (Lumi)
Contemporary AI lacks the imagination to diverge or negate in science
Jun 6, 2026Honglin Bao, Siyang Wu, Xiao Liu +3LLM-as-a-Judge Scientific Reasoning
Data Science Institute, University of Chicago · Knowledge Lab, University of Chicago
Illusions of the Gold Standard: A Large-scale Analysis of Human Evaluation Protocols for Long-form Text Generation
Jun 6, 2026Katelyn Xiaoying Mei, Yi-Li Hsu, Minjoon Choi +5Human-in-the-Loop Evaluation Scientific Reproducibility
University of Washington · National Tsing Hua University · Seoul National University +2
Towards Healthy Evolution: Exploring the Role and Mechanisms of Human-Agent Interaction in Self-Evolving Systems
Jun 4, 2026Dianxing Shi, Bowen Wang, Junqi He +2Human-in-the-Loop Evaluation LLM Safety Alignment
The University of Osaka · Independent researcher
The Saturation Trap and the Subjectivity of Intervention Timing: Why Affect-Based Triggers and LLM Judges Fail to Time Interventions on Autonomous Agents
Jun 2, 2026Manvendra ModgilLLM-as-a-Judge Human-in-the-Loop Evaluation
Hierarchical Online Prompt Mutation with Dual-Loop Feedback for Guardrailed Evidence Document Generation: A Production-Evaluation Case Study
May 31, 2026Nataraj Agaram Sundar, Tejas MorabiaLLM Guardrails Human-in-the-Loop Evaluation
eBay Inc. · San Jose, CA, USA
CV-Arena: An Open Benchmark for Instructional Computer Vision Problem Solving with Human-AI Collaborative Preferences
May 30, 2026Fangzhou Lin, Peiran Li, Lingyu Xu +12Image Editing Evaluation Text-Guided Image Editing
Texas A&M University · Worcester Polytechnic Institute · Tohoku University +4
Bridging Chemists and AI: An Expert-Augmented Framework for Interpretable Route Evaluation
May 27, 2026Yujia Guo, Mikhail Kabeshov, Tat Hong Duong Le +6Human-in-the-Loop Evaluation Deep Sets
Department of Computer Science, Aalto University, Espoo, Finland. · Discovery Sciences R&D, AstraZeneca, Gothenburg, Sweden. · Department of Computer Science and Engineering,May Chalmers University of Technology and University of Gothenburg, Gothenburg, Sweden. +1
Human-in-the-Loop Swarms: A Bionic Swarm Approach to Real-World Soil Mapping
May 27, 2026Petras Swissler, Mohammadali Rashidioun, Nicholas Sahu +3Multi-Robot Systems Swarm Robotics
Department of Mechanical and Industrial Engineering, New Jersey Institute of Technology, Newark, New Jersey, United States of America · Department of Data Science, New Jersey Institute of Technology, Newark, New Jersey, United States of America · University of Washington, Seattle, Washington, United States of America +1
GrowLoop: Self-Evolving Conversation Evaluation Seeded by Human
May 26, 2026Yihang Lin, Yunze Gao, Zeyang Lin +3Benchmark Design Human-in-the-Loop Evaluation
Amap, Alibaba Group · The Chinese University of Hong Kong, Shenzhen
Grounding Text Embeddings in Stakeholder Associations
May 26, 2026Jonathan Rystrøm, Sofie Burgos-Thorsen, Zihao Fu +3Text Embedding Evaluation Human-in-the-Loop Evaluation
University of Oxford, UK · Institute for Wicked Problems, Denmark · The Chinese University of Hong Kong, China +2
Workflow Closure Is Not Scientific Closure in Auto-Research Systems
May 25, 2026Shuai Wang, Xinyuan Tian, Pangpang Liu +1Human-in-the-Loop Evaluation Autonomous Scientific Discovery
School of Public Health, Yale University, USA
From Automation to Collaboration: Human-in-the-Loop Methods for Safe and Trustworthy NLP
May 24, 2026Most. Sharmin Sultana Samu, MD. Tanvir Ahmed Seum, Md. Rakibul IslamLLM Auditing Human-in-the-Loop Annotation
Department of Computer Science and Engineering, BRAC University, Dhaka-1212. · Department of Electrical and Electronic Engineering, Rajshahi University of Engineering and Technology, Rajshahi-6204.
Healthcare LLM Benchmarks Are Only as Good as Their Explicit Assumptions
May 21, 2026Naveen Raman, Santiago Cortes-Gomez, Mateo Dulce Rubio +2Healthcare Benchmark Design
New York University
Addressing the Synergy Gap: The Six Elements of the Design Space
May 20, 2026Tommaso Turchi, Ben Wilson, Matt Roach +2Human-in-the-Loop Evaluation Human-Centered AI
Department of Computer Science University of Pisa Pisa, Italy · Computational Foundry Swansea University Swansea, United Kingdom · Cardiff Metropolitan University, United Kingdom & Computational Foundry, Swansea University +1
Can Conversational XAI Improve User Performance? An Experimental Study
May 19, 2026Sven Kruschel, Julian Rosenberger, Lasse Bohlen +2Explainable Artificial Intelligence Human-in-the-Loop Evaluation
TU Dresden, Dresden, Germany · University of Regensburg, Regensburg, Germany
QQJ: Quantifying Qualitative Judgment for Scalable and Human-Aligned Evaluation of Generative AI
May 17, 2026Marjan Veysi, Pirooz Shamsinejadbabaki, Mohammad Zare +1LLM-as-a-Judge Human-in-the-Loop Evaluation
AI Lab, Arioobarzan Engineering Team, Shiraz, Iran · Department of Computer Engineering and Information Technology, Shiraz University of Technology, Shiraz, Iran · Department of Informatics, Bioengineering, Robotics and Systems Engineering, University of Genoa, Genoa, Italy
Improving Reproducibility in Evaluation through Multi-Level Annotator Modeling
May 13, 2026Deepak Pandita, Flip Korn, Chris Welty +1Inter-Rater Reliability LLM Evaluation
Rochester Institute of Technology · Google Research
Skill-Aligned Annotation for Reliable Evaluation in Text-to-Image Generation
May 13, 2026Abdelrahman Eldesokey, Merey Ramazanova, Ahmad Sait +4Human-in-the-Loop Evaluation Automated Evaluation
King Abdullah University of Science and Technology (KAUST) · Thuwal, Saudi Arabia
DisaBench: A Participatory Evaluation Framework for Disability Harms in Language Models
May 12, 2026Eugenia Kim, Ioana Tanase, Christina MallonLLM Safety Benchmarks LLM Evaluation
Microsoft
Elicitation-Augmented Bayesian Optimization
May 12, 2026Alvar Haltia, Ville Hyvönen, Samuel KaskiBayesian Optimization Human-in-the-Loop Evaluation
Aalto University · University of Helsinki · Aalto University, University of Manchester
LLARS: Enabling Domain Expert & Developer Collaboration for LLM Prompting, Generation and Evaluation
May 11, 2026Philipp Steigerwald, Mara Stieler, Jennifer Burghardt +2Language Model Generation Evaluation LLM Prompting
Technische Hochschule Nürnberg Georg Simon Ohm, Nürnberg, Germany · Faculty of Computer Science, Centre for Artificial Intelligence (KIZ) · Faculty of Social Sciences, Institute for E-Counselling
Do Benchmarks Underestimate LLM Performance? Evaluating Hallucination Detection With LLM-First Human-Adjudicated Assessment
May 8, 2026I. F. Atasoy, B. Mutlu, E. A. Sezer +1LLM Evaluation LLM-Assisted Annotation
Department of Computer Engineering, Hacettepe University, Ankara, 06800, Türkiye · Department of Computer Engineering, Ankara University, Ankara, 06830, Türkiye · Zephlen AI and Information Technologies Inc., Kocaeli, 41400, Türkiye
Augmenting Human Evaluation with LLM Judges: How Many Human Reviews Do You Need?
May 8, 2026Jane Paik KimLLM Evaluation LLM-as-a-Judge
Department of Psychiatry and Behavioral Sciences Stanford University Stanford, CA 94304
PersonaKit (PK): A Plug-and-Play Platform for User Testing Diverse Roles in Full-Duplex Dialogue
May 7, 2026Hyunbae Jeon, Jinho D. ChoiVoice Agent Evaluation Spoken Dialogue Systems
Department of Computer Science Emory University Atlanta, GA, USA
Why Expert Alignment Is Hard: Evidence from Subjective Evaluation
May 6, 2026Tzu-Mi Lin, Wataru Hirota, Tatsuya Ishigaki +2Human Preference Evaluation LLM Alignment
National Yang Ming Chiao Tung University, Taiwan · Stockmark, Japan · National Institute of Advanced Industrial Science and Technology (AIST), Japan +1
STABLEVAL: Disagreement-Aware and Stable Evaluation of AI Systems
May 4, 2026Akash Bonagiri, Gerard Janno Anderias, Saee Patil +6Annotator Disagreement AI Agent Evaluation
Department of Computer Science, University of California, Davis, Davis, CA, USA.
Principles and Guidelines for Randomized Controlled Trials in AI Evaluation
May 3, 2026Christopher Kelly, Angelica Chowdhury, Alexandra Campili +5Human-in-the-Loop Evaluation Rubric-Based Evaluation
Independent · Dalhousie University · Northeastern University London +4
Linking Behaviour and Perception to Evaluate Meaningful Human Control over Partially Automated Driving
May 1, 2026Ashwin George, Lucas Elbert Suryana, Lorenzo Flipse +5Autonomous Driving Human-in-the-Loop Evaluation
Centre for Meaningful Human Control, Delft University of Technology, Mekelweg 2, 2628 CD, Delft, Zuid Holland, The Netherlands · Department of Cognitive Robotics, Delft University of Technology, Mekelweg 2, 2628 CD, Delft, Zuid Holland, The Netherlands · Department of Transport & Planning, Delft University of Technology, Mekelweg 2, 2628 CD, Delft, Zuid Holland, The Netherlands +1
HATS: An Open data set Integrating Human Perception Applied to the Evaluation of Automatic Speech Recognition Metrics
Apr 30, 2026Thibault Bañeras Roux, Jane Wottawa, Mickael Rouvier +2ASR Evaluation Human-in-the-Loop Evaluation
Nantes University, France · Le Mans University, France · Avignon University, France

Jun 9, 2026, cs.CLJ/K move · Enter open · S save

Catching One in Five: LLM-as-Judge Blind Spots in Production Multi-Turn Transaction Agents

Sawyer Zhang, Alexander Wang, Sophie Lei

Lumivate (Lumi)

LLM-as-a-Judge LLM Agent Evaluation Human-in-the-Loop Evaluation LLM Agent Reliability

Open paper AskPDF arXiv

LLM-as-judge is the default instrument for evaluating conversational agents, yet its reliability is almost always reported as agreement with human ratings, not recall of real defects. We study a deployed multi-turn food-and-beverage ordering agent and measure how many genuine quality problems its built-in LLM judge catches, using exhaustive human transcript review as ground truth. Across three batches the judge surfaces well under a quarter of human-confirmed systematic problems -- 2 of 9 patterns (22%) in one batch, and its operational gate flagged zero of 100 rounds in a batch where humans confirmed 23 distinct defects and 7 new cross-cutting patterns. Our blind-spot taxonomy shows the failure is structured, not random: the judge catches turn-local issues (a fabricated statistic, a wrong language) but misses cross-turn state issues (confirm-gate lockout, cart hallucination, escalation lockout, stale referents). The mechanism: the scoring rubric exposes only three coarse axes (intent, brand-voice, personalization) and has no category for the behavioural dimensions -- state-tracking, guardrails, recovery -- where most defects cluster. The failure is routing, not perception: 113 of 114 rounds whose raw judge note describes a confirm-gate or cart-state defect are scored "brand voice", and none reach an operational failure -- the gate is wired to hangs and hard assertions, not the rubric -- so the 0% is a routing-and-wiring failure, not blindness. The consequence for prevalence estimation is sharp: when the apparent defect rate is zero the Rogan-Gladen correction degenerates -- no signal can recover the true rate -- while where the gate reports a nonzero rate the same estimator implies a 3-6x undercount under our measured sensitivity. For production multi-turn agents, automated judging is a regression floor, not a substitute for human review.

Momentum

Related topics

Latest papers 102