Human-in-the-Loop Annotation

Ask about this topic

Overview

Human-in-the-loop annotation studies how people and AI can work together to label data for training, evaluation, and analysis. Common approaches use active learning to select informative examples, model-generated drafts or propagated labels for people to correct, and uncertainty-based routing to send difficult cases to experts. Tools and workflows span text, images, video, audio, and clinical data, while measurement methods examine annotator differences, disagreement, and label quality. A central tension is how to reduce time and cost without obscuring errors, bias, or genuine differences in judgment; automation still needs careful validation, especially on subjective and high-stakes tasks.

Latest papers

CardsList

135 papers12 papers in the last four weeks, down 29% on the four weeks before. 0.1% of all new papers.

  1. nnInteractive Agent: General-Purpose AI Agents as 3D Image Annotators

    Oct 22, 2026Fabian Isensee, Constantin Ulrich Harsy, Maximilian Rokuss +3Medical AI AgentsTool-Using Agents

    Division of Medical Image Computing (MIC), German Cancer Research Center (DKFZ), Heidelberg, Germany · National Center for Tumor Diseases (NCT) Heidelberg, a partnership between the German Cancer Research Center (DKFZ) and Heidelberg University Hospital (UKHD), Heidelberg, Germany · Helmholtz Imaging, German Cancer Research Center (DKFZ), Heidelberg, Germany +4

  2. GAIDA: Generative AI for Human-in-the-Loop Document Annotation

    Oct 19, 2026Islam Mesabah, Ali Elhosary, Priyabanta Sandulu +2Human-in-the-Loop AnnotationLLM-Assisted Annotation

    German Research Center for Artificial Intelligence (DFKI) · RPTU Kaiserslautern-Landau, Kaiserslautern, Germany

  3. CoDA: A Human-LM Collaborative Codebook Development Assistant for Domain-Specific Text Annotation

    Oct 16, 2026Jaspreet Ranjit, Swaminathan Chellappa, Augusto Rivas Costante +5LLM-Assisted AnnotationHuman-in-the-Loop Annotation

    Thomas Lord Dept. of Computer Science, University of Southern California · My Friend’s Place, Los Angeles, California · School of Social Work, University of North Carolina

  4. Annotrace: A Workbench for Step-Level Annotation of Agent Traces

    Oct 16, 2026Chenyi Huang, Arun Josephraj Arokiaraj, Bin Wu +1Human-in-the-Loop AnnotationLLM-Assisted Annotation

    Department of Computer Science, University College London, London, United Kingdom · Centre for Artificial Intelligence, University College London, United Kingdom

  5. Partial-Revelation Panel Forecasting: Predictive Information Gain, Allocation, and Coarsening-Consistent Operators

    Oct 16, 2026Bin Wen, Dai-Qiao Zhang, Tien-Ping TanHuman-in-the-Loop AnnotationConditional Distribution Estimation

    School of Computer Sciences, Universiti Sains Malaysia, Gelugor 11800, Penang, Malaysia · School of Mathematical Sciences, Universiti Sains Malaysia, Gelugor 11800, Penang, Malaysia

  6. From Human Corrections to Learned Segmentations: A Critical Mathematical Review of Information, Learning, and Human Effort

    Oct 14, 2026Jon SporringHuman-in-the-Loop AIHuman-in-the-Loop Annotation

    Department of Computer Science, University of Copenhagen, DK-2100 Copenhagen, Denmark

  7. Activating the Annotator: Operationalising Lived Experience in Data Annotation

    Oct 8, 2026Sheza Munir, Ramaravind Kommiya Mothilal, Syed Ishtiaque AhmedHuman-in-the-Loop AnnotationAnnotator Disagreement

    Department of Computer Science University of Toronto · Faculty of Information University of Toronto

  8. Goldsmith: Gold-Loss-Guided Definition Optimization with an Agentic Annotation Harness

    Oct 7, 2026Yihan Li, Hanyi Zhang, Xiaoxi Jiang +1LLM-Assisted AnnotationLarge Language Model-Guided Optimization

    Sun Yat-sen University, Guangzhou, China · South China University of Technology, Guangzhou, China

  9. An automated pipeline for standardised speech-unit annotation in spontaneous dialogue

    Oct 2, 2026Hanlu He, Harald Vilhelm Skat-Rørdam, Ingvi Örnólfsson +1Speech ProcessingTurn-Taking in Dialogue

    Department of Applied Mathematics and Computer Science, Technical University of Denmark

  10. Uncertainty-Guided Handshake: Efficient Human-in-the-Loop Refinement for Surgical-Grade Glioma Segmentation

    Oct 1, 2026Samuel Hart, Ahmad Yahya, Ahmed Karam EldalyInteractive Medical Image SegmentationHuman-in-the-Loop Annotation

    Department of Computer Science, University of Exeter, Exeter, EX4 4QF, United Kingdom. · Department of Nuclear Engineering, Faculty of Engineering, King Abdulaziz University, Jeddah, Saudi Arabia. · UCL Hawkes Institute, University College London, Gower St., London, WC1E 6AE, United Kingdom.

  11. HaPRL: Human-Anchored Process Reinforcement Learning for Visual Search Agent

    Sep 29, 2026Zhangquan Chen, Yaoxin Niu, Xiang An +5Human-in-the-Loop AnnotationProcess Supervision

    Tsinghua University · Peng Cheng Laboratory · LMMs-Lab +2

  12. QuanReview: Offline, Auditable Reconciliation of Human and LLM Span Annotations

    Sep 28, 2026Matteo Musacchio, Juan Cruz Giner Pulero, Isabel Castañeda +4Annotator DisagreementLLM-Assisted Annotation

    Universidad de San Andrés, Buenos Aires, Argentina · ISI Foundation, Turin, Italy · CONICET, Buenos Aires, Argentina +1

  13. Human-in-the-Loop Geospatial Annotation for Rapid Dataset Construction in Field-Deployed UAV Systems

    Sep 23, 2026Morgan Masters, Adam Korycki, Nikolaas Bender +4Human-in-the-Loop AnnotationUncertainty Propagation

    Department of Electrical and Computer Engineering, University of California Santa Cruz

  14. Can Vision-Language Models Analyze Human-Centered Video? Mapping Model Capabilities and Human-AI Collaborative Workflows

    Sep 23, 2026Xiyuan Shen, Jiuyang Lyu, Seokhyun Hwang +4Video UnderstandingVideo-Language Model Evaluation

    University of Washington, USA · Georgia Institute of Technology, USA · Google DeepMind, USA

  15. Experts Rise Where LLMs Disagree: Using Cross-Model Disagreement to Target Expert Effort in LLM Codebook Revision for Large-Scale Annotation

    Sep 22, 2026Zeyu He, Zhuqian Zhou, Kirk Vanacore +2LLM-Assisted AnnotationHuman-in-the-Loop Annotation

    The Pennsylvania State University, University Park, PA, USA · Cornell University, Ithaca, NY, USA

  16. Optimal Sequential Annotations for Off-Policy Evaluation

    Sep 22, 2026Woojin Chae, Ezinne Nwankwo, Haitong Qin +1Off-Policy EvaluationLearning with Missing Data

    University of Southern California · University of California, Berkeley · University of Washington, Seattle

  17. MICRO: Multi-Fidelity Active Search for Severe Error Discovery

    Sep 22, 2026Orlando Leone, Niclas Pokel, Pehuén Moure +2Human-in-the-Loop AnnotationMT Evaluation

    Institute of Neuroinformatics, University of Zurich and ETH Zurich, Switzerland · ETH AI Center, Switzerland · Department of Computational Linguistics, University of Zurich, Switzerland

  18. mbariml: a curation pipeline for turning deep-sea imagery and video into object-detection training data

    Sep 21, 2026Lonny Lundsten, Kevin Barnard, Dave CaressTraining Data CurationHuman-in-the-Loop Annotation

    Monterey Bay Aquarium Research Institute

  19. onPanda: Efficient Annotation of On-Policy Alignment Data for LLMs and Agents via Token-Level Correction

    Sep 21, 2026Lei Yang, Mengyin Liu, Jia Wang +9LLM AlignmentLLM-Assisted Annotation

    StepFun · Xiamen University

  20. Object Detection Benchmarks are Incomplete: The Role of Label Errors and Annotation Uncertainty

    Sep 18, 2026Sarina Penquitt, Jonathan Klees, Antonia van Betteray +4Object DetectionHuman-in-the-Loop Annotation

    Institute of Computer Science Osnabrück University Osnabrück, Germany · Department of Mathematics University of Wuppertal Wuppertal, Germany · Wayve London, United Kingdom

  21. IMVS: Interactive Medical Volume Segmentation with Test-Time Adaptation - A New Method for Annotating Radiology Datasets

    Sep 15, 2026Abhilaksh Singh Reen, Kushal Borkar, Ritvik MahapatraInteractive Medical Image SegmentationTest-Time Adaptation

    Independent Researcher, Delhi, India · Independent Researcher, Kolkata, India · California State University, Fresno, CA, USA

  22. CALICO: A Human-Centered, Codebook-Aligned System for Annotation

    Sep 13, 2026Boqin Yuan, Xiaoyi Gu, Fiona Li +3LLM-Assisted AnnotationHuman-in-the-Loop Annotation

    UC San Diego · University of Southern California

  23. What Counts as a Mistake? Annotating Recitation Events in Quran Memorization Transcripts

    Sep 10, 2026Mohamad Al Mdfaa, Nursultan Askarbekuly, Ahmed Helaly +2ASR EvaluationLLM-Assisted Annotation

    LemoniLab FZCO (lemonilab.com) · Innopolis University (innopolis.ru)

  24. TimeCues Studio: A Workspace for Music Annotation and Algorithm Prototyping

    Sep 9, 2026Sapir Caduri, Yoav GoldbergHuman-in-the-Loop AnnotationMusic Information Retrieval

    Bar-Ilan University Ramat Gan, Israel

  25. Decomposing LLM-Judge Uncertainty to Target Expert Labels

    Sep 6, 2026Ryan LailAnnotator DisagreementLLM-as-a-Judge

    Composo AI

  26. Correction as Annotation: Bootstrapping a Dependency Parser for Documentary Medieval Latin

    Sep 5, 2026Gabriel H. PizzornoDependency ParsingLow-Resource Language Processing

    Harvard University · Department of History | Robinson Hall, Office 210 | 35 Quincy Street, Cambridge, MA 02138

  27. Beyond Majority Vote: Multi-Perspective Adjudication for Medical Hallucination Detection

    Sep 3, 2026Joe Cecil, Marjorie FreedmanHallucination in RAGLLM-as-a-Judge

    Information Sciences Institute, University of Southern California

  28. DropClick: Semi-Automated One-Click Segmentation for Agricultural Robotic Data

    Sep 3, 2026Patrick Zimmer, Michael Halstead, Chris McCoolAgricultural Image AnalysisPromptable Image Segmentation

    University of Bonn, Germany · CSIRO, Australia

  29. The Differential Reasoning Router: Operationalizing Cost-Aware LLM Annotation in E-commerce

    Aug 31, 2026Cheng Lyu, Jingyue Zhang, Vinny DeGenova +2LLM-Assisted AnnotationHuman-in-the-Loop Annotation

    Wayfair

  30. QUORUM: QUality-Optimized Routing Using Multiple annotators

    Aug 28, 2026Antonio Purificato, Maria Sofia Bucarelli, Andrea Bacciu +2LLM-Assisted AnnotationLLM Routing

    Amazon · Sapienza University of Rome · Université Côte d’Azur, CNRS, Inria, I3S