Human-in-the-Loop Annotation

Overview

Human-in-the-loop annotation studies how people and AI can work together to label data for training, evaluation, and analysis. Common approaches use active learning to select informative examples, model-generated drafts or propagated labels for people to correct, and uncertainty-based routing to send difficult cases to experts. Tools and workflows span text, images, video, audio, and clinical data, while measurement methods examine annotator differences, disagreement, and label quality. A central tension is how to reduce time and cost without obscuring errors, bias, or genuine differences in judgment; automation still needs careful validation, especially on subjective and high-stakes tasks.

Latest papers

CardsList

149 papers14 papers in the last four weeks, down 18% on the four weeks before. 0.1% of all new papers.

  1. MedImgAnnotator: batch wise medical image annotation and preprocessing tool

    Nov 17, 2026Ewunate Assaye Kassaw, Amit Mehndiratta, Peush Sahni +1Automated Data AnnotationHuman-in-the-Loop Annotation

  2. Anytime-Valid Empirical Likelihood Inference from LLM Annotations with Adaptive Human Auditing

    Nov 11, 2026Hongfu Gao, Jiang Hu, Pengfei Liu +1Anytime-Valid InferenceLLM-Assisted Annotation

  3. What Factors Matter for Collection-Level Counting with Human-AI Collaboration?

    Nov 10, 2026Seoyun Jeong, Logan Lawrence, Owen L. Martin +3Object CountingHuman-AI Collaboration

  4. Beyond Hard Labels: LLM Probabilities Improve Human-Assisted Regression

    Nov 8, 2026Xinyue Wang, Qunhua LiLLM-Assisted AnnotationSoft-Label Learning

  5. TripleGrader: A Web-Based Platform Supporting Collaborative Triple Annotation with Quality Grading and Inter-Rater Agreement Assessment

    Nov 6, 2026Rajan Lamichhane, Songhui YueHuman-in-the-Loop AnnotationKnowledge Graphs

  6. ReviewMap: Mapping Evaluative Acts and Automation Boundaries in Peer-Review Comments

    Nov 5, 2026Kihyun Na, Injung KimScholarly Peer ReviewAutomated Peer Review

  7. TACT: A Pragmatic Approach to Qualitative Analysis with an AI Council

    Nov 3, 2026Rudrajit Choudhuri, Christian Bird, Carmen Badea +1Automated Data AnnotationHuman-in-the-Loop Annotation

  8. ARIDA - Active railway infrastructure dataset annotation using neural network predictions

    Nov 3, 2026Maximilian Dornik, Christian Sieberichs, Simon Geerkens +1Automated Data AnnotationHuman-in-the-Loop Annotation

  9. A Human-Verified AI Workflow for Annotating Environmental Sound Data

    Nov 2, 2026Michael Barasa Murunga, Allan VikiruAutomated Data AnnotationHuman-in-the-Loop Annotation

  10. JABER: Sequential Quality Control for Human-AI Data Annotation under Shared Bias

    Nov 2, 2026Fengxu Yang, Minghui Liwang, Yang Lu +3Human-in-the-Loop AnnotationBayesian Decision Theory

  11. Conformal Annotation with Randomized Expert Review: Application to Online Cell-Type Labeling

    Oct 29, 2026Xuhe Zhang, Zhi Huang, Ying JinHuman-in-the-Loop AnnotationConformal Prediction

  12. nnInteractive Agent: General-Purpose AI Agents as 3D Image Annotators

    Oct 22, 2026Fabian Isensee, Constantin Ulrich Harsy, Maximilian Rokuss +3Medical AI AgentsTool-Using Agents

  13. GAIDA: Generative AI for Human-in-the-Loop Document Annotation

    Oct 19, 2026Islam Mesabah, Ali Elhosary, Priyabanta Sandulu +2Automated Data AnnotationHuman-in-the-Loop Annotation

  14. CoDA: A Human-LM Collaborative Codebook Development Assistant for Domain-Specific Text Annotation

    Oct 16, 2026Jaspreet Ranjit, Swaminathan Chellappa, Augusto Rivas Costante +5LLM-Assisted AnnotationAutomated Data Annotation

  15. Annotrace: A Workbench for Step-Level Annotation of Agent Traces

    Oct 16, 2026Chenyi Huang, Arun Josephraj Arokiaraj, Bin Wu +1Human-in-the-Loop AnnotationLLM-Assisted Annotation

  16. Partial-Revelation Panel Forecasting: Predictive Information Gain, Allocation, and Coarsening-Consistent Operators

    Oct 16, 2026Bin Wen, Dai-Qiao Zhang, Tien-Ping TanHuman-in-the-Loop AnnotationConditional Distribution Estimation

  17. Activating the Annotator: Operationalising Lived Experience in Data Annotation

    Oct 8, 2026Sheza Munir, Ramaravind Kommiya Mothilal, Syed Ishtiaque AhmedHuman-in-the-Loop AnnotationAnnotator Disagreement

  18. Goldsmith: Gold-Loss-Guided Definition Optimization with an Agentic Annotation Harness

    Oct 7, 2026Yihan Li, Hanyi Zhang, Xiaoxi Jiang +1Automated Data AnnotationLLM-Assisted Annotation

  19. An automated pipeline for standardised speech-unit annotation in spontaneous dialogue

    Oct 2, 2026Hanlu He, Harald Vilhelm Skat-Rørdam, Ingvi Örnólfsson +1Speech ProcessingTurn-Taking in Dialogue

  20. Uncertainty-Guided Handshake: Efficient Human-in-the-Loop Refinement for Surgical-Grade Glioma Segmentation

    Oct 1, 2026Samuel Hart, Ahmad Yahya, Ahmed Karam EldalyInteractive Medical Image SegmentationHuman-in-the-Loop Annotation

  21. HaPRL: Human-Anchored Process Reinforcement Learning for Visual Search Agent

    Sep 29, 2026Zhangquan Chen, Yaoxin Niu, Xiang An +5Human-in-the-Loop AnnotationProcess Supervision

  22. QuanReview: Offline, Auditable Reconciliation of Human and LLM Span Annotations

    Sep 28, 2026Matteo Musacchio, Juan Cruz Giner Pulero, Isabel Castañeda +4Annotator DisagreementLLM-Assisted Annotation

  23. Human-in-the-Loop Geospatial Annotation for Rapid Dataset Construction in Field-Deployed UAV Systems

    Sep 23, 2026Morgan Masters, Adam Korycki, Nikolaas Bender +4Human-in-the-Loop AnnotationUncertainty Propagation

  24. Can Vision-Language Models Analyze Human-Centered Video? Mapping Model Capabilities and Human-AI Collaborative Workflows

    Sep 23, 2026Xiyuan Shen, Jiuyang Lyu, Seokhyun Hwang +4Video UnderstandingVideo-Language Model Evaluation

  25. Experts Rise Where LLMs Disagree: Using Cross-Model Disagreement to Target Expert Effort in LLM Codebook Revision for Large-Scale Annotation

    Sep 22, 2026Zeyu He, Zhuqian Zhou, Kirk Vanacore +2LLM-Assisted AnnotationHuman-in-the-Loop Annotation

  26. Optimal Expert Annotation for Evaluating Sequential Decision Policies from Text Records

    Sep 22, 2026Woojin Chae, Ezinne Nwankwo, Haitong Qin +1Off-Policy EvaluationLearning with Missing Data

  27. MICRO: Multi-Fidelity Active Search for Severe Error Discovery

    Sep 22, 2026Orlando Leone, Niclas Pokel, Pehuén Moure +2Human-in-the-Loop AnnotationMT Evaluation