Human-in-the-Loop Annotation

Ask about this topic

Overview

Human-in-the-loop annotation studies how people and AI can work together to label data for training, evaluation, and analysis. Common approaches use active learning to select informative examples, model-generated drafts or propagated labels for people to correct, and uncertainty-based routing to send difficult cases to experts. Tools and workflows span text, images, video, audio, and clinical data, while measurement methods examine annotator differences, disagreement, and label quality. A central tension is how to reduce time and cost without obscuring errors, bias, or genuine differences in judgment; automation still needs careful validation, especially on subjective and high-stakes tasks.

Latest papers

CardsList

135 papers12 papers in the last four weeks, down 29% on the four weeks before. 0.1% of all new papers.

  1. Sandpiper: Orchestrated AI-Annotation for Educational Discourse at Scale

    Mar 9, 2026Daryl Hedley, Doug Pietrzak, Jorge Dias +9LLM-Assisted AnnotationHuman-in-the-Loop Annotation

    National Tutoring Observatory · FreshCognate · Cornell University +2

  2. Evaluating and Correcting Human Annotation Bias in Dynamic Micro-Expression Recognition

    Mar 5, 2026Feng Liu, Bingyu Nan, Xuezhong Qian +1Micro-Expression RecognitionFacial Expression Recognition

    School of Psychology, Shanghai Jiao Tong University, Shanghai 200030, China · School of Artificial Intelligence and Computer Science, Jiangnan University, Wuxi 214122, China

  3. LabelBuddy: An Open Source Music and Audio Language Annotation Tagging Tool Using AI Assistance

    Mar 4, 2026Ioannis Prokopiou, Ioannis Sina, Agisilaos Kounelis +2Human-in-the-Loop AnnotationMusic Information Retrieval

    Athens University of Economics and Business · Orfium · University of Patras +1

  4. AnnoABSA: A Web-Based Annotation Tool for Aspect-Based Sentiment Analysis with Retrieval-Augmented Suggestions

    Mar 2, 2026Nils Constantin Hellwig, Jakob Fehle, Udo Kruschwitz +1Aspect-Based Sentiment AnalysisLLM-Assisted Annotation

    Media Informatics Group, University of Regensburg, Regensburg, Germany · Information Science Group, University of Regensburg, Regensburg, Germany

  5. A Human-Centered Workflow for Using Large Language Models in Content Analysis

    Feb 27, 2026Ivan ZupicLLM-Assisted AnnotationHuman-in-the-Loop Annotation

    School of Creative Management Goldsmiths, University of London

  6. Interactive Medical-SAM2 GUI: A Napari-based semi-automatic annotation tool for medical images

    Feb 26, 2026Woojae Hong, Jong Ha Hwang, Jiyong Chung +3Interactive Medical Image SegmentationHuman-in-the-Loop Annotation

    Department of Biomechatronic Engineering, Sungkyunkwan University, Suwon, Gyeonggi, Republic of Korea · Department of Neurosurgery, Seoul National University Hospital, Seoul National University College of Medicine, Seoul, Republic of Korea

  7. Understanding Annotation Error Propagation and Learning an Adaptive Policy for Expert Intervention in Barrett's Video Segmentation

    Feb 25, 2026Lokesha Rasanjalee, Jin Lin Tan, Dileepa Pitawela +2Interactive Medical Image SegmentationHuman-in-the-Loop Annotation

    School of Computer Science and Information Technology, Adelaide University, Australia · The Australian Institute for Machine Learning (AIML), Australia · Adelaide Medical School, Faculty of Health and Medical Sciences, Adelaide University, Australia +1

  8. AuditoryHuM: Auditory Scene Label Generation and Clustering using Human-MLLM Collaboration

    Feb 23, 2026Henry Zhong, Jörg M. Buchholz, Julian Maclaren +2ClusteringLLM-Assisted Annotation

    Australian Hearing Hub, Macquarie University, Sydney, Australia · Google Research Australia, Sydney, Australia

  9. Beyond Labels: Information-Efficient Human-in-the-Loop Learning using Ranking and Selection Queries

    Feb 17, 2026Belén Martín-Urcelay, Yoonsang Lee, Matthieu R. Bloch +1Human-in-the-Loop AnnotationActive Learning

    School of Electrical and Computer Engineering, Georgia Institute of Technology, Atlanta, GA 30332 USA.

  10. CREMD: Crowd-Sourced Emotional Multimodal Dogs Dataset

    Feb 17, 2026Jinho Baek, Houwei Cao, Kate BlackwellEmotion RecognitionMultimodal Emotion Recognition

    dept. of Computer Science, New York Institute of Technology, New York, USA

  11. Active Learning with Imperfect Labels: Optimal Labeler Assignment and Sample Selection

    Dec 14, 2025Pouya Ahadi, Blair Winograd, Camille Zaug +3Human-in-the-Loop AnnotationActive Learning

    School of Industrial and Systems Engineering Georgia Institute of Technology Atlanta, GA 30332, USA · Ford Motor Company

  12. SiLVi: Simple Interface for Labeling Video Interactions

    Nov 5, 2025Ozan Kanbertay, Richard Vogg, Elif Karakoc +3Animal Behavior AnalysisHuman-in-the-Loop Annotation

    Institute of Computer Science and Campus Institute Data Science, University of Göttingen · Behavioral Ecology & Sociobiology Unit, German Primate Center, Göttingen, Germany · Department of Sociobiology/Anthropology, University of Göttingen, Göttingen, Germany

  13. Interactive In-Meeting Speaker Correction with Human Feedback

    Sep 22, 2025Xinlu He, Yiwen Guan, Badrivishal Paurana +3LLM-Assisted AnnotationHuman-in-the-Loop Annotation

    Worcester Polytechnic Institute

  14. The Re-Label Method For Data-Centric Machine Learning

    Feb 9, 2023Tong GuoLearning with Noisy LabelsHuman-in-the-Loop Annotation

  15. Learning From How Humans Correct

    Jan 30, 2021Tong GuoLearning with Noisy LabelsTraining Data Curation