Human-in-the-Loop Annotation

Ask about this topic

Momentum

13 papers in the last four weeks, up 117% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 95

All topics
CardsList

  1. Goldsmith: Gold-Loss-Guided Definition Optimization with an Agentic Annotation Harness

    Oct 7, 2026Yihan Li, Hanyi Zhang, Xiaoxi Jiang +1LLM-Assisted AnnotationLarge Language Model-Guided Optimization

    Sun Yat-sen University, Guangzhou, China · South China University of Technology, Guangzhou, China

  2. Uncertainty-Guided Handshake: Efficient Human-in-the-Loop Refinement for Surgical-Grade Glioma Segmentation

    Oct 1, 2026Samuel Hart, Ahmad Yahya, Ahmed Karam EldalyInteractive Medical Image SegmentationHuman-in-the-Loop Annotation

    Department of Computer Science, University of Exeter, Exeter, EX4 4QF, United Kingdom. · Department of Nuclear Engineering, Faculty of Engineering, King Abdulaziz University, Jeddah, Saudi Arabia. · UCL Hawkes Institute, University College London, Gower St., London, WC1E 6AE, United Kingdom.

  3. HaPRL: Human-Anchored Process Reinforcement Learning for Visual Search Agent

    Sep 29, 2026Zhangquan Chen, Yaoxin Niu, Xiang An +5Human-in-the-Loop AnnotationProcess Supervision

    Tsinghua University · Peng Cheng Laboratory · LMMs-Lab +2

  4. QuanReview: Offline, Auditable Reconciliation of Human and LLM Span Annotations

    Sep 28, 2026Matteo Musacchio, Juan Cruz Giner Pulero, Isabel Castañeda +4Annotator DisagreementLLM-Assisted Annotation

    Universidad de San Andrés, Buenos Aires, Argentina · ISI Foundation, Turin, Italy · CONICET, Buenos Aires, Argentina +1

  5. Human-in-the-Loop Geospatial Annotation for Rapid Dataset Construction in Field-Deployed UAV Systems

    Sep 23, 2026Morgan Masters, Adam Korycki, Nikolaas Bender +4Human-in-the-Loop AnnotationUncertainty Propagation

    Department of Electrical and Computer Engineering, University of California Santa Cruz

  6. Can Vision-Language Models Analyze Human-Centered Video? Mapping Model Capabilities and Human-AI Collaborative Workflows

    Sep 23, 2026Xiyuan Shen, Jiuyang Lyu, Seokhyun Hwang +4Video UnderstandingVideo-Language Model Evaluation

    University of Washington, USA · Georgia Institute of Technology, USA · Google DeepMind, USA

  7. Experts Rise Where LLMs Disagree: Using Cross-Model Disagreement to Target Expert Effort in LLM Codebook Revision for Large-Scale Annotation

    Sep 22, 2026Zeyu He, Zhuqian Zhou, Kirk Vanacore +2LLM-Assisted AnnotationHuman-in-the-Loop Annotation

    The Pennsylvania State University, University Park, PA, USA · Cornell University, Ithaca, NY, USA

  8. Optimal Sequential Annotations for Off-Policy Evaluation

    Sep 22, 2026Woojin Chae, Ezinne Nwankwo, Haitong Qin +1Off-Policy EvaluationLearning with Missing Data

    University of Southern California · University of California, Berkeley · University of Washington, Seattle

  9. MICRO: Multi-Fidelity Active Search for Severe Error Discovery

    Sep 22, 2026Orlando Leone, Niclas Pokel, Pehuén Moure +2Human-in-the-Loop AnnotationMT Evaluation

    Institute of Neuroinformatics, University of Zurich and ETH Zurich, Switzerland · ETH AI Center, Switzerland · Department of Computational Linguistics, University of Zurich, Switzerland

  10. mbariml: a curation pipeline for turning deep-sea imagery and video into object-detection training data

    Sep 21, 2026Lonny Lundsten, Kevin Barnard, Dave CaressTraining Data CurationHuman-in-the-Loop Annotation

    Monterey Bay Aquarium Research Institute

  11. onPanda: Efficient Annotation of On-Policy Alignment Data for LLMs and Agents via Token-Level Correction

    Sep 21, 2026Lei Yang, Mengyin Liu, Jia Wang +9LLM AlignmentLLM-Assisted Annotation

    StepFun · Xiamen University

  12. IMVS: Interactive Medical Volume Segmentation with Test-Time Adaptation - A New Method for Annotating Radiology Datasets

    Sep 15, 2026Abhilaksh Singh Reen, Kushal Borkar, Ritvik MahapatraInteractive Medical Image SegmentationTest-Time Adaptation

    Independent Researcher, Delhi, India · Independent Researcher, Kolkata, India · California State University, Fresno, CA, USA

  13. What Counts as a Mistake? Annotating Recitation Events in Quran Memorization Transcripts

    Sep 14, 2026Mohamad Al Mdfaa, Nursultan Askarbekuly, Ahmed Helaly +2ASR EvaluationLLM-Assisted Annotation

    LemoniLab FZCO (lemonilab.com) · Innopolis University (innopolis.ru)

  14. TimeCues Studio: A Workspace for Music Annotation and Algorithm Prototyping

    Sep 9, 2026Sapir Caduri, Yoav GoldbergHuman-in-the-Loop AnnotationMusic Information Retrieval

    Bar-Ilan University Ramat Gan, Israel

  15. Beyond Majority Vote: Multi-Perspective Adjudication for Medical Hallucination Detection

    Sep 3, 2026Joe Cecil, Marjorie FreedmanLLM-as-a-JudgeFactual Consistency Evaluation

    Information Sciences Institute, University of Southern California

  16. DropClick: Semi-Automated One-Click Segmentation for Agricultural Robotic Data

    Sep 3, 2026Patrick Zimmer, Michael Halstead, Chris McCoolAgricultural Image AnalysisPromptable Image Segmentation

    University of Bonn, Germany · CSIRO, Australia

  17. The Differential Reasoning Router: Operationalizing Cost-Aware LLM Annotation in E-commerce

    Aug 31, 2026Cheng Lyu, Jingyue Zhang, Vinny DeGenova +2LLM-Assisted AnnotationHuman-in-the-Loop Annotation

    Wayfair

  18. Attune: A Self-Annotation Tool for Understanding Robot Operator Attention Profiles

    Aug 12, 2026Puqi Zhou, Sungsoo Ray Hong, David PorfirioMulti-Robot SystemsHuman-in-the-Loop Annotation

    Department of Computer Science George Mason University Fairfax, VA, USA · Department of Information Sciences and Technology George Mason University Fairfax, VA, USA

  19. Personalized Scorer Modeling: A Learning-Based Framework for Deriving Robust Sleep Stage Labels from Multiple Experts

    Aug 12, 2026Seyyed Ali Hoseini, Javad Baseri, Hamid Saadatfar +2Sleep Stage ClassificationHuman-in-the-Loop Annotation

    Department of Electrical and Computer Engineering, University of Birjand, Birjand, South Khorasan, Iran

  20. Click2Poly: A VLM for vector mapping buildings and walls

    Aug 11, 2026Nicolas Girard, Jawher Ben Abdallah, Arno Gobbin +2Vision-Language ModelsBuilding Footprint Extraction

    LuxCarta

  21. H2: A Dual Hybrid Semantic Data Lake Architecture for Medical Data Harmonization with Human-In-the-Loop verified, LLM Driven Metadata Annotation System

    Aug 8, 2026Ioannis N. Tzortzis, Georgia Kapetadimitri, Agapi Davradou +6LLM-Assisted AnnotationHuman-in-the-Loop Annotation

    Institute of Communication and Computer Systems (ICCS), 15780 Zografou, Athens, Greece. · University of Macedonia (UOM), 54636, Thessaloniki, Greece.

  22. From Benchmark Performance to Tool Deployment: Human-in-the-Loop Anomaly Detection

    Aug 7, 2026Mike Szklarzewski, CJ George, Gavin Smithson +10Human-in-the-Loop AnnotationIndustrial Anomaly Detection

    Department of Computing Sciences, Coastal Carolina University, Conway, SC, USA · High Performance Computing Design, Los Alamos National Laboratory, Los Alamos, NM, USA · Academy of Arts Science and Technology, Horry County School District, Myrtle Beach, SC, USA +3

  23. ArtAnno: Annotating Implicit Semantics in Artworks through LLM Agent-Driven Bidirectional Human-AI Augmentation

    Aug 5, 2026Xiaoyan Gu, Yifang Wang, Wenqing Zheng +6Multi-Agent LLM SystemsLLM-Assisted Annotation

    School of Computer and Computing Science, Hangzhou City University Hangzhou, Zhejiang, China

  24. Oh Deer, How Should I Handle This? Seasonal Priors for Selective Wildlife Annotation and Classification

    Aug 3, 2026Hugo Markoff, Christoph Praschl, Anton Hjalte Jørgensen +5Selective PredictionHuman-in-the-Loop Annotation

    Aalborg University, Department of Chemistry and Bioscience, Aalborg, Denmark · University of Applied Sciences Upper Austria, Hagenberg, Austria · Massachusetts Institute of Technology, Cambridge, MA, USA

  25. Human-LLM Collaborative Inductive Coding for Conceptualizing K-12 Educator AI Use

    Jul 30, 2026Alex Liu, Min Sun, Lief Esbenshade +4LLM-Assisted AnnotationHuman-in-the-Loop Annotation

    College of Education, University of Washington, Seattle, WA, United States · Colleague AI

  26. Challenges in annotations by humans and LLMs: A case study of evaluative language

    Jul 30, 2026Mirela Imamovic, Aenne Cecilia Kristine Knierim, Khushi Pitroda +1LLM EvaluationLLM-Assisted Annotation

    University of Hildesheim, Institute of Translation Studies and Specialised Communication, Germany · University of Paris 8 Vincennes-Saint Denis, TransCrit, France · University of Hildesheim, Institute of Information Science and Language Technology, Germany

  27. Annotating Topical Legal Insights from Case Proceedings

    Jul 30, 2026Subinay Adhikary, Dwaipayan Roy, Debasis Ganguly +2Legal NLPHuman-in-the-Loop Annotation

    Indian Institute of Science Education and Research Kolkata, India · University of Glasgow, United Kingdom · West Bengal National University of Juridical Sciences, India

  28. Contrastive ESA: Human Evaluation of Multiple Translations at Once

    Jul 29, 2026Vilém Zouhar, Roman Grundkiewicz, Sara Rajaee +6Human-in-the-Loop AnnotationMT Evaluation

    ETH Zurich · Microsoft · University of Amsterdam +6

  29. Human-in-the-Loop Signature Bootstrapping for UAV Hyperspectral PFM-1 Mine Detection

    Jul 28, 2026Sagar Lekhak, Prasanna Reddy Pulakurthi, Emmett J. IentilucciHuman-in-the-Loop AnnotationHyperspectral Imaging

    Rochester Institute of Technology, Rochester, NY 14623, USA

  30. Plausibility-Driven Prioritization of Candidate Biomedical Annotations

    Jul 22, 2026Emanuele Cavalleri, Miad Alavinezhad, Dario Malchiodi +1Human-in-the-Loop AnnotationKnowledge Graphs

    Department of Computer Science, University of Milano, Milano, Italy

  31. Foundation-Assisted Active Learning for Object Detection Annotation

    Jul 18, 2026Jinchang Zhang, Arnold Zumbrun, Jing Lin +1Remote Sensing Image UnderstandingRemote Sensing Small Object Detection

    Intelligent Vision and Sensing (IVS) Lab at Indiana University Bloomington

  32. Self-Evolving Human-Centered Framework for Explainable Depression Symptom Annotation

    Jul 16, 2026Hoang-Loc Cao, Van Pham, Truong Thanh Hung Nguyen +4Explainable Artificial IntelligenceLLM-Assisted Annotation

    University of Science, VNU-HCM, Vietnam · University of New Brunswick, Canada · Friedrich-Alexander-Universität Erlangen-Nürnberg, Germany

  33. From Prediction to Collaboration: Interactive Symbolic Music Analysis

    Jul 15, 2026Emmanouil Karystinaios, Johannes Hentschel, Markus Neuwirth +1Human-in-the-Loop AnnotationMusic Information Retrieval

    Department of Computational Percepetion, Johannes Kepler University Linz, Austria · Anton Brukner University Linz, Austria

  34. Active Learning for Efficient Annotation of Surgical Videos with Weak Supervision

    Jul 14, 2026Manasa Dendukuri, Matjaz Jogan, Daniel A. Hashimoto +1Video Object SegmentationHuman-in-the-Loop Annotation

    1*GRASP Laboratory, University of Pennsylvania, 3330 Walnut Street, Philadelphia, PA, 19104, USA. · 2PCASO Laboratory, Dept. of Surgery, University of Pennsylvania, 3400 Spruce Street, Philadelphia, PA, 19104, USA. · 3Dept. of Computer and Information Science, University of Pennsylvania, 3330 Walnut Street, Philadelphia, PA, 19104, USA.

  35. Spotted: Location-informed Reidentification of Hyenas and Leopards in Camera Trap Surveys

    Jul 1, 2026Halil Sina Kelebek, Julia Hindel, Kobus Hoffman +9Animal Re-IdentificationHuman-in-the-Loop Annotation

    Oxford Robotics Institute, University of Oxford, Oxford, UK · Department of Computer Science, University of Freiburg, Germany · Bubye Valley Conservancy, Zimbabwe +3

  36. A Hybrid Framework for Song Lyric Annotation Based on Human-LLM Alignment

    Jun 28, 2026Rashini Liyanarachchi, Frank Tran, Md Mahmudul Hasan +2Annotator DisagreementLLM-Assisted Annotation

    School of Computer Science and Engineering, University of New South Wales, Sydney, Australia

  37. voxmap-studio: An open-source speaker diarization annotation tool with built-in cost instrumentation

    Jun 25, 2026Fumiaki YamaguchiHuman-in-the-Loop AnnotationSpeaker Diarization

    Independent Researcher

  38. Mask to Concept: Auto-Promptable SAM3 via Efficient Test-Time Concept Embedding Search for Few-Shot Annotation

    Jun 25, 2026Quan Zhou, Shaoqing Zhai, Qiang Hu +3Medical Imaging Foundation ModelsTest-Time Adaptation

    Wuhan University of Technology · Huazhong University of Science and Technology · Changzhou United lmaging Healthcare Surgical Technology Co. Ltd.

  39. Can Large Language Models Reliably Code Qualitative Humanitarian Data? A Benchmark Study Against Human Expert Adjudication

    Jun 25, 2026Jerome Marston, Tino Kreutzer, Salomé Garnier +3LLM EvaluationHuman-in-the-Loop Annotation

    Kobo Inc · McGill University · Harvard University

  40. Task Decomposition for Efficient Annotation

    Jun 23, 2026Nupoor Gandhi, Emma StrubellHuman-in-the-Loop AnnotationTask Decomposition

    Carnegie Mellon University

  41. Human and AI collaboration for pulmonary nodule segmentation

    Jun 21, 2026Hongqiao Dong, Wenhao Chi, Ruobing Liang +9Interactive Medical Image SegmentationMedical Imaging Foundation Models

    State Key Laboratory of Mathematical Sciences, Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing, China · School of Mathematical Sciences, University of Chinese Academy of Sciences, Beijing, China · Yau Mathematical Sciences Center, Tsinghua University, Beijing, China +5

  42. Speeding up the annotation process in semantic segmentation industrial applications

    Jun 18, 2026Marta Fernandez-Moreno, Margarita Guerrero, Rosalia Rementeria +2Human-in-the-Loop AnnotationSemantic Segmentation

    Department of Computer Science and Artificial Intelligence, Andalusian Research Institute in Data Science and Computational Intelligence, DaSCI, University of Granada, 18071, Granada, Spain · Ingénica STS, Av. Pablo Iglesias 38, Bj, 33205, Gijón, Spain · ArcelorMittal Global R&D, SLab—Steel Labs, Calle Marineros 4, 33490, Avilés, Spain +1

  43. A Clinician-Centered Pipeline for Annotation and Evaluation in Ultrasound AI Studies

    Jun 17, 2026Fangyijie Wang, Jianjun Yu, Wentao Shi +4Human-in-the-Loop AnnotationHuman-in-the-Loop Evaluation

    Research Ireland Centre for Research Training in Machine Learning · School of Medicine, University College Dublin, Dublin, Ireland · The Third People’s Hospital of Zhenjiang City, Zhenjiang, China +3

  44. From Bounding Boxes to Visual Reasoning: An On-Policy Data Annotation Tool for Vision-Language Models

    Jun 17, 2026Like Zhang, Runliang Niu, Shiqi Wang +5Human-in-the-Loop AnnotationVLM Reasoning

    School of Artificial Intelligence, Jilin University · College of Computer Science, Jilin University · OPPO

  45. ARGUSTRACK: A Multi-View Annotation System for Multi-Object Tracking

    Jun 14, 2026Hao Vo, Duc Nguyen, Ngan LeMulti-View ConsistencyHuman-in-the-Loop Annotation

    AICV Lab, University of Arkansas, USA

  46. Surprise-Guided MergeSort: Budget-Efficient Human-in-the-Loop Ranking via Adaptive Comparison Scheduling

    Jun 14, 2026Yujin Park, Haejun Chung, Ikbeom JangPairwise Preference LearningPairwise Comparison

    Hanyang University, Seoul, Republic of Korea · Hankuk University of Foreign Studies, Yongin, Republic of Korea

  47. Metric Match: A Subset Selection Approach to Evaluating LLM Judge Reliability

    Jun 12, 2026Alyssa Unell, Natalie Dullerud, Naomi Boneh +4LLM-as-a-JudgeLanguage Model Generation Evaluation

    Department of Computer Science Stanford University · Department of Medicine Stanford University

  48. A Resource for Enthymeme Detection in Controversial Political Discourse

    Jun 10, 2026Martial Pastor, Nelleke OostdijkAnnotator DisagreementHuman-in-the-Loop Annotation

    Centre for Language Studies, Radboud University, Nijmegen

  49. Measuring Human Value Expression in Social Media Texts: Calibrated LLM Annotation and Encoder Transfer

    Jun 9, 2026Maria Milkova, Maksim RudnevSoft-Label LearningLLM-Assisted Annotation

    Independent researcher, Lisbon, Portugal · University of Waterloo, ON, Canada

  50. iSAGE: A Human-in-the-Loop Framework for Remote Sensing Semantic Segmentation via Sparse Point Supervision

    Jun 8, 2026Osmar Luiz Ferreira de Carvalho, Osmar Abilio de Carvalho Junior, Anesmar Olino de Albuquerque +1Remote Sensing Image SegmentationHuman-in-the-Loop Annotation

  51. Deep Active Re-Labeling: Toward Noise-Resilient Annotation Efficiency

    Jun 7, 2026Md Abdullah Al Forhad, Weishi ShiHuman-in-the-Loop AnnotationActive Learning

    Active Machine Learning Lab · Learning · Language · Vision (LLaVi) Lab · Department of Computer Science and Engineering +1

  52. Knowledge Index of Noah's Ark

    Jun 3, 2026Sheng Jin, Minghao Liu, Yunze Xiao +24LLM EvaluationHuman-in-the-Loop Annotation

    12077AI · 2M-A-P · 3Carnegie Mellon University +7

  53. IMPose: Interactive Multi-person Pose Estimation with Dynamic Correction Propagation

    Jun 3, 2026Haoyang Ge, Jian Ma, Ziwen Wang +5Human-in-the-Loop AnnotationHuman Pose Estimation

    Tianjin University, Tianjin 300350, China · Tiandi, Tianjin, China · Zhongguancun Academy, Beijing 100094, China

  54. ACAT: A Collaborative Platform for Efficient Aspect-Based Sentiment Dataset Annotation

    Jun 2, 2026Ana-Maria Luisa Mocanu, Ciprian-Octavian Truica, Elena-Simona ApostolHuman-in-the-Loop AnnotationAspect-Based Sentiment Analysis

    1National University of Science and Technology POLITEHNICA Bucharest, Romania · 2Academy of Romanian Scientists, Ilfov 3, Bucharest, 050044, Romania

  55. Framing Migration News with LLMs: Structured CoT as a Support for Human Interpretation

    Jun 2, 2026David Alonso del Barrio, Jing Wen, Daniel Gatica-PerezLLM InterpretabilityCoT Reasoning

    Idiap Research Institute and EPFL Switzerland

  56. Fixing FOLIO and MALLS: Verified Annotations and an LLM-assisted Framework to Focus Human Relabeling

    Jun 1, 2026Andrea Brunello, Cristian Curaba, Luca Geatti +3LLM-Assisted AnnotationHuman-in-the-Loop Annotation

    University of Udine, Italy

  57. Who Annotates in NLP? A Large-scale Assessment of Human Annotation Reporting between 2018 and 2025

    Jun 1, 2026Maria Kunilovskaya, Gagan Bhatia, Lisa Sophie Albertelli +10Human-in-the-Loop Annotation

    NLLG Lab University of Technology Nuremberg · Interdisciplinary Transformation University, Austria

  58. Machine Learning for Coding Retail Product Names to Consumer-Price Categories: A Rule-plus-Bag-of-Words Pipeline with Reliability-Weighted Human-in-the-Loop Labeling

    Jun 1, 2026Vladimir BeskorovainyiHuman-in-the-Loop AnnotationText Classification

    Besk Tech · Moscow Institute of Physics and Technology (MIPT)

  59. Beyond Agreement: Scoring Panel-Surfaced Biomedical Entity Candidates for Curator Triage

    May 29, 2026Shuheng Cao, Ruiqi Chen, Renjie Cao +3Named Entity RecognitionLLM-Assisted Annotation

    University of California, San Diego · University of Michigan, Ann Arbor · The Hong Kong University of Science and Technology, Guangzhou +2

  60. Motion-guided sparse correction enables expert-quality point tracking across diverse microscopy regimes

    May 28, 2026Leonidas Zimianitis, Pasindu Thenahandi, Kai Buckhalter +10Point TrackingHuman-in-the-Loop Annotation

    Department of Computer Science, Old Dominion University, Norfolk, VA 23529, USA · Department of Biology, Massachusetts Institute of Technology, Cambridge, MA 02139, USA · The Picower Institute for Learning and Memory, Massachusetts Institute of Technology, Cambridge, MA 02139, USA +4