Human-in-the-Loop Annotation

Follow this topic Ask about this topic

Momentum

13 papers in the last four weeks, up 117% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 95

All topics

Multi-Perspective LLM Annotations for Valid Analyses in Subjective Tasks
Mar 22, 2026Navya Mehrotra, Adam Visokay, Kristina GligorićAnnotator Disagreement LLM-Assisted Annotation
University of Washington
AuditoryHuM: Auditory Scene Label Generation and Clustering using Human-MLLM Collaboration
Feb 23, 2026Henry Zhong, Jörg M. Buchholz, Julian Maclaren +2Clustering LLM-Assisted Annotation
Australian Hearing Hub, Macquarie University, Sydney, Australia · Google Research Australia, Sydney, Australia
Active Learning with Imperfect Labels: Optimal Labeler Assignment and Sample Selection
Dec 14, 2025Pouya Ahadi, Blair Winograd, Camille Zaug +3Human-in-the-Loop Annotation Active Learning
School of Industrial and Systems Engineering Georgia Institute of Technology Atlanta, GA 30332, USA · Ford Motor Company
Interactive In-Meeting Speaker Correction with Human Feedback
Sep 22, 2025Xinlu He, Yiwen Guan, Badrivishal Paurana +3LLM-Assisted Annotation Human-in-the-Loop Annotation
Worcester Polytechnic Institute
Decomposing LLM-Judge Uncertainty to Target Expert Labels
Date pendingRyan LailAnnotator Disagreement LLM-as-a-Judge
Composo AI

Mar 22, 2026, cs.CLJ/K move · Enter open · S save

Multi-Perspective LLM Annotations for Valid Analyses in Subjective Tasks

Navya Mehrotra, Adam Visokay, Kristina Gligorić

University of Washington

Annotator Disagreement LLM-Assisted Annotation Human-in-the-Loop Annotation Active Learning Demographic Bias in Language Models

Open paper AskPDF arXiv

Large language models are increasingly used to annotate texts, but their outputs reflect some human perspectives better than others. Existing methods for correcting LLM annotation error assume a single ground truth. However, this assumption fails in subjective tasks where disagreement across demographic groups is meaningful. Here we introduce Perspective-Driven Inference, a method that treats the distribution of annotations across groups as the quantity of interest, and estimates it using a small human annotation budget. We contribute an adaptive sampling strategy that concentrates human annotation effort on groups where LLM proxies are least accurate. We evaluate on politeness and offensiveness rating tasks, showing targeted improvements for harder-to-model demographic groups relative to uniform sampling baselines, while maintaining coverage.

Momentum

Related topics

Latest papers 95

Multi-Perspective LLM Annotations for Valid Analyses in Subjective Tasks

AuditoryHuM: Auditory Scene Label Generation and Clustering using Human-MLLM Collaboration

Active Learning with Imperfect Labels: Optimal Labeler Assignment and Sample Selection

Interactive In-Meeting Speaker Correction with Human Feedback

Decomposing LLM-Judge Uncertainty to Target Expert Labels