LLM-Assisted Annotation

Ask about this topic

Overview

This field studies how large language models can help create labels and structured metadata for text, images, audio, video, and specialized records. Common approaches include prompt and codebook design, retrieval-augmented examples, model ensembles, confidence-based review, and training smaller classifiers on generated labels. These methods can reduce annotation time and cost, especially in low-resource or domain-specific settings, but their usefulness depends on careful validation against expert judgments and downstream tasks. A central challenge is that agreement and repeatability do not guarantee valid labels: results can shift with prompts, models, task definitions, and demographic cues, so human oversight and transparent error analysis remain important.

Latest papers

CardsList

202 papers21 papers in the last four weeks, down 16% on the four weeks before. 0.1% of all new papers.

  1. What Is Actually Being Annotated? Inter-Prompt Reliability as a Measurement Problem in LLM-Based Social Science Labeling

    Apr 2, 2026Jingyuan LiuLLM-Assisted AnnotationLLM Reliability

    Boston University

  2. More Human, More Efficient: Aligning Annotations with Quantized SLMs

    Apr 1, 2026Jiayu Wang, Junyoung LeeLLM-Assisted AnnotationLLM Fine-Tuning

    Home Team Science and Technology Agency

  3. Magic Words or Methodical Work? Challenging Conventional Wisdom in LLM-Based Political Text Annotation

    Mar 27, 2026Lorcan McLaren, James Cross, Zuzanna Krakowska +2LLM-Assisted AnnotationLLM Evaluation

    University College Dublin · University of Groningen

  4. When Hate Meets Facts: LLMs-in-the-Loop for Check-worthiness Detection in Hate Speech

    Mar 26, 2026Nicolás Benjamín Ocampo, Tommaso Caselli, Davide CeolinLLM-Assisted AnnotationHate Speech Detection

    Centrum Wiskunde & Informatica · Amsterdam, The Netherlands · University of Groningen +1

  5. LLMAR: A Tuning-Free Recommendation Framework for Sparse and Text-Rich Industrial Domains

    Mar 25, 2026Ryogo Hishikawa, Ichiro Kataoka, Shinya YudaLLM-Assisted AnnotationRecommender Systems

    Hitachi, Ltd. Research and Development Group Hitachi, Ibaraki, Japan · Hitachi Power Solutions Co., Ltd. Hitachi, Ibaraki, Japan

  6. From Content to Audience: A Multimodal Annotation Framework for Broadcast Television Analytics

    Mar 24, 2026Paolo Cupini, Francesco PierriLLM-Assisted AnnotationMultimodal Large Language Models

    Politecnico di Milano

  7. Hierarchical Retrieval Augmented Generation for Adversarial Technique Annotation in Cyber Threat Intelligence Text

    Mar 24, 2026Filippo Morbiato, Markus Keller, Priya Nair +1Retrieval-Augmented GenerationCyber Threat Intelligence

    University of Padua Italy

  8. Multi-Perspective LLM Annotations for Valid Analyses in Subjective Tasks

    Mar 22, 2026Navya Mehrotra, Adam Visokay, Kristina GligorićAnnotator DisagreementLLM-Assisted Annotation

    University of Washington

  9. ReasonScaffold: A Scaffolded Reasoning-based Annotation Protocol for Human-AI Co-Annotation

    Mar 22, 2026Smitha Muthya Sudheendra, Jaideep SrivastavaLLM-Assisted AnnotationHuman-in-the-Loop Annotation

    University of Minnesota, Twin Cities

  10. From Instructions to Assistance: a Dataset Aligning Instruction Manuals with Assembly Videos for Evaluating Multimodal LLMs

    Mar 20, 2026Federico Toschi, Nicolò Brunello, Andrea Sassella +2Multimodal Large Language ModelsMultimodal Model Evaluation

    DEIB, Politecnico di Milano, Via Ponzio 34/5, 20133, Milano (MI), Italy · KASTEL, Karlsruhe Institute of Technology, Am Fasanengarten, 576131, Karlsruhe, Germany

  11. SignAgent: Agentic LLMs for Linguistically-Grounded Sign Language Annotation and Dataset Curation

    Mar 19, 2026Oliver Cory, Ozge Mercanoglu Sincan, Richard BowdenLLM-Assisted AnnotationSign Language Datasets

    Centre for Vision, Speech and Signal Processing, University of Surrey, Guildford, UK

  12. RadAnnotate: Large Language Models for Efficient and Reliable Radiology Report Annotation

    Mar 16, 2026Saisha Pradeep Shetty, Roger Eric Goldman, Vladimir FilkovLLM-Assisted AnnotationHuman-in-the-Loop Annotation

    Department of Computer Science, University of California, Davis, CA, USA · Department of Radiology, University of California, Davis, CA, USA

  13. AgriChat: A Multimodal Large Language Model for Agriculture Image Understanding

    Mar 14, 2026Abderrahmene Boudiaf, Irfan Hussain, Sajid JavedAgricultural Image AnalysisVision-Language Models

    Department of Computer Science, Khalifa University of Science and Technology, Abu Dhabi, United Arab Emirates

  14. Large Language Models Reproduce Racial Stereotypes When Used for Text Annotation

    Mar 14, 2026Petter TörnbergSocial Bias in Language ModelsDemographic Bias in Language Models

    Institute for Logic, Language and Computation (ILLC), University of Amsterdam„ Science Park 107, 1098 XG Amsterdam, Netherlands

  15. Is Human Annotation Necessary? Iterative MBR Distillation for Error Span Detection in Machine Translation

    Mar 13, 2026Boxuan Lyu, Haiyue Song, Zhi QuMBR DecodingLLM-Assisted Annotation

    Institute of Science Tokyo · National Institute of Information and Communications Technology, Japan

  16. ZTab: Domain-based Zero-shot Annotation for Table Columns

    Mar 12, 2026Ehsan Hoseinzade, Ke WangZero-Shot LearningLLM-Assisted Annotation

    School of Computing Science, Simon Fraser University, Burnaby, Canada

  17. Interpretable Chinese Metaphor Identification via LLM-Assisted MIPVU Rule Script Generation: A Comparative Protocol Study

    Mar 11, 2026Weihang Huang, Mengna LiuFigurative Language UnderstandingInterpretable ML

    University of Birmingham Birmingham, UK · Guangdong University of Foreign Studies Guangzhou, China

  18. Large Language Models as Annotators for Machine Translation Quality Estimation

    Mar 11, 2026Sidi Wang, Sophie Arnoult, Amir KamranMT EvaluationLLM-Assisted Annotation

    Maastricht university · Vrije Universiteit Amsterdam · Taus

  19. Quantifying and extending the coverage of spatial categorization data sets

    Mar 10, 2026Wanchun Li, Alexandra Carstensen, Yang Xu +2Spatial Reasoning BenchmarksLLM-Assisted Annotation

    Faculty of Engineering and Information Technology, The University of Melbourne · Psychology, Arizona State University · Department of Computer Science, Cognitive Science Program, University of Toronto +2

  20. Automated Thematic Analysis for Clinical Qualitative Data: Iterative Codebook Refinement with Full Provenance

    Mar 9, 2026Seungjun Yi, Joakim Nguyen, Huimin Xu +9LLM-Assisted AnnotationIterative Refinement

    Department of Biomedical Engineering, University of Texas at Austin, Austin, TX, USA · Department of Computer Science, University of Texas at Austin, Austin, TX, USA · School of Information, University of Texas at Austin, Austin, TX, USA +6

  21. Sandpiper: Orchestrated AI-Annotation for Educational Discourse at Scale

    Mar 9, 2026Daryl Hedley, Doug Pietrzak, Jorge Dias +9LLM-Assisted AnnotationHuman-in-the-Loop Annotation

    National Tutoring Observatory · FreshCognate · Cornell University +2

  22. LAMUS: A Large-Scale Corpus for Legal Argument Mining from U.S. Caselaw using LLMs

    Mar 9, 2026Serene Wang, Lavanya Pobbathi, Haihua ChenLegal NLPArgument Mining

    University of North Texas Denton, Texas, USA

  23. Optimizing LLM Annotation of Classroom Discourse through Multi-Agent Orchestration

    Mar 8, 2026Bakhtawar Ahtisham, Kirk Vanacore, Rene F. KizilcecLLM-Assisted AnnotationAI in Education

    Cornell University, USA

  24. Wisdom of the AI Crowd (AI-CROWD) for Ground Truth Approximation in Content Analysis: A Research Protocol & Validation Using Eleven Large Language Models

    Mar 6, 2026Luis de-Marcos, Manuel Goyanes, Adrián Domínguez-DíazLLM-Assisted AnnotationLanguage Model Ensembles

    Computer Science Department, Polytechnic School, University of Alcalá. Madrid, Spain. · MITCOM Research Group, Department of Communication, Carlos III University of Madrid, Spain

  25. Building an Ensemble LLM Semantic Tagger for UN Security Council Resolutions

    Mar 6, 2026Hussein GhalyLLM-Assisted AnnotationLanguage Model Ensembles

    Independent Researcher, Rego Park, New York

  26. When Weak LLMs Speak with Confidence, Preference Alignment Gets Stronger

    Mar 5, 2026Amirabbas Afzali, Myeongho Jeon, Maria BrbicPreference AlignmentPreference Optimization

    EPFL · Sharif University of Technology

  27. LLM-as-an-Annotator: Training Lightweight Models with LLM-Annotated Examples for Aspect Sentiment Tuple Prediction

    Mar 2, 2026Nils Constantin Hellwig, Jakob Fehle, Udo Kruschwitz +1Aspect-Based Sentiment AnalysisLLM-Assisted Annotation

    Media Informatics Group, University of Regensburg, Regensburg, Germany · Information Science Group, University of Regensburg, Regensburg, Germany

  28. AnnoABSA: A Web-Based Annotation Tool for Aspect-Based Sentiment Analysis with Retrieval-Augmented Suggestions

    Mar 2, 2026Nils Constantin Hellwig, Jakob Fehle, Udo Kruschwitz +1Aspect-Based Sentiment AnalysisLLM-Assisted Annotation

    Media Informatics Group, University of Regensburg, Regensburg, Germany · Information Science Group, University of Regensburg, Regensburg, Germany

  29. Hybrid Neural-LLM Pipeline for Morphological Glossing in Endangered Language Documentation: A Case Study of Jungar Tuvan

    Mar 1, 2026Siyu Liang, Talant Mawkanuli, Gina-Anne LevowLLM-Assisted AnnotationLow-Resource Language Processing

    Department of Linguistics, University of Washington · Department of Middle Eastern Languages and Cultures, University of Washington

  30. A Human-Centered Workflow for Using Large Language Models in Content Analysis

    Feb 27, 2026Ivan ZupicLLM-Assisted AnnotationHuman-in-the-Loop Annotation

    School of Creative Management Goldsmiths, University of London