Overview

This field studies how large language models can help create labels and structured metadata for text, images, audio, video, and specialized records. Common approaches include prompt and codebook design, retrieval-augmented examples, model ensembles, confidence-based review, and training smaller classifiers on generated labels. These methods can reduce annotation time and cost, especially in low-resource or domain-specific settings, but their usefulness depends on careful validation against expert judgments and downstream tasks. A central challenge is that agreement and repeatability do not guarantee valid labels: results can shift with prompts, models, task definitions, and demographic cues, so human oversight and transparent error analysis remain important.

Latest papers

CardsList

217 papers18 papers in the last four weeks, down 28% on the four weeks before. 0.1% of all new papers.

  1. Anytime-Valid Empirical Likelihood Inference from LLM Annotations with Adaptive Human Auditing

    Nov 11, 2026Hongfu Gao, Jiang Hu, Pengfei Liu +1Anytime-Valid InferenceLLM-Assisted Annotation

  2. Measuring Definition Sensitivity in LLM Annotation of Social Science Concepts

    Nov 9, 2026Bogdan Mamaev, Qizhou Wang, Christopher Leckie +1LLM-Assisted AnnotationPrompt Sensitivity

  3. Beyond Hard Labels: LLM Probabilities Improve Human-Assisted Regression

    Nov 8, 2026Xinyue Wang, Qunhua LiLLM-Assisted AnnotationSoft-Label Learning

  4. Definitions, Label Names, or Demonstrations? Auditing What Governs LLM Annotation

    Nov 6, 2026Viraj Mishra, Shubham Gupta, Dhruv Gupta +2LLM-Assisted AnnotationLLM Auditing

  5. When F1 Is Not Enough: Auditing Codebooks for LLM-Assisted Annotation

    Nov 6, 2026Yuehong Cassandra Tai, Qunhua LiLLM-Assisted AnnotationLLM Auditing

  6. Veterans' Mental Health Crisis Severity Annotation: Do LLMs Agree More Than Human Experts?

    Nov 5, 2026K M Sajjadul Islam, Pranav Yaramalla, Nadiyah Johnson +3LLM-Assisted AnnotationMental Health

  7. LLM-Assisted Label Noise Correction for EEG-Based Sleep Stage Classification

    Nov 5, 2026Elmira Salari, Shruti KshirsagarSleep Stage ClassificationNoisy-Label Learning

  8. ReviewMap: Mapping Evaluative Acts and Automation Boundaries in Peer-Review Comments

    Nov 5, 2026Kihyun Na, Injung KimScholarly Peer ReviewAutomated Peer Review

  9. Distilling Literary Sentiment from LLM Annotations into an Ordinal BERT Model

    Nov 2, 2026Artem Suslov, Takehiro HashimotoSentiment AnalysisKnowledge Distillation

  10. Taurus: LLM-in-the-Loop CT Annotation Matches Coding Agents at Half the Cost

    Oct 30, 2026Bing Hu, Rishabh SharmaAutomated Data AnnotationLLM-Assisted Annotation

  11. Well-Specified and Well-Tested? Assessing SWE-bench-Style Datasets with a Self-Evolving Agent

    Oct 29, 2026Zhonghao Jiang, Xuyang Zheng, Yanfu Yan +3LLM-Assisted AnnotationSoftware Engineering Benchmarks

  12. High Scores, Hidden Failures: Evaluating Codebook-Based LLM Content Analysis

    Oct 26, 2026Elena Senger, Yuri Campbell, Rob van der Goot +1LLM-Assisted AnnotationLLM Reliability

  13. From Presence Labels to Bounding Boxes: Can Multimodal Large Language Models Scale Surgical Instrument Localization?

    Oct 25, 2026Mohamed Hamdy, Muraam Abdel-Ghani, Fatmaelzahraa Ahmed +5LLM-Assisted AnnotationMedical Image Localization

  14. Scaling Qualitative Analysis with Large Language Models: Development and Validation of the Smart Coder Framework for Human-LLM Collaborative Coding

    Oct 23, 2026Cheng Tang, Kun Wang, Yaxuan Yang +4LLM-Assisted AnnotationLLM Prompting

  15. GAIDA: Generative AI for Human-in-the-Loop Document Annotation

    Oct 19, 2026Islam Mesabah, Ali Elhosary, Priyabanta Sandulu +2Automated Data AnnotationHuman-in-the-Loop Annotation

  16. Can This LLM Annotate My Data? Estimating Zero-Shot LLM Performance

    Oct 16, 2026Tong Lin, Grant Schoenebeck, Sabina TomkinsLanguage Model Performance PredictionLLM-Assisted Annotation

  17. CoDA: A Human-LM Collaborative Codebook Development Assistant for Domain-Specific Text Annotation

    Oct 16, 2026Jaspreet Ranjit, Swaminathan Chellappa, Augusto Rivas Costante +5LLM-Assisted AnnotationAutomated Data Annotation

  18. Annotrace: A Workbench for Step-Level Annotation of Agent Traces

    Oct 16, 2026Chenyi Huang, Arun Josephraj Arokiaraj, Bin Wu +1Human-in-the-Loop AnnotationLLM-Assisted Annotation

  19. LLM-Based Weak Annotation for Cybersecurity Tweet Classification: A Teacher-Student Framework

    Oct 15, 2026Ferry Afit Kurniawan, Kazushi Okamoto, Koki Karube +2LLM-Assisted AnnotationText Classification

  20. Towards Cost-Efficient Video Dataset Construction with Large Language Models

    Oct 12, 2026Mohammed Shabot, Jonathan Adams, Jan Van den Stock +1Training Data CurationLLM-Assisted Annotation

  21. Towards Autopilot Labeling for Short Video Content Governance

    Oct 11, 2026Junwen Chen, Hongwei Wang, Zixuan Wang +12LLM-Assisted AnnotationVideo Content Moderation

  22. Studying Comics at Scale with Artificial Intelligence: The PAGE Framework and Three Studies of American Comics and Japanese Manga

    Oct 9, 2026Emilio FerraraLLM-Assisted Annotation

  23. Goldsmith: Gold-Loss-Guided Definition Optimization with an Agentic Annotation Harness

    Oct 7, 2026Yihan Li, Hanyi Zhang, Xiaoxi Jiang +1Automated Data AnnotationLLM-Assisted Annotation

  24. Agreement Is Not Validity: Cross-Model LLM Consensus in Diagnosing Student Failure Modes in K-12 Math Tutoring Dialogue

    Oct 6, 2026Clayton Cohn, Joyce Fonteles, Kirk Vanacore +5LLM EvaluationLLM-Assisted Annotation

  25. Verb-ICL: Rethinking In-Context Learning for Structured Prediction

    Oct 3, 2026Fan Bai, Hengshuo Miao, Sanjit S Batra +3In-Context LearningStructured Prediction