Overview

This field studies how large language models can help create labels and structured metadata for text, images, audio, video, and specialized records. Common approaches include prompt and codebook design, retrieval-augmented examples, model ensembles, confidence-based review, and training smaller classifiers on generated labels. These methods can reduce annotation time and cost, especially in low-resource or domain-specific settings, but their usefulness depends on careful validation against expert judgments and downstream tasks. A central challenge is that agreement and repeatability do not guarantee valid labels: results can shift with prompts, models, task definitions, and demographic cues, so human oversight and transparent error analysis remain important.

Latest papers

CardsList

217 papers18 papers in the last four weeks, down 28% on the four weeks before. 0.1% of all new papers.

  1. AnnoCaseLaw: Legal Annotation Benchmark of Civil Negligence

    Feb 28, 2025Rilton Franzone, Magnus Sesodia, Alina Petrova +7Legal NLPLLM-Assisted Annotation

  2. Harmful YouTube Video Detection: A Taxonomy of Online Harm and MLLMs as Alternative Annotators

    Nov 6, 2024Claire Wonjeong Jo, Miki Wesołowska, Magdalena WojcieszakVideo Content ModerationLLM-Assisted Annotation

  3. Transferring Natural Language Datasets Between Languages Using Large Language Models for Modern Decision Support and Sci-Tech Analytical Systems

    Oct 17, 2024Dmitrii Popov, Egor Terentev, Danil Serenko +2LLM-Assisted AnnotationLow-Resource Language Processing

  4. Which Demographics do LLMs Default to During Annotation?

    Oct 11, 2024Johannes Schäfer, Aidan Combs, Christopher Bagdon +9LLM PromptingLLM-Assisted Annotation

  5. "You are an expert annotator": Automatic Best-Worst-Scaling Annotations for Emotion Intensity Modeling

    Mar 26, 2024Christopher Bagdon, Prathamesh Karmalker, Harsha Gurulingappa +1LLM-Assisted AnnotationEmotion Recognition