LLM-Assisted Annotation

Ask about this topic

Overview

This field studies how large language models can help create labels and structured metadata for text, images, audio, video, and specialized records. Common approaches include prompt and codebook design, retrieval-augmented examples, model ensembles, confidence-based review, and training smaller classifiers on generated labels. These methods can reduce annotation time and cost, especially in low-resource or domain-specific settings, but their usefulness depends on careful validation against expert judgments and downstream tasks. A central challenge is that agreement and repeatability do not guarantee valid labels: results can shift with prompts, models, task definitions, and demographic cues, so human oversight and transparent error analysis remain important.

Latest papers

CardsList

202 papers21 papers in the last four weeks, down 16% on the four weeks before. 0.1% of all new papers.

  1. TableNet A Large-Scale Table Dataset with LLM-Powered Autonomous

    Feb 27, 2026Ruilin Zhang, Kai YangTable Structure RecognitionLLM-Assisted Annotation

    School of Computer Science and Technology, Tongji University, 4800 Cao’an Road, Shanghai, 201804, China

  2. Are Multimodal Large Language Models Good Annotators for Image Tagging?

    Feb 24, 2026Ming-Kun Xie, Jia-Hao Xiao, Zhiqiang Kou +3LLM-Assisted AnnotationMultimodal Large Language Models

    RIKEN Center for Advanced Intelligence Project, Japan · Southeast University, China · China University of Mining and Technology, China +1

  3. Can Large Language Models Replace Human Coders? Introducing ContentBench

    Feb 23, 2026Michael HamanLLM-Assisted AnnotationLLM Evaluation

    Department of Humanities, Faculty of Economics and Management, Czech University of Life Sciences Prague, Czech Republic

  4. AuditoryHuM: Auditory Scene Label Generation and Clustering using Human-MLLM Collaboration

    Feb 23, 2026Henry Zhong, Jörg M. Buchholz, Julian Maclaren +2ClusteringLLM-Assisted Annotation

    Australian Hearing Hub, Macquarie University, Sydney, Australia · Google Research Australia, Sydney, Australia

  5. "How Do I ...?": Procedural Questions Predominate Student-LLM Chatbot Conversations

    Feb 20, 2026Alexandra Neagu, Marcus Messer, Peter Johnson +1Generative AI in EducationEducational Dialogue Systems

    Imperial College London, London, United Kingdom

  6. Context-Aware Mapping of 2D Drawing Annotations to 3D CAD Features Using LLM-Assisted Reasoning for Manufacturing Automation

    Feb 20, 2026Muhammad Tayyab Khan, Lequn Chen, Wenhe Feng +1Industrial AutomationManufacturing

    Singapore Institute of Manufacturing Technology (SIMTech), A*STAR, Singapore 636732, Republic of Singapore · School of Mechanical and Aerospace Engineering, Nanyang Technological University, Singapore 639798 · Advanced Remanufacturing and Technology Centre (ARTC), A*STAR, Singapore 637143, Republic of Singapore

  7. Using LLMs for Knowledge Component-level Correctness Labeling in Open-ended Coding Problems

    Feb 19, 2026Zhangqi Duan, Arnav Kankaria, Dhruv Kartik +1Programming EducationUser Modeling

    University of Massachusetts Amherst

  8. Supercharging Agenda Setting Research: The ParlaCAP Dataset of 28 European Parliaments and a Scalable Multilingual LLM-Based Classification

    Feb 18, 2026Taja Kuzman Pungeršek, Peter Rupnik, Daniela Širinić +1Political Discourse AnalysisLLM-Assisted Annotation

    ∗Jožef Stefan Institute; ∗†‡Ljubljana, Slovenia · §Faculty of Political Science, University of Zagreb; §Zagreb, Croatia · †Faculty of Computer and Information Science, University of Ljubljana; ∗†‡Ljubljana, Slovenia +1

  9. Are LLMs Ready to Replace Bangla Annotators?

    Feb 18, 2026Md. Najib Hasan, Touseef Hasan, Souvika SarkarLLM-Assisted AnnotationLLM Reliability

    A2I Lab, School of Computing, Wichita State University

  10. Measuring Social Integration Through Participation: Categorizing Organizations and Leisure Activities in the Displaced Karelians Interview Archive using LLMs

    Feb 17, 2026Joonatan Laato, Veera Schroderus, Jenna Kanerva +3Computational Social ScienceLLM-Assisted Annotation

    TurkuNLP, University of Turku, Finland · Department of Biology, University of Turku, Finland · ELLIS Institute Finland

  11. BETA-Labeling for Multilingual Dataset Construction in Low-Resource IR

    Feb 16, 2026Md. Najib Hasan, Mst. Jannatun Ferdous Rain, Fyad Mohammed +1Low-Resource Language ProcessingLLM-Assisted Annotation

    Wichita State University · Begum Rokeya University, Rangpur · Khulna University of Engineering & Technology +1

  12. A Large-Scale Dataset for Molecular Structure-Language Description via a Rule-Regularized Method

    Feb 2, 2026Feiyang Cai, Guijuan He, Yi Hu +7LLM-Assisted Annotation

    Clemson University · Independent Researcher · University of Delaware +1

  13. Standardizing Longitudinal Radiology Report Evaluation via Large Language Model Annotation

    Jan 23, 2026Xinyi Wang, Grazziela Figueredo, Ruizhe Li +1Medical Report GenerationLLM-Assisted Annotation

    School of Computer Science, The University of Nottingham, Nottingham, NG7 2RD, United Kingdom. · School of Medicine, The University of Nottingham, Nottingham, NG7 2RD, United Kingdom.

  14. A large-scale pipeline for LLM-assisted corpus annotation: variation and change in the English consider construction

    Oct 14, 2025Cameron Morin, Matti Marttinen LarssonLLM-Assisted Annotation

    Université Paris-Cité · University of Gothenburg

  15. Interactive In-Meeting Speaker Correction with Human Feedback

    Sep 22, 2025Xinlu He, Yiwen Guan, Badrivishal Paurana +3LLM-Assisted AnnotationHuman-in-the-Loop Annotation

    Worcester Polytechnic Institute

  16. LLMs and their Limited Theory of Mind: Evaluating Mental State Annotations in Situated Dialogue

    Sep 2, 2025Katharine Kowalyshyn, Matthias ScheutzLLM EvaluationTheory of Mind

    Department of Computer Science Tufts University

  17. Introducing HALC: A general pipeline for the systematic and reliable construction of prompts for automated coding with LLMs in the computational social sciences

    Jul 29, 2025Andreas Reich, Claudia Thoms, Tobias SchrimpfLLM PromptingLLM-Assisted Annotation

  18. AnnoCaseLaw: Legal Annotation Benchmark of Civil Negligence

    Feb 28, 2025Rilton Franzone, Magnus Sesodia, Alina Petrova +7Legal NLPLLM-Assisted Annotation

    University of Oxford · Thomson Reuters Labs · Vienna University of Technology +1

  19. Harmful YouTube Video Detection: A Taxonomy of Online Harm and MLLMs as Alternative Annotators

    Nov 6, 2024Claire Wonjeong Jo, Miki Wesołowska, Magdalena WojcieszakVideo Content ModerationLLM-Assisted Annotation

    University of California, Davis · University of Warsaw

  20. Transferring Natural Language Datasets Between Languages Using Large Language Models for Modern Decision Support and Sci-Tech Analytical Systems

    Oct 17, 2024Dmitrii Popov, Egor Terentev, Danil Serenko +2LLM-Assisted AnnotationLow-Resource Language Processing

    Federal Research Center “Computer Science and Control” of the Russian Academy of Sciences (FRC CSC RAS), Moscow 119333, Russia · Faculty of Physics and Mathematics and Natural Sciences, RUDN University, Moscow 117198, Russia · Institute for Information Transmission Problems of the Russian Academy of Sciences (IITP RAS), Moscow 127051, Russia +1

  21. Which Demographics do LLMs Default to During Annotation?

    Oct 11, 2024Johannes Schäfer, Aidan Combs, Christopher Bagdon +9LLM PromptingLLM-Assisted Annotation

    Fundamentals of Natural Language Processing, University of Bamberg, Germany

  22. "You are an expert annotator": Automatic Best-Worst-Scaling Annotations for Emotion Intensity Modeling

    Mar 26, 2024Christopher Bagdon, Prathamesh Karmalker, Harsha Gurulingappa +1LLM-Assisted AnnotationEmotion Recognition

    Merck Data & AI Organization, Merck Group, Darmstadt, Germany · Institut für Maschinelle Sprachverarbeitung, University of Stuttgart, Germany · Fundamentals of Natural Language Processing, University of Bamberg, Germany +1