LLM-Assisted Annotation

Follow this topic Ask about this topic

This field studies how large language models can help create labels and structured metadata for text, images, audio, video, and specialized records. Common approaches include prompt and codebook design, retrieval-augmented examples, model ensembles, confidence-based review, and training smaller classifiers on generated labels. These methods can reduce annotation time and cost, especially in low-resource or domain-specific settings, but their usefulness depends on careful validation against expert judgments and downstream tasks. A central challenge is that agreement and repeatability do not guarantee valid labels: results can shift with prompts, models, task definitions, and demographic cues, so human oversight and transparent error analysis remain important.

Latest papers

202 papers21 papers in the last four weeks, down 16% on the four weeks before. 0.1% of all new papers.

High Scores, Hidden Failures: Evaluating Codebook-Based LLM Content Analysis
Oct 26, 2026Elena Senger, Yuri Campbell, Rob van der Goot +1LLM-Assisted Annotation LLM Reliability
MaiNLP, Center for Information and Language Processing, LMU Munich & MCML, Germany · Fraunhofer Institute for Systems and Innovation Research ISI, Germany · Department of Computer Science, IT University of Copenhagen, Denmark
From Presence Labels to Bounding Boxes: Can Multimodal Large Language Models Scale Surgical Instrument Localization?
Oct 25, 2026Mohamed Hamdy, Muraam Abdel-Ghani, Fatmaelzahraa Ahmed +5LLM-Assisted Annotation Medical Image Localization
Department of Computer Science and Engineering, Qatar University, Doha, Qatar · Hamad Medical Corporation, Doha, Qatar · College of Medicine, Qatar University, Doha, Qatar
Scaling Qualitative Analysis with Large Language Models: Development and Validation of the Smart Coder Framework for Human-LLM Collaborative Coding
Oct 23, 2026Cheng Tang, Kun Wang, Yaxuan Yang +4LLM-Assisted Annotation LLM Prompting
Department of Educational Psychology, University of Georgia, Athens, GA, USA · Rethink Learning Labs, Woodcliff Lake, NJ, USA · Department of Early Childhood/Elementary Education, Penn State Abington, Abington, PA, USA
Can LLM Annotators Match Human Consistency? Benchmarking Human Agreement and Optimizing Multi-Agent with Contrastive Knowledge
Oct 23, 2026Xiaopeng Ke, Zhixin Wang, Xuebo Liu +5LLM-Assisted Annotation Multi-Agent LLM Systems
ANT Group · Harbin Institute of Technology (Shenzhen)
GAIDA: Generative AI for Human-in-the-Loop Document Annotation
Oct 19, 2026Islam Mesabah, Ali Elhosary, Priyabanta Sandulu +2Human-in-the-Loop Annotation LLM-Assisted Annotation
German Research Center for Artificial Intelligence (DFKI) · RPTU Kaiserslautern-Landau, Kaiserslautern, Germany
SLA-LENS: From Classroom Transcripts to CSLA-Informed and Evidence-Linked L2 Interaction Analytics
Oct 16, 2026Nemika Tyagi, Olga Kellert, Phanos Kareen +1Second Language Acquisition LLM-Assisted Annotation
Arizona State University
Can This LLM Annotate My Data? Estimating Zero-Shot LLM Performance
Oct 16, 2026Tong Lin, Grant Schoenebeck, Sabina TomkinsLanguage Model Performance Prediction LLM-Assisted Annotation
School of Information, University of Michigan
CoDA: A Human-LM Collaborative Codebook Development Assistant for Domain-Specific Text Annotation
Oct 16, 2026Jaspreet Ranjit, Swaminathan Chellappa, Augusto Rivas Costante +5LLM-Assisted Annotation Human-in-the-Loop Annotation
Thomas Lord Dept. of Computer Science, University of Southern California · My Friend’s Place, Los Angeles, California · School of Social Work, University of North Carolina
Annotrace: A Workbench for Step-Level Annotation of Agent Traces
Oct 16, 2026Chenyi Huang, Arun Josephraj Arokiaraj, Bin Wu +1Human-in-the-Loop Annotation LLM-Assisted Annotation
Department of Computer Science, University College London, London, United Kingdom · Centre for Artificial Intelligence, University College London, United Kingdom
LLM-Based Weak Annotation for Cybersecurity Tweet Classification: A Teacher-Student Framework
Oct 15, 2026Ferry Afit Kurniawan, Kazushi Okamoto, Koki Karube +2LLM-Assisted Annotation Text Classification
Department of Informatics, Graduate School of Informatics and Engineering, The University of Electro-Communications 1-5-1 Chofugaoka, Chofu, Tokyo 182-8585, Japan
Towards Cost-Efficient Video Dataset Construction with Large Language Models
Oct 12, 2026Mohammed Shabot, Jonathan Adams, Jan Van den Stock +1Training Data Curation LLM-Assisted Annotation
AI Lab, Vrije Universiteit Brussel, Pleinlaan 2, 1050 Brussels, Belgium · Neuropsychiatry, KU Leuven, ON5/B Herestraat 49 - box 1029, 3000 Leuven, Belgium
Towards Autopilot Labeling for Short Video Content Governance
Oct 11, 2026Junwen Chen, Hongwei Wang, Zixuan Wang +12LLM-Assisted Annotation Video Content Moderation
TikTok Inc., San Jose, CA, USA
Studying Comics at Scale with Artificial Intelligence: The PAGE Framework and Three Studies of American Comics and Japanese Manga
Oct 9, 2026Emilio FerraraLLM-Assisted Annotation
Thomas Lord Department of Computer Science, University of Southern California, Los Angeles, CA, USA.
Large Language Models for Machine Translation Quality Annotation: Humans and Models Are Both Challenged
Oct 7, 2026Hala Almaghout, Christian Federmann, Qin GaoMT Evaluation LLM Reliability
Apple
Goldsmith: Gold-Loss-Guided Definition Optimization with an Agentic Annotation Harness
Oct 7, 2026Yihan Li, Hanyi Zhang, Xiaoxi Jiang +1LLM-Assisted Annotation Large Language Model-Guided Optimization
Sun Yat-sen University, Guangzhou, China · South China University of Technology, Guangzhou, China
Agreement Is Not Validity: Cross-Model LLM Consensus in Diagnosing Student Failure Modes in K-12 Math Tutoring Dialogue
Oct 6, 2026Clayton Cohn, Joyce Fonteles, Kirk Vanacore +5LLM Evaluation LLM-Assisted Annotation
College of Computing and Information Science, Cornell University, Ithaca, USA. · College of Connected Computing, Vanderbilt University, Nashville, USA. · Third Space Learning, Swindon, UK.
Verb-ICL: Rethinking In-Context Learning for Structured Prediction
Oct 3, 2026Fan Bai, Hengshuo Miao, Sanjit S Batra +3In-Context Learning Structured Prediction
Johns Hopkins University · Optum
SoftGene: Protein Language Model-Enhanced Soft Prompting for Interpretable Gene Set Annotation
Oct 2, 2026Drew Ross, Arya Hadizadeh Moghaddam, Dongjie Wang +2LLM-Assisted Annotation Genomics
Electrical Engineering and Computer Science, University of Kansas, USA · Cell Biology and Physiology, University of Kansas Medical Center, USA
Can LLMs Reliably Annotate Bioassay Metadata to Improve Data Readiness?
Oct 1, 2026Laura van Weesep, Riccardo Tedoldi, Jens Sjölund +6LLM Auditing LLM-Assisted Annotation
Molecular AI, Discovery Sciences R&D, AstraZeneca · Department of Information Technology, Uppsala University · Robotics, Perception & Learning, KTH Royal Institute of Technology +3
LLM-Assisted Discovery of Typed Semantic Links for Ontology Network Construction
Oct 1, 2026Nouha Hayouni, Sheeba Samuel, Alsayed AlgergawyRelation Extraction KG Construction
Chair of Data and Knowledge Engineering, University of Passau, Germany · Distributed and Self-organizing Systems, Chemnitz University of Technology, Germany
Paying for Too Many Tokens? Valid and Cost-Efficient Multimodal LLM Annotation with Simple Heuristics
Sep 30, 2026Zhixi Zhu, Kristina GligoricVLM Evaluation Video-Language Model Evaluation
Johns Hopkins University
QuanReview: Offline, Auditable Reconciliation of Human and LLM Span Annotations
Sep 28, 2026Matteo Musacchio, Juan Cruz Giner Pulero, Isabel Castañeda +4Annotator Disagreement LLM-Assisted Annotation
Universidad de San Andrés, Buenos Aires, Argentina · ISI Foundation, Turin, Italy · CONICET, Buenos Aires, Argentina +1
Beyond Keywords: Leveraging Generative LLMs and Label Aggregation to Classify Economic Policy Uncertainty in News Articles
Sep 25, 2026Paul TrustHierarchical Classification Text Classification
Modeling Student Sensemaking with LLMs and Knowledge-Graph-Guided Inference
Sep 25, 2026Özge Alacam, Zübeyde Demet Kirbulut Güneş, Funda Ekici +7Science Education Educational Technology
Center for Information and Language Processing, LMU München, Munich, Germany · Department of Chemistry Education, Gazi Faculty of Education, Gazi University, Ankara, Türkiye
EduBehaviors: Assertion-based Schemas for Auditable Coding of Educational Dialogues
Sep 22, 2026Julian Bernado, Ana Trindade Ribeiro, Xander Beberman +1LLM-Assisted Annotation AI in Education
SCALE Initiative, Stanford University
Experts Rise Where LLMs Disagree: Using Cross-Model Disagreement to Target Expert Effort in LLM Codebook Revision for Large-Scale Annotation
Sep 22, 2026Zeyu He, Zhuqian Zhou, Kirk Vanacore +2LLM-Assisted Annotation Human-in-the-Loop Annotation
The Pennsylvania State University, University Park, PA, USA · Cornell University, Ithaca, NY, USA
Reliable but Design-Sensitive: Instrument Uncertainty in LLM Annotation
Sep 22, 2026Thomas Reiter, Christoph Kern, Fedor Miasnikov +4LLM-Assisted Annotation LLM Reliability
Amazon · Ludwig Maximilian University of Munich, Germany · RTI International, North Carolina, USA +1
onPanda: Efficient Annotation of On-Policy Alignment Data for LLMs and Agents via Token-Level Correction
Sep 21, 2026Lei Yang, Mengyin Liu, Jia Wang +9LLM Alignment LLM-Assisted Annotation
StepFun · Xiamen University
Evaluating Decision Models for Text Annotation in Computational Social Science
Sep 21, 2026Hazem Ibrahim, Yasir ZakiLLM Evaluation LLM-Assisted Annotation
Computer Science, New York University Abu Dhabi, Abu Dhabi, UAE
QLoRA Fine-Tuning of Ministral LLM for Sequence-to-Function Protein Annotation
Sep 21, 2026Demian Pavlyshenko, Bohdan PavlyshenkoLLM Fine-Tuning LLM-Assisted Annotation

Oct 26, 2026J/K move · Enter open · S save

High Scores, Hidden Failures: Evaluating Codebook-Based LLM Content Analysis

Elena Senger, Yuri Campbell, Rob van der Goot, Barbara Plank

MaiNLP, Center for Information and Language Processing, LMU Munich & MCML, Germany · Fraunhofer Institute for Systems and Innovation Research ISI, Germany · Department of Computer Science, IT University of Copenhagen, Denmark

LLM-Assisted Annotation LLM Reliability Language Model Robustness

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly used for codebook-based content analysis in computational social science, where codebooks translate theoretical concepts into labeling schemes. While prior work reports strong predictive performance, it remains unclear whether these systems produce reliable and robust text annotations. We conduct a large-scale empirical failure-mode analysis of LLM-based coding pipelines across four datasets and five models by systematically evaluating their stability, robustness, and conceptual validity under controlled perturbations, domain shifts, and optimization procedures. We uncover multiple failure modes that are largely invisible under standard metrics, showing that high aggregate scores frequently coexist with severe label-level disparities, instability across runs, domain sensitivity, and shifts in label interpretation. Importantly, these failure modes persist under widely used optimization strategies and remain largely invisible under standard metrics such as accuracy and micro-F1. Our results demonstrate that current LLM-based coding pipelines cannot be treated as stable measurement tools without targeted validation beyond aggregate performance.