LLM-Assisted Annotation

Follow this topic Ask about this topic

Momentum

17 papers in the last four weeks, up 143% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 125

All topics

Large Language Models for Machine Translation Quality Annotation: Humans and Models Are Both Challenged
Oct 7, 2026Hala Almaghout, Christian Federmann, Qin GaoMT Evaluation LLM Reliability
Apple
Goldsmith: Gold-Loss-Guided Definition Optimization with an Agentic Annotation Harness
Oct 7, 2026Yihan Li, Hanyi Zhang, Xiaoxi Jiang +1LLM-Assisted Annotation Large Language Model-Guided Optimization
Sun Yat-sen University, Guangzhou, China · South China University of Technology, Guangzhou, China
Agreement Is Not Validity: Cross-Model LLM Consensus in Diagnosing Student Failure Modes in K-12 Math Tutoring Dialogue
Oct 6, 2026Clayton Cohn, Joyce Fonteles, Kirk Vanacore +5LLM Evaluation LLM-Assisted Annotation
College of Computing and Information Science, Cornell University, Ithaca, USA. · College of Connected Computing, Vanderbilt University, Nashville, USA. · Third Space Learning, Swindon, UK.
Can LLMs Reliably Annotate Bioassay Metadata to Improve Data Readiness?
Oct 1, 2026Laura van Weesep, Riccardo Tedoldi, Jens Sjölund +6LLM Auditing LLM-Assisted Annotation
Molecular AI, Discovery Sciences R&D, AstraZeneca · Department of Information Technology, Uppsala University · Robotics, Perception & Learning, KTH Royal Institute of Technology +3
LLM-Assisted Discovery of Typed Semantic Links for Ontology Network Construction
Oct 1, 2026Nouha Hayouni, Sheeba Samuel, Alsayed AlgergawyRelation Extraction KG Construction
Chair of Data and Knowledge Engineering, University of Passau, Germany · Distributed and Self-organizing Systems, Chemnitz University of Technology, Germany
Paying for Too Many Tokens? Valid and Cost-Efficient Multimodal LLM Annotation with Simple Heuristics
Sep 30, 2026Zhixi Zhu, Kristina GligoricVLM Evaluation Video-Language Model Evaluation
Johns Hopkins University
QuanReview: Offline, Auditable Reconciliation of Human and LLM Span Annotations
Sep 28, 2026Matteo Musacchio, Juan Cruz Giner Pulero, Isabel Castañeda +4Annotator Disagreement LLM-Assisted Annotation
Universidad de San Andrés, Buenos Aires, Argentina · ISI Foundation, Turin, Italy · CONICET, Buenos Aires, Argentina +1
EduBehaviors: Assertion-based Schemas for Auditable Coding of Educational Dialogues
Sep 22, 2026Julian Bernado, Ana Trindade Ribeiro, Xander Beberman +1LLM-Assisted Annotation AI in Education
SCALE Initiative, Stanford University
Experts Rise Where LLMs Disagree: Using Cross-Model Disagreement to Target Expert Effort in LLM Codebook Revision for Large-Scale Annotation
Sep 22, 2026Zeyu He, Zhuqian Zhou, Kirk Vanacore +2LLM-Assisted Annotation Human-in-the-Loop Annotation
The Pennsylvania State University, University Park, PA, USA · Cornell University, Ithaca, NY, USA
onPanda: Efficient Annotation of On-Policy Alignment Data for LLMs and Agents via Token-Level Correction
Sep 21, 2026Lei Yang, Mengyin Liu, Jia Wang +9LLM Alignment LLM-Assisted Annotation
StepFun · Xiamen University
Evaluating Decision Models for Text Annotation in Computational Social Science
Sep 21, 2026Hazem Ibrahim, Yasir ZakiLLM Evaluation LLM-Assisted Annotation
Computer Science, New York University Abu Dhabi, Abu Dhabi, UAE
QLoRA Fine-Tuning of Ministral LLM for Sequence-to-Function Protein Annotation
Sep 21, 2026Demian Pavlyshenko, Bohdan PavlyshenkoLLM Fine-Tuning LLM-Assisted Annotation
Beyond Relevance: Structured Semantic Supervision for Product Search with LLM-Augmented Annotations
Sep 20, 2026Girish A. Koushik, Swapnil Bhosale, Samarth Agrawal +4Learning to Rank LLM-Assisted Annotation
Computer Science, University of Surrey, Guildford, United Kingdom · People-Centred AI, University of Surrey, Guildford, United Kingdom · eBay USA +2
Reproducibility is not construct validity: LLM measurement of institutionally situated communication
Sep 17, 2026Veronika Batzdorfer, Carlo Romano Marcello Alessandro SantagiustinaLLM Evaluation LLM-Assisted Annotation
KIT · Department of Sociology and Computational Sociology, Karlsruhe Institute of Technology (KIT), Douglasstr. 24, 76133 Karlsruhe, Germany · ALMAnaCH, médialab, Sciences Po +2
CliniCIRCA: A Modular LLM Framework for Constructing Longitudinal Mental Health Patient Journeys from Raw EHR Narratives
Sep 17, 2026Aiwei Ivy Zhang, Nimra Ishfaq, Mohit Chandra +5Temporal Reasoning in Language Models Mental Health
Georgia Institute of Technology · University of Texas at Austin · Northwell Health
What Counts as a Mistake? Annotating Recitation Events in Quran Memorization Transcripts
Sep 14, 2026Mohamad Al Mdfaa, Nursultan Askarbekuly, Ahmed Helaly +2ASR Evaluation LLM-Assisted Annotation
LemoniLab FZCO (lemonilab.com) · Innopolis University (innopolis.ru)
Extracting Dataset Mentions in Forced Displacement and FCV Documents: A Weakly Supervised Framework with LLM-Based Label Refinement
Sep 14, 2026Rafael Macalaba, Aivin V. Solatorio, Patrick Michael Brock +1LLM-Assisted Annotation Document Information Extraction
Development Data Group Office of the World Bank Group Chief Statistician The World Bank 1818 H Street N.W., Washington, 20433 District of Columbia, USA · World Bank UNHCR Joint Data Center UN City Marmorvej 51, 2100 Copenhagen, Denmark
Debate-to-Skill: Capability-Bound Process Supervision for Industrial Query-to-Agent Annotation
Sep 12, 2026Shiyu Zhang, Leisheng Cheng, Huifu LiLLM-Assisted Annotation Process Supervision
Baidu, Inc.
Cross-Lingual Clinical Annotation Projection as Constrained Text Generation: A Six-Language Study
Sep 12, 2026Álvaro Rey-Blanes, Francisco J. Moreno-Barea, Francisco J. VeredasLLM-Assisted Annotation Controllable Text Generation
Department of Programming Languages and Computer Sciences, Universidad de Málaga, Málaga, Spain · Research Institute of Multilingual Language Technologies, Universidad de Málaga, Málaga, Spain · IBIMA Plataforma BIONAND, Instituto de Investigación Biomédica de Málaga, Málaga, Spain
SpeechAnnotator: A Context-Aware Multi-Agent Framework and Benchmark for Multidimensional Speech Annotation
Sep 9, 2026Qirui Zhan, Shuiyuan Wang, Jingbin Hu +10LLM-Assisted Annotation Multi-Agent Systems
Northwestern Polytechnical University, China · yutuzhineng, China · Shenzhen Pimei Technology Co., Ltd., Guangdong, China +1
PersianAnonymizer: Evaluating LLM-Labeled Training for Efficient NER-based Anonymization in Persian
Sep 1, 2026Mohammad Hossein Shalchian, Mostafa Amiri, Amir Mahdi SadeghzadehNamed Entity Recognition LLM-Assisted Annotation
1Sharif University of Technology · University of Tehran
Bridging Lexical Divergence: LLM-Assisted, Cost-Efficient, Zero-shot Scientific Entity Linking
Aug 31, 2026Md Rasel Khondokar, Qiao Qiao, Farjana Sultana Samia +3LLM-Assisted Annotation
Department of Computer Science, Iowa State University, Ames, Iowa, USA
Error-Type-Aware Loss Reweighting for Robust Named Entity Recognition with Noisy LLM Labels
Aug 31, 2026Elena Merdjanovska, Jonas Golde, Alan AkbikNamed Entity Recognition LLM-Assisted Annotation
Humboldt-Universität zu Berlin · Science of Intelligence
Designing an Auditable LLM-Supported Workflow for Qualitative Thematic Analysis
Aug 31, 2026Nadia Jul Jeldtoft, Tariq YousefLLM-Assisted Annotation
Department of Culture and Language 2Department of Mathematics and Computer Science University of Southern Denmark
The Differential Reasoning Router: Operationalizing Cost-Aware LLM Annotation in E-commerce
Aug 31, 2026Cheng Lyu, Jingyue Zhang, Vinny DeGenova +2LLM-Assisted Annotation Human-in-the-Loop Annotation
Wayfair
Homo-RAG: Homology-Guided Retrieval-Augmented Generation for Cross-Species Gene Function Prediction
Aug 26, 2026Azrin SultanaRetrieval-Augmented Generation LLM-Assisted Annotation
Department of Computer Science, American International University-Bangladesh, 408/1, Kuratoli, Khilkhet, 1229, Dhaka, Bangladesh.
Debiased Inference for AI-Generated Data without Gold-Standard Labels: Identification via Multiple Imperfect Measurements
Aug 18, 2026Naoki Egami, Sooahn ShinSemiparametric Inference Latent Variable Models
Department of Political Science and Statistics and Data Science Center, Massachusetts Institute of Technology, Cambridge, MA 02139 · Department of Political Science, Massachusetts Institute of Technology, Cambridge, MA 02139
H2: A Dual Hybrid Semantic Data Lake Architecture for Medical Data Harmonization with Human-In-the-Loop verified, LLM Driven Metadata Annotation System
Aug 8, 2026Ioannis N. Tzortzis, Georgia Kapetadimitri, Agapi Davradou +6LLM-Assisted Annotation Human-in-the-Loop Annotation
Institute of Communication and Computer Systems (ICCS), 15780 Zografou, Athens, Greece. · University of Macedonia (UOM), 54636, Thessaloniki, Greece.
ArtAnno: Annotating Implicit Semantics in Artworks through LLM Agent-Driven Bidirectional Human-AI Augmentation
Aug 5, 2026Xiaoyan Gu, Yifang Wang, Wenqing Zheng +6Multi-Agent LLM Systems LLM-Assisted Annotation
School of Computer and Computing Science, Hangzhou City University Hangzhou, Zhejiang, China
Agreement Is Not Quality: Blind Expert Verification of Human and LLM Qualitative Coding When Human Consensus Is Not Ground Truth
Jul 30, 2026Alex Liu, Lief Esbenshade, Michael Xiao +4LLM-Assisted Annotation Human-in-the-Loop Evaluation
University of Washington · Colleague AI/ · Colleague AI

Oct 7, 2026, cs.CLJ/K move · Enter open · S save

Large Language Models for Machine Translation Quality Annotation: Humans and Models Are Both Challenged

Hala Almaghout, Christian Federmann, Qin Gao

Apple

MT Evaluation LLM Reliability LLM-Assisted Annotation Multilingual Language Model Evaluation

Open paper AskPDF arXiv

Large Language Models (LLMs) are considered to be a more efficient and cost-effective alternative to human judgment for Machine Translation (MT) evaluation. With MT evaluation spanning a large number of language pairs, domains and levels of annotation granularity, LLMs must be thoroughly evaluated across these dimensions before being reliably used as alternatives to human evaluation. In this paper, we evaluate the performance of LLMs for two prominent MT quality evaluation schemes: Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA) by comparing their agreement with human annotators. We present results on a long-context test set of 70 language pairs and the publicly available WMT23 and WMT25 data, investigating both score and error span annotation agreement across a variety of language pairs and domains. Our results show that while LLM agreement with human annotators exceeds agreement between human annotators for some evaluation tasks, both vary substantially across annotation schemes, language pairs and domains and remain unreliable for most settings. Furthermore, we identify challenges facing both human and LLM annotators: humans are particularly challenged by fine-grained MQM annotations and low-resource language pairs, while LLMs struggle with minor errors, wrong language variants and error span annotation. Our results highlight the potential for improvement for both human and LLM annotation performance, possibly through human-LLM collaborative annotation pipelines that address the reliability issues identified in this work.

Momentum

Related topics

Latest papers 125