LLM-Assisted Annotation

Follow this topic Ask about this topic

Momentum

17 papers in the last four weeks, up 143% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 124

All topics

BEATS: Bootstrapping E-commerce Attribute Taxonomies for Search through Iterative Human-AI Collaboration
Jun 3, 2026Yung-Yu Shih, Shang-Yu Su, Tzu-I Ho +2Semantic Search LLM-Assisted Annotation
National Taiwan University · Rakuten Group, Inc. · Taiwan Rakuten Ichiba, Inc. +1
Framing Migration News with LLMs: Structured CoT as a Support for Human Interpretation
Jun 2, 2026David Alonso del Barrio, Jing Wen, Daniel Gatica-PerezLLM Interpretability CoT Reasoning
Idiap Research Institute and EPFL Switzerland
Fixing FOLIO and MALLS: Verified Annotations and an LLM-assisted Framework to Focus Human Relabeling
Jun 1, 2026Andrea Brunello, Cristian Curaba, Luca Geatti +3LLM-Assisted Annotation Human-in-the-Loop Annotation
University of Udine, Italy
TalkTag: Fine-Grained Morphosyntactic Error Annotation for Transcribed Speech
Jun 1, 2026Shamira Venturini, Oliver Hennhöfer, Steffen Kinkel +1LLM Fine-Tuning LLM-Assisted Annotation
Karlsruhe Institute of Technology · Karlsruhe University of Applied Sciences · Karlsruhe Institute of Technology, Germany +1
EvoPool: Evolutionary Programmatic Annotation for Label-Efficient Specialized Supervision
Jun 1, 2026Tianyi Xu, Yaolun Zhang, Xuan Ouyang +1Multi-Agent LLM Systems LLM-Assisted Annotation
Oregon State University · University of Wisconsin–Madison
On the Limits of LLM Adaptability: Impact of Model-Internalized Priors on Annotation Task Performance
May 30, 2026Etienne Casanova, Rafal Kocielnik, R. Michael AlvarezLLM Evaluation Zero-Shot Text Classification
California Institute of Technology, Pasadena, CA, USA.
Wind Turbine Maintenance Log Labelling Framework: LLM-Driven Data Correction and Enrichment via Semantic Extraction of Reliability Intelligence
May 29, 2026Max Malyi, Jonathan Shek, Alasdair McDonald +1LLM-Assisted Annotation Document Information Extraction
Institute for Energy Systems, School of Engineering, The University of Edinburgh, Edinburgh, UK · 2Nadara, Lisbon, Portugal
Industrializing Prediction-Powered Inference: The GLIDE Library for Reliable GenAI and Agentic Systems Evaluation
May 29, 2026Grégoire Martinon, Ibrahim Merad, Mohammed RakiLLM-Assisted Annotation LLM Agent Evaluation
Emerton Data, 16 Avenue Hoche, 75008, Paris, France.
Traceable by Design: An LLM Pipeline and Dashboard for EU Regulatory Consultation Analysis
May 29, 2026Thales Bertaglia, Haoyang Gui, Catalina Goanta +1LLM-Assisted Annotation Document Information Extraction
Utrecht University, The Netherlands · Maastricht University, The Netherlands
Beyond Agreement: Scoring Panel-Surfaced Biomedical Entity Candidates for Curator Triage
May 29, 2026Shuheng Cao, Ruiqi Chen, Renjie Cao +3Named Entity Recognition LLM-Assisted Annotation
University of California, San Diego · University of Michigan, Ann Arbor · The Hong Kong University of Science and Technology, Guangzhou +2
When Models Disagree: Rethinking LLM Evaluation for Public Comment Analysis
May 27, 2026Aisha Najera, Alvin Moon, Vedant Srinivasan +1LLM Evaluation Annotator Disagreement
AI Lab, Princeton University · Engineering and Applied Sciences, RAND Corporation · Science, Technology, and International Affairs, Georgetown University
Frontier LLM-based agents can overcome the ontology curation bottleneck for natural phenotypes
May 27, 2026James P. Balhoff, Hilmar LappLLM-Assisted Annotation LLM Agent Evaluation
Renaissance Computing Institute, University of North Carolina, Chapel Hill, North Carolina, USA · Neuromatch, USA
Human Label Variation as Stable Signal: Learning Annotator-Specific Explanation Behavior via Cross-Annotator Preference Optimization
May 27, 2026Beiduo Chen, Pingjun Hong, Ziyun Zhang +3LLM-Assisted Annotation Preference Optimization
U MaiNLP, Center for Information and Language Processing, LMU Munich, Germany · Æ Munich Center for Machine Learning (MCML), Munich, Germany · [ University of Vienna, Austria +1
From Learning Resources to Competencies: LLM-Based Tagging with Evidence and Graph Constraints
May 27, 2026Ngoc Luyen Le, Marie-Hélène Abel, Bertrand LaforgeLLM-Assisted Annotation LLM Information Extraction
Gamaizer, 93340 Le Raincy, France · Université de technologie de Compiègne, CNRS, Heudiasyc (Heuristics and Diagnosis of Complex Systems), CS 60319 - 60203 Compiègne Cedex, France · Sorbonne Université, CNRS UMR 7585, LPMHE (Laboratoire de Physique Nucléaire et des Hautes Énergies), 75252 Paris cedex 05, France
Where LLM Annotators Fail: Label-Free Learning on Graphs with LLMs
May 27, 2026Safal Thapaliya, Jiatan Huang, Chuxu ZhangLLM-Assisted Annotation Text-Attributed Graph Learning
University of Connecticut, USA
Attribute-Based Diagnosis of LLM Alignment with Hate Speech Annotations
May 26, 2026Mohammad Amine Jradi, Faeze Ghorbanpour, Alexander FraserLLM Evaluation LLM Alignment
School of Computation, Information and Technology, TU Munich Munich Center for Machine Learning (MCML)
Slide Deck Q&A Quality Assurance App: A Multi-Stage Pipeline for Pedagogical Question Generation
May 26, 2026Jim SalsmanLLM-Assisted Annotation Generative AI in Education
TalkNicer, Inc.
WhoSaidIt: Human-LLM Collaborative Annotation for Text-Based Multilingual Speaker-Attribute Classification
May 25, 2026Lingyu Gao, Will Monroe, David Smith +2Multilingual Language Model Evaluation LLM-Assisted Annotation
Duolingo
Double Triangle Annotation: A Scalable Human-in-the-Loop Framework for High-Precision Historical Document Annotation
May 25, 2026Yi RenHistorical Document Analysis LLM-Assisted Annotation
École Polytechnique Fédérale de Lausanne
Agent-as-Peer-Debriefer: A Multi-Agent Framework with Perspective-Based Refinement for Qualitative Analysis
May 23, 2026Zhimin Lin, Kun Cheng, Zhiyao Shu +4Multi-Agent LLM Systems LLM-Assisted Annotation
Soochow University · George Mason University · Johns Hopkins University
Improving Labeling Consistency with Detailed Constitutional Definitions and AI-Driven Evaluation
May 22, 2026Konstantin Berlin, Adam SwandaAnnotator Disagreement Language Model Safety Evaluation
Cisco AI Defense
Refining and Reusing Annotation Guidelines for LLM Annotation
May 20, 2026Kon Woo Kim, Jin-Dong Kim, Akiko AizawaLLM Alignment LLM-Assisted Annotation
The Graduate University for Advanced Studies, SOKENDAI · National Institute of Informatics (NII) · BioData Science Initiative (BSI), National Institute of Genetics (NIG)
iPOE: Interpretable Prompt Optimization via Explanations
May 18, 2026Jiahui Li, Yarik Menchaca Resendiz, Sean Papay +1LLM Interpretability LLM-Assisted Annotation
Fundamentals of Natural Language Processing, University of Bamberg, Germany · Leibniz-Institut für Psychologie (ZPID), Trier, Germany
LLMs for automatic annotation of Mandarin narrative transcripts
May 17, 2026Qingwen Zhao, Hongao Zhu, Yunqi He +3Multilingual Language Model Evaluation LLM-Assisted Annotation
Shanghai Jiao Tong University, China · University of California San Diego, USA · The Hong Kong Polytechnic University, China +2
How to Instruct Your Robot: Dense Language Annotations Power Robot Policy Learning
May 16, 2026Bosung Kim, Ruiyi Wang, David Acuna +5Robot Policy Learning Robot Skill Learning
University of California, San Diego · 2NVIDIA
LLMs as annotators of credibility assessment in Danish asylum decisions: evaluating classification performance and errors beyond aggregated metrics
May 13, 2026Galadrielle Humblot-Renaux, Mohammad N. S. Jahromi, Rohat Bakuri-Jørgensen +7Legal NLP LLM Evaluation
Visual Analysis and Perception Lab, Aalborg University · Pioneer Center for AI, Denmark · Center of Excellence for Global Mobility Law, University of Copenhagen +1
Two Wrongs, No Right: Auditing Social-Desirability Bias in LLM Annotators for Computational Social Science
May 12, 2026Varun KotteSocial Bias in Language Models LLM-Assisted Annotation
Independent Researcher
Do Benchmarks Underestimate LLM Performance? Evaluating Hallucination Detection With LLM-First Human-Adjudicated Assessment
May 8, 2026I. F. Atasoy, B. Mutlu, E. A. Sezer +1LLM Evaluation LLM-Assisted Annotation
Department of Computer Engineering, Hacettepe University, Ankara, 06800, Türkiye · Department of Computer Engineering, Ankara University, Ankara, 06830, Türkiye · Zephlen AI and Information Technologies Inc., Kocaeli, 41400, Türkiye
Prompt Engineering Strategies for LLM-based Qualitative Coding of Psychological Safety in Software Engineering Communities: A Controlled Empirical Study
May 8, 2026Moaath Alshaikh, Tasneem Alshaher, Ricardo Vieira +7Software Engineering LLM Evaluation
Federal University of Bahia · State University of Feira de Santana · Federal University of Sergipe +1
MultiSoc-4D: A Benchmark for Diagnosing Instruction-Induced Label Collapse in Closed-Set LLM Annotation of Bengali Social Media
May 7, 2026Souvik Pramanik, S. M. Riaz Rahman Antu, Shak Mohammad Abyad +2LLM Evaluation LLM-Assisted Annotation
North South University, Dhaka, Bangladesh

Jun 3, 2026, cs.IRJ/K move · Enter open · S save

BEATS: Bootstrapping E-commerce Attribute Taxonomies for Search through Iterative Human-AI Collaboration

Yung-Yu Shih, Shang-Yu Su, Tzu-I Ho, Dongzhe Wang+1

National Taiwan University · Rakuten Group, Inc. · Taiwan Rakuten Ichiba, Inc.+1

Semantic Search LLM-Assisted Annotation E-Commerce Search LLM Information Extraction Information Retrieval

Open paper AskPDF arXiv

E-commerce platforms in emerging markets often operate with underdeveloped product catalogs that contain only category taxonomies but lack structured attribute schemas. This absence of fine-grained product attributes limits search capabilities -- preventing faceted filtering, degrading query understanding, and weakening semantic representations used by search systems. We present BEATS, a human-in-the-loop LLM framework for bootstrapping product attribute taxonomies entirely from scratch. Our approach extends a multi-stage LLM generation pipeline with two critical production stages: (1) proactive quality checking by model developers to filter erroneous outputs, and (2) human annotation by domain-expert local staff to validate generated attributes. The framework operates iteratively -- prompts at each generation stage are refined based on quality check observations and annotator feedback across successive rounds, progressively improving attribute quality. Once the attribute taxonomy is established, we employ LLMs to perform structured attribute tagging on individual product items, enriching their contextual representations. The enriched catalog directly benefits multiple components of the search system: enabling granular attribute-based filtering, providing structured features for ranking models, and improving semantic representations for dense retrieval. We validate the generated taxonomy by training dense retrieval models on attribute-enriched product data, demonstrating consistent improvements over baselines using original catalog information. Our system has been deployed at Rakuten Taiwan, enriching 9 major categories spanning 2,694 sub-categories with 67,277 generated attributes, and over 5.4 million products have been tagged with the generated attributes, with plans to enrich the entire product catalog.

Momentum

Related topics

Latest papers 124