LLM Annotation

LLM annotation leverages large language models (LLMs) to automate the labeling of data for various natural language processing (NLP) tasks, aiming to reduce the cost and time associated with manual annotation. Current research focuses on optimizing LLM prompting strategies, developing methods to assess and improve LLM annotation accuracy (e.g., confidence-driven inference, multi-fidelity learning), and exploring the integration of LLM annotations with human expertise in a collaborative workflow. This approach holds significant promise for improving the efficiency and scalability of NLP research and development, particularly in resource-constrained settings and for tasks involving large or complex datasets.

Papers

February 16, 2024

AFaCTA: Assisting the Annotation of Factual Claim Detection with Reliable LLM Annotators
Jingwei Ni, Minjing Shi, Dominik Stammbach, Mrinmaya Sachan, Elliott Ash, Markus Leippold
Annotation Rather LLM Annotation Factuality Detection Claim Detection

February 14, 2024

Leveraging Large Language Models for Enhanced NLP Task Performance through Knowledge Distillation and Optimized Training Strategies
Yining Huang, Keke Tang, Meilian Chen
Natural Language Processing Knowledge Distillation Entity Recognition BERT Model LLM Annotation

February 5, 2024

Best Practices for Text Annotation with Large Language Models
Petter Törnberg
Best Practice Text Annotation Structured Prompting LLM Annotation

January 14, 2024

Inroads to a Structured Data Natural Language Bijection and the role of LLM annotation
Blake Vente
Multi Task Integral Role Sequence to Sequence Cross Task Generalization Inverse Task LLM Annotation Training Bottleneck

October 31, 2023

Interactive Multi-fidelity Learning for Cost-effective Adaptation of Language Model with Sparse Human Supervision
Jiaxin Zhang, Zhuohang Li, Kamalika Das, Sricharan Kumar
Large Language Model Language Model Human Annotation Multi Fidelity Point Annotation Sparse Learning LLM Annotation Effective Adaptation

October 27, 2023

Modeling Legal Reasoning: LM Annotation at the Edge of Human Agreement
Rosamond Thalken, Edward H. Stiglitz, David Mimno, Matthew Wilkens
Language Model Generative Model Extreme Edge Generative Language Model Legal Reasoning Human Agreement LLM Annotation

September 29, 2023

Using Large Language Models for Qualitative Analysis can Introduce Serious Bias
Julian Ashwin, Aditya Chhabra, Vijayendra Rao
Absolute Stance Bias Human Annotation Qualitative Analysis LLM Annotation High Quality Annotation

June 27, 2023

Large Language Models as Annotators: Enhancing Generalization of NLP Models at Minimal Cost
Parikshit Bansal, Amit Sharma
Large Language Model Strong Generalization Active Learning NLP Model Individual Annotator LLM Annotation Minimal Cost

LLM Annotation

Papers

AFaCTA: Assisting the Annotation of Factual Claim Detection with Reliable LLM Annotators

Leveraging Large Language Models for Enhanced NLP Task Performance through Knowledge Distillation and Optimized Training Strategies

Best Practices for Text Annotation with Large Language Models

Inroads to a Structured Data Natural Language Bijection and the role of LLM annotation

Interactive Multi-fidelity Learning for Cost-effective Adaptation of Language Model with Sparse Human Supervision

Modeling Legal Reasoning: LM Annotation at the Edge of Human Agreement

Using Large Language Models for Qualitative Analysis can Introduce Serious Bias

Large Language Models as Annotators: Enhancing Generalization of NLP Models at Minimal Cost