Can LLM Annotators Match Human Consistency? Benchmarking Human Agreement and Optimizing Multi-Agent with Contrastive Knowledge

Xiaopeng Ke; Zhixin Wang; Xuebo Liu; Xinyu Ma; Xing Zheng; Bowen Song; Weiqiang Wang; Min Zhang

cs.CLOct 23, 2026

Can LLM Annotators Match Human Consistency? Benchmarking Human Agreement and Optimizing Multi-Agent with Contrastive Knowledge

Authors: Xiaopeng Ke, Zhixin Wang, Xuebo Liu, Xinyu Ma, Xing Zheng, Bowen Song, Weiqiang Wang, Min Zhang

Organizations: ANT Group · Harbin Institute of Technology (Shenzhen)

Abstract

LLM-based annotation has advanced rapidly, yet existing benchmarks remain misaligned with real-world practice. Actual annotation often involves lengthy, task-specific guidelines, and different annotators may produce divergent but reasonable judgments, especially on boundary cases, where single-labeled dataset fail to reflect this disagreement. This gap also limits existing single-agent and workflow-based annotation systems, which often lack adaptability to complex, evolving guidelines. To bridge this gap, we introduce ANT-A Benchmark, a real-world multi-annotator benchmark spanning 7 datasets across domains and modalities, with task-specific guidelines and 5 independent human labels per instance. ANT-A enables a distribution-aware evaluation, using Fleiss kappa to assess how closely model annotations align with human labeling distributions. We further propose ANT-OR, a multi-agent annotation framework that refines prompts by distilling contrastive decision rules from positive and negative feedback. Experiments show that ANT-OR improves agreement@>=2 accuracy by 4.5 points over the strongest baseline and achieves the highest alignment with human annotator distributions as measured by Fleiss kappa within 2 points. Code and benchmark are available at https://github.com/Krueske/ANT-OR.

Figures & tables

Figure 1: Overview of our contributions. We construct ANT-A , a benchmark with detailed guidelines and multiple independent human annotations, and develop ANT-OR , a multi-agent framework that improves automated annotation through iterative optimization with contrastive knowledge.

Figure 2: Construction of ANT-A . We curate domain-relevant samples, establish annotation guidelines through a pre-annotation pilot, and collect 5 independent annotations per sample to measure inter-annotator agreement.

Dataset	Domain	Modality	#Samples	#Classes	Fleiss’ $\kappa$
Disease Privacy Assessment	Medical	Text	1,500	2	38.77
Billing Scenario Classification	E-commerce	Text	1,500	31	83.50
Medical Entity Extraction	Medical	Text	1,500	22	82.09
Social Text Classification	General	Text	1,500	15	44.62
Image Attribution Recognition	General	Vision-Language	1,500	3	58.59
Image Classification	E-commerce	Vision-Language	1,500	24	81.27

Table 1: Summary statistics of ANT-A

Figure 3: Workflow of ANT-OR . Agents construct contrastive knowledge bases (§ 4.2 ), perform retrieval-augmented annotation (§ 4.3 ), and optimize prompts via failure attribution with iterative knowledge updates (§ 4.4 ).

Single-Agent Method for Data Annotation
Method	MedEE		DisPA		BillSC		SocialTC		ImageAR		ImageCl		ImageTR		Average
	Acc	F1	Acc	$\kappa$	Acc	$\kappa$	Acc	$\kappa$	Acc	$\kappa$	Acc	$\kappa$	Acc	$\kappa$	Acc	$\kappa$
Human	82.76	82.09	93.86	38.77	94.66	83.50	76.82	44.62	94.06	58.59	83.54	81.27	96.86	56.37	88.94	63.60
Vanilla	16.10	73.21	91.80	36.28	79.80	80.12	54.20	41.63	49.10	41.58	40.20	76.19	88.40	48.37	59.94	56.77
CoT Wei et al. (2022)	24.30	74.07	92.40	38.59	83.30	81.03	57.80	42.92	57.80	45.03	45.10	77.20	88.20	48.22	64.13	58.15
ICL Liu et al. (2022)	27.60	75.59	91.60	40.13	89.40	82.75	63.30	44.26	59.20	44.91	54.60	78.37	87.20	47.61	67.56	59.09

Table 2: Performance comparison of different methods on data annotation tasks. We report Accuracy (agreement@

\geq

2) (%) as the metric for annotation correctness, with Fleiss’ Kappa (

\kappa

) measuring inter-annotator consistency. For Medical Entity Extraction, where Fleiss’

\kappa

is not applicable due to span-level entity annotations, we use Entity-Level F1 Zhuang et al. (2024) instead. The best results are highlighted in bold .

I. Component Ablation
Method	MedEE		DisPA		BillSC		SocialTC		Avg.
	Acc	F1	Acc	$\kappa$	Acc	$\kappa$	Acc	$\kappa$	Acc	$\kappa$
ANT-OR (Full)	35.90	76.10	93.50	41.05	92.60	83.82	65.00	44.50	71.75	61.37
w/o Dimension Mining	34.90	75.93	91.20	40.34	87.60	82.01	58.50	42.59	68.05	60.22
w/o Contrastive Triples	32.80	75.70	89.50	40.35	91.90	83.69	63.70	43.18	69.47	60.74
w/o Coupled Injection	27.40	75.18	93.40	40.90	93.50	84.04	63.80	44.07	69.53	61.04

Table 3: Ablation and sensitivity analysis of ANT-OR on four text datasets.

Figure 4: Average accuracy under different levels of human agreement.

Figure 5: Generalization comparison of ANT-OR and SPO across datasets.

# Pos	BillSC		MedEE		Average
Samples	Agree $\geq$ 2	Fleiss’ $\kappa$	Agree $\geq$ 2	Fleiss’ $\kappa$	Agree $\geq$ 2	Fleiss’ $\kappa$
1	91.70	83.51	33.90	75.81	62.80	79.66
2	92.60	83.82	35.90	76.10	64.25	79.96
3	92.70	83.78	36.20	76.21	64.45	79.99

Table 4: Ablation study on the number of positive samples in contrastive mining.

Method	MedEE		DisPA		BillSC		SocialTC		Avg.
	Acc	Cost	Acc	Cost	Acc	Cost	Acc	Cost	Acc	Cost
SPO	27.70	73.87	91.10	54.32	90.50	33.86	59.00	101.71	67.08	65.94
ANT-OR	35.90	116.70	93.50	74.87	92.60	47.15	65.00	166.99	71.75	101.43

Table 5: Optimization Cost Analysis.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Method	MedEE		DisPA		BillSC		Average
	Acc	Cost	Acc	Cost	Acc	Cost	Acc	Cost
CoT	24.3	7.00	92.4	6.83	83.3	8.18	66.67	7.34
Debate	16.7	66.65	86.5	78.34	76.1	77.36	59.77	74.12
Planner-Worker	23.2	37.07	88.3	34.11	82.9	33.31	64.80	34.83
Annotator-Verifier	25.1	29.16	92.8	22.08	84.6	22.80	67.50	24.68

Appendix

Table 6: Performance comparison on different MAS (w/o Prompt Optimization ).

Task	Billing Scenario Classification
Input	Merchant Name : Leisure Tea House Bill Title : Leisure Tea House Counterparty : —
Human Annotations	[Food & Dining, Culture & Leisure, Food & Dining, Food & Dining, Culture & Leisure]
Observation	This case captures the inherent ambiguity of crossover business formats at the semantic boundary. According to the annotation rules, Food & Dining covers “beverages,” while Culture & Leisure encompasses “leisure venue consumption, including cinemas, bars, chess and card rooms, etc.” A tea house simultaneously operates as a beverage provider (Food & Dining) and a leisure venue (Culture & Leisure). Three annotators prioritized the beverage service aspect, whereas two annotators categorized it as a leisure venue analogous to a bar or chess room. This 3:2 split reflects the difficulty of assigning a single category to establishments whose core offering inherently spans multiple semantic boundaries.

Appendix

Table 7: A representative boundary case with human disagreement in Billing Scenario Classification Task.

Dataset	Domain	Modality	Task Usage	Query Source
BillSC	E-commerce	Text	Identifying bill types in payment scenarios	Anonymized billing records
MedEE	Medical	Text	Medical Content Analysis and Structuring	Huatuo-26M Li et al. (2023b)
DisPA	Medical	Text	Privacy Protection and Content Review	QABasedOnMedicaKnowledgeGraph (GitHub)
SocialTC	General	Text	Knowledge Classification and Content Layering	Zhihu Website
ImageTR	General	Vision-Language	Search images by text	Paraphrased from MUGE Lin et al. (2021) via LLM; images via Bing Image Search API
ImageCI	E-commerce	Vision-Language	Sorting and managing poster materials	CGL-Dataset Li et al. (2023a)

Appendix

Table 8: Overview of the seven tasks in the ANT-A benchmark.

Dataset	Avg Guideline Length (chars)	Avg Query Length (chars)	4–5 Agree (%)	2–3 Agree (%)	No Agree (%)
BillSC	9,236	160	86.1	13.9	0.0
MedEE	4,624	261	61.1	33.9	5.0
DisPA	4,499	22	80.1	19.9	0.0
SocialTC	27,529	2,700	44.8	53.3	1.9
ImageTR	3,907	32	89.9	10.1	0.0
ImageCI	7,668	N/A	82.5	17.5	0.0

Appendix

Table 9: Statistics of annotation guidelines, query lengths, and inter-annotator agreement distributions. We define 4–5 agreement as cases where at least four annotators selected the same label, 2–3 agreement as cases where the majority label was selected by two or three annotators, and no agreement as cases without a dominant label.

ID	Age	Degree	Expertise	Years of Experience
1	37	Bachelor	Law	6
2	33	Bachelor	Computer Application Technology	2.5
3	29	Bachelor	Business Administration	2.5
4	25	Bachelor	Electronic Information Engineering Technology	2
5	27	Bachelor	Computer Science and Technology	2
6	26	Bachelor	E-Commerce	2

Appendix

Table 10: Demographics of the human annotators.

Task	Billing Scenario Classification
Input	Merchant Name: Convenience Retail Store Bill Title: Personal Transfer Receipt Counterparty: Individual Customer
Human Annotations	[Daily Necessities, Daily Necessities, Daily Necessities, Daily Necessities, Daily Necessities]
MAS	Result: Transfer & Red Envelope Reasoning: Focused on the surface-level keyword “transfer” in the bill title and treated the transaction as a peer-to-peer fund transfer between individuals, completely overlooking the merchant’s industry attribute implied by “Retail Store.”
ANT-OR	Result: Daily Necessities Reasoning: Applied the dimension-based rule that merchant names containing explicit industry keywords (e.g., “retail store”) should take priority over payment-behavior descriptions in the title. Referenced analogous cases where merchants with clear industry labels were correctly classified by business type despite transfer/receipt descriptors.
Observation	This case demonstrates a classic failure mode of literal keyword matching. MAS is misled by the word “transfer” in the bill title and ignores the structural priority between merchant industry and transaction behavior. In contrast, ANT-OR’s multi-dimensional reasoning framework correctly elevates the merchant name’s explicit industry signal (“retail store” $\rightarrow$ Daily Necessities ) over the payment-operation description. The unanimous human consensus (5/5) on Daily Necessities confirms that ANT-OR’s reasoning aligns with human judgment while MAS falls into the trap of superficial pattern matching.

Appendix

Table 11: Qualitative comparison between MAS and ANT-OR on Billing Scenario Classification Task.

LLM-Assisted Annotation

Large language model-assisted annotation uses AI to label, classify, and structure data, often with human review to improve scale and consistency.

See all 202 papers

More Human, More Efficient: Aligning Annotations with Quantized SLMs
Apr 1, 2026Jiayu Wang, Junyoung LeeLLM-Assisted Annotation LLM Fine-Tuning
Home Team Science and Technology Agency
Rethinking Ground Truth: A Case Study on Human Label Variation in MLLM Benchmarking
Mar 20, 2026Tomas Ruiz, Tanalp Agustoslu, Carsten SchwemmerMultimodal Model Evaluation Multimodal Large Language Models
Computational Social Sciences LMU Munich Munich, Germany · Computational Linguistics LMU Munich Munich, Germany
Multi-Label Agreement: A Flexible Framework for Inter-Annotator Reliability
Oct 27, 2026Tim Menzner, Jochen L. LeidnerInter-Rater Reliability
Center for Responsible Artificial Intelligence (CRAI), Coburg University of Applied Sciences

Apr 1, 2026J/K move · Enter open · S save

More Human, More Efficient: Aligning Annotations with Quantized SLMs

Jiayu Wang, Junyoung Lee

Home Team Science and Technology Agency

LLM-Assisted Annotation LLM Fine-Tuning LLM Evaluation LLM Alignment Small Language Models

Open paper AskPDF arXiv

As Large Language Model (LLM) capabilities advance, the demand for high-quality annotation of exponentially increasing text corpora has outpaced human capacity, leading to the widespread adoption of LLMs in automatic evaluation and annotation. However, proprietary LLMs often exhibit systematic biases that diverge from human expert consensus, lacks reproducibility, and raises data privacy concerns. Our work examines the viability of finetuning a quantized Small Language Model of 1.7B parameter size on limited human-annotated data to serve as a highly aligned, deterministic evaluator and annotator. By implementing a custom, multi-dimensional rubric framework and simple augmentation and regularization techniques, the proposed approach achieves higher inter-annotator agreement (0.23 points increase in Krippendorff's

α

) than the best performing state-of-the-art proprietary LLM. We also demonstrate the generalizability of the proposed training pipeline on a separate emotion classification task. The results show that task-specific alignment and efficient 4-bit quantized fine-tuning provide superior open-source alternative to using proprietary models for evaluation and annotation. Our finetuning approach is publicly available at https://github.com/jylee-k/slm-judge.

Abstract

LLM-Assisted Annotation

Explore similar work

More Human, More Efficient: Aligning Annotations with Quantized SLMs

Rethinking Ground Truth: A Case Study on Human Label Variation in MLLM Benchmarking

Multi-Label Agreement: A Flexible Framework for Inter-Annotator Reliability