QUORUM: QUality-Optimized Routing Using Multiple annotators

Antonio Purificato; Maria Sofia Bucarelli; Andrea Bacciu; Amin Mantrach; Fabrizio Silvestri

cs.CLAug 28, 2026

QUORUM: QUality-Optimized Routing Using Multiple annotators

Authors: Antonio Purificato, Maria Sofia Bucarelli, Andrea Bacciu, Amin Mantrach, Fabrizio Silvestri

Organizations: Amazon · Sapienza University of Rome · Université Côte d’Azur, CNRS, Inria, I3S

Abstract

Data annotation remains a central bottleneck in natural language processing, requiring human effort to obtain high-quality labels at scale. While Large Language Models (LLMs) offer a fast and cost-effective alternative, their reliability is highly instance-dependent: they perform well on simple inputs but often fail on examples requiring nuanced reasoning or contextual understanding. In this work, we address this challenge with QUORUM (QUality-Optimized Routing Using Multiple annotators), a budget-aware routing framework that dynamically assigns each instance to human or LLM annotators under a fixed annotation budget. Unlike prior approaches relying on model confidence or uncertainty estimates, QUORUM leverages feature-based signals to estimate instance difficulty and supports multiple annotations per instance, combining them through agreement-based rewards to improve reliability. We evaluate QUORUM across diverse closed- and open-ended annotation tasks in English and multilingual settings, and QUORUM improves annotation quality by up to 34.4% while reducing costs by 8.8% over competing methods. Code can be found at https://github.com/amazon-science/QUORUM.

Figures & tables

Panel of Figure 1 — Figure 1: Pareto frontier of cost versus performance on AG’s News dataset with 5,320 human annotations. The standalone LLM achieves 0.834 average accuracy. Costs denote cumulative annotation expenses.

Figure 2: QUORUM, a budget-aware annotation router estimating instance difficulty using linguistic and semantic features, assigning samples to human or LLM annotators to maximize annotation quality while minimizing cost.

Dataset

Method

10%

30%

50%

70%

Accuracy/ROUGE

MCA

Cost

Accuracy/ROUGE

MCA

Cost

Accuracy/ROUGE

MCA

Cost

Accuracy/ROUGE

MCA

Cost

AG’s News (LLM: 0.834) 7,600 samples

Random

0.729\pm 0.001

0.163\pm 0.012

144

0.805\pm 0.002

0.146\pm 0.005

281

0.794\pm 0.002

0.145\pm 0.004

418

0.804\pm 0.001

0.154\pm 0.002

555

ARAIDA

0.787\pm 0.010

\mathbf{0.255\pm 0.050}

152

0.834\pm 0.013

0.218\pm 0.012

304

0.861\pm 0.035

0.217\pm 0.014

456

0.886\pm 0.055

0.215\pm 0.010

608

CDI

0.800\pm 0.001

0.229\pm 0.012

152

\underline{0.841\pm 0.001}

0.215\pm 0.005

304

0.887\pm 0.003

0.222\pm 0.005

456

0.921\pm 0.002

0.220\pm 0.002

608

CoAnnotating

\underline{0.800\pm 0.001}

\underline{0.233\pm 0.001}

152

0.844\pm 0.001

0.223\pm 0.001

304

0.889\pm 0.001

0.224\pm 0.001

456

\underline{0.922\pm 0.001}

0.221\pm 0.001

608

HyPAC

0.794\pm 0.007

0.220\pm 0.016

152

0.840\pm 0.013

\mathbf{0.226\pm 0.007}

304

\underline{0.875\pm 0.033}

\underline{0.225\pm 0.009}

456

0.903\pm 0.052

\underline{0.224\pm 0.008}

608

Table 1: Mean

\pm

std in the auditor style setting. For each dataset, we report the number of samples, Accuracy or ROUGE-1 depending on the task (

\uparrow

), Machine Cumulative Accuracy (

\uparrow

), and total annotation cost (

\downarrow

) for varying percentages of human-labeled data. Bold indicates the best-performing method, while underlined the second best. Value in parentheses represents the average performance obtained by LLM annotators on that dataset.

Dataset	Method	10%		30%		50%		70%
Dataset	Method	Accuracy/ROUGE	MCA	Accuracy/ROUGE	MCA	Accuracy/ROUGE	MCA	Accuracy/ROUGE	MCA
AG’s News (LLM: 0.834) Human Cost: 760	Random	$0.678\pm 0.001$	$0.059\pm 0.070$	$0.679\pm 0.001$	$0.064\pm 0.050$	$0.681\pm 0.001$	$0.050\pm 0.028$	$0.688\pm 0.001$	$0.064\pm 0.013$
	ARAIDA	$0.778\pm 0.001$	$\mathbf{0.120\pm 0.001}$	$0.783\pm 0.012$	$0.060\pm 0.001$	$0.786\pm 0.018$	$\mathbf{0.110\pm 0.001}$	$0.792\pm 0.030$	$\mathbf{0.095\pm 0.001}$
	CDI	$0.778\pm 0.001$	$\underline{0.120\pm 0.001}$	$0.779\pm 0.001$	$0.060\pm 0.001$	$0.780\pm 0.001$	$\underline{0.110\pm 0.001}$	$0.783\pm 0.001$	$\underline{0.095\pm 0.001}$
	CoAnnotating	$0.778\pm 0.001$	$0.069\pm 0.001$	$0.778\pm 0.001$	$0.069\pm 0.001$	$0.778\pm 0.001$	$0.069\pm 0.001$	$0.778\pm 0.001$	$0.069\pm 0.001$
	HyPAC	$\underline{0.783\pm 0.003}$	$0.100\pm 0.017$	$\underline{0.785\pm 0.005}$	$\underline{0.094\pm 0.016}$	$\underline{0.790\pm 0.008}$	$0.097\pm 0.015$	$\underline{0.797\pm 0.013}$	$0.094\pm 0.010$

Table 2: Mean

\pm

std in the dollars setting. For each dataset, we report the full human annotation cost, Accuracy or ROUGE-1 depending on the task (

\uparrow

), and Machine Cumulative Accuracy (

\uparrow

) under varying budget constraints. Bold indicates the best-performing method, underlined the second best. Row values indicate the available annotation budget, expressed as a percentage of the cost required to annotate the entire dataset using human annotators.

Budget	Dataset	LLM-Uncertainty Features	Random Features	Ours
10%	AG’s News	$0.800\pm 0.002$	$0.697\pm 0.009$	$\mathbf{0.800\pm 0.001}$
	SST2	$0.948\pm 0.001$	$0.901\pm 0.012$	$\mathbf{0.968\pm 0.001}$
	Global-MMLU (es)	$0.741\pm 0.001$	$0.518\pm 0.012$	$\mathbf{0.749\pm 0.002}$
70%	AG’s News	$0.932\pm 0.002$	$0.712\pm 0.007$	$\mathbf{0.936\pm 0.001}$
	SST2	$0.978\pm 0.002$	$0.921\pm 0.010$	$\mathbf{0.990\pm 0.001}$
	Global-MMLU (es)	$0.910\pm 0.001$	$0.692\pm 0.009$	$\mathbf{0.912\pm 0.001}$

Table 3: Mean

\pm

95% confidence intervals accuracy (

\uparrow

) with uncertainty-based routing, routing based on random features and our feature-based difficulty estimation. Budget is expressed as a percentage of dataset size.

#Annotations	AG’s News	PubMed	MMLU-Redux	CNN
1	0.852	0.740	0.721	0.387
2	0.873	0.756	0.754	0.398
3	0.891	0.791	0.766	0.410

Table 4: Effect of multiple annotations per instance on QUORUM’s accuracy/ROUGE-1 across datasets.

Labeller	QUORUM	CDI	CoAnnotating	Random
Labeller 1 (8%)	0.7458	0.7292	0.7375	0.7313
Labeller 2 (8%)	0.7500	0.7271	0.7250	0.7375
Labeller 3 (17%)	0.7271	0.6854	0.7125	0.6917

Table 5: Accuracy of different routing strategies on SentiMP-En under the Auditor Style setting with a budget of 200 human annotations. The noise rate is reported for each annotator. Best results for each annotator are shown in bold.

Figure 3: Performance under multilingual settings across Spanish (es) and Japanese (ja) benchmarks for classification (Global-MMLU) and summarization (XLSum) at different annotation budgets (30% and 70%).

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Name	#Classes	Class Distribution (%)	#Samples
AG’s News	4	[25,25,25,25]	7,600
SST2	2	[49.9,51.1]	870
Global-MMLU ∗	4	[23, 25, 25, 27]	14,300
MMLU-Redux ∗	4	[22, 25, 25, 28]	3,000
IMDB ∗	2	[50,50]	25,000
PubMed ∗	5	[10.4,15.45, 33.42,7.89,32.84]	30,000

Appendix

Table 6: Number of classes, class distribution and number of samples of the selected datasets. NA indicates summarization datasets with no classes.

Figure 4: Analysis of the learned bandit routing policy on the AG’s News dataset (window of 2000 samples) in the dollars setting. Left: arm selection proportions over time, showing convergence from exploration to exploitation. Center: heatmap of posterior weight vectors across features and arms (blue=negative, red=positive). Right: feature importance computed as mean absolute weight across arms.

Table 7: Sensitivity analysis of QUORUM with respect to key routing and aggregation hyperparameters. We report downstream accuracy while varying the human escalation threshold (

\tau_{H}

), human voting weight (

w_{H}

), initial exploration rate (

\epsilon_{0}

), exploration decay factor (

\gamma

) and calibration size.

Method	AG’s News	PubMed	Global-MMLU
Random	0.007	0.035	0.014
ARAIDA	4.988+TT	19.879+TT	12.250+TT
CDI	0.503+CT	0.547+CT	0.661+CT
CoAnnotating	0.006+CT	0.038+CT	0.014+CT
HyPAC	0.046+CT	0.150+CT	0.075+CT
SANT	18.052+TT	161.014+TT	40.853+TT

Appendix

Table 8: Routing overhead (seconds) required before routing decisions across datasets (

\downarrow

). ET denotes embedding extraction time, CT confidence estimation time via additional LLM prompting, and TT model training time.

Dataset	Method	10%			30%			50%			70%
Dataset	Method	Accuracy/ROUGE	MCA	Cost	Accuracy/ROUGE	MCA	Cost	Accuracy/ROUGE	MCA	Cost	Accuracy/ROUGE	MCA	Cost
Global-MMLU (ja) (LLM: 0.640) 14,040 samples	Random	$0.641\pm 0.001$	$0.305\pm 0.013$	$\underline{267}$	$0.729\pm 0.002$	$0.313\pm 0.007$	$\underline{519}$	$0.729\pm 0.002$	$0.165\pm 0.003$	$\underline{772}$	$0.796\pm 0.002$	$0.243\pm 0.002$	$\underline{1025}$
	ARAIDA	$0.679\pm 0.014$	$0.383\pm 0.109$	$281$	$0.732\pm 0.018$	$0.312\pm 0.012$	$562$	$0.759\pm 0.052$	$0.315\pm 0.015$	$842$	$0.782\pm 0.082$	$0.318\pm 0.018$	$1123$
	CDI	$0.691\pm 0.001$	$0.355\pm 0.014$	$281$	$0.751\pm 0.002$	$0.318\pm 0.006$	$562$	$0.823\pm 0.002$	$\underline{0.335\pm 0.004}$	$842$	$0.893\pm 0.002$	$0.338\pm 0.002$	$1123$
	CoAnnotating	$0.692\pm 0.001$	$0.358\pm 0.001$	$281$	$\underline{0.760\pm 0.001}$	$\underline{0.348\pm 0.001}$	$562$	$0.818\pm 0.001$	$0.323\pm 0.001$	$842$	$\underline{0.909\pm 0.001}$	$\underline{0.361\pm 0.001}$	$1123$
	HyPAC	$\underline{0.715\pm 0.001}$	$0.349\pm 0.009$	$281$	$\mathbf{0.784\pm 0.001}$	$\mathbf{0.350\pm 0.002}$	$562$	$\mathbf{0.838\pm 0.001}$	$0.331\pm 0.002$	$842$	$0.899\pm 0.010$	$0.342\pm 0.008$	$1123$

Appendix

Table 9: Performance in the auditor style setting under different human annotation budgets. We report Accuracy or ROUGE-1 depending on the task (

\uparrow

), Machine Cumulative Accuracy (

\uparrow

), and total annotation cost (

\downarrow

Dataset	Method	10%		30%		50%		70%
Dataset	Method	Accuracy/ROUGE	MCA	Accuracy/ROUGE	MCA	Accuracy/ROUGE	MCA	Accuracy/ROUGE	MCA
Global-MMLU (ja) (LLM: 0.640) Human Cost: 1,404	Random	$0.665\pm 0.002$	$0.313\pm 0.008$	$0.659\pm 0.002$	$0.343\pm 0.007$	$0.729\pm 0.002$	$0.245\pm 0.003$	$0.766\pm 0.002$	$0.243\pm 0.002$
	ARAIDA	$0.696\pm 0.028$	$0.354\pm 0.109$	$0.732\pm 0.018$	$0.312\pm 0.012$	$0.759\pm 0.052$	$0.315\pm 0.015$	$0.782\pm 0.082$	$0.318\pm 0.018$
	CDI	$0.726\pm 0.002$	$0.350\pm 0.008$	$0.751\pm 0.002$	$0.318\pm 0.006$	$0.823\pm 0.002$	$\underline{0.335\pm 0.004}$	$0.893\pm 0.002$	$0.338\pm 0.002$
	CoAnnotating	$0.727\pm 0.001$	$0.353\pm 0.001$	$0.760\pm 0.001$	$0.348\pm 0.001$	$0.818\pm 0.001$	$0.323\pm 0.001$	$\underline{0.909\pm 0.001}$	$\underline{0.361\pm 0.001}$
	HyPAC	$\underline{0.749\pm 0.001}$	$0.351\pm 0.004$	$\underline{0.784\pm 0.001}$	$\underline{0.350\pm 0.002}$	$\mathbf{0.838\pm 0.001}$	$0.331\pm 0.002$	$0.899\pm 0.010$	$0.342\pm 0.008$

Appendix

Table 10: Performance in the dollars setting under a fixed monetary budget. We report Accuracy or ROUGE-1 depending on the task (

\uparrow

) and Machine Cumulative Accuracy (

\uparrow

) under varying budget constraints. Bold indicates the best-performing method, while underlined the second best. Row values indicate the available annotation budget, expressed as a percentage of the cost required to annotate the entire dataset using human annotators.

LLM-Assisted Annotation

Large language model-assisted annotation uses AI to label, classify, and structure data, often with human review to improve scale and consistency.

See all 202 papers

More Human, More Efficient: Aligning Annotations with Quantized SLMs
Apr 1, 2026Jiayu Wang, Junyoung LeeLLM-Assisted Annotation LLM Fine-Tuning
Home Team Science and Technology Agency
Can LLM Annotators Match Human Consistency? Benchmarking Human Agreement and Optimizing Multi-Agent with Contrastive Knowledge
Oct 23, 2026Xiaopeng Ke, Zhixin Wang, Xuebo Liu +5LLM-Assisted Annotation Multi-Agent LLM Systems
ANT Group · Harbin Institute of Technology (Shenzhen)
Are LLMs Ready to Replace Bangla Annotators?
Feb 18, 2026Md. Najib Hasan, Touseef Hasan, Souvika SarkarLLM-Assisted Annotation LLM Reliability
A2I Lab, School of Computing, Wichita State University

Apr 1, 2026J/K move · Enter open · S save

More Human, More Efficient: Aligning Annotations with Quantized SLMs

Jiayu Wang, Junyoung Lee

Home Team Science and Technology Agency

LLM-Assisted Annotation LLM Fine-Tuning LLM Evaluation LLM Alignment Small Language Models

Open paper AskPDF arXiv

As Large Language Model (LLM) capabilities advance, the demand for high-quality annotation of exponentially increasing text corpora has outpaced human capacity, leading to the widespread adoption of LLMs in automatic evaluation and annotation. However, proprietary LLMs often exhibit systematic biases that diverge from human expert consensus, lacks reproducibility, and raises data privacy concerns. Our work examines the viability of finetuning a quantized Small Language Model of 1.7B parameter size on limited human-annotated data to serve as a highly aligned, deterministic evaluator and annotator. By implementing a custom, multi-dimensional rubric framework and simple augmentation and regularization techniques, the proposed approach achieves higher inter-annotator agreement (0.23 points increase in Krippendorff's

α

) than the best performing state-of-the-art proprietary LLM. We also demonstrate the generalizability of the proposed training pipeline on a separate emotion classification task. The results show that task-specific alignment and efficient 4-bit quantized fine-tuning provide superior open-source alternative to using proprietary models for evaluation and annotation. Our finetuning approach is publicly available at https://github.com/jylee-k/slm-judge.

Abstract

LLM-Assisted Annotation

Explore similar work

More Human, More Efficient: Aligning Annotations with Quantized SLMs

Can LLM Annotators Match Human Consistency? Benchmarking Human Agreement and Optimizing Multi-Agent with Contrastive Knowledge

Are LLMs Ready to Replace Bangla Annotators?