Large language models (LLMs) are increasingly used as judges and graders to replace human annotators, raising the question of when such replacement is statistically justified. To support such decisions, the Alternative Annotator Test (Alt-Test) was introduced for dense settings where all items are annotated by the same annotators. However, in many large-scale annotation settings, items are annotated by different subsets of annotators. We therefore propose the Alternative Annotator Test with Resampled Annotations (AltRA-Test), a statistical test that does not require that all items are annotated by the same annotators. AltRA-Test uses a bootstrap that resamples items, draws k human annotations for each item, and substitutes one of them with an LLM annotation. This produces paired all-human and mixed human-LLM panels. A replacement is supported when the inter-annotator agreement (IAA) of the mixed panel is statistically non-inferior to the IAA of the all-human panel, i.e., when the LLM preserves the signal that the human annotators share. We establish that AltRA-Test is a consistent test of asymptotic level alpha. We then evaluate it through simulations and real-world experiments in which we test 18 LLMs and 5 prompting techniques on 20 datasets across text, image, audio, and video. Across these evaluations, about 100 items often suffice for the test. In simulations, where we know whether the LLM is a better or a worse annotator than the humans, AltRA-Test makes more correct replacement decisions than existing baselines. In real-world data experiments, LLMs pass the test in about half of the setups, depending mainly on the task. Since replacement is not always justified, we encourage researchers to test it for their task.
Figures & tables
Figure 1: An illustration of the AltRA-Test . The two left boxes present the paired-bootstrap procedure: for each of B iterations, we sample N items and k=3 annotations per item, replace one annotation with the LLM annotation, and compute IAA for the human-only and human–LLM panels. The substitution gap Δ(b) and adaptive margin ε(b) are then computed. The two right boxes present the decision stage, based on a non-inferiority test. Here, qα∗ is the α -quantile statistic; if it is positive, we conclude that the LLM is a good alternative to a human annotator.
Rh(b)=R(Yh(b)),Rm(b)=R(Ym(b)).
Algorithm 1 AltRA-Test
Figure 2: Left: The AltRA-Test against the baselines, simulated with N=200 items and k=3 , varying each procedure’s hyperparameter: c , ε , or frac . The X-axis shows the worse-model pass rate and the Y-axis the better-model pass rate. The green region marks the valid regime, where the worse-model pass rate is below 0.05 . As shown, the AltRA-Test dominates the baselines. Right: Using the calibration in the left panel, we set c=0.1 and vary the number of items N . From N=100 on, the AltRA-Test achieves reliable better-model pass rates ( >0.85 ).
Gemini
GPT
Qwen
Gemma
Llama
* Text
Rˉh
Gem-P
Gem-F
Gem-L
GPT-T
GPT-L
Q3-8b
Q3-14b
G3-4b
G3-12b
L3.2-3b
L3.1-8b
DICES-350 ( Aroyo et al., 2023 )
0.20
0.14
0.06
0.10
0.16
0.16
0.09
0.16
0.12
0.22
0.14
0.15
EPIC ( Frenda et al., 2023 )
0.20
0.22
0.19
0.24
0.22
0.21
0.20
0.22
−0.06
0.19
0.01
0.02
MultiPref ( Miranda et al., 2025 )
0.28
0.33
0.30
0.28
0.24
0.24
0.19
0.26
0.14
0.25
0.20
0.12
ChaosNLI ( Nie et al., 2020 )
0.28
0.31
0.30
0.29
0.31
0.26
0.26
0.32
0.28
0.29
0.23
0.28
Persuasion
0.31
0.06
0.07
0.01
0.15
0.10
0.12
0.18
0.18
0.17
0.10
0.09
Table 1: Main results. Rows are datasets with their human-only agreement Rˉh ; columns are LLM judges, grouped by family. Each cell reports the mean mixed-panel agreement Rˉm . Cells in green mean the judge passes the AltRA-Test ; bold marks the highest Rˉm for that dataset.
Gemini
GPT
Qwen
Gemma
Llama
Gem-P
Gem-F
Gem-L
GPT-T
GPT-L
Q3-8b
Q3-14b
Q2.5V-3b
Q2.5V-7b
G3-4b
G3-12b
L3.2-3b
L3.1-8b
# passed
16/20
14/20
14/20
8/17
9/17
4/12
5/12
1/6
3/6
2/17
8/17
2/12
2/12
Zero-shot
65%
55%
50%
41%
41%
33%
42%
17%
33%
12%
29%
8%
17%
Few-shot
80%
50%
50%
41%
29%
33%
33%
0%
33%
12%
24%
8%
17%
CoT
60%
65%
55%
35%
29%
33%
33%
0%
17%
12%
29%
8%
17%
Ensemble
70%
65%
50%
41%
53%
33%
42%
17%
17%
12%
41%
17%
17%
Table 2: Prompting techniques and minimum item count. For each judge, the table reports the number of datasets on which it passes under at least one prompting technique, the pass rate (in %) of each technique, and the average minimum number of items needed to pass among datasets where it passes at N=300 . Bold marks the judge’s best prompting technique.
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
Notation
Meaning
D={x1,…,xN}
Dataset of N items.
Hi={hi,1,…,hi,Mi}
The Mi human annotations observed for item xi .
f(xi)
LLM annotation for item xi .
k,kc
Panel size used by the test, with k≥3 and Mi≥k , and the panel size at which c is calibrated.
c
Margin constant; c=0.1 by default.
yi,h,yi,m
Human-only panel of k annotations for item xi , and the mixed panel formed by replacing one of them with f(xi) ; stacking them over items gives the matrices Yh and Ym .
Appendix
Table 3: Main notation used in the paper.
Alt-Test ( Calderon et al., 2025b )
STT ( He et al., 2025a )
AltRA-Test (ours)
Question
can the LLM replace a human annotator?
is the LLM indistinguishable from a human annotator?
can the LLM replace a human annotator?
Hypothesis
non-inferiority, one-sided
equivalence, two-sided
non-inferiority, one-sided
Compared against
the excluded annotator, head to head
the all-human mean
the annotator pool
Annotator identities
✗ required
✗ required
✓ not required
Annotator structure
✗ repeated annotators across items
✗ two disjoint groups, same items
✓ a single pool, no overlap needed
Annotators per item
≥3
≥4 ( ≥2 per group)
≥3
Appendix
Table 4: Comparison of the three procedures evaluated in the paper. ✓ marks a property the test has; ✗ marks a limitation or requirement.
Table 5: Dataset descriptions. The table reports the task, label type, number of labels, number of usable items, number of human annotations per item, and human-only agreement Rˉh . Here, ord. means ordinal, nom. means nominal, L is the number of labels, N is the number of items with at least three labels, and Mi is the number of human annotations for item xi . Setup is the annotation setup, in which the same annotators label all items (same), different annotators label different items (varying), or the annotators are not identified (no IDs).
Figure 3: Simulations with two groups of annotators, with (left) and without (right) a personal bias of the annotators, at c=0.1 and N=200 . Top: the pass rate of each LLM profile against the noise difference ηf−ηh , averaged over all group sizes from 9+1 to 5+5 . Bottom: the pass rate against the group sizes, averaged over every noise pair with ηf=ηh . As shown, the test favors an LLM that labels like the larger group.
Attribute
Group
Annotators
Rˉh
G3-12b
Gem-P
GPT-T
GPT-L
All
Whole pool
123
0.20
✓
✓
✓
✓
Age
Gen X+
31
0.17
✓
✓
✓
✗
Gen Z
56
0.19
✓
✓
✓
✓
Millennial
36
0.22
✓
✓
✓
✓
Education
College degree or higher
75
0.19
✓
✓
✓
✓
High school or below
41
0.20
✓
✓
✓
✓
Appendix
Table 6: Group-wise testing on DICES-350. For each group, the table reports the number of annotators, the human-only agreement Rˉh , and whether each of the four LLMs that pass with pooled panels also passes within the group.
Figure 4: Left: Comparison of the Eq - AltRA-Test (see Appendix B.1 ) against the STT for the equivalence question. Both apply two one-sided tests: the Eq - AltRA-Test to bootstrap quantiles, and the STT as t -tests. Simulations use N=200 items and k=3 , while varying c for the Eq - AltRA-Test and frac for the STT. The X-axis is the pass rate when not equal, over conditions with ∣ηf−ηh∣≥0.2 , and the Y-axis is the pass rate when equal, over conditions with ηf=ηh . Right: Using the calibration in the left panel, we set c=0.125 and vary the number of items N .
Figure 5: Comparison with the baselines in other settings, using k=3 and varying the hyperparameter of each test ( c , ε , or frac ). (a) and (b): other generative families. (c): N=50 , 100 , 150 , and 300 items. All plots share the legend of (c). The AltRA-Test dominates the baselines in all settings, with the largest gains at smaller N .
Figure 6: Calibration and behavior of the AltRA-Test . (a) Each curve is the mean over the conditions of its regime and the shaded band is one standard deviation across them; the dashed line marks the recommended c=0.1 . (b) Per-condition pass rate at that setting; passing is correct where the LLM is less noisy, an error where it is noisier, and neither on the diagonal.
Figure 7: Simulations of the AltRA-Test with c=0.1 , varying the panel size k , the number of annotators per item m , and the number of labels L , with N=200 (top) and N=100 (bottom). For the panel size, the margin is scaled by kc/k . In all cases, the worse-model pass rate is below 0.05 .
Figure 8: Additional simulations with N=200 items. (a) The AltRA-Test with c=0.1 and different agreement metrics. (b) Worse-model and better-model pass rates when the bias is shared by the humans and the LLM, and when only the humans have it. Each test uses its default setting ( c=0.1 , ε=0.1 , frac=0.5 ).
Figure 9: Simulations in which every human annotator has a personal bias, using N=200 items and k=3 . (a) Comparison with the baselines, while varying the hyperparameter of each test ( c , ε , or frac ). The green region is the valid region, and the red triangle is c=0.1 . The AltRA-Test outperforms the baselines within the valid region. (b) The AltRA-Test with c=0.1 and sparse annotations. The X-axis is the number of items that each annotator labels, where 200 means that every annotator labels all items. The pass rates do not change with it.
Accuracy ≥ human
Accuracy < human
Passes the test
8
0
Fails the test
1
36
Appendix
Table 7: Validation on CIFAR-10H with the CIFAR-10 labels as gold labels. The table counts the 45 combinations of LLMs and prompting techniques, according to the result of the AltRA-Test and to whether the accuracy of the LLM is at least the average human accuracy ( 0.94 ).
Family
Judge
Zero
Few
CoT
Ens
GEPA
Zero
Few
CoT
Ens
GEPA
DICES-350 ( Rˉh=0.20 )
EPIC ( Rˉh=0.20 )
Gemini
Gem-P
0.14
0.20
0.14
0.15
✗
0.21
0.23
0.22
0.22
✗
Gem-F
0.05
0.10
0.06
0.05
✗
0.19
0.16
0.19
0.19
✗
Gem-L
0.10
0.12
0.10
0.10
✗
0.20
0.20
0.24
0.22
✗
GPT
GPT-T
0.14
0.19
0.16
0.17
✗
0.22
0.23
0.22
0.23
✗
GPT-L
0.16
0.19
0.16
0.18
✗
0.23
0.28
0.21
0.24
✗
Appendix
Table 8: * Text. Test outcomes and mixed-panel agreement Rˉm for every text judge and prompting technique. Dark green marks the judge’s best technique when it passes; light green marks a passing technique that is not the judge’s best; gray marks the judge’s best technique when it fails. Bold marks the highest Rˉm for each dataset.
Family
Judge
Zero
Few
CoT
Ens
GEPA
Zero
Few
CoT
Ens
GEPA
T2I-DPO ( Rˉh=0.12 )
Vibe-Arena ( Rˉh=0.12 )
Gemini
Gem-P
0.15
0.15
0.14
0.15
✗
0.12
0.11
0.13
0.12
✗
Gem-F
0.15
0.14
0.15
0.14
✗
0.13
0.11
0.11
0.12
✗
Gem-L
0.13
0.12
0.13
0.12
✗
0.14
0.12
0.12
0.14
✗
GPT
GPT-T
0.12
0.13
0.14
0.13
✗
0.14
0.12
0.14
0.13
✗
GPT-L
0.11
0.11
0.12
0.11
✗
0.11
0.08
0.11
0.11
✗
Appendix
Table 9: Image. Test outcomes and mixed-panel agreement Rˉm for every image judge and prompting technique. See the caption of Table 8 .
Family
Judge
Zero
Few
CoT
Ens
GEPA
Zero
Few
CoT
Ens
GEPA
TTS-Preference ( Rˉh=0.01 )
SOMOS ( Rˉh=0.05 )
Gemini
Gem-P
0.01
0.00
0.02
0.03
✗
−0.14
0.03
−0.12
−0.10
✗
Gem-F
−0.02
−0.03
0.01
0.01
✗
0.06
0.04
0.04
0.03
✗
Gem-L
−0.09
−0.03
−0.00
0.02
✗
−0.10
−0.04
0.05
−0.06
✗
GPT
GPT-A
0.02
−0.04
−0.03
−0.05
✗
0.04
0.04
0.05
0.04
✗
GPT-A-M
−0.01
−0.00
−0.00
−0.00
✗
0.01
0.05
0.04
0.04
✗
Appendix
Table 10: Audio. Test outcomes and mixed-panel agreement Rˉm for every audio judge and prompting technique. See the caption of Table 8 .
Family
Judge
Zero
Few
CoT
Ens
T2V-Preference ( Rˉh=0.04 )
Gemini
Gem-P
0.07
0.07
0.08
0.08
Gem-F
0.08
0.08
0.08
0.08
Gem-L
0.06
0.06
0.06
0.07
Qwen
Q2.5V-3b
0.03
−0.00
−0.01
0.03
Q2.5V-7b
0.08
0.08
0.07
0.08
Appendix
Table 11: Video. Test outcomes and mixed-panel agreement Rˉm for every video judge and prompting technique. See the caption of Table 8 .
Gemini
GPT
Qwen
Gemma
Llama
* Text
Gem-P
Gem-F
Gem-L
GPT-T
GPT-L
Q3-8b
Q3-14b
G3-4b
G3-12b
L3.2-3b
L3.1-8b
* DICES-350 ( Rˉh=0.20 )
Alt-Test ω
0.89
0.81
0.78
0.88
0.87
0.12
0.70
0.49
0.98
0.87
0.80
AltRA-Test qα∗
+0.03
−0.07
−0.06
+0.01
+0.01
−0.09
−0.02
−0.06
+0.04
−0.03
−0.02
STT pTOST
0.01
1.00
1.00
0.15
0.02
1.00
0.94
1.00
0.08
1.00
1.00
STT-Alt plower
<0.01
1.00
1.00
0.15
0.02
1.00
0.94
1.00
<0.01
1.00
1.00
Appendix
Table 12: Outcomes of the three tests on datasets with shared annotators. For each dataset and judge, the table reports the Alt-Test winning rate ω at ε=0.1 , the AltRA-Test decision statistic qα∗ , the STT pTOST , and the STT-Alt plower at frac=0.5 , using each judge’s best prompting technique. For each dataset, all tests are applied to the items that satisfy the requirements of every test. Green marks a passing cell. Because each test depends on its own hyperparameter, the comparison should not be read as strictly apples-to-apples.
Figure 10: Varying the sample size N for real-world text datasets. The Y-axis of each plot presents the decision statistic qα∗ against the number of evaluated items N , with one curve per LLM, using each judge’s best prompting technique. When qα∗>0 , the LLM passes the test. The dashed line marks the smallest N at which any judge first passes the test.
Figure 11: Varying the sample size N for real-world image, audio, and video datasets. See the caption of Figure 10 .
Figure 12: Evaluating the impact of the number of random few-shot examples (X-axis) on the decision statistic qα∗ . As shown, the effect is mixed and depends on both the model and the dataset.
Figure 13: Evaluating the impact of reasoning effort in CoT prompting (X-axis) on the decision statistic qα∗ . Reasoning effort is controlled by the Gemini API thinking-effort parameter. As shown, the effect is mixed, but higher effort generally leads to better judges.
Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.
LLM-based annotation has advanced rapidly, yet existing benchmarks remain misaligned with real-world practice. Actual annotation often involves lengthy, task-specific guidelines, and different annotators may produce divergent but reasonable judgments, especially on boundary cases, where single-labeled dataset fail to reflect this disagreement. This gap also limits existing single-agent and workflow-based annotation systems, which often lack adaptability to complex, evolving guidelines. To bridge this gap, we introduce ANT-A Benchmark, a real-world multi-annotator benchmark spanning 7 datasets across domains and modalities, with task-specific guidelines and 5 independent human labels per instance. ANT-A enables a distribution-aware evaluation, using Fleiss kappa to assess how closely model annotations align with human labeling distributions. We further propose ANT-OR, a multi-agent annotation framework that refines prompts by distilling contrastive decision rules from positive and negative feedback. Experiments show that ANT-OR improves agreement@>=2 accuracy by 4.5 points over the strongest baseline and achieves the highest alignment with human annotator distributions as measured by Fleiss kappa within 2 points. Code and benchmark are available at https://github.com/Krueske/ANT-OR.
Xiaopeng Ke, Zhixin Wang, Xuebo Liu +5
ANT Group · Harbin Institute of Technology (Shenzhen)
LLM judges are used to reduce the need for costly human labor in evaluating open-ended text generation. However, the reliability of these judges depends critically on their alignment with human raters -- a property that itself depends on costly human annotations. In this work, we develop a method (Metric Match) for estimating correlation-based reliability metrics of LLM judges from limited annotations. Metric Match selects a subset of samples for human annotation such that the subset matches the population reliability metric with respect to acquired synthetic labels. We empirically show that Metric Match achieves a win-rate of 0.838 against random subset selection across four different correlation metrics and 15 datasets, with an 18.7% decrease in average estimation error and reduces annotation needs by 32.5%. We provide a cost model and highlight a medical case study where our method saves $1,041.67 compared to random selection for expert annotation. Further, we shift our task from reliability estimation to reliability classification of whether a given judge is above a deployment threshold, outperforming random selection with Metric Match. All project code is publicly available, and we additionally provide an installable package for ease of use.
Alyssa Unell, Natalie Dullerud, Naomi Boneh +4
Department of Computer Science Stanford University · Department of Medicine Stanford University
Although large language models (LLMs) are increasingly used as annotators at scale, they are typically treated as a pragmatic fallback rather than a faithful estimator of human perspectives. This work challenges that presumption. By framing perspective-taking as the estimation of a latent group-level judgment, we characterize the conditions under which modern LLMs can outperform human annotators, including in-group humans, when predicting aggregate subgroup opinions on subjective tasks, and show that these conditions are common in practice. This advantage arises from structural properties of LLMs as estimators, including low variance and reduced coupling between representation and processing biases, rather than any claim of lived experience. Our analysis identifies clear regimes where LLMs act as statistically superior frontline estimators, as well as principled limits where human judgment remains essential. These findings reposition LLMs from a cost-saving compromise to a principled tool for estimating collective human perspectives.
Hasan Amin, Harry Yizhou Tian, Xiaoni Duan +3
Purdue University · Washington University in St. Louis