Reliable but Design-Sensitive: Instrument Uncertainty in LLM Annotation

Thomas Reiter; Christoph Kern; Fedor Miasnikov; Sofiia Nikolenko; Rob Chew; Stephanie Eckman; Frauke Kreuter

cs.CLSep 22, 2026

Reliable but Design-Sensitive: Instrument Uncertainty in LLM Annotation

Authors: Thomas Reiter, Christoph Kern, Fedor Miasnikov, Sofiia Nikolenko, Rob Chew, Stephanie Eckman, Frauke Kreuter

Organizations: Amazon · Ludwig Maximilian University of Munich, Germany · RTI International, North Carolina, USA · University of Maryland, Social Data Science Center, USA

Abstract

Large language models (LLMs) can give reliable labels under one setup yet change those labels when researchers make other reasonable design choices. We tested seven LLMs, 12 task designs, three independent runs, and 3,000 tweets labeled for offensive language and hate speech. Repeating the same model and task design produced high agreement (median Fleiss' $κ= 0.91$ ). Agreement fell when we changed the task design for the same tweets (median Cohen's $κ= 0.76$ ). Task design and model choice increased the variance of estimated prevalence by factors of 76.7 for offensive language and 110.6 for hate speech compared with sampling variance alone. Variation across LLM task designs reached 560-572 basis points, compared with 270-331 basis points across five human instrument versions. Confidence scores did not solve this problem. They tracked repeated model outputs more closely than agreement with human labels, and grouping six tweets in one prompt lowered mean offensive-language confidence by 660 basis points. We call the variation caused by task design and model choice instrument uncertainty. Researchers can measure it only by comparing reasonable task designs. Repeating one setup or relying on confidence scores cannot replace that test.

Figures & tables

Criterion	Comparison
Validity	Outputs vs. intended construct
Performance	Model vs. human benchmark
Reliability	Same model-design, repeated runs
Sensitivity	Across task designs and models

Table 1: The four evaluation criteria adapted from AAPOR Task Force on Responsible AI Integration in Survey Research (2026) and the comparison required for each.

	Offensive language				Hate speech
Variance source added	Component SD	SD cum	$\pm$ 95%	$deff$	Component SD	SD cum	$\pm$ 95%	$deff$
Sampling only (nominal)	75	75	$\pm$ 150	1.00	86	86	$\pm$ 170	1.00
Run-to-run variation	36	83	$\pm$ 160	1.23	38	94	$\pm$ 180	1.19
Task design, incl. Model $\times$ Design	560	567	$\pm$ 1,110	57.7	572	579	$\pm$ 1,140	45.5
Model choice	325	653	$\pm$ 1,280	76.7	693	903	$\pm$ 1,770	110.6

Table 2: Cumulative uncertainty in estimated label prevalence. Component standard deviations are estimated from crossed random-effects models of run-level prevalence. Task-design variation combines the task-design and Model

\times

Design variance components. Cumulative standard deviations add independently estimated components as variances. The 95% column reports

1.96\times\mathrm{SD}_{\mathrm{cum}}

, and

deff

is cumulative variance divided by nominal binomial sampling variance. Standard deviations and interval half-widths are reported in basis points.

Figure 3: Stated confidence tracks repeated model outputs more closely than external agreement and changes with presentation format. (a) Observed agreement against stated confidence for another run, another task design, and an individual human annotator. The diagonal marks perfect calibration. (b) Mean stated confidence under individual and batched presentation. All seven models report lower confidence when the same tweets are presented six per call, showing that confidence depends on the instrument used to elicit it.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Task Design Effects
Term	OL Task	OL Models	OL Random Slopes	HS Task	HS Models	HS Random Slopes
(Intercept)	0.794***	0.765***	0.722***	0.322***	0.389***	0.434***
Joint: OL first (vs. Separate)	0.027***	0.027***	0.027	0.023***	0.023***	0.023
Joint: HS first (vs. Separate)	0.011***	0.011***	0.011	$-$ 0.007***	$-$ 0.007***	$-$ 0.007
With Confidence (vs. Without)	0.023***	0.023***	0.023*	$-$ 0.010***	$-$ 0.010***	$-$ 0.010
Batch Prompt (vs. Individual)	$-$ 0.059***	$-$ 0.059***	$-$ 0.059	0.018***	0.018***	0.018

Appendix

Table 3: Pooled mixed-effects LPMs for OL and HS across three nested specifications. Task estimates task-design effects pooled across models; Models adds model identity; Random Slopes allows task-design effects to vary by model.

Task Design Effects
Term	OL Task	OL Models	OL Task Int.	OL Design Int.	HS Task	HS Models	HS Task Int.	HS Design Int.
(Intercept)	2.608***	2.234***	2.147***	1.998***	$-$ 1.786***	$-$ 1.116***	$-$ 1.169***	$-$ 1.004***
Joint: OL first (vs. Separate)	0.401***	0.417***	0.711***	0.432***	0.235***	0.257***	0.259***	0.265***
Joint: HS first (vs. Separate)	0.156***	0.162***	0.162***	0.167***	$-$ 0.075***	$-$ 0.083***	0.073***	$-$ 0.085***
With Confidence (vs. Without)	0.338***	0.351***	0.352***	0.645***	$-$ 0.105***	$-$ 0.115***	$-$ 0.115***	0.220***
Batch Prompt (vs. Individual)	$-$ 0.882***	$-$ 0.917***	$-$ 0.919***	$-$ 0.675***	0.192***	0.210***	0.211***	$-$ 0.366***

Appendix

Table 4: Pooled mixed-effects logistic regressions for OL and HS across four nested specifications: Task, Models, Task Interaction, and Design Interaction.

Offensive Language
Term	GPT-4o-mini	GPT-5.4	Mistral Large 3	Mistral Medium 3.5	Llama 3.1 8B	Llama 3.1 70B	Llama 4	SD
Intercept (Separate, individual, no confidence)	.721	.751	.831	.791	.830	.802	.831	.043
Joint: OL first (vs. Separate)	.068	.002	.032	.003	.064	$-$ .001	.020	.029
Joint: HS first (vs. Separate)	.050	$-$ .022	.027	$-$ .005	.016	.002	.006	.023
With Confidence (vs. Without)	.043	.020	.046	.010	$-$ .003	.028	.014	.018
Batch Prompt (vs. Individual)	$-$ .045	$-$ .001	$-$ .055	.003	$-$ .200	$-$ .053	$-$ .059	.067

Appendix

Table 5: Model-specific mixed-effects linear probability model estimates for offensive language and hate speech. Each model column reports a separate model with a random intercept for tweet. The intercept is the labeling probability under separate calls, individual presentation, and no confidence elicitation. The remaining coefficients are changes relative to that condition.

Model	Within a design	Across designs	Against humans
	(Fleiss’ $\kappa$ )	(Cohen’s $\kappa$ )	(Cohen’s $\kappa$ )
GPT-4o-mini	.84–.97 (.94)	.57–.94 (.79)	.41–.62 (.54)
GPT-5.4	.91–.96 (.93)	.69–.92 (.84)	.50–.66 (.59)
Mistral Large 3	.89–.95 (.92)	.46–.90 (.68)	.22–.60 (.47)
Mistral Medium 3.5	.81–.92 (.87)	.61–.91 (.77)	.36–.67 (.56)
Llama 3.1 8B	.19–.80 (.46)	$-$ .00–.81 (.35)	.14–.54 (.34)

Appendix

Table 6: Agreement by model. Min–max (mean) over task designs or design pairs. Within a design is Fleiss’

\kappa

over three runs; across designs is Cohen’s

\kappa

between modal labels from two designs of the same model; against humans is Cohen’s

\kappa

against the human majority label. The pooled rows give min–max (median).

	ECE against agreement with…
Outcome	Another run	Another design	Individual human	Human majority
Offensive language	0.074	0.050	0.218	0.160
Hate speech	0.096	0.059	0.163	0.111

Appendix

Table 7: Calibration and change detection. Panel A reports Expected Calibration Error (ECE) pooled across all seven models for the four agreement targets discussed in Section 4.4 . Panel B reports model-specific ECE for the first three targets, mean stated confidence, the share of items whose modal label changes across designs, and the AUC of confidence as a detector of those changes.

Condition	Calls/run	Task	Tweet	Out	4o-mini	5.4
	API	Tokens per call			Cost ($), 3 runs
Indiv., no conf.	3000	517	19	2	0.73	12.33
Indiv., w/ conf.	3000	580	19	4	0.83	14.02
Batch, no conf.	500	639	112	12	0.18	3.09
Batch, w/ conf.	500	724	112	24	0.21	3.67

Appendix

Table 8: Annotation costs for three runs of 3,000 tweets. Batch presentation reduces API calls sixfold and cost by approximately 76% within each model; GPT-5.4 costs approximately 17 times more than GPT-4o-mini.

Explore similar work

When Linguistic and Internal Confidence Diverge in Large Language Models
Aug 28, 2026Hefan Zhang, Bingquan Zhang, Ming Cheng +3Confidence Estimation in Language Models LLM Reliability
Dartmouth College · Oakland University
How Much Do LLM-as-a-Judge Design Choices Matter? A Systematic Comparison of Prompt Designs, Rating Scales, and Models
Oct 4, 2026Laurène Vaugrante, Thilo HagendorffLLM Evaluation LLM-as-a-Judge
AI Safety Research Group, University of Stuttgart
Two Wrongs, No Right: Auditing Social-Desirability Bias in LLM Annotators for Computational Social Science
May 12, 2026Varun KotteSocial Bias in Language Models LLM-Assisted Annotation
Independent Researcher

Aug 28, 2026, cs.CLJ/K move · Enter open · S save

When Linguistic and Internal Confidence Diverge in Large Language Models

Hefan Zhang, Bingquan Zhang, Ming Cheng, Saeed Hassanpour+2

Dartmouth College · Oakland University

Confidence Estimation in Language Models LLM Reliability LLM Evaluation

Open paper AskPDF arXiv

Users often ask large language models (LLMs) to report how confident they are, but it is unclear whether such linguistic confidence tracks the model's internal confidence. We study this question across 8 classification tasks, 2 generation tasks and 30 models from three families. For classification, we compare linguistic confidence with logits-based confidence along three axes: association, magnitude agreement and calibration. For generation, we test whether linguistic confidence tracks semantic-entropy-based uncertainty. The axes frequently diverge. Instance-level association is weak on average, although it improves on easier items and for stronger base models. Instruction-tuned models often report higher confidence and sometimes show higher association, but they also have larger confidence gaps and worse calibration. Prompt design mostly changes the distribution of reported confidence. Attitude cues inflate confidence without improving alignment, while score exemplars can preserve rank-order signal when they avoid collapsed confidence values. Regression analyses show that distributional properties of confidence scores explain much of the observed alignment pattern, with model metadata playing a smaller role after controls. These results support a lossy-channel view of linguistic confidence. A more dispersed verbal confidence distribution can carry useful rank information, but it does not make the scores calibrated. Linguistic confidence should therefore be evaluated with multi-axis diagnostics before being used in downstream reliability pipelines.

Abstract

Explore similar work

When Linguistic and Internal Confidence Diverge in Large Language Models

How Much Do LLM-as-a-Judge Design Choices Matter? A Systematic Comparison of Prompt Designs, Rating Scales, and Models

Two Wrongs, No Right: Auditing Social-Desirability Bias in LLM Annotators for Computational Social Science