Can This LLM Annotate My Data? Estimating Zero-Shot LLM Performance

Tong Lin; Grant Schoenebeck; Sabina Tomkins

cs.CLOct 16, 2026

Can This LLM Annotate My Data? Estimating Zero-Shot LLM Performance

Authors: Tong Lin, Grant Schoenebeck, Sabina Tomkins

Organizations: School of Information, University of Michigan

Abstract

Large language models (LLMs) are increasingly used in annotation tasks with varying success. While several studies have demonstrated the opportunities and shortcomings of LLMs for annotation, a critical question remains unanswered: is it possible to predict how well they will perform on a new dataset? To answer this and related questions, we construct two task families which induce 461 text classification datasets. We find that it is possible to predict LLM performance - to a certain degree - and that contrary to other work which highlights the usefulness of LLM verbalizations of self-confidence, internal log-probabilities are the most predictive signal.

Figures & tables

Figure 1: Two-stage estimator: from instance-level correctness predictions to a dataset-level reliability decision.

		Accuracy						F1
$\kappa$		10	50	90	130	170	210	10	50	90	130	170	210
Baseline		.740						.850
ATC ( Garg et al., 2022 )		$.717_{.015}$	$.750_{.016}$	$.766_{.004}$	$.774_{.011}$	$.781_{.012}$	$.782_{.005}$	$.815_{.011}$	$.830_{.012}$	$.841_{.003}$	$.847_{.008}$	$.851_{.008}$	$.852_{.003}$
Meta-model ( Fu et al., 2023 )		$.666_{.015}$	$\mathbf{.772_{.016}}$	$\mathbf{.790_{.017}}$	$.795_{.015}$	$\mathbf{.810_{.008}}$	$.806_{.019}$	$.760_{.009}$	$.846_{.013}$	$.855_{.013}$	$.861_{.011}$	$\mathbf{.871_{.005}}$	$.866_{.013}$
Internal probability	$\mathrm{uncertainty}_{\mathrm{first}}$	$.726_{.020}$	$.767_{.019}$	$.787_{.005}$	$\mathbf{.808_{.008}}$	$.808_{.003}$	$\mathbf{.814_{.006}}$	$.822_{.015}$	$.844_{.015}$	$.856_{.005}$	$\mathbf{.872_{.006}}$	$\mathbf{.871_{.002}}$	$\mathbf{.875_{.004}}$
Internal probability	$\mathrm{uncertainty}_{\mathrm{avg}}$	$.714_{.014}$	$.762_{.011}$	$.762_{.011}$	$.781_{.006}$	$.787_{.005}$	$.790_{.002}$	$.813_{.010}$	$.841_{.008}$	$.840_{.008}$	$.854_{.005}$	$.857_{.003}$	$.859_{.000}$

Table 1: Accuracy and F1 of the estimator for GPT-4o mini at

\tau=0.9

, with HGB as instance-level classifier, using each signal on its own or all five signals together, for each number of sampled instances

\kappa

. Values are means over three random seeds with standard errors as subscripts. The baseline predicts the majority class for accuracy and every dataset as reliable for F1. ATC and the meta-model use average-token uncertainty. Bold: best in column.

Figure 2: (a) F1 of the reliability estimator for GPT-4o mini at

\tau=0.9

as signals are added one at a time (left to right), in three orders that each start from a different signal family; each line is one number of sampled instances

\kappa

. (b) Number of datasets in each bin of true accuracy that the estimator with the first-token uncertainty alone predicts reliable or unreliable at

\kappa=210

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Genre A	Genre B	$n_{A}$	$n_{B}$
mystery & suspense	drama	90	194
drama	comedy	144	137
drama	action & adventure	143	130
mystery & suspense	action & adventure	141	133
science fiction & fantasy	action & adventure	73	189
romance	drama	38	249

Appendix

Table 2: The 11 genre pairs and the number of usable movies of each genre, after dropping movies whose scraped script is shorter than 500 characters. Each pair gives ten datasets, one per script length, with the same movies.

	NYT topic pair	Sliding-window genre pair
Task description	Given this article, which of these topics does it belong to: ‘Health’ or ‘Agriculture’?	Given this movie script, which genre does the movie belong to: ‘drama’ or ‘comedy’?
Label definitions	Health: The article belongs to the topic ‘Health’. Agriculture: The article belongs to the topic ‘Agriculture’.	drama: The script comes from a movie with genre ‘drama’ and not genre ‘comedy’. comedy: The script comes from a movie with genre ‘comedy’ and not genre ‘drama’.
Example text	Patients Surge And Hospitals Hunt for Beds	SHREK Written by William Steig & Ted Elliott SHREK Once upon a time there was a lovely princess. But she had an enchantment upon her of a fearful sort which could only be broken by love’s first kiss. […] (1,295 words)
Gold label	Health	comedy

Appendix

Table 3: Task description, label definitions, and one instance with its gold label for a dataset of each family, as filled into the prompt of Figure 3 . These are the two training datasets of Figure 1 ; the genre dataset uses the first 10% of each script.

Figure 4: Dataset accuracy

R(D)

of the 461 datasets for each LLM, stacked by task family. The dashed line marks

\tau=0.9

, and the text gives the share of datasets that reach it, overall and in each family.

Figure 5: Datasets in each bin of true accuracy (bars, left axis) and the share of each bin that each method predicts reliable (lines, right axis) at

\kappa=210

, averaged over three seeds. Our estimator uses HGB with the first-token uncertainty. Lower lines are better in the shaded bins, which hold the truly unreliable datasets (

R(D)<\tau

), and higher lines are better to the right of the dashed line; the dotted line marks the ideal.

	GPT-4o mini		Llama-3.1-8B		Qwen3-8B
Method	Acc	F1	Acc	F1	Acc	F1
All datasets	461 datasets		461 datasets		461 datasets
Baseline	.740	.850	.638	.532	.544	.705
ATC	$.782_{.005}$	$.852_{.003}$	$.539_{.005}$	$.470_{.005}$	$.711_{.004}$	$.718_{.004}$
Meta-model	$.806_{.019}$	$.866_{.013}$	$.750_{.014}$	$.599_{.025}$	$.705_{.003}$	$.718_{.005}$
Ours	$\mathbf{.814_{.006}}$	$\mathbf{.875_{.004}}$	$\mathbf{.802_{.010}}$	$\mathbf{.730_{.013}}$	$\mathbf{.729_{.005}}$	$\mathbf{.740_{.006}}$

Appendix

Table 4: Accuracy and F1 at

\tau=0.9

and

\kappa=210

after excluding the datasets whose true accuracy

R(D)

lies within a band around

\tau

, with the number of remaining datasets for each LLM. The baseline is recomputed on the remaining datasets and predicts the majority class for accuracy and every dataset as reliable for F1. Values are means over three random seeds with standard errors as subscripts.

		Accuracy						F1
Classifier $g$	Signals	10	50	90	130	170	210	10	50	90	130	170	210
Baseline		.740						.850
ATC ( Garg et al., 2022 )		$.717_{.015}$	$.750_{.016}$	$.766_{.004}$	$.774_{.011}$	$.781_{.012}$	$.782_{.005}$	$.815_{.011}$	$.830_{.012}$	$.841_{.003}$	$.847_{.008}$	$.851_{.008}$	$.852_{.003}$
Meta-model ( Fu et al., 2023 )		$.666_{.015}$	$.772_{.016}$	$\mathbf{.790_{.017}}$	$.795_{.015}$	$.810_{.008}$	$.806_{.019}$	$.760_{.009}$	$.846_{.013}$	$.855_{.013}$	$.861_{.011}$	$.871_{.005}$	$.866_{.013}$
HGB	$\mathrm{uncertainty}_{\mathrm{first}}$	$.726_{.020}$	$.767_{.019}$	$.787_{.005}$	$\mathbf{.808_{.008}}$	$.808_{.003}$	$\mathbf{.814_{.006}}$	$.822_{.015}$	$.844_{.015}$	$.856_{.005}$	$\mathbf{.872_{.006}}$	$.871_{.002}$	$\mathbf{.875_{.004}}$
	$\mathrm{uncertainty}_{\mathrm{avg}}$	$.714_{.014}$	$.762_{.011}$	$.762_{.011}$	$.781_{.006}$	$.787_{.005}$	$.790_{.002}$	$.813_{.010}$	$.841_{.008}$	$.840_{.008}$	$.854_{.005}$	$.857_{.003}$	$.859_{.000}$

Appendix

Table 5: Accuracy and F1 of the estimator for GPT-4o mini at

\tau=0.9

with each instance-level classifier

g

, using each signal on its own or all five signals together, for each

\kappa

. Values are means over three random seeds with standard errors as subscripts.

		Accuracy						F1
Classifier $g$	Signals	10	50	90	130	170	210	10	50	90	130	170	210
Baseline		.638						.532
ATC ( Garg et al., 2022 )		$.521_{.015}$	$.549_{.011}$	$.547_{.005}$	$.538_{.005}$	$.542_{.004}$	$.539_{.005}$	$.521_{.012}$	$.495_{.013}$	$.484_{.005}$	$.474_{.005}$	$.477_{.006}$	$.470_{.005}$
Meta-model ( Fu et al., 2023 )		$\mathbf{.643_{.005}}$	$.731_{.003}$	$.734_{.005}$	$.751_{.005}$	$.754_{.011}$	$.750_{.014}$	$.496_{.013}$	$.591_{.012}$	$.578_{.016}$	$.608_{.008}$	$.620_{.019}$	$.599_{.025}$
HGB	$\mathrm{uncertainty}_{\mathrm{first}}$	$.607_{.016}$	$.735_{.013}$	$.770_{.002}$	$.785_{.002}$	$.803_{.002}$	$.802_{.010}$	$.597_{.013}$	$.672_{.015}$	$.701_{.003}$	$.712_{.002}$	$.734_{.003}$	$.730_{.013}$
	$\mathrm{uncertainty}_{\mathrm{avg}}$	$.580_{.019}$	$.641_{.002}$	$.648_{.010}$	$.667_{.010}$	$.672_{.015}$	$.651_{.013}$	$.571_{.021}$	$.566_{.011}$	$.566_{.008}$	$.588_{.016}$	$.591_{.014}$	$.564_{.011}$

Appendix

Table 6: Accuracy and F1 of the estimator for Llama-3.1-8B at

\tau=0.9

with each instance-level classifier

g

, using each signal on its own or all five signals together, for each

\kappa

. Values are means over three random seeds, with standard errors as subscripts.

		Accuracy						F1
Classifier $g$	Signals	10	50	90	130	170	210	10	50	90	130	170	210
Baseline		.544						.705
ATC ( Garg et al., 2022 )		$.641_{.020}$	$.695_{.006}$	$.698_{.014}$	$.706_{.004}$	$.704_{.006}$	$.711_{.004}$	$.708_{.013}$	$.710_{.013}$	$.710_{.021}$	$.716_{.002}$	$.712_{.009}$	$.718_{.004}$
Meta-model ( Fu et al., 2023 )		$.586_{.022}$	$.657_{.011}$	$.683_{.011}$	$.695_{.010}$	$.711_{.015}$	$.705_{.003}$	$.590_{.034}$	$.658_{.021}$	$.684_{.016}$	$.699_{.010}$	$.724_{.017}$	$.718_{.005}$
HGB	$\mathrm{uncertainty}_{\mathrm{first}}$	$\mathbf{.664_{.027}}$	$.698_{.005}$	$.714_{.009}$	$\mathbf{.729_{.005}}$	$\mathbf{.726_{.008}}$	$\mathbf{.729_{.005}}$	$\mathbf{.726_{.022}}$	$.721_{.005}$	$.728_{.015}$	$\mathbf{.744_{.009}}$	$\mathbf{.738_{.010}}$	$\mathbf{.740_{.006}}$
	$\mathrm{uncertainty}_{\mathrm{avg}}$	$.627_{.023}$	$\mathbf{.710_{.008}}$	$\mathbf{.720_{.007}}$	$.718_{.006}$	$.715_{.004}$	$.713_{.005}$	$.700_{.022}$	$.730_{.012}$	$\mathbf{.735_{.013}}$	$.731_{.003}$	$.728_{.003}$	$.725_{.006}$

Appendix

Table 7: Accuracy and F1 of the estimator for Qwen3-8B at

\tau=0.9

with each instance-level classifier

g

, using each signal on its own or all five signals together, for each

\kappa

. Values are means over three random seeds, with standard errors as subscripts.

Figure 6: F1 of the estimator (HGB) for each LLM at

\tau=0.9

as signals are added one at a time (left to right), in three orders that each start from a different signal family; each line is one number of sampled instances

\kappa

. The dashed line is the baseline that predicts every dataset as reliable.

Language Model Performance Prediction

See all 27 papers

On the Limits of LLM Adaptability: Impact of Model-Internalized Priors on Annotation Task Performance
May 30, 2026Etienne Casanova, Rafal Kocielnik, R. Michael AlvarezLLM Evaluation Zero-Shot Text Classification
California Institute of Technology, Pasadena, CA, USA.
When Do LLMs Actually Help? Evaluating LLMs as Data Quality Annotators
Aug 13, 2026Praphulla Lal ShresthaLLM-Assisted Annotation LLM Evaluation
Independent Researcher Kathmandu, Nepal
More Human, More Efficient: Aligning Annotations with Quantized SLMs
Apr 1, 2026Jiayu Wang, Junyoung LeeLLM-Assisted Annotation LLM Fine-Tuning
Home Team Science and Technology Agency

May 30, 2026J/K move · Enter open · S save

On the Limits of LLM Adaptability: Impact of Model-Internalized Priors on Annotation Task Performance

Etienne Casanova, Rafal Kocielnik, R. Michael Alvarez

California Institute of Technology, Pasadena, CA, USA.

LLM Evaluation Zero-Shot Text Classification LLM-Assisted Annotation Language Model Robustness

Open paper AskPDF arXiv

Large Language Models (LLMs) are increasingly used for zero-shot annotation and LLM-as-a-judge tasks, yet their reliability hinges on how model-internalized priors interact with user-provided instructions. We investigate three dimensions of this interaction: (1) how an LLM's familiarity with data and task definitions affects performance, (2) the extent to which additional information in prompts can correct zero-shot errors ("decision stickiness"), and (3) model susceptibility to misaligned task definitions. Through experiments on toxicity detection across diverse datasets (spanning social media, gaming, news, and forums) using both dense and mixture-of-experts models, we find that nearly two-thirds of zero-shot errors are resistant to correction, with an overall rescue rate (fraction of initial errors corrected by prompting) of only 34.8%. High-confidence errors prove especially resistant to correction. When given misaligned definitions, LLMs follow them while maintaining confidence levels unchanged from the aligned condition. Crucially, we introduce Definition-Specific Familiarity (DSF), which measures alignment between a model's internal concept and the task definition. After controlling for dataset-level confounds, DSF shows a positive association with model performance (partial r = +0.41), while three distinct memorization metrics (ROUGE-L, BERTScore, and embedding cosine similarity) all fail to show a positive association. These findings show the limitations of prompt-based correction in annotation tasks, highlighting the importance of definition alignment over text-level memorization.

Abstract

Language Model Performance Prediction

Explore similar work

On the Limits of LLM Adaptability: Impact of Model-Internalized Priors on Annotation Task Performance

When Do LLMs Actually Help? Evaluating LLMs as Data Quality Annotators

More Human, More Efficient: Aligning Annotations with Quantized SLMs