Political scientists are rapidly adopting large language models (LLMs) for text annotation, yet the sensitivity of annotation results to implementation choices remains poorly understood. Most evaluations test a single model or configuration; how model choice, model size, learning approach, and prompt style interact, and whether popular "best practices" survive controlled comparison, are largely unexplored. We present a controlled evaluation of these pipeline choices, testing six open-weight models across four political science annotation tasks under identical quantisation, hardware, and prompt-template conditions. Our central finding is methodological: interaction effects dominate main effects, so seemingly reasonable pipeline choices can become consequential researcher degrees of freedom. No single model, prompt style, or learning approach is uniformly superior, and the best-performing model varies across tasks. Two corollaries follow. First, model size is an unreliable guide both to cost and to performance: cross-family efficiency differences are so large that some larger models are less resource-intensive than much smaller alternatives, while within model families mid-range variants often match or exceed larger counterparts. Second, widely recommended prompt engineering techniques yield inconsistent and sometimes negative effects on annotation performance. We use these benchmark results to develop a validation-first framework - with a principled ordering of pipeline decisions, guidance on prompt freezing and held-out evaluation, reporting standards, and open-source tools - to help researchers navigate this decision space transparently.
Figures & tables
Task
Text Type
Unit of Analysis
Annotation Type(s)
N
No. Coders
Aggregation Strategy
Source
Approval
Council of EU deliberations
Debate participation
5-pt Likert
1,110
1 (25% double-coded)
Single coder
[ 55 ]
Psych. Distance
EP plenary speeches
Sentence
Binary, categorical (2 classes)
1,037
3
Majority agreement
Original
Econ. Sentiment
Newspaper articles
Article segment (first 5 sent.)
Binary
420
≥ 7
Averaged 9-pt scale, dichotomised
[ 6 ]
Manifesto Topic
NZ parl. speeches
Speech
Categorical (8 classes)
4,165
1 (validated against 3)
Single coder
[ 38 ]
Table 1: Overview of the four annotation tasks. N = number of units in the ground truth dataset.
Annotation Type
Standard Prompt Template
Binary
< section name >< section instructions >< annotation name >< tooltip > Respond with 1 if “Yes” or 0 if “No”. Return your response in JSON format, with the key “response”. < example(s) > — Text: “ < text to annotate > ” Response:
Categorical
< section name >< section instructions >< annotation name >< tooltip > Respond with < option1 > , or < option2 > , or … Return your response in JSON format, with the key “response”. < example(s) > — Text: “ < text to annotate > ” Response:
Likert scale
< section name >< section instructions >< annotation name >< tooltip > Respond with a whole number from < min value > to < max value > (inclusive), where < min value > means lowest and < max value > means highest. Return your response in JSON format, with the key “response”. < example(s) > — Text: “ < text to annotate > ” Response:
Table 2: Standardised prompt templates by annotation type.
Model
Params.
Architecture
Release
Reason.
Temp.
Top- k
Top- p
Qwen 3
32B
Dense transformer
May 2025
Yes
0.6
20
0.95
Qwen 2.5
72B
Dense transformer
Sep. 2024
No
0.8*
40*
0.9*
Gemma 3
27B
Dense transformer
Feb. 2025
No
1.0
64
0.95
GPT OSS
20B (3.6B active)
MoE transformer
Aug. 2025
Yes
1.0
40*
0.9*
DeepSeek R1
70B
Dense transformer †
Jan. 2025
Yes
0.8*
40*
0.9*
Llama 3.1
70B
Dense transformer
Jul. 2024
No
0.8*
40*
0.9*
Table 3: Comparison of the six language models used in this study. All models are run in Q4_K_M quantisation. † Distilled from DeepSeek R1 (671B MoE); the 8B and 70B variants use Llama base models, while smaller variants use Qwen 2.5 bases. * = Ollama default (parameter not specified in model card).
Figure 1: Structure of different prompt styles for LLM annotation
Figure 2: Model choice performance–energy trade-off. Points show average F1 score and energy consumption for each model across the four annotation tasks, with point size indicating model parameters.
Figure 3: Model choice performance and efficiency metrics. Panels compare average F1 score, inference time, energy consumption, and output length for each model, aggregated across tasks.
Figure 4: Model choice performance by task. Task-level F1 scores are shown for each model across the four annotation tasks.
Figure 5: Model size performance–energy trade-off within model families. Panels show average F1 score against model size for Gemma 3, DeepSeek R1, and Qwen 3 variants, with point size indicating energy consumption.
Figure 6: Model size efficiency metrics within model families. Panels compare average inference time, energy consumption, and output length across Gemma 3, DeepSeek R1, and Qwen 3 size variants.
Figure 7: Model size performance by task. Panels show task-level F1 scores for model variants within the Gemma 3, DeepSeek R1, and Qwen 3 families.
Figure 8: Few-shot performance–energy trade-off. Points show the change in average F1 score and energy consumption when moving from zero-shot to few-shot prompting for each model, with point size indicating model parameters.
Figure 9: Few-shot performance and efficiency metrics. Panels compare average F1 score, inference time, energy consumption, and output length under zero-shot and few-shot prompting across models.
Figure 10: Few-shot performance by task. Panels show task-level F1 scores for each model under zero-shot and few-shot prompting.
Figure 11: Prompt style performance–energy trade-off. The left panel shows the change in average F1 score and energy consumption from standard to persona prompting, and the right panel shows the same comparison for standard to chain-of-thought prompting.
Figure 12: Prompt style performance and efficiency metrics. Panels compare average F1 score, inference time, energy consumption, and output length across standard, persona, and chain-of-thought prompting.
Figure 13: Prompt style performance by task. Panels show task-level F1 scores for each model across standard, persona, and chain-of-thought prompting.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Task
F1
Accuracy
Precision
Recall
κ
α
DeepSeek-R1
approval
0.357
0.414
0.476
0.355
0.225
0.538
DeepSeek-R1
psych-dist
0.393
0.948
0.415
0.376
0.643
0.656
DeepSeek-R1
sentiment
0.735
0.743
0.732
0.741
0.471
0.470
DeepSeek-R1
topic-8
0.234
0.496
0.282
0.248
0.378
0.385
GPT-OSS
approval
0.427
0.477
0.481
0.418
0.285
0.571
GPT-OSS
psych-dist
0.699
0.935
0.726
0.681
0.547
0.564
Appendix
Table B.1 : Task-level performance metrics for the model choice comparison, reporting F1, accuracy, precision, recall, Cohen’s κ , and Krippendorff’s α for each model.
Model
Task
F1
Accuracy
Precision
Recall
κ
α
Deepseek-R1 1.5B
approval
0.177
0.219
0.217
0.220
0.024
-0.008
Deepseek-R1 1.5B
psych-dist
0.048
0.779
0.050
0.047
0.066
0.031
Deepseek-R1 1.5B
sentiment
0.015
0.486
0.018
0.013
0.183
-0.056
Deepseek-R1 1.5B
topic-8
0.002
0.153
0.004
0.003
0.071
0.025
Deepseek-R1 8B
approval
0.335
0.416
0.435
0.338
0.216
0.503
Deepseek-R1 8B
psych-dist
0.305
0.908
0.322
0.301
0.437
0.411
Appendix
Table B.2 : Task-level performance metrics for the model size comparison, reporting F1, accuracy, precision, recall, Cohen’s κ , and Krippendorff’s α for each model variant.
Model
Learning Approach
Task
F1
Accuracy
Precision
Recall
κ
α
DeepSeek-R1
Few-Shot
approval
0.379
0.391
0.477
0.375
0.216
0.532
DeepSeek-R1
Few-Shot
psych-dist
0.677
0.937
0.646
0.719
0.640
0.649
DeepSeek-R1
Few-Shot
sentiment
0.713
0.729
0.713
0.712
0.426
0.427
DeepSeek-R1
Few-Shot
topic-8
0.163
0.465
0.212
0.181
0.334
0.366
DeepSeek-R1
Zero-Shot
approval
0.357
0.414
0.476
0.355
0.225
0.538
DeepSeek-R1
Zero-Shot
psych-dist
0.393
0.948
0.415
0.376
0.643
0.656
Appendix
Table B.3 : Task-level performance metrics for the zero-shot and few-shot comparison, reporting F1, accuracy, precision, recall, Cohen’s κ , and Krippendorff’s α by model and learning approach.
Model
Prompt Style
Task
F1
Accuracy
Precision
Recall
κ
α
DeepSeek-R1
CoT
approval
0.429
0.433
0.520
0.418
0.261
0.563
DeepSeek-R1
CoT
psych-dist
0.626
0.905
0.594
0.678
0.516
0.499
DeepSeek-R1
CoT
sentiment
0.474
0.719
0.479
0.469
0.418
0.401
DeepSeek-R1
CoT
topic-8
0.166
0.468
0.215
0.181
0.337
0.465
DeepSeek-R1
persona
approval
0.389
0.394
0.462
0.384
0.215
0.525
DeepSeek-R1
persona
psych-dist
0.744
0.935
0.716
0.783
0.609
0.634
Appendix
Table B.4 : Task-level performance metrics for the prompt style comparison, reporting F1, accuracy, precision, recall, Cohen’s κ , and Krippendorff’s α by model and prompt style.
Choosing a Large Language Model (LLM) for a given task requires comparing many strong candidates, yet standard evaluation relies on costly annotations over fixed evaluation sets. To address this challenge, we develop SELECT-LLM, the first framework for active model selection of LLMs. SELECT-LLM aims to find a small set of queries whose annotations are most informative for identifying the best LLM for a given task. To this end, we introduce a query selection rule based on expected information gain, computed from pairwise similarities between candidate model outputs. Because this rule only uses generated model responses, SELECT-LLM can be applied across candidate models without assumptions about their architecture or access to model weights. This makes it suitable for both open-weight and black-box LLMs. We evaluate SELECT-LLM across 23 datasets, 156 evaluated models, diverse task families, and multiple text evaluation metrics. Across all experiments, SELECT-LLM improves over the strongest baseline in every setting, with annotation cost reductions up to 81.8% for best model selection and up to 84.78% for near-best model selection.
Yavuz Durmazkeser, Patrik Okanovic, Andreas Kirsch +2
Choosing a Large Language Model (LLM) for a given task requires comparing many strong candidates, yet standard evaluation relies on costly annotations over fixed evaluation sets. To address this challenge, we develop SELECT-LLM, the first framework for active model selection of LLMs. SELECT-LLM aims to find a small set of queries whose annotations are most informative for identifying the best LLM for a given task. To this end, we introduce a query selection rule based on expected information gain, computed from pairwise similarities between candidate model outputs. Because this rule only uses generated model responses, SELECT-LLM can be applied across candidate models without assumptions about their architecture or access to model weights. This makes it suitable for both open-weight and black-box LLMs. We evaluate SELECT-LLM across 23 datasets, 156 evaluated models, diverse task families, and multiple text evaluation metrics. Across all experiments, SELECT-LLM improves over the strongest baseline in every setting, with annotation cost reductions up to 81.8% for best model selection and up to 84.78% for near-best model selection.
Yavuz Durmazkeser, Patrik Okanovic, Andreas Kirsch +2
As Large Language Model (LLM) capabilities advance, the demand for high-quality annotation of exponentially increasing text corpora has outpaced human capacity, leading to the widespread adoption of LLMs in automatic evaluation and annotation. However, proprietary LLMs often exhibit systematic biases that diverge from human expert consensus, lacks reproducibility, and raises data privacy concerns. Our work examines the viability of finetuning a quantized Small Language Model of 1.7B parameter size on limited human-annotated data to serve as a highly aligned, deterministic evaluator and annotator. By implementing a custom, multi-dimensional rubric framework and simple augmentation and regularization techniques, the proposed approach achieves higher inter-annotator agreement (0.23 points increase in Krippendorff's α) than the best performing state-of-the-art proprietary LLM. We also demonstrate the generalizability of the proposed training pipeline on a separate emotion classification task. The results show that task-specific alignment and efficient 4-bit quantized fine-tuning provide superior open-source alternative to using proprietary models for evaluation and annotation. Our finetuning approach is publicly available at https://github.com/jylee-k/slm-judge.