High Scores, Hidden Failures: Evaluating Codebook-Based LLM Content Analysis

Elena Senger; Yuri Campbell; Rob van der Goot; Barbara Plank

cs.CLOct 26, 2026

High Scores, Hidden Failures: Evaluating Codebook-Based LLM Content Analysis

Authors: Elena Senger, Yuri Campbell, Rob van der Goot, Barbara Plank

Organizations: MaiNLP, Center for Information and Language Processing, LMU Munich & MCML, Germany · Fraunhofer Institute for Systems and Innovation Research ISI, Germany · Department of Computer Science, IT University of Copenhagen, Denmark

Abstract

Large language models (LLMs) are increasingly used for codebook-based content analysis in computational social science, where codebooks translate theoretical concepts into labeling schemes. While prior work reports strong predictive performance, it remains unclear whether these systems produce reliable and robust text annotations. We conduct a large-scale empirical failure-mode analysis of LLM-based coding pipelines across four datasets and five models by systematically evaluating their stability, robustness, and conceptual validity under controlled perturbations, domain shifts, and optimization procedures. We uncover multiple failure modes that are largely invisible under standard metrics, showing that high aggregate scores frequently coexist with severe label-level disparities, instability across runs, domain sensitivity, and shifts in label interpretation. Importantly, these failure modes persist under widely used optimization strategies and remain largely invisible under standard metrics such as accuracy and micro-F1. Our results demonstrate that current LLM-based coding pipelines cannot be treated as stable measurement tools without targeted validation beyond aggregate performance.

Figures & tables

Figure 1: Illustrative examples of LLM failure modes in inference and optimization hidden beneath metrics.

Dataset	Multi-Label	#Labels	#Train	#Val	#Test
CCC	$\times$	4	4,706	1,007	1,010
Minimum Wage	$\checkmark$	12	758	194	194
Value Creation Science	$\checkmark$	8	2,149	268	269
Value Creation News	$\checkmark$	8	–	–	60

Table 1: Dataset statistics. Data split size is in number of documents.

Codebook Variant	Minimum Wage					CCC					Value Creation Science
Codebook Variant	Acc	mi. F1	ma. F1	ma. P	WLF1	Acc	mi. F1	ma. F1	ma. P	WLF1	Acc	mi. F1	ma. F1	ma. P	WLF1
Unchanged (§ 4.1 , § 4.1 )	$18.2_{16.8}^{19.9}$	$59.1_{58.2}^{59.9}$	$54.8_{53.2}^{56.4}$	$47.5_{45.8}^{49.2}$	$28.9_{19.1}^{38.7}$	$33.0_{32.7}^{33.2}$	$34.0_{33.8}^{34.3}$	$33.4_{33.1}^{33.7}$	$45.6_{45.4}^{45.8}$	$6.4_{6.4}^{6.5}$	$1.9_{1.6}^{2.1}$	$44.9_{44.6}^{45.1}$	$39.6_{39.0}^{40.1}$	$30.5_{29.9}^{31.0}$	$18.8_{15.9}^{21.7}$
Generic Label (§ 4.1 )	$\underline{\mathbf{23.4_{22.0}^{24.9}}}$	$\underline{\mathbf{55.7_{55.1}^{56.2}}}$	$\underline{\mathbf{49.5_{48.5}^{50.6}}}$	$\underline{\mathbf{39.7_{38.4}^{41.0}}}$	$32.2_{24.5}^{39.9}$	$\underline{\mathbf{42.9_{42.6}^{43.1}}}$	$\underline{\mathbf{43.8_{43.5}^{44.1}}}$	$\underline{\mathbf{27.8_{27.6}^{28.0}}}$	$45.8_{45.1}^{46.5}$	$\underline{\mathbf{5.8_{5.7}^{6.0}}}$	$\mathbf{1.0_{0.7}^{1.2}}$	$\underline{\mathbf{41.6_{41.2}^{42.1}}}$	$\mathbf{38.7_{38.1}^{39.2}}$	$\mathbf{27.5_{27.1}^{27.9}}$	$19.3_{17.3}^{21.4}$
Swapped Label (§ 4.1 )	$\underline{\mathbf{2.9_{2.4}^{3.4}}}$	$\underline{\mathbf{37.7_{37.0}^{38.5}}}$	$\underline{\mathbf{29.9_{28.6}^{31.2}}}$	$\underline{\mathbf{25.2_{24.0}^{26.4}}}$	$\underline{\mathbf{1.4_{0.0}^{4.7}}}$	$\underline{\mathbf{12.3_{12.2}^{12.4}}}$	$\underline{\mathbf{12.5_{12.4}^{12.7}}}$	$\underline{\mathbf{10.6_{10.3}^{10.8}}}$	$\underline{\mathbf{17.4_{17.1}^{17.6}}}$	$\underline{\mathbf{2.5_{2.3}^{2.6}}}$	$2.0_{1.8}^{2.2}$	$\underline{\mathbf{39.9_{39.4}^{40.4}}}$	$\underline{\mathbf{35.1_{34.6}^{35.7}}}$	$\mathbf{28.5_{28.0}^{28.9}}$	$20.6_{19.6}^{21.6}$
Label-Only (§ 4.1 )	$\underline{\mathbf{6.9_{6.4}^{7.3}}}$	$\underline{\mathbf{45.7_{45.5}^{45.9}}}$	$\underline{\mathbf{42.1_{41.8}^{42.4}}}$	$\underline{\mathbf{36.2_{35.9}^{36.5}}}$	$\underline{\mathbf{14.2_{13.3}^{15.1}}}$	$\underline{\mathbf{15.9_{15.6}^{16.1}}}$	$\underline{\mathbf{16.8_{16.6}^{17.0}}}$	$\underline{\mathbf{24.0_{23.7}^{24.7}}}$	$\mathbf{46.5_{46.0}^{46.9}}$	$\underline{\mathbf{5.8_{5.8}^{5.8}}}$	$\mathbf{0.6_{4.5}^{8.1}}$	$\mathbf{43.5_{43.2}^{43.8}}$	$\mathbf{38.4_{37.9}^{38.8}}$	$\mathbf{29.2_{28.8}^{29.6}}$	$\underline{\mathbf{24.2_{22.0}^{26.4}}}$
Simple Codebook (§ 4.2 .i)	$\mathbf{21.6_{21.1}^{22.1}}$	$\mathbf{61.2_{60.7}^{61.7}}$	$\mathbf{56.8_{55.9}^{57.8}}$	$48.9_{47.9}^{49.9}$	$\underline{\mathbf{39.1_{38.0}^{40.1}}}$	$\underline{\mathbf{30.0_{26.7}^{27.2}}}$	$\underline{\mathbf{28.7_{28.4}^{28.9}}}$	$\underline{\mathbf{31.2_{31.0}^{31.4}}}$	$45.6_{45.5}^{45.8}$	$\underline{\mathbf{6.1_{5.9}^{6.3}}}$	$1.7_{1.5}^{2.0}$	$\underline{\mathbf{42.5_{42.3}^{42.8}}}$	$\underline{\mathbf{35.9_{35.5}^{36.3}}}$	$\underline{\mathbf{26.7_{26.2}^{27.2}}}$	$\underline{\mathbf{10.9_{8.6}^{13.3}}}$

Table 2: LLM-Inference results with baseline Unchanged Codebook vs. other codebook variants. We report accuracy (Acc), micro-F1 (mi. F1), macro-F1 (ma. F1), macro precision (ma. P), and worst-label-F1 (WLF1). For all metrics, we show their mean value and the 95% confidence interval over 10 runs. Bold values indicate statistically significant differences over baseline (

p<0.05

) under paired

t

-tests. Underlined values indicate significance under paired bootstrap tests. Codebook variant numbers (e.g., § 4.1 ) indicate where each variant is discussed. Results of an encoder baseline can be found in Appendix E for comparison.

Figure 2: Aggregated mean F 1 scores over 15 runs, illustrating the stabilizing effect of averaging multiple runs for each label.

Figure 3: An overview of model performance for the different datasets.

	Acc	mi. F1	macro-F1	ma. P	WLF1
Value Creation Science	$1.9_{1.6}^{2.1}$	$44.9_{44.6}^{45.1}$	$39.6_{39.0}^{40.1}$	$30.5_{29.9}^{31.0}$	$18.8_{15.9}^{21.7}$
Value Creation News	$\underline{\mathbf{0.0_{0.0}^{0.1}}}$	$\underline{\mathbf{62.2_{61.7}^{62.8}}}$	$\underline{\mathbf{59.6_{58.9}^{60.2}}}$	$\underline{\mathbf{54.1_{53.5}^{54.7}}}$	$\underline{\mathbf{32.9_{32.2}^{33.5}}}$

Table 3: Metrics over ten runs for Value Creation Science and Value Creation News , shown with their mean value and the 95% confidence interval over 10 runs. Bold values indicate statistically significant differences under paired

t

-tests (

p<0.05

). Underlined values indicate significance under paired bootstrap tests.

Optimization	Minimum Wage					CCC					Value Creation Science
Optimization	Acc	mi. F1	ma. F1	ma. P	WLF1	Acc	mi. F1	ma. F1	ma. P	WLF1	Acc	mi. F1	ma. F1	ma. P	WLF1
Unchanged Codebook	17.5	59.8	54.4	47.1	25.0	38.3	40.9	38.1	44.4	7.1	0.7	44.5	39.4	29.8	18.2
Label Names (§ 5.1 , § 5.1 )	26.8	56.0	51.0	44.8	22.2	45.4	47.2	40.1	46.3	7.4	3.4	42.4	35.3	29.3	8.5
Label Definition (§ 5.1 , § 5.1 )	19.1	60.0	55.7	46.1	28.6	52.9	55.0	44.4	45.0	13.0	1.9	45.5	39.8	30.8	14.8
Pos Clarification (§ 5.1 , § 5.1 )	21.1	58.1	53.7	46.6	25.0	35.4	37.3	35.7	44.8	7.5	2.6	44.5	39.0	29.4	12.3
Pos + Neg Clarification (§ 5.1 , § 5.1 )	24.7	61.1	57.5	53.9	40.0	45.5	47.2	40.1	46.3	7.4	2.2	44.8	40.4	30.6	23.5

Table 4: Results for different codebook adaptations across datasets. Each dataset reports accuracy (Acc), micro F1 (mi. F1), macro F1 (ma. F1), marco precision (ma. P), and worst label F1 (WLF1). For the structured experiment Mistral-7B-Instruct-v0.3 was used. GPT-4 was used for FULL, APO and SCALE experiments.

Method	Definition for the REVENUE label
Original	The benefit or reward an actor receives for contributing to value creation. Can be tangible or intangible. Examples: wages, profits, royalties, increases in social capital.
Full	The reward or advantage received by an actor for participating in value creation, either tangible or intangible. Examples: wages, profits, royalties, increases in social capital.
Apo	The benefit or compensation received by an actor for their involvement in value creation. This can be either tangible or intangible. Examples: wages, profits, royalties, increases in social capital.
Scale	The benefit or reward an actor receives for contributing to value creation. Can be tangible or intangible. Examples: wages, profits, royalties, increases in social capital, financial implications of operational changes, and benefits from educational initiatives. Clarification: Revenue can be seen as the return on investment for actors involved in the value creation process.

Table 5: Optimized versions of the label REVENUE across three approaches.

Figure 4: Overview of mean Likert scores across dimensions for each optimization method.

Dataset	Full	APO	SCALE
Minimum Wage (§ 5.3 )	1	113,200	856
CCC (§ 5.3 )	1	113,200	688
Value Creation Science (§ 5.3 )	1	113,200	1,044

Table 6: Number of API calls required for each optimization method across datasets (§ 5.3 ).

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Acc.	Micro-F1	Macro-F1	Macro-Prec.
CCC	72.67	73.95	49.51	51.32
Minimum Wage	52.58	67.67	36.65	53.29
Value Creation Science	11.15	31.38	13.46	18.21

Appendix

Table 7: Performance of the fine-tuned RoBERTa-large baseline across datasets.

Figure 5: Overview of relevance, clarity, and distinctiveness across optimization methods.

LLM-Assisted Annotation

Large language model-assisted annotation uses AI to label, classify, and structure data, often with human review to improve scale and consistency.

See all 202 papers

Feb 23, 2026

Can Large Language Models Replace Human Coders? Introducing ContentBench

Can low-cost large language models (LLMs) take over the interpretive coding work that still anchors much of empirical content analysis? This paper introduces ContentBench, a public benchmark suite that helps answer this replacement question by tracking how much agreement low-cost LLMs achieve and what they cost on the same interpretive coding tasks. The suite uses versioned tracks that invite researchers to contribute new benchmark datasets. I report results from the first track, ContentBench-ResearchTalk v1.0: 1,000 synthetic, social-media-style posts about academic research labeled into five categories spanning praise, critique, sarcasm, questions, and procedural remarks. Reference labels are assigned only when three state-of-the-art reasoning models (GPT-5, Gemini 2.5 Pro, and Claude Opus 4.1) agree unanimously, and all final labels are checked by the author as a quality-control audit. Among the 59 evaluated models, the best low-cost LLMs reach roughly 97-99% agreement with these jury labels, far above GPT-3.5 Turbo, the model behind early ChatGPT and the initial wave of LLM-based text annotation. Several top models can code 50,000 posts for only a few dollars, pushing large-scale interpretive coding from a labor bottleneck toward questions of validation, reporting, and governance. At the same time, small open-weight models that run locally still struggle on sarcasm-heavy items (for example, Llama 3.2 3B reaches only 4% agreement on hard-sarcasm). ContentBench is released with data, documentation, and an interactive quiz at contentbench.github.io to support comparable evaluations over time and to invite community extensions.

Michael Haman

Department of Humanities, Faculty of Economics and Management, Czech University of Life Sciences Prague, Czech Republic

LLM-Assisted Annotation LLM Evaluation Text Classification

Ask PDF

Sep 22, 2026

Experts Rise Where LLMs Disagree: Using Cross-Model Disagreement to Target Expert Effort in LLM Codebook Revision for Large-Scale Annotation

Large-scale text annotation brings expert insight to millions of documents, often through a codebook that AI annotators follow. Developing a robust codebook, however, takes months. Large language models (LLMs) could speed this process by applying an early codebook to the data, surfacing cases with strong LLM disagreement, and eliciting expert feedback to address them. We examined three ways experts can provide feedback for LLM codebook revision: (i) editing LLM-generated revisions driven by cross-LLM disagreement (Codebook Verifying), (ii) answering questions about LLM disagreements (Question Answering), and (iii) labeling disagreement cases with rationales (Rationale Labeling). Experiments on thousands of tutoring-session transcripts show that Rationale Labeling yielded the highest LLM-labeling accuracy (64.9%) against expert labels, outperforming the expert-revised codebook (57.8%). The best Question Answering setting also outperformed it (60.5%). Our work shows that LLMs can be used to strategically target expert attention, shortening months of codebook revision to days without sacrificing labeling performance.

Zeyu He, Zhuqian Zhou, Kirk Vanacore +2

The Pennsylvania State University, University Park, PA, USA · Cornell University, Ithaca, NY, USA

LLM-Assisted Annotation Human-in-the-Loop Annotation

Ask PDF

Feb 21, 2026

Operational Robustness of LLMs on Code Generation

It is now common practice in software development for large language models (LLMs) to be used to generate program code. It is desirable to evaluate the robustness of LLMs for this usage. This paper is concerned in particular with how sensitive LLMs are to variations in descriptions of the coding tasks. However, existing techniques for evaluating this robustness are unsuitable for code generation because the input data space of natural language descriptions is discrete. To address this problem, we propose a robustness evaluation method called scenario domain analysis, which aims to find the expected minimal change in the natural language descriptions of coding tasks that would cause the LLMs to produce incorrect outputs. We have formally proved the theoretical properties of the method and also conducted extensive experiments to evaluate the robustness of four state-of-the-art art LLMs: Gemini-pro, Codex, Llamma2 and Falcon 7B, and have found that we are able to rank these with confidence from best to worst. Moreover, we have also studied how robustness varies in different scenarios, including the variations with the topic of the coding task and with the complexity of its sample solution, and found that robustness is lower for more complex tasks and also lower for more advanced topics, such as multi-threading and data structures.

Debalina Ghosh Paul, Hong Zhu, Ian Bayley

School of Engineering, Computing and Mathematics, Oxford Brookes University Oxford, OX3 0BP, United Kingdom

Language Model Robustness LLM Code Generation Code Generation Evaluation

Ask PDF

Abstract

LLM-Assisted Annotation

Explore similar work

Can Large Language Models Replace Human Coders? Introducing ContentBench

Experts Rise Where LLMs Disagree: Using Cross-Model Disagreement to Target Expert Effort in LLM Codebook Revision for Large-Scale Annotation

Operational Robustness of LLMs on Code Generation