High Scores, Hidden Failures: Evaluating Codebook-Based LLM Content Analysis

Elena Senger; Yuri Campbell; Rob van der Goot; Barbara Plank

cs.CLOct 26, 2026

High Scores, Hidden Failures: Evaluating Codebook-Based LLM Content Analysis

Authors: Elena Senger, Yuri Campbell, Rob van der Goot, Barbara Plank

Organizations: MaiNLP, Center for Information and Language Processing, LMU Munich & MCML, Germany · Fraunhofer Institute for Systems and Innovation Research ISI, Germany · Department of Computer Science, IT University of Copenhagen, Denmark

Abstract

Large language models (LLMs) are increasingly used for codebook-based content analysis in computational social science, where codebooks translate theoretical concepts into labeling schemes. While prior work reports strong predictive performance, it remains unclear whether these systems produce reliable and robust text annotations. We conduct a large-scale empirical failure-mode analysis of LLM-based coding pipelines across four datasets and five models by systematically evaluating their stability, robustness, and conceptual validity under controlled perturbations, domain shifts, and optimization procedures. We uncover multiple failure modes that are largely invisible under standard metrics, showing that high aggregate scores frequently coexist with severe label-level disparities, instability across runs, domain sensitivity, and shifts in label interpretation. Importantly, these failure modes persist under widely used optimization strategies and remain largely invisible under standard metrics such as accuracy and micro-F1. Our results demonstrate that current LLM-based coding pipelines cannot be treated as stable measurement tools without targeted validation beyond aggregate performance.

Figures & tables

Figure 1: Illustrative examples of LLM failure modes in inference and optimization hidden beneath metrics.

Dataset	Multi-Label	#Labels	#Train	#Val	#Test
CCC	$\times$	4	4,706	1,007	1,010
Minimum Wage	$\checkmark$	12	758	194	194
Value Creation Science	$\checkmark$	8	2,149	268	269
Value Creation News	$\checkmark$	8	–	–	60

Table 1: Dataset statistics. Data split size is in number of documents.

Codebook Variant	Minimum Wage					CCC					Value Creation Science
Codebook Variant	Acc	mi. F1	ma. F1	ma. P	WLF1	Acc	mi. F1	ma. F1	ma. P	WLF1	Acc	mi. F1	ma. F1	ma. P	WLF1
Unchanged (§ 4.1 , § 4.1 )	$18.2_{16.8}^{19.9}$	$59.1_{58.2}^{59.9}$	$54.8_{53.2}^{56.4}$	$47.5_{45.8}^{49.2}$	$28.9_{19.1}^{38.7}$	$33.0_{32.7}^{33.2}$	$34.0_{33.8}^{34.3}$	$33.4_{33.1}^{33.7}$	$45.6_{45.4}^{45.8}$	$6.4_{6.4}^{6.5}$	$1.9_{1.6}^{2.1}$	$44.9_{44.6}^{45.1}$	$39.6_{39.0}^{40.1}$	$30.5_{29.9}^{31.0}$	$18.8_{15.9}^{21.7}$
Generic Label (§ 4.1 )	$\underline{\mathbf{23.4_{22.0}^{24.9}}}$	$\underline{\mathbf{55.7_{55.1}^{56.2}}}$	$\underline{\mathbf{49.5_{48.5}^{50.6}}}$	$\underline{\mathbf{39.7_{38.4}^{41.0}}}$	$32.2_{24.5}^{39.9}$	$\underline{\mathbf{42.9_{42.6}^{43.1}}}$	$\underline{\mathbf{43.8_{43.5}^{44.1}}}$	$\underline{\mathbf{27.8_{27.6}^{28.0}}}$	$45.8_{45.1}^{46.5}$	$\underline{\mathbf{5.8_{5.7}^{6.0}}}$	$\mathbf{1.0_{0.7}^{1.2}}$	$\underline{\mathbf{41.6_{41.2}^{42.1}}}$	$\mathbf{38.7_{38.1}^{39.2}}$	$\mathbf{27.5_{27.1}^{27.9}}$	$19.3_{17.3}^{21.4}$
Swapped Label (§ 4.1 )	$\underline{\mathbf{2.9_{2.4}^{3.4}}}$	$\underline{\mathbf{37.7_{37.0}^{38.5}}}$	$\underline{\mathbf{29.9_{28.6}^{31.2}}}$	$\underline{\mathbf{25.2_{24.0}^{26.4}}}$	$\underline{\mathbf{1.4_{0.0}^{4.7}}}$	$\underline{\mathbf{12.3_{12.2}^{12.4}}}$	$\underline{\mathbf{12.5_{12.4}^{12.7}}}$	$\underline{\mathbf{10.6_{10.3}^{10.8}}}$	$\underline{\mathbf{17.4_{17.1}^{17.6}}}$	$\underline{\mathbf{2.5_{2.3}^{2.6}}}$	$2.0_{1.8}^{2.2}$	$\underline{\mathbf{39.9_{39.4}^{40.4}}}$	$\underline{\mathbf{35.1_{34.6}^{35.7}}}$	$\mathbf{28.5_{28.0}^{28.9}}$	$20.6_{19.6}^{21.6}$
Label-Only (§ 4.1 )	$\underline{\mathbf{6.9_{6.4}^{7.3}}}$	$\underline{\mathbf{45.7_{45.5}^{45.9}}}$	$\underline{\mathbf{42.1_{41.8}^{42.4}}}$	$\underline{\mathbf{36.2_{35.9}^{36.5}}}$	$\underline{\mathbf{14.2_{13.3}^{15.1}}}$	$\underline{\mathbf{15.9_{15.6}^{16.1}}}$	$\underline{\mathbf{16.8_{16.6}^{17.0}}}$	$\underline{\mathbf{24.0_{23.7}^{24.7}}}$	$\mathbf{46.5_{46.0}^{46.9}}$	$\underline{\mathbf{5.8_{5.8}^{5.8}}}$	$\mathbf{0.6_{4.5}^{8.1}}$	$\mathbf{43.5_{43.2}^{43.8}}$	$\mathbf{38.4_{37.9}^{38.8}}$	$\mathbf{29.2_{28.8}^{29.6}}$	$\underline{\mathbf{24.2_{22.0}^{26.4}}}$
Simple Codebook (§ 4.2 .i)	$\mathbf{21.6_{21.1}^{22.1}}$	$\mathbf{61.2_{60.7}^{61.7}}$	$\mathbf{56.8_{55.9}^{57.8}}$	$48.9_{47.9}^{49.9}$	$\underline{\mathbf{39.1_{38.0}^{40.1}}}$	$\underline{\mathbf{30.0_{26.7}^{27.2}}}$	$\underline{\mathbf{28.7_{28.4}^{28.9}}}$	$\underline{\mathbf{31.2_{31.0}^{31.4}}}$	$45.6_{45.5}^{45.8}$	$\underline{\mathbf{6.1_{5.9}^{6.3}}}$	$1.7_{1.5}^{2.0}$	$\underline{\mathbf{42.5_{42.3}^{42.8}}}$	$\underline{\mathbf{35.9_{35.5}^{36.3}}}$	$\underline{\mathbf{26.7_{26.2}^{27.2}}}$	$\underline{\mathbf{10.9_{8.6}^{13.3}}}$

Table 2: LLM-Inference results with baseline Unchanged Codebook vs. other codebook variants. We report accuracy (Acc), micro-F1 (mi. F1), macro-F1 (ma. F1), macro precision (ma. P), and worst-label-F1 (WLF1). For all metrics, we show their mean value and the 95% confidence interval over 10 runs. Bold values indicate statistically significant differences over baseline (

p<0.05

) under paired

t

-tests. Underlined values indicate significance under paired bootstrap tests. Codebook variant numbers (e.g., § 4.1 ) indicate where each variant is discussed. Results of an encoder baseline can be found in Appendix E for comparison.

Figure 2: Aggregated mean F 1 scores over 15 runs, illustrating the stabilizing effect of averaging multiple runs for each label.

Figure 3: An overview of model performance for the different datasets.

	Acc	mi. F1	macro-F1	ma. P	WLF1
Value Creation Science	$1.9_{1.6}^{2.1}$	$44.9_{44.6}^{45.1}$	$39.6_{39.0}^{40.1}$	$30.5_{29.9}^{31.0}$	$18.8_{15.9}^{21.7}$
Value Creation News	$\underline{\mathbf{0.0_{0.0}^{0.1}}}$	$\underline{\mathbf{62.2_{61.7}^{62.8}}}$	$\underline{\mathbf{59.6_{58.9}^{60.2}}}$	$\underline{\mathbf{54.1_{53.5}^{54.7}}}$	$\underline{\mathbf{32.9_{32.2}^{33.5}}}$

Table 3: Metrics over ten runs for Value Creation Science and Value Creation News , shown with their mean value and the 95% confidence interval over 10 runs. Bold values indicate statistically significant differences under paired

t

-tests (

p<0.05

). Underlined values indicate significance under paired bootstrap tests.

Optimization	Minimum Wage					CCC					Value Creation Science
Optimization	Acc	mi. F1	ma. F1	ma. P	WLF1	Acc	mi. F1	ma. F1	ma. P	WLF1	Acc	mi. F1	ma. F1	ma. P	WLF1
Unchanged Codebook	17.5	59.8	54.4	47.1	25.0	38.3	40.9	38.1	44.4	7.1	0.7	44.5	39.4	29.8	18.2
Label Names (§ 5.1 , § 5.1 )	26.8	56.0	51.0	44.8	22.2	45.4	47.2	40.1	46.3	7.4	3.4	42.4	35.3	29.3	8.5
Label Definition (§ 5.1 , § 5.1 )	19.1	60.0	55.7	46.1	28.6	52.9	55.0	44.4	45.0	13.0	1.9	45.5	39.8	30.8	14.8
Pos Clarification (§ 5.1 , § 5.1 )	21.1	58.1	53.7	46.6	25.0	35.4	37.3	35.7	44.8	7.5	2.6	44.5	39.0	29.4	12.3
Pos + Neg Clarification (§ 5.1 , § 5.1 )	24.7	61.1	57.5	53.9	40.0	45.5	47.2	40.1	46.3	7.4	2.2	44.8	40.4	30.6	23.5

Table 4: Results for different codebook adaptations across datasets. Each dataset reports accuracy (Acc), micro F1 (mi. F1), macro F1 (ma. F1), marco precision (ma. P), and worst label F1 (WLF1). For the structured experiment Mistral-7B-Instruct-v0.3 was used. GPT-4 was used for FULL, APO and SCALE experiments.

Method	Definition for the REVENUE label
Original	The benefit or reward an actor receives for contributing to value creation. Can be tangible or intangible. Examples: wages, profits, royalties, increases in social capital.
Full	The reward or advantage received by an actor for participating in value creation, either tangible or intangible. Examples: wages, profits, royalties, increases in social capital.
Apo	The benefit or compensation received by an actor for their involvement in value creation. This can be either tangible or intangible. Examples: wages, profits, royalties, increases in social capital.
Scale	The benefit or reward an actor receives for contributing to value creation. Can be tangible or intangible. Examples: wages, profits, royalties, increases in social capital, financial implications of operational changes, and benefits from educational initiatives. Clarification: Revenue can be seen as the return on investment for actors involved in the value creation process.

Table 5: Optimized versions of the label REVENUE across three approaches.

Figure 4: Overview of mean Likert scores across dimensions for each optimization method.

Dataset	Full	APO	SCALE
Minimum Wage (§ 5.3 )	1	113,200	856
CCC (§ 5.3 )	1	113,200	688
Value Creation Science (§ 5.3 )	1	113,200	1,044

Table 6: Number of API calls required for each optimization method across datasets (§ 5.3 ).

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Acc.	Micro-F1	Macro-F1	Macro-Prec.
CCC	72.67	73.95	49.51	51.32
Minimum Wage	52.58	67.67	36.65	53.29
Value Creation Science	11.15	31.38	13.46	18.21

Appendix

Table 7: Performance of the fine-tuned RoBERTa-large baseline across datasets.

Figure 5: Overview of relevance, clarity, and distinctiveness across optimization methods.

LLM-Assisted Annotation

Large language model-assisted annotation uses AI to label, classify, and structure data, often with human review to improve scale and consistency.

See all 202 papers

Can Large Language Models Replace Human Coders? Introducing ContentBench
Feb 23, 2026Michael HamanLLM-Assisted Annotation LLM Evaluation
Department of Humanities, Faculty of Economics and Management, Czech University of Life Sciences Prague, Czech Republic
Experts Rise Where LLMs Disagree: Using Cross-Model Disagreement to Target Expert Effort in LLM Codebook Revision for Large-Scale Annotation
Sep 22, 2026Zeyu He, Zhuqian Zhou, Kirk Vanacore +2LLM-Assisted Annotation Human-in-the-Loop Annotation
The Pennsylvania State University, University Park, PA, USA · Cornell University, Ithaca, NY, USA
Operational Robustness of LLMs on Code Generation
Feb 21, 2026Debalina Ghosh Paul, Hong Zhu, Ian BayleyLanguage Model Robustness LLM Code Generation
School of Engineering, Computing and Mathematics, Oxford Brookes University Oxford, OX3 0BP, United Kingdom

Feb 23, 2026J/K move · Enter open · S save

Can Large Language Models Replace Human Coders? Introducing ContentBench

Michael Haman

Department of Humanities, Faculty of Economics and Management, Czech University of Life Sciences Prague, Czech Republic

LLM-Assisted Annotation LLM Evaluation Text Classification

Open paper AskPDF arXiv

Can low-cost large language models (LLMs) take over the interpretive coding work that still anchors much of empirical content analysis? This paper introduces ContentBench, a public benchmark suite that helps answer this replacement question by tracking how much agreement low-cost LLMs achieve and what they cost on the same interpretive coding tasks. The suite uses versioned tracks that invite researchers to contribute new benchmark datasets. I report results from the first track, ContentBench-ResearchTalk v1.0: 1,000 synthetic, social-media-style posts about academic research labeled into five categories spanning praise, critique, sarcasm, questions, and procedural remarks. Reference labels are assigned only when three state-of-the-art reasoning models (GPT-5, Gemini 2.5 Pro, and Claude Opus 4.1) agree unanimously, and all final labels are checked by the author as a quality-control audit. Among the 59 evaluated models, the best low-cost LLMs reach roughly 97-99% agreement with these jury labels, far above GPT-3.5 Turbo, the model behind early ChatGPT and the initial wave of LLM-based text annotation. Several top models can code 50,000 posts for only a few dollars, pushing large-scale interpretive coding from a labor bottleneck toward questions of validation, reporting, and governance. At the same time, small open-weight models that run locally still struggle on sarcasm-heavy items (for example, Llama 3.2 3B reaches only 4% agreement on hard-sarcasm). ContentBench is released with data, documentation, and an interactive quiz at contentbench.github.io to support comparable evaluations over time and to invite community extensions.

Abstract

LLM-Assisted Annotation

Explore similar work

Can Large Language Models Replace Human Coders? Introducing ContentBench

Experts Rise Where LLMs Disagree: Using Cross-Model Disagreement to Target Expert Effort in LLM Codebook Revision for Large-Scale Annotation

Operational Robustness of LLMs on Code Generation