When F1 Is Not Enough: Auditing Codebooks for LLM-Assisted Annotation
Organizations: Center for Social Data Analytics, Pennsylvania State University, United States · Department of Statistics, Pennsylvania State University, United States
Abstract
Large language models (LLMs) are increasingly used for text annotation in social science, but standard performance metrics do not explain why errors occur. Low F1 may reflect limitations of the annotation system, defects in the written codebook, or inconsistency in reference labels. We propose a pre-deployment codebook audit for LLM-assisted annotation. The audit uses class-specific metrics and confusion matrices to identify problematic classes and boundaries, structured document-level review to attribute error sources, and targeted codebook revision followed by held-out evaluation. We apply the audit to two published multi-class codebooks from distinct domains. In both applications, the audit identifies under-specified or conflicting coding rules, and targeted revisions reduce the diagnosed errors on held-out data. Analyses with two LLM annotators show that some problematic boundaries and codebook revisions transfer across models, whereas other revision effects are annotator-specific. The results show that LLM annotation errors can help identify weaknesses in the written measurement instrument, not only in the model. Codebook auditing complements prompting, retrieval, and fine-tuning by clarifying whether persistent errors call for model improvement, codebook revision, or renewed scrutiny of reference labels.
Figures & tables
Appendix figures & tables31 assets
Supplementary material from the paper’s appendix.
Appendix
| Class | precision | recall | F1 | 95% CI | |
| drone_assassination | 43 | 1.00 | 1.00 | 1.000 | |
| attack_on_state | 50 | 0.89 | 0.96 | 0.923 | |
| threat_of_violence | 30 | 0.96 | 0.70 | 0.808 | |
| violent_political_demonstration | 50 | 0.71 | 0.92 | 0.800 | |
| guerilla_attack_on_gov_forces | 50 | 0.87 | 0.68 | 0.764 | |
| gov_attack_on_nonstate_combatants | 50 | 0.68 | 0.78 | 0.729 |
| Quantity | Estimate | 95% CI |
|---|---|---|
| BFRS (dev, ) | ||
| Level gap: F1( gov_attack_on_civilians ) F1( other ) | ||
| Asymmetry : gov_attack_on_civilians | ||
| Asymmetry : riot | ||
| Asymmetry : threat_of_violence | ||
| Asymmetry : conventional_attack_on_gov_forces | ||
| Class | precision | recall | F1 | 95% CI | |
| drone_assassination | 43 | 1.00 | 1.00 | 1.000 | |
| attack_on_state | 50 | 0.89 | 0.96 | 0.923 | |
| threat_of_violence | 30 | 0.95 | 0.70 | 0.808 | |
| violent_political_demonstration | 50 | 0.69 | 0.90 | 0.783 | |
| guerilla_attack_on_gov_forces | 50 | 0.89 | 0.68 | 0.773 | |
| gov_attack_on_nonstate_combatants | 50 | 0.71 | 0.80 | 0.755 |
| Class | precision | recall | F1 | 95% CI | |
| drone_assassination | 43 | 1.00 | 1.00 | 1.000 | |
| attack_on_state | 50 | 0.89 | 0.96 | 0.923 | |
| threat_of_violence | 30 | 0.95 | 0.70 | 0.808 | |
| guerilla_attack_on_gov_forces | 50 | 0.88 | 0.72 | 0.791 | |
| violent_political_demonstration | 50 | 0.68 | 0.90 | 0.776 | |
| riot | 50 | 0.87 | 0.68 | 0.764 |
| Class | |||||||
|---|---|---|---|---|---|---|---|
| other | 50 | 0.39 | 0.42 | 0.46 | 0.52 | 0.53 | 0.62 |
| gov_attack_on_civilians | 50 | 0.85 | 0.46 | 0.88 | 0.42 | 0.91 | 0.40 |
| assassination | 50 | 0.60 | 0.64 | 0.65 | 0.66 | 0.67 | 0.70 |
| terrorism | 50 | 0.58 | 0.70 | 0.62 | 0.70 | 0.67 | 0.68 |
| riot | 50 | 0.87 | 0.54 | 0.87 | 0.66 | 0.87 | 0.68 |
| Class | |||||||
|---|---|---|---|---|---|---|---|
| other | 100 | 0.44 | 0.42 | 0.51 | 0.47 | 0.54 | 0.53 |
| gov_attack_on_civilians | 78 | 0.81 | 0.33 | 0.80 | 0.26 | 0.83 | 0.26 |
| threat_of_violence | 32 | 1.00 | 0.56 | 1.00 | 0.56 | 1.00 | 0.56 |
| terrorism | 100 | 0.63 | 0.84 | 0.63 | 0.83 | 0.63 | 0.81 |
| riot | 100 | 0.80 | 0.66 | 0.80 | 0.75 | 0.80 | 0.74 |
| F1 | ( ) | ( ) | |
|---|---|---|---|
| Dev ( , /class) | |||
| other | ( ) | ( ) | |
| riot | ( ) | ( ) | |
| assassination | ( ) | ( ) | |
| macro | ( ) | ( ) | |
| weighted | ( ) | ( ) | |
| Category | precision | recall | F1 | 95% CI | |
|---|---|---|---|---|---|
| Blame-shifting | 22 | 0.83 | 0.68 | 0.750 | |
| Claiming | 50 | 0.75 | 0.72 | 0.735 | |
| Descriptive | 50 | 0.59 | 0.82 | 0.683 | |
| Sharing | 50 | 0.81 | 0.58 | 0.674 |
| Quantity | Estimate | 95% CI |
|---|---|---|
| Credit claiming (dev, ) | ||
| Level gap: F1(Descriptive) F1(Sharing) | ||
| Asymmetry : Descriptive | ||
| Asymmetry : Sharing | ||
| Asymmetry : Blame-shifting | ||
| Asymmetry : Claiming | ||
| Category | precision | recall | F1 | 95% CI | |
|---|---|---|---|---|---|
| Sharing | 50 | 0.87 | 0.66 | 0.750 | |
| Blame-shifting | 22 | 0.83 | 0.68 | 0.750 | |
| Claiming | 50 | 0.76 | 0.74 | 0.747 | |
| Descriptive (residual) | 50 | 0.60 | 0.80 | 0.684 |
| Class | |||||
| Sharing | 50 | 0.81 | 0.58 | 0.87 | 0.66 |
| Descriptive (residual) | 50 | 0.59 | 0.82 | 0.60 | 0.80 |
| Claiming | 50 | 0.75 | 0.72 | 0.76 | 0.74 |
| Blame-shifting | 22 | 0.83 | 0.68 | 0.83 | 0.68 |
| Notes. At Credit-sharing is nominally the lowest-F1 class ( against Descriptive’s , a gap of ), while Descriptive shows the strongest over-prediction pattern: it has the lowest precision ( ) and highest recall ( ) in the table, triggering the asymmetry criterion. The revision trades that recall for precision by design: at Credit-sharing rises to while Descriptive stays flat ( ) and remains the residual. | |||||
| Class | |||||
|---|---|---|---|---|---|
| Sharing | 100 | 0.66 | 0.53 | 0.67 | 0.64 |
| Descriptive (residual) | 100 | 0.58 | 0.67 | 0.62 | 0.59 |
| Blame-shifting | 21 | 0.65 | 0.71 | 0.67 | 0.76 |
| Claiming | 100 | 0.70 | 0.72 | 0.73 | 0.77 |
| F1 | ( ) | |
|---|---|---|
| Dev ( ) | ||
| Descriptive | ( ) | |
| Claiming | ( ) | |
| Sharing | ( ) | |
| Blame-shifting | ( ) | |
| macro | ( ) | |
| Class | id ( CI) | ||||
|---|---|---|---|---|---|
| other | 50 | ||||
| guerilla_attack_on_gov_forces | 50 | ||||
| terrorism | 50 | ||||
| conventional_attack_on_gov_forces | 50 | ||||
| assassination | 50 | ||||
| attack_on_state | 50 |
| Class | id ( CI) | |||
|---|---|---|---|---|
| Blame-shifting | 22 | |||
| Claiming | 50 | |||
| Descriptive | 50 | |||
| Sharing | 50 | |||
| macro | 172 |
| ft (fine-tuned) | full codebook (no RAG) | base (+RAG) | other_rev (+RAG) | other_assass (+RAG) | ||||||||||||
| Category | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | |
| drone_assassination | 43 | 1.000 | 1.00 | 1.00 | 1.000 | 1.00 | 1.00 | 1.000 | 1.00 | 1.00 | 1.000 | 1.00 | 1.00 | 1.000 | 1.00 | 1.00 |
| attack_on_state | 50 | 0.970 | 0.96 | 0.98 | 0.923 | 0.89 | 0.96 | 0.923 | 0.89 | 0.96 | 0.923 | 0.89 | 0.96 | 0.923 | 0.89 | 0.96 |
| riot | 50 | 0.854 | 0.83 | 0.88 | 0.709 | 0.97 | 0.56 | 0.667 | 0.87 | 0.54 | 0.750 | 0.87 | 0.66 | 0.764 | 0.87 | 0.68 |
| guerilla_attack_on_gov_forces | 50 | 0.851 | 0.91 | 0.80 | 0.760 | 0.76 | 0.76 | 0.764 | 0.87 | 0.68 | 0.773 | 0.89 | 0.68 | 0.791 | 0.88 | 0.72 |
| gov_attack_on_nonstate_combatants | 50 | 0.841 | 0.79 | 0.90 | 0.702 | 0.75 | 0.66 | 0.729 | 0.68 | 0.78 | 0.755 | 0.71 | 0.80 | 0.757 | 0.74 | 0.78 |
| Codebook | Audit decision | Reproduced |
|---|---|---|
| BFRS | other (lowest F1) | |
| Localize: gold other flows to assassination | ||
| Credit-claim | Descriptive (largest recall-over-precision excess) | |
| Localize: Descriptive receives more than it loses |
| F1 | base | other_rev ( ) | other_assass ( ) |
|---|---|---|---|
| Dev ( , /class) | |||
| other | ( ) | ( ) | |
| riot | ( ) | ( ) | |
| assassination | ( ) | ( ) | |
| macro | ( ) | ( ) | |
| weighted | ( ) | ( ) | |
| Class | P | R | F1 | P | R | F1 | P | R | F1 | |
|---|---|---|---|---|---|---|---|---|---|---|
| drone_assassination | 43 | 1.00 | 1.00 | 1.000 | 1.00 | 1.00 | 1.000 | 1.00 | 1.00 | 1.000 |
| attack_on_state | 50 | 0.94 | 0.96 | 0.950 | 0.94 | 0.96 | 0.950 | 0.94 | 0.96 | 0.950 |
| threat_of_violence | 30 | 0.96 | 0.73 | 0.830 | 0.96 | 0.73 | 0.830 | 0.96 | 0.73 | 0.830 |
| violent_political_demonstration | 50 | 0.80 | 0.80 | 0.800 | 0.80 | 0.78 | 0.788 | 0.80 | 0.78 | 0.788 |
| riot | 50 | 0.78 | 0.76 | 0.768 | 0.75 | 0.82 | 0.781 | 0.75 | 0.82 | 0.781 |
| Class | P | R | F1 | P | R | F1 | P | R | F1 | |
|---|---|---|---|---|---|---|---|---|---|---|
| drone_assassination | 35 | 1.00 | 0.97 | 0.986 | 0.97 | 0.97 | 0.971 | 0.97 | 0.97 | 0.971 |
| attack_on_state | 100 | 0.93 | 0.96 | 0.946 | 0.93 | 0.96 | 0.946 | 0.93 | 0.96 | 0.946 |
| violent_political_demonstration | 100 | 0.78 | 0.83 | 0.802 | 0.79 | 0.85 | 0.821 | 0.79 | 0.85 | 0.817 |
| gov_attack_on_nonstate_combatants | 100 | 0.75 | 0.82 | 0.781 | 0.75 | 0.85 | 0.798 | 0.75 | 0.86 | 0.800 |
| riot | 100 | 0.76 | 0.76 | 0.760 | 0.78 | 0.89 | 0.832 | 0.79 | 0.89 | 0.840 |
| Class | P | R | F1 | P | R | F1 | F1 ( CI) | |
|---|---|---|---|---|---|---|---|---|
| Blame-shifting | 22 | 0.69 | 0.91 | 0.784 | 0.71 | 0.91 | 0.800 | |
| Claiming | 50 | 0.67 | 0.80 | 0.727 | 0.69 | 0.84 | 0.757 | |
| Descriptive | 50 | 0.70 | 0.64 | 0.667 | 0.71 | 0.64 | 0.674 | |
| Sharing | 50 | 0.73 | 0.54 | 0.621 | 0.74 | 0.56 | 0.636 | |
| F1 | ( vs. ) | ( vs. ) | vs. | |
|---|---|---|---|---|
| Llama-3.3-70B (deployment) | ||||
| Dev ( ) | ||||
| Sharing | ( ) | ( ) | ||
| Claiming | ( ) | ( ) | ||
| Blame-shifting | ( ) | ( ) | ||
| Descriptive | ( ) | ( ) | ||
LLM-Assisted Annotation
Large language model-assisted annotation uses AI to label, classify, and structure data, often with human review to improve scale and consistency.
See all 217 papers