Large language models (LLMs) are increasingly used to annotate social-science texts, but high accuracy does not establish that they apply researcher-supplied category definitions: predictions may instead reflect prior associations with category labels or labeled demonstrations. We introduce a counterfactual audit that reassigns definitions among output labels, distinguishing adherence to supplied definitions from preservation of a dataset's original mapping. Across three political-text datasets, models rarely follow reassigned definitions, and standard fine-tuning strengthens the original mapping without reliably increasing definition adherence. We then study retrieval-augmented annotation. Retrieving the nearest training examples generally outperforms retrieving examples separately within each category despite using fewer demonstrations. On the two tasks with substantial gains, direct k-nearest-neighbor classification recovers 74--93% of the improvement, indicating that neighborhood labels provide much of the predictive signal. Relabeling fixed demonstrations redirects predictions, whereas aligning demonstrations with reassigned definitions substantially increases adherence to the new mapping. Thus, accuracy-improving interventions need not increase codebook adherence; definitions, category labels, demonstrations, retrieval, and training data should be aligned and validated jointly.
Figures & tables
Condition
Definitions
Demo labels
Original
Original
Original
Reassigned definitions
Reassigned
–
Reassigned demos
–
Reassigned
Definition conflict
Reassigned
Original
Demo conflict
Original
Reassigned
Aligned
Reassigned
Reassigned
Table 1: Conditions crossing the mappings encoded by definitions and demonstrations. A dash indicates that the corresponding source is not supplied.
Dataset
Definition
Original
Other
Invalid
CCC
10.75 (4.0)
55.80 (6.4)
32.35 (6.9)
1.10 (0.5)
BFRS
12.87 (6.5)
20.93 (4.0)
65.34 (6.1)
0.86 (0.9)
Manifesto
3.24 (1.9)
30.25 (2.5)
57.94 (2.3)
8.57 (1.3)
Table 2: Zero-shot audit outcomes for Mistral-7B, in percent, across ten definition–label rearrangements (mean with standard deviation in parentheses; 1,000 documents per dataset).
Δ Definition
Δ Original
Mistral-7B
CCC
− 3.92 [ − 8.13, − 0.25]
+ 10.81 [ + 6.90, + 15.41]
BFRS
− 0.18 [ − 3.20, + 2.84]
+ 29.44 [ + 24.13, + 34.46]
Manifesto
− 0.12 [ − 0.99, + 0.73]
+ 7.78 [ + 5.89, + 9.60]
Llama-3.1-8B
CCC
− 2.97 [ − 4.80, − 1.21]
+ 9.93 [ + 7.99, + 11.88]
Table 3: Paired percentage-point changes from base to fine-tuned models across ten definition–label rearrangements, with 95% confidence intervals in brackets.
Prompt information
Manifesto
CCC
BFRS
dev/test
dev/test
dev/test
Mistral-7B
Label only
.394/.394
.647/.665
.441/.449
Codebook only
.416/.404
.639/.658
.545/.548
Global: demo only
.563/.555
.696/.710
.782/.778
Global: codebook+demo
.580/.551
.691/.677
.794/.774
Table 4: Development/test weighted F1 under the original mapping. Global conditions use 8, 10, and 20 demonstrations for Manifesto, CCC, and BFRS, respectively. Class-wise retrieval uses 98, 10, and 24 demonstrations. Matched-count Global comparisons appear in Table 11 . Bold marks the best condition within each model, dataset, and split.
Figure 1: Weighted- F1 difference between Global codebook-plus-demonstrations prompting and direct k NN using the same demonstrations. Positive values favor the LLM prompt; error bars show paired 95% confidence intervals. The numeric results are in Table 16
Original target
Reassigned target
Dataset
Orig.
Demo conflict
Reassigned demos
Def. conflict
Aligned
Mistral-7B
Manifesto
.551
.466
.439
.415
.529
CCC
.677
.515
.488
.221
.567
BFRS
.774
.455
.407
.370
.739
Llama-70B
Table 5: Test weighted F1 under original, conflicting, and aligned mappings. Targets follow the column group; Table 1 defines the conditions. Reassigned demo omits codebook information. Retrieved texts remain fixed.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Outcome
Mistral-7B
Llama-3-8B
Gemma-7B
Llama-3.3-70B
CCC
Definition
16.5
28.0
17.5
24.5
Original
56.0
45.5
29.0
57.0
Other
27.5
26.5
53.5
18.5
Invalid
0.0
0.0
0.0
0.0
BFRS
Definition
6.0
18.0
4.0
24.5
Original
15.5
32.5
8.5
45.0
Appendix
Table 6: Zero-shot audit outcomes for four models, as percentages of 200 documents under one rearrangement. Definition denotes agreement with the reassigned target; Original denotes agreement with the original human label; Other denotes another valid label; Invalid denotes an unparseable response.
Manifesto
CCC
BFRS
Document is a
sentence
article
report
Median words per document
17
444
27
Codebook entries
142
8
12
Labels ever used in the data
55
5
12
Labels scored
49
5
12
Train rows, usable
5,755
4,710
20,978
Appendix
Table 7: Dataset sizes and retrieval pools. Usable rows have human labels. Manifesto is scored on 49 retained output labels; CCC and BFRS use all observed labels. Retrieval pools contain usable training rows, restricted on Manifesto to those 49 labels, after removing texts also present in the evaluation split.
Figure 2: Weighted F1 by the number of retrieved examples k for Mistral-7B with codebook information; k=0 supplies no examples. Dotted vertical lines mark the development-selected values used for test evaluation. Shading shows the 95% bootstrap interval for development F1 at the selected value. Beyond k=8 , development F1 varies by at most 0.021 within each dataset.
Dataset
k=0
k=3
k=4
k=5
k=8
k=10
k=15
k=20
Manifesto
.416/.404
.559 ∗ /.549
.566/.557
.566/.555
.580/.551
.568/.540
.578/.542
.579/.546
CCC
.639/.658
.654 ∗ /.672
.650 ∗ /.679
.662 ∗ /.649 ∗
.671 ∗ /.687
.691/.677
.679/.685
.679/.691
BFRS
.545/.548
.765 ∗ /.744 ∗
.775 ∗ /.757 ∗
.778 ∗ /.761 ∗
.786 ∗ /.767 ∗
.789/.769
.792/.775
.794/.774
Appendix
Table 8: Values plotted in Figure 2
Condition
Manifesto
CCC
BFRS
dev/test
dev/test
dev/test
Mistral-7B
1 Label only
.394 / .394
.647 / .665
.441 / .449
2 Codebook only
.416 / .404
.639 / .658
.545 / .548
3 Global: demo only
.563 / .555
.696 / .710
.782 / .778
4 Global: labels+demo
.518 / .519
.711 / .703
.764 / .757
Appendix
Table 9: Development/test weighted F1 under the original mapping for all six prompt conditions. Conditions 3 to 5 use the selected k (8/10/20); condition 6 supplies two examples per represented category (98/10/24). Individual 95% confidence intervals are approximately ±0.03 on Manifesto and CCC and ±0.012 on BFRS. Paired bootstrap comparisons appear in Figure 3 .
Figure 3: Paired differences in weighted F1 for four prompt comparisons. Circles show development-set results and squares test-set results; positive values favor the later condition. Conditions are numbered as in Table 9 . The Manifesto comparisons that add codebook information use the same 49 output labels in both conditions.
Comparison, dataset
Dev [95% CI]
Test [95% CI]
Mistral-7B
1 → 2: add codebook, no demos
Manifesto †
−.017 [ −.045 , +.010 ]
−.007 [ −.034 , +.019 ]
CCC
−.008 [ −.035 , +.019 ]
−.007 [ −.035 , +.020 ]
BFRS
+.104∗ [ +.090 , +.118 ]
+.100∗ [ +.085 , +.114 ]
2 → 3: demos instead of codebook
Appendix
Table 10: Values plotted in Figure 3 : paired change in weighted F1 from the earlier to the later prompt condition, numbered as in Table 9 , with 95% paired bootstrap intervals (2,000 resamples). ∗ marks an interval excluding zero. † marks Manifesto comparisons in which the codebook side uses all entries, so both conditions show the same labels.
Manifesto
CCC
BFRS
dev/test
dev/test
dev/test
Examples in prompt
98
10
24
Mistral-7B
Class-wise
.538∗ / .513∗
.667∗ / .668
.738∗ / .719∗
Global, same count
.567 / .543
.691 / .677
.794 / .782
Llama-3.3-70B
Appendix
Table 11: Class-wise Example-RAG against Global Example-RAG with the same number of examples; both prompts include codebook entries. Weighted F1, development/test. ∗ marks a Class-wise score whose paired difference from Global has a 95% bootstrap interval excluding zero; every starred difference favors Global.
Prompt
Manifesto
CCC
BFRS
dev/test
dev/test
dev/test
Mistral-7B
Label only
.394 / .394
.647 / .665
.441 / .449
Global: demo only
.563 / .555
.696 / .710
.782 / .778
Global: codebook+demo
.580 / .551
.691 / .677∗
.794∗ / .774
Llama-3.1-8B
Appendix
Table 12: Three prompts on four models, weighted F1, development/test, k = 8/10/20. ∗ marks a codebook+demo score that differs from the same model’s demo-only score (95% paired bootstrap interval excludes zero). Models are Mistral-7B-Instruct-v0.2, Llama-3.1-8B-Instruct, Gemma-7B-it, and Llama-3.3-70B-Instruct.
Manifesto
CCC
BFRS
dev/test
dev/test
dev/test
Free-text answers that fail to parse (%), Mistral
1 Label only
5.1 / 6.6
0.9 / 1.0
4.1 / 3.9
2 Codebook only
6.0 / 5.6
0.8 / 0.9
5.1 / 4.7
3 Demo only
15.9 / 16.4
2.8 / 1.3
3.5 / 3.8
4 Labels+demo
3.9 / 4.7
0.5 / 0.0
1.4 / 1.3
Appendix
Table 13: Constrained decoding against free text on the same documents. Top: share of free-text answers from which no valid label can be read; constrained decoding makes this zero. Middle and bottom: weighted F1 with decoding restricted to valid labels minus weighted F1 with free text. ∗ marks a difference whose 95% paired bootstrap interval excludes zero. Conditions are numbered as in Table 9 .
Manifesto
CCC
BFRS
dev/test
dev/test
dev/test
Mistral-7B
Codebook only
.318 / .283
.382 / .437
.453 / .438
Demo only
.439 / .432
.357 / .472
.724 / .706
Codebook+demo
.457 / .396
.343 / .430
.732 / .696
Llama-3.3-70B
Appendix
Table 14: Macro-F1, development/test, giving equal weight to each category. Demo only and codebook+demo are the Global conditions. The direct k NN vote uses the same retrieved examples as the Global conditions.
Balanced
Remove
Remove
Dataset, model
Nearest
Random
random
sim. ≥.85
same source
Manifesto, Mistral
.580/.551
.432*/.396*
.414*/.382*
.575*/.549*
.518*/.483*
CCC, Mistral
.691/.677
.602*/.615*
.626*/.659
.685/.680
—
BFRS, Mistral
.794/.774
.642*/.635*
.611*/.623*
.789*/.769*
—
Manifesto, 70B
.583/.570
.485*/.490*
.482*/.481*
.581/.568
.543*/.524*
CCC, 70B
.641/.639
.606*/.610*
.638/.646
.637/.634
—
Appendix
Table 15: Development/test weighted F1 for example-selection controls with codebook information. Stars mark paired differences from nearest-neighbor retrieval whose bootstrap confidence intervals exclude zero.
Model
Dataset
Split
LLM
k NN vote
Difference [95% CI]
Mistral-7B
Manifesto
dev
.580
.538
+.042∗ [ +.012 , +.071 ]
test
.551
.515
+.037∗ [ +.009 , +.068 ]
CCC
dev
.691
.639
+.053∗ [ +.028 , +.078 ]
test
.677
.649
+.029∗ [ +.001 , +.054 ]
BFRS
dev
.794
.765
+.029∗ [ +.018 , +.040 ]
test
.774
.760
+.015∗ [ +.005 , +.025 ]
Appendix
Table 16: Values plotted in Figure 1 .
Unanimous
≥4 labels
Closest support
Dataset, model
LLM / vote
LLM / vote
LLM / vote
Manifesto, Mistral
83.1 / 93.8
48.5 / 41.9
47.6 / 28.9
Manifesto, 70B
89.2 / 93.8
50.7 / 41.9
49.2 / 28.9
CCC, Mistral
79.4 / 79.4
71.2 / 61.0
66.7 / 52.4
CCC, 70B
79.4 / 79.4
69.5 / 61.0
59.5 / 52.4
BFRS, Mistral
96.9 / 97.7
66.8 / 61.9
58.4 / 49.3
Appendix
Table 17: Development-set accuracy (%) of the Global codebook+demo prompt and of the direct k NN vote within retrieved-example subsets. Closest support is the quarter of documents with the smallest difference in summed similarity between the two leading labels. These are descriptive subsets without subgroup significance tests.
Manifesto
CCC
BFRS
Mistral
70B
Mistral
70B
Mistral
70B
Condition
dev
test
dev
test
dev
test
dev
test
dev
test
dev
test
Original codebook
Original
.580
.551
.583
.570
.691
.677
.641
.639
.794
.774
.813
.806
Demo conflict
.491
.466
.565
.543
.511
.515
.557
.566
.476
.455
.430
.430
Reassigned codebook
Appendix
Table 18: Development/test weighted F1 under the two-label reassignment. Conditions follow Table 1 ; neutral names replaces familiar category names under Definition conflict. Original and Demo conflict are scored against original human labels; reassigned-codebook conditions are scored against the reassigned target. Label-list conditions are scored against original human labels, and swapping their demonstration labels significantly lowers performance except for the 70B Manifesto comparisons. Aligned significantly exceeds Definition conflict in all twelve comparisons and falls significantly below Original except for the two 70B Manifesto comparisons. The table extends the five conditions in Table 5 with development results and additional controls.
Table 19: Development/test switch rates after the pairwise label swap (%). The denominator contains documents predicted as either swapped label before the intervention; the numerator counts predictions that move to the paired label. Swapped demos only supplies demonstrations without a label list or codebook. Demo conflict supplies the original codebook with swapped demonstrations. Weighted-F1 losses for Manifesto, CCC, and BFRS are .125/.127, .316/.326, and .364/.374 for Mistral and .133/.149, .272/.314, and .413/.407 for the 70B under swapped demos only. Corresponding losses with the codebook are .089/.086, .180/.162, .318/.319 and .018/.027, .083/.072, .384/.376; every loss is statistically significant.
Original codebook
Reassigned codebook
Demo
Definition
Dataset, model
Original
conflict
conflict
Aligned
Manifesto, Mistral
.580/.551
.267/.263
.022/.021
.512/.485
CCC, Mistral
.691/.677
.452/.451
.028/.023
.501/.497
BFRS, Mistral
.794/.774
.119/.119
.022/.025
.705/.704
Manifesto, 70B
.583/.570
.461/.447
.185/.190
.574/.536
Appendix
Table 20: Development/test weighted F1 when every codebook entry receives a different label. Original repeats the unchanged Global codebook+demo scores. Mistral scores average five reassignments; the 70B uses one.
In social science research, text annotation is a fundamental task that is traditionally a costly, hard-to-scale manual process. Researchers have used large language models (LLMs) to perform annotation by writing a definition of a concept into a prompt - a rule that determines whether a label should be attributed to a text depending on its content. However, concepts such as hate speech, fake news, and terrorism typically have several theoretically defensible competing definitions. It is also unknown whether the model actually applied the definition to provide the label or the label was based on the model's pretrained notion of the concept. To test this, we have the model annotate the same texts twice, each time with a different definition. We call the disagreement between the two sets of annotations definition sensitivity. As a control, we paraphrase one definition but leave its requirements unchanged - the disagreement between the annotations with the two wordings becomes the paraphrase floor. Across six concepts and three LLMs, a swap of competing definitions always changes more annotations, on average, than a paraphrase. This shows that the annotations depend on the prompted definition, so any set of annotations should be interpreted together with the definition that produced it. The LLM annotator's definition sensitivity and its paraphrase floor should both be reported in such work.
Large language models (LLMs) are increasingly used for annotation in computational social science, yet their methodological reliability under prompt variation remains unclear. This paper introduces Inter-Prompt Reliability (IPR), a framework for evaluating the stability of LLM outputs across semantically equivalent but linguistically varied prompts. Drawing on Inter-Rater Reliability, IPR is measured by Pairwise Agreement Rate (PAR) and its distribution to capture both consistency and stochasticity in model behavior. We evaluate this framework on two tasks with distinct properties: TREC (interpretative) and Politifact (knowledge-anchored). Results show that LLM annotation exhibits substantial stochastic variation in interpretative tasks, while appearing more stable in knowledge-based tasks. We further show that majority voting across prompts significantly improves reproducibility and reduces variance. These findings suggest that LLM prompt acts as an instrumental measurement while its wording exhibits methodological uncertainty. For future LLM-based CSS studies, we suggest that researchers move beyond single-prompt evaluation toward distributional stability and prompt aggregation within our IPR framework.
LLM benchmark labels are frozen at release and silently propagated into downstream benchmarks, errors and all. We introduce an Item Response Theory-based indicator that surfaces likely mislabels at 95% precision in the top 200 examples across seven preference and multiple-choice benchmarks using responses from 114 models, outperforming a supervised classifier. We trace these errors to mechanical labeling heuristics, upstream annotation mistakes inherited unchanged from source datasets, and fundamentally ambiguous items without a defensible single label. The same model fit reveals that reward models specialize in stylistic preference rather than factual knowledge, and identifies one frontier reward model that agrees with detected mislabels at 78% accuracy versus 38% for its peers, consistent with benchmark contamination or benchmark-specific over-optimization.