In social science research, text annotation is a fundamental task that is traditionally a costly, hard-to-scale manual process. Researchers have used large language models (LLMs) to perform annotation by writing a definition of a concept into a prompt - a rule that determines whether a label should be attributed to a text depending on its content. However, concepts such as hate speech, fake news, and terrorism typically have several theoretically defensible competing definitions. It is also unknown whether the model actually applied the definition to provide the label or the label was based on the model's pretrained notion of the concept. To test this, we have the model annotate the same texts twice, each time with a different definition. We call the disagreement between the two sets of annotations definition sensitivity. As a control, we paraphrase one definition but leave its requirements unchanged - the disagreement between the annotations with the two wordings becomes the paraphrase floor. Across six concepts and three LLMs, a swap of competing definitions always changes more annotations, on average, than a paraphrase. This shows that the annotations depend on the prompted definition, so any set of annotations should be interpreted together with the definition that produced it. The LLM annotator's definition sensitivity and its paraphrase floor should both be reported in such work.
Figures & tables
Figure 1: Study design. Competing definitions of one concept, and paraphrases of one definition with the rule preserved, are each paired with the same query in a separate prompt. The same model annotates the same texts with every prompt and returns include , exclude , or indeterminate . The mean pairwise disagreement over cross-definition pairs is definition sensitivity, and the mean over wording pairs of one definition is the paraphrase floor. We report their difference as a gap and their quotient as a ratio.
Definition
Paraphrase
Concept
Model
sensitivity
floor
Gap
Ratio
Protest
Gemma 4 31B
39.5
5.6
33.8
7.0
Qwen 3.6 27B
44.1
8.7
35.4
5.1
Claude Sonnet 5
47.1
12.0
35.1
3.9
Assassination
Gemma 4 31B
53.7
11.7
42.0
4.6
Qwen 3.6 27B
40.3
15.2
25.1
2.7
Table 1: Definition sensitivity against the paraphrase floor, by concept and model. Each sensitivity and floor cell is the share of texts on which two annotations disagree. The rate is averaged over pairs of competing definitions for sensitivity, and over pairs of wordings of one definition for the floor. Gap is definition sensitivity minus the floor, in percentage points. Ratio is definition sensitivity divided by the floor.
Concept
Pair
Definitions
Gemma
Qwen
Sonnet
Protest
Closest
CCC 2021, CCC 2024
9.7
13.0
20.3
Farthest
CCC 2021, Cross-National Time-Series Data Archive (CNTS)
60.8
71.9
70.3
Assassination
Closest
GTD, GTD with successful attacks only
25.4
19.5
29.2
Farthest
GTD, International Committee of the Red Cross
64.2
48.1
57.1
Terrorism
Closest
Enders , GTD
32.0
18.4
24.1
Farthest
Schmid , Weinberg et al.
58.1
27.0
52.3
Table 2: Disagreement rate of the closest and the farthest pair of definitions in each concept, in percent of the texts. Closest and farthest are by the mean rate across the three models.
Text
Labels
Difference
Reasons for exclude
An event note on about a dozen people who protested outside a high school to ask a school board member to resign.
ACLED include , MMPD exclude
ACLED counts a demonstration of three or more participants. MMPD requires 50 or more people.
All three cite the minimum of 50 people.
An event note on dozens of nurses who picketed a hospital at the start of a strike for staffing and pay.
CCC 2021 include , CCC 2024 exclude
The 2024 edition no longer covers labour strikes or the pickets of striking workers.
All three cite the exclusion of workers’ pickets.
A news report on militants who tried to kill a government minister. The minister was stabbed but survived.
Global Terrorism Database (GTD) include , GTD with successful attacks only exclude
The second definition counts assassinations only when the target is killed.
All three cite the survival of the minister.
A news report on gunmen who shot dead a senior army general outside his home. A newly formed militant group claimed responsibility for the attack online.
Weinberg et al. include , Schmid exclude
Schmid states that the main direct victims of terrorism are in general civilians, not armed forces. Weinberg et al. do not distinguish combatants from non-combatants.
All three cite the military role of the victim.
Table 3: Four texts on which all three models changed their label with the definition. Each row names the clause that differs and what the models’ rationales cite for the exclude label.
include share
indeterminate share
Concept
Gemma
Qwen
Sonnet
Gemma
Qwen
Sonnet
Protest
9.1–68.6
5.5–66.1
8.8–72.0
18.6–63.2
10.1–56.4
8.6–41.8
Assassination
22.8–68.0
13.4–41.1
25.6–60.5
10.3–22.4
2.3–13.1
7.7–19.4
Terrorism
8.7–23.2
11.9–28.3
29.0–52.5
17.1–71.2
8.2–20.9
10.7–47.0
Hate speech
0.3–2.8
0.3–2.3
0.7–3.3
0.0–1.5
1.8–5.2
0.0–0.5
Climate contrarianism
18.0–30.7
13.8–25.2
28.0–46.8
4.0–19.3
5.5–35.5
10.3–12.5
Table 4: Range of the include and indeterminate shares across the definitions of each concept, in percent of the texts, for the original wording of each definition.
Figure 2: Definitions change label shares on the same texts. In the left panel, the share of the 3,249 texts labelled include is shown for each protest definition. The right panel shows the change in this share per model from CCC 2021 to CCC 2024, in percentage points.
Large language models (LLMs) are increasingly used to annotate social-science texts, but high accuracy does not establish that they apply researcher-supplied category definitions: predictions may instead reflect prior associations with category labels or labeled demonstrations. We introduce a counterfactual audit that reassigns definitions among output labels, distinguishing adherence to supplied definitions from preservation of a dataset's original mapping. Across three political-text datasets, models rarely follow reassigned definitions, and standard fine-tuning strengthens the original mapping without reliably increasing definition adherence. We then study retrieval-augmented annotation. Retrieving the nearest training examples generally outperforms retrieving examples separately within each category despite using fewer demonstrations. On the two tasks with substantial gains, direct k-nearest-neighbor classification recovers 74--93% of the improvement, indicating that neighborhood labels provide much of the predictive signal. Relabeling fixed demonstrations redirects predictions, whereas aligning demonstrations with reassigned definitions substantially increases adherence to the new mapping. Thus, accuracy-improving interventions need not increase codebook adherence; definitions, category labels, demonstrations, retrieval, and training data should be aligned and validated jointly.
Viraj Mishra, Shubham Gupta, Dhruv Gupta +2
Department of Computer Science and Engineering, Pennsylvania State University · Department of Statistics, Pennsylvania State University
Large language models (LLMs) are increasingly used for annotation in computational social science, yet their methodological reliability under prompt variation remains unclear. This paper introduces Inter-Prompt Reliability (IPR), a framework for evaluating the stability of LLM outputs across semantically equivalent but linguistically varied prompts. Drawing on Inter-Rater Reliability, IPR is measured by Pairwise Agreement Rate (PAR) and its distribution to capture both consistency and stochasticity in model behavior. We evaluate this framework on two tasks with distinct properties: TREC (interpretative) and Politifact (knowledge-anchored). Results show that LLM annotation exhibits substantial stochastic variation in interpretative tasks, while appearing more stable in knowledge-based tasks. We further show that majority voting across prompts significantly improves reproducibility and reduces variance. These findings suggest that LLM prompt acts as an instrumental measurement while its wording exhibits methodological uncertainty. For future LLM-based CSS studies, we suggest that researchers move beyond single-prompt evaluation toward distributional stability and prompt aggregation within our IPR framework.
LLM annotators are increasingly used in computational social science (CSS), but it is unclear whether their alignment-shaped errors preserve the empirical conclusions a researcher would report. We audit three open-source 7B instruction-tuned models (Zephyr, Mistral-Instruct, Qwen2.5-Instruct) across six TweetEval tasks under four prompt conditions (72 cells) and find that social-desirability failures do not run in a single direction. Zephyr exhibits leniency bias, systematically under-applying harmful labels (offensive language: false benign rate 0.729, false alarm rate 0.031). Mistral and Qwen exhibit overcorrection, over-applying the same labels (Mistral hate-speech FAR = 0.604). All three models exhibit neutrality bias on abortion stance, underestimating opposition prevalence by 24 to 40 percentage points and inflating the neutral label. None of the four prompting interventions we test (neutral, safety framing, depersonalized, chain-of-thought) corrects these failures across models; safety framing can worsen stance distortion. Strikingly, Zephyr's hate-speech prevalence estimate matches the gold rate exactly while its class-conditional errors are large in both directions, an accidental cancellation that misleads aggregate validation. We translate these patterns into a three-part taxonomy with diagnostic FBR/FAR signatures and a lightweight gold-sample validation protocol. The headline for trustworthy CSS: a model that looks calibrated on aggregate metrics can still flip the substantive empirical conclusion a researcher would report.