cs.CLAug 24, 2026
SaveObservational Equivalence of LLM and Human Annotation
Organizations: Ph.D. student at John F. Kennedy School of Government, Harvard University · Ph.D. student at Department of Government, Harvard University
Abstract
In this paper, we show that LLM and human coding are observationally equivalent in terms of annotation quality: recent LLMs agree with expert coders at rates comparable to those observed among experts themselves. We demonstrate this through replications of text-classification tasks from 14 peer-reviewed political science studies, in which ten LLMs, three human experts, and 165 crowdsourced workers independently classify the same texts using identical codebooks. We find that this equivalence is driven by ambiguity in the texts and coding rules. When LLMs disagree with experts, experts are also more likely to disagree with one another, and clarifying coding rules reduces disagreement among both experts and sufficiently capable LLMs. Thus, there is little empirical basis for preferring human coding on the basis of annotation quality alone, while LLMs offer substantial advantages in speed and cost. We therefore argue that the central challenge of text annotation is no longer choosing between human and machine coders, but developing coding rules that minimize ambiguity and accounting for the ambiguity that remains. To this end, we propose using disagreement across LLMs to identify difficult cases and refine codebooks, and we develop ambiguity-aware bounds for downstream inference when a unique annotation cannot be defined for every text.
Figures & tables
| Model | Developer | Release date |
|---|---|---|
| Mistral 7B (Instruct v0.3) | Mistral AI | May 2024 |
| LLaMA 3.1-8B | Meta | July 2024 |
| Qwen2.5-7B | Alibaba | September 2024 |
| Qwen2.5-14B | Alibaba | September 2024 |
| Gemma 3-12B | March 2025 | |
| Gemma 3-27B | March 2025 |
Table 1: Language models used in the replication, with release dates.
Figure 1: Average LLM–expert agreement by model, separately for relatively easy studies (expert ICR ; 9 studies) and relatively difficult studies (expert ICR ; 5 studies). Points are model-specific means, and error bars are 95% between-study confidence intervals. The red dotted line indicates mean expert–expert agreement within each panel, and the gray dashed line indicates 0.50 agreement.
Figure 2: Agreement with a held-out expert, by model and by whether the two remaining experts agree. The red dotted line indicates the held-out expert’s agreement with the common label of the other two experts on clear texts. Coral and teal points report LLM agreement with the held-out expert on clear and ambiguous texts, respectively. Error bars are 95% between-study confidence intervals.
Figure 3: LLM–expert agreement under the original (red) and revised codebook (blue) for the Schub, 2022 annotation task. Each point reports a model’s average agreement with the three experts. The colored dotted lines indicate mean expert–expert agreement.
Figure 4: (Left) For each paper, texts are grouped as the 10 least contested texts across LLM coding versus 10 most contested ones, based on our stratified sampling. (Right) Texts are grouped into those which the three expert coders are unanimous versus split. In red are the group means with 95% confidence intervals; gray points are individual texts. We exclude the revised codebook version of Schub, 2022 for consistency.
Figure 5: Survey extension of Figure 1 , restricted to the 20-text-sample. The red dotted line is mean pairwise expert-expert agreement, black dot and bars are LLM-expert agreement, and blue the crowd-expert agreement. Each unit is the per-paper mean agreement over 20 texts by coder type, averaged across all coders.
Figure 6: Proposed analysis procedure for text-as-data using LLMs.
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
| Paper | Texts (unit) | Classification method | Reanalysis target | Use in original analysis |
|---|---|---|---|---|
| Arias (2022) | UN General Assembly General Debate speech | Structural topic model | Climate security | Analyzed as dependent variable |
| Blaydes et al. (2018) | Political advice texts | Structural topic model | Multiple political content | Describes topic trends over time |
| Blumenau and Lauderdale (2018) | European Parliament roll-call voting records | Structural topic model | Finances | Distinguishes crisis-related votes to estimate how crises altered voting cleavages |
| Bush & Clayton (2023) | Open-ended survey responses (response) | Structural topic model | Material-costs topic | Used directly; topic prevalence modeled on gender and income |
| Clark & Dolan (2021) | World Bank policy conditionality | Pre-existing categories | Fiscal policy | Reanalyzed as dependent variable |
| Feltovich & Giovannoni (2024) | Lab campaign messages (message) | Human coders, majority of three | Negative campaigning | Used directly; dependent variable on experimental conditions |
Table S1 : Summary of Text Classification Tasks for 14 Studies
| Component | Role in Prompt | Justification |
|---|---|---|
| Role | Assigns the model a specific analytical perspective | Directs the model to adopt the relevant domain expertise and evaluative stance, improving task alignment and reducing inconsistent interpretations. |
| Definition | Provides a formal conceptualization of the construct | Aligns the model with the intended theoretical meaning, ensuring consistency with the research concept rather than colloquial usage. |
| Classification Rules (Yes / No / Tricky Cases) | Specifies inclusion (Yes), exclusion (No), and ambiguous (Tricky) conditions for classification | Operationalizes the abstract concept into an explicit decision boundary: inclusion rules define positive cases, exclusion rules prevent category leakage, and tricky cases guide decisions under ambiguity. This structured separation improves consistency, reduces false positives, and enhances robustness in edge cases. |
| Label Wording | Defines the vocabulary of allowable outputs | Standardizes the semantic space of labels, ensuring consistency across observations and enabling aggregation and downstream analysis. |
| Summary of Paper/Data | Provides contextual information about the substantive setting and the dataset | Anchors interpretation in the research context, helping the model infer relevance when signals are indirect or mechanism-based rather than explicit. Clarifies the distribution of topics and potential co-occurrence of issue areas, helping calibrate expectations and avoid overgeneralization. |
| Reasoning | Specifies the inferential process for classification | Encourages the model to evaluate the text against the definition systematically and articulate the basis for its decision, improving transparency and conceptual consistency. |
Table S2 : General Prompt Components and Their Justification (Codebook-Style)
| Study | Prompt |
|---|---|
| Arias (2022) | Topic : Climate security Definition : A text should be labeled Yes for climate security when Climate change is framed as an international security threat. Classification Rules Yes - Climate described as security, conflict, instability, or UNSC issue. No - Climate discussed without security, conflict, threat, or UNSC framing. Tricky/borderline - Existential harm or sea-level threats without security jurisdiction. Label Wording Climate security Climate securitization Climate as security threat Threat to international peace and security Climate-linked conflict, instability, terrorism, or state failure Summary of Paper Arias asks why some states frame climate change as a security issue in the United Nations, and which states are most likely to make such securitizing moves. The paper argues that securitization is shaped by agenda-control incentives: P5 states benefit from moving issues toward the Security Council, while highly vulnerable states such as SIDS may resist losing agenda control. Empirically, the paper finds that UN climate discourse overall is not securitized, P5 states are more likely to securitize climate change, and SIDS are less likely to do so. Summary of Data The text data are speeches by state representatives in the UN General Assembly General Debate, filtered to speech segments discussing climate change. The analysis uses 4,525 climate-related speech segments from 1,987 speeches, with the earliest climate-related speech segment appearing in 1984. |
| Blaydes et al. (2018) | Topic : Art of Rulership Definition Practical guidance on governing, exercising power, and ruling effectively. Classification Rules YES - Describes ruler’s duties, governance practices, or statecraft strategies. NO - Focuses on religion, personal morality, or non-political life. Tricky / borderline - Moral advice tied directly to governing decisions or authority. Label Wording Statecraft, Governance advice, Ruling practices, Political management, Exercise of power / kingship guidance Summary of Paper The paper compares medieval Christian and Islamic political advice texts (“mirrors for princes”) to examine how ideas of governance evolved across regions. Using text-as-data methods, it identifies shared thematic structures and tracks their evolution over time. It finds broad similarities across traditions but key divergences, including declining religious emphasis in Europe and shifting ruler-centered discourse in the Islamic world. Summary of Data The data consist of translated medieval political advice texts addressed to rulers in Christian Europe and the Islamic world. These texts include guidance on governance, morality, religion, and social order, often written by elites for kings, courts, or political audiences. |
| Bush & Clayton (2023) | Topic : taxes Definition Tax-related material costs of climate action. Classification rules Yes -Mentions taxes, fees, prices, or cost increases. No - Mentions non-tax harms, benefits, or general climate concern. Tricky/borderline - Code broad “cost” claims only if tax-like or monetary. Label wording Taxes, Tax burden, Carbon tax / fossil-fuel tax, Fees, penalties, mandatory payments, Higher costs / cost-of-living increases Summary of Paper Bush and Clayton study why the gender gap in climate concern is larger in wealthier countries. They argue that as countries become wealthier, climate mitigation is perceived as more costly and less beneficial, especially among men, who are more likely to associate decarbonization with material and psychological costs. Using cross-national surveys, an original 10-country survey, and focus groups in Peru and the United States, they find that men’s climate concern declines more sharply with economic development and that men in wealthier countries more often emphasize costs of climate action. Summary of Data The relevant text data are open-ended survey responses from citizens in 10 countries in the Americas and Western Europe, collected through Netquest in 2019–2020, asking how acting to stop climate change might personally harm respondents. The “Taxes” category appears in a structural topic model of these open-ended responses; the paper labels Topic 2 as “Taxes,” associated with words such as “increase,” “tax,” and “cost,” and finds it more common in wealthier countries. |
| Clark & Dolan (2021) | Topic : fiscal policy Definition Government taxation, spending, debt, and budgetary policy actions Classification rules Yes - Mentions taxes, spending, deficits, debt, or fiscal measures No - Discusses non-fiscal policies (e.g., monetary, foreign, social) Tricky / borderline - Economic discussion without explicit fiscal instruments or government action Label wording fiscal policy / budget policy, taxation and government spending, deficit, debt, public finance, revenues, expenditures, austerity, government budget management Summary of Paper The paper examines how World Bank policy conditionality reflects the preferences of powerful states, especially the United States. It shows that countries politically aligned with the U.S. receive fewer and less stringent policy conditions. The mechanism is primarily indirect, with World Bank staff designing programs consistent with U.S. preferences rather than explicit bargaining. Summary of Data The data consist of World Bank Development Policy Financing (DPF) loan conditions from 2005–2018, including detailed text and categorical coding of policy requirements. Each observation captures policy conditions (prior actions) imposed on borrowing countries as part of loan agreements. |
| Feltovich & Giovannoni (2024) | Topic : Contains negative campaigning Definition: Criticizes opponent’s behavior, quality, or performance. Yes - Attacks opponent’s behavior, quality, competence, or record. No - Promotes sender without criticizing the opponent. Tricky/borderline - Neutral opponent references without criticism are borderline. Label wording attack; criticism; opponent failure; poor performance; low quality Summary of Paper Feltovich and Giovannoni study how politicians’ track records and campaign messages interact to shape elections and voter welfare. They develop and test a laboratory election model where incumbents and challengers vary in quality, voters observe performance imperfectly, and candidates may send campaign messages before elections. The paper finds that both higher quality variability and campaigning improve voter welfare, while challengers are more likely to use negative campaigning when incumbents perform poorly. Summary of Data The text data are short, free-form campaign announcements written by candidates in a laboratory election experiment after a straw poll and before the final election. Messages were 0–140 characters, written in English, and sent by incumbents and challengers; research assistants coded whether each message was positive, negative, about quality, about past performance, or about future decisions. |
Table S3 : Text classification across the studies
Figure S1 : Example of coding task for the Arias, 2022 paper.
| Item | Question |
|---|---|
| Baseline and moderator items | |
| Political knowledge | Which of the following best describes a political party manifesto or platform? Options: A document describing a party’s policy positions (correct); A court ruling; A private tax document; A voter registration form; Not sure |
| News | How often do you follow news about politics or public policy? Options: Never; Once or a few times a month; A few times a week; Daily |
| Coursework | Have you ever taken a political science, government, public policy, or international relations course? Options: No; Yes, one; Yes, more than one; Not sure |
| Demographic items | |
Table S4: Survey Questionnaire
| Paper | Topic | Text |
|---|---|---|
| Arias, 2022 | Climate security | Climate change is rightfully described as the most urgent threat facing mankind, and, at least for the next several months, it will remain at the top of the global diplomatic and negotiating agenda. But what is the challenge of climate change if not a risk to development, security and peace? What is the threat of climate change, if not a threat to the very notion of human survival and ecological balance? |
| Blumenau and Lauderdale, 2018 | Finance | PURPOSE: to establish a new instrument for pre-accession, IPA II, in the framework of the reform of the EU external action financial instruments and following on from the unified Instrument for Pre-Accession Assistance, IPA, for the potential candidate countries to accession. PHILOSOPHY AND ACTION PLAN FOR EXTERNAL AID: what happens outside the borders of the EU can and does directly affect the prosperity and security of EU citizens. It is therefore in the interest of the EU to be actively engaged in influencing the world around us, including through the use of financial instruments. |
| Bush and Clayton, 2023 | Taxes for climate policy | It would harm me through the increases in costs of energy. |
| Clark and Dolan, 2021 | Fiscal policy | The Recipient has demonstrated enhanced predictability of budget execution as reflected by the fact that, during Fiscal Year 2009, at least 85% of Budget Heads expended between 95% and 105% of their total budgetary allocated funding for said year, as maintained to the date of this Agreement. |
| Feltovich and Giovannoni, 2024 | Negative campaigning | The other challenger either failed in maths or lied to you. |
Table S6 : Examples of unambiguous texts
| Paper | Topic | Text |
|---|---|---|
| Arias, 2022 | Climate security | Climate change must be further mainstreamed into the work of the whole United Nations system, with a view to supporting efforts to help the transition to low-carbon economies consistent with sustainable development, to strengthen countries’ adaptation and resilience in the face of climate change, and to minimize the possible security implications. In the light of diminishing natural resources, environmental degradation, extreme poverty, hunger and diseases, and social unrest, we agree with others that sustainable development has become the defining issue of our time. Our highly globalized and interdependent world means that we share not only the same challenges but a common fate. |
| Arias, 2022 | Climate security | 08-53141 40 being especially vulnerable. the recent hurricanes that left such a trail of destruction across the caribbean once again brought into sharp relief the acute vulnerabilities of small island states, such as the maldives, to global warming and climate change. for the maldives, climate change is not a distant possibility. it is happening now and is a reality that we are experiencing on a daily basis. the continuing degradation of the global environment is not only undermining our development process, but also seriously threatening the very survival of our people and the existence of our tiny country. |
| Blaydes et al., 2018 | Art of rulership | the Church of Christ. [5] We must make the same judgment about the end of the whole multitude as we do of one person.170 If, therefore, the end of a person were some good existing in that one alone, and the ultimate end of the multitude to be governed were similar in that the multitude should acquire such a good and preserve it, and if indeed such an ultimate end were a corporal one, either of one person or of the multitude, then the life and health of that body would be the duty of a physician. If the ultimate end were affluence and riches, a steward should be king of the multitude. If the good of knowing the truth were of the kind which the multitude could attain, the king would have the same duty as a professor. [ 6] It seems that the end of a multitude gathered together is to live according. |
| Blaydes et al., 2018 | Art of rulership | Any other person, have revealed to us the true sense of the oracle. Why do we then delay to offer him the crown, whom the Fates have ordained to be our king? The old men immediately quitted the sacred grove, and the chief of them, taking me by the hand, acquainted the people, who waited with impatience for their decision, that I had gained the prize. Scarcely had he done speaking, when a confused noise ran through the whole assembly. Everyone shouted for joy. The whole coast, and neighboring mountains, echoed with these words: “May the son of Ulysses, who resembles Minos, reign over the Cretans.” After waiting a while, I made a sign with my hand, to ask to be heard. In the meantime, Mentor whispered thus in my ear: “Are you going to renounce your country? Will the ambition of reigning make you forget Penelope, who longs for you as her only …” |
| Clark and Dolan, 2021 | Fiscal policy | The Recipient has reduced administrative costs arising from (A) the creation of a business and (B) the transfer of urban commercial real property (mutation d’immeubles urbains bâtis et non-bâtis) by amending the Recipient’s Registration Code (Code de l’enregistrement) so as to reduce the registration fees in connection with: (i) the creation of a business; and (2) the transfer of urban commercial real property (mutation d’immeubles urbains bâtis et non-bâtis), each by at least fifty percent (50%). |
Table S7 : Examples of ambiguous texts
Figure S2 : LLM–expert Krippendorff’s by coding difficulty. Points report the mean pairwise Krippendorff’s between each model and the individual expert coders, averaged first within papers and then across papers. Error bars show 95% confidence intervals calculated across papers. The red dotted line reports mean pairwise expert–expert Krippendorff’s within each difficulty group.
| Model | Study difficulty | Experts agree | Experts disagree | Difference | -value |
|---|---|---|---|---|---|
| Gemma3 (12B) | Easy ( ) | 0.060 | 0.136 | 0.076 | |
| Gemma3 (12B) | Hard ( ) | 0.150 | 0.200 | 0.049 | |
| Gemma3 (27B) | Easy ( ) | 0.048 | 0.168 | 0.120 | |
| Gemma3 (27B) | Hard ( ) | 0.112 | 0.123 | 0.011 | |
| Llama3.1 (8B) | Easy ( ) | 0.066 | 0.163 | 0.097 | |
| Llama3.1 (8B) | Hard ( ) | 0.182 | 0.248 | 0.066 |
Table S9 : Within-model prompt disagreement by model, expert agreement, and study difficulty. For each model and text, Gini impurity is calculated across the 64 prompt-specific predictions. Higher values indicate greater sensitivity to prompt specification.
Figure S3 : Marginal effects of prompt engineering components and text length on mean LLM–expert agreement (OLS coefficient estimates with paper-clustered 95% confidence intervals).
Figure S4 : Instability of Text Annotations across seven open-source models for each paper. Instability is measured as the proportion of 100 randomly sampled texts that receive more than one classification label across 10 repeated runs under stochastic decoding (temperature = 1.0). The seven open-source models are GPT-OSS (20B), Gemma 3 (12B and 27B), Llama 3.1 (8B), Mistral (7B), and Qwen 2.5 (7B and 14B).
Figure S5 : Instability of Text Annotations for each open-source model across 14 papers. Instability is measured as the proportion of 100 randomly sampled texts that receive more than one classification label across 10 repeated runs under stochastic decoding (temperature = 1.0).
| Model | Experts agree | Experts disagree | Difference | -value |
|---|---|---|---|---|
| GPT-OSS (20B) | 0.112 | 0.406 | 0.293 | ¡0.001 |
| Llama3.1 (8B) | 0.131 | 0.308 | 0.177 | 0.003 |
| Qwen2.5 (14B) | 0.010 | 0.060 | 0.050 | 0.002 |
| Mistral (7B) | 0.119 | 0.150 | 0.031 | 0.436 |
| Gemma3 (12B) | 0.021 | 0.033 | 0.012 | 0.257 |
| Gemma3 (27B) | 0.009 | 0.020 | 0.011 | 0.142 |
Table S10 : Annotation Variation for each model under Stochastic Decoding (Temperature = 1) Across 10 Runs, Stratified by Expert Agreement.
| Study difficulty | Experts agree | Experts disagree | Difference | -value | agree | disagree |
|---|---|---|---|---|---|---|
| Easy (ICR ) | 0.156 | 0.525 | 0.369 | 753 | 147 | |
| Hard (ICR ) | 0.323 | 0.559 | 0.237 | 252 | 248 |
Table S11 : Cross-model disagreement by expert agreement and study difficulty using entropy. Higher values indicate greater disagreement among the 10 LLMs.
| Model | Study difficulty | Experts agree | Experts disagree | Difference | -value |
|---|---|---|---|---|---|
| Gemma3 (12B) | Easy ( ) | 0.140 | 0.314 | 0.175 | |
| Gemma3 (12B) | Hard ( ) | 0.332 | 0.448 | 0.115 | |
| Gemma3 (27B) | Easy ( ) | 0.113 | 0.366 | 0.253 | |
| Gemma3 (27B) | Hard ( ) | 0.256 | 0.280 | 0.025 | |
| Llama3.1 (8B) | Easy ( ) | 0.153 | 0.370 | 0.217 | |
| Llama3.1 (8B) | Hard ( ) | 0.396 | 0.537 | 0.140 |
Table S12 : Within-model prompt disagreement by model, expert agreement, and study difficulty. For each model and text, normalized Shannon entropy is calculated across the 64 prompt-specific predictions. Higher values indicate greater sensitivity to prompt specification.
Figure S6 : Agreement with expert coders on the 20-text survey sample, for each LLM, their majority vote, and the crowd workers (in blue). Points are mean pairwise agreement with the three experts; the red dotted line is the mean expert–expert agreement. Levels are lower than in the full corpus because the survey sample oversamples contested texts.
Figure S7 : (Left:) Gini impurity, measure of within-source disagreement; fall in Gini corresponds to H3a (Better codebook, less disagreement). (Right:) agreement of crowd workers with experts and experts with one another; directionally consistent with H3b (Better codebook, closer to experts).
Figure S8 : H4/H5, Preregistered univariate associations between coder characteristics and crowd–expert agreement. Every association is indistinguishable from zero once task fixed effects are added.
| (1) Univariate (preregistered) | (2) Joint, task fixed effects | |||||
| Moderator | Est. | SE | Est. | SE | ||
| Panel A: Respondent’s agreement with experts ( respondents) | ||||||
| H4a: Political knowledge (correct) | ||||||
| H4b: News consumption (high) | ||||||
| H4c: Education (BA or higher) | ||||||
| H4d: Coursework (any) | ||||||
Table S18 : Exploratory moderators of crowd coding quality (H4/H5)
Figure S9 : H6, Crowd-worker reliability by LLM sampling stratum and by expert agreement. Points are Krippendorff’s over the crowd workers within each group of texts, with bootstrap 95% confidence intervals; the dashed line marks the chance level of zero.
Figure S10 : H2, Krippendorff’s between each source and the individual experts (mean pairwise, averaged within papers then across papers), with 95% confidence intervals across papers. The ten LLMs are collapsed into one panel average; the dotted line is the experts’ own mean pairwise ; the dashed line marks chance ( ); crowd workers in blue.
Figure S11 : H3, Crowd–expert agreement (left) and within-source Krippendorff’s (right) under the original and improved Schub (2022) codebook.
Figure S12 : Coefficients of (P5) and (SIDS), along with their 95% confidence intervals, based on (i) the proposed ambiguity-aware bounds, (ii) PPI / DSL correction (which corrects the measurement error in LLM annotations by regarding human annotation as ground truth), and (iii) the naive LLM regression (which uses LLM predictions as the outcome without accounting for the underlying ambiguity of the concepts).
More in this topic
LLM-Assisted Annotation
Large language model-assisted annotation uses AI to label, classify, and structure data, often with human review to improve scale and consistency.
See all 197 papers