cs.CLAug 24, 2026

Observational Equivalence of LLM and Human Annotation

Authors: Kentaro Nakamura, Jing Ling Tan, George Yean

Organizations: Ph.D. student at John F. Kennedy School of Government, Harvard University · Ph.D. student at Department of Government, Harvard University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

In this paper, we show that LLM and human coding are observationally equivalent in terms of annotation quality: recent LLMs agree with expert coders at rates comparable to those observed among experts themselves. We demonstrate this through replications of text-classification tasks from 14 peer-reviewed political science studies, in which ten LLMs, three human experts, and 165 crowdsourced workers independently classify the same texts using identical codebooks. We find that this equivalence is driven by ambiguity in the texts and coding rules. When LLMs disagree with experts, experts are also more likely to disagree with one another, and clarifying coding rules reduces disagreement among both experts and sufficiently capable LLMs. Thus, there is little empirical basis for preferring human coding on the basis of annotation quality alone, while LLMs offer substantial advantages in speed and cost. We therefore argue that the central challenge of text annotation is no longer choosing between human and machine coders, but developing coding rules that minimize ambiguity and accounting for the ambiguity that remains. To this end, we propose using disagreement across LLMs to identify difficult cases and refine codebooks, and we develop ambiguity-aware bounds for downstream inference when a unique annotation cannot be defined for every text.

Figures & tables

ModelDeveloperRelease date
Mistral 7B (Instruct v0.3)Mistral AIMay 2024
LLaMA 3.1-8BMetaJuly 2024
Qwen2.5-7BAlibabaSeptember 2024
Qwen2.5-14BAlibabaSeptember 2024
Gemma 3-12BGoogleMarch 2025
Gemma 3-27BGoogleMarch 2025
Table 1: Language models used in the replication, with release dates.
Figure 1: Average LLM–expert agreement by model, separately for relatively easy studies (expert ICR >0.8>0.8 ; 9 studies) and relatively difficult studies (expert ICR ≤0.8\leq 0.8 ; 5 studies). Points are model-specific means, and error bars are 95% between-study confidence intervals. The red dotted line indicates mean expert–expert agreement within each panel, and the gray dashed line indicates 0.50 agreement.
Figure 2: Agreement with a held-out expert, by model and by whether the two remaining experts agree. The red dotted line indicates the held-out expert’s agreement with the common label of the other two experts on clear texts. Coral and teal points report LLM agreement with the held-out expert on clear and ambiguous texts, respectively. Error bars are 95% between-study confidence intervals.
Figure 3: LLM–expert agreement under the original (red) and revised codebook (blue) for the Schub, 2022 annotation task. Each point reports a model’s average agreement with the three experts. The colored dotted lines indicate mean expert–expert agreement.
Figure 4: (Left) For each paper, texts are grouped as the 10 least contested texts across LLM coding versus 10 most contested ones, based on our stratified sampling. (Right) Texts are grouped into those which the three expert coders are unanimous versus split. In red are the group means with 95% confidence intervals; gray points are individual texts. We exclude the revised codebook version of Schub, 2022 for consistency.
Figure 5: Survey extension of Figure 1 , restricted to the 20-text-sample. The red dotted line is mean pairwise expert-expert agreement, black dot and bars are LLM-expert agreement, and blue the crowd-expert agreement. Each unit is the per-paper mean agreement over 20 texts by coder type, averaged across all coders.
Figure 6: Proposed analysis procedure for text-as-data using LLMs.
Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

PaperTexts (unit)Classification methodReanalysis targetUse in original analysis
Arias (2022)UN General Assembly General Debate speechStructural topic modelClimate securityAnalyzed as dependent variable
Blaydes et al. (2018)Political advice textsStructural topic modelMultiple political contentDescribes topic trends over time
Blumenau and Lauderdale (2018)European Parliament roll-call voting recordsStructural topic modelFinancesDistinguishes crisis-related votes to estimate how crises altered voting cleavages
Bush & Clayton (2023)Open-ended survey responses (response)Structural topic modelMaterial-costs topicUsed directly; topic prevalence modeled on gender and income
Clark & Dolan (2021)World Bank policy conditionalityPre-existing categoriesFiscal policyReanalyzed as dependent variable
Feltovich & Giovannoni (2024)Lab campaign messages (message)Human coders, majority of threeNegative campaigningUsed directly; dependent variable on experimental conditions
Appendix
Table S1 : Summary of Text Classification Tasks for 14 Studies
ComponentRole in PromptJustification
RoleAssigns the model a specific analytical perspectiveDirects the model to adopt the relevant domain expertise and evaluative stance, improving task alignment and reducing inconsistent interpretations.
DefinitionProvides a formal conceptualization of the constructAligns the model with the intended theoretical meaning, ensuring consistency with the research concept rather than colloquial usage.
Classification Rules (Yes / No / Tricky Cases)Specifies inclusion (Yes), exclusion (No), and ambiguous (Tricky) conditions for classificationOperationalizes the abstract concept into an explicit decision boundary: inclusion rules define positive cases, exclusion rules prevent category leakage, and tricky cases guide decisions under ambiguity. This structured separation improves consistency, reduces false positives, and enhances robustness in edge cases.
Label WordingDefines the vocabulary of allowable outputsStandardizes the semantic space of labels, ensuring consistency across observations and enabling aggregation and downstream analysis.
Summary of Paper/DataProvides contextual information about the substantive setting and the datasetAnchors interpretation in the research context, helping the model infer relevance when signals are indirect or mechanism-based rather than explicit. Clarifies the distribution of topics and potential co-occurrence of issue areas, helping calibrate expectations and avoid overgeneralization.
ReasoningSpecifies the inferential process for classificationEncourages the model to evaluate the text against the definition systematically and articulate the basis for its decision, improving transparency and conceptual consistency.
Appendix
Table S2 : General Prompt Components and Their Justification (Codebook-Style)
  
StudyPrompt
Arias (2022)Topic : Climate security Definition : A text should be labeled Yes for climate security when Climate change is framed as an international security threat. Classification Rules Yes - Climate described as security, conflict, instability, or UNSC issue. No - Climate discussed without security, conflict, threat, or UNSC framing. Tricky/borderline - Existential harm or sea-level threats without security jurisdiction. Label Wording Climate security Climate securitization Climate as security threat Threat to international peace and security Climate-linked conflict, instability, terrorism, or state failure Summary of Paper Arias asks why some states frame climate change as a security issue in the United Nations, and which states are most likely to make such securitizing moves. The paper argues that securitization is shaped by agenda-control incentives: P5 states benefit from moving issues toward the Security Council, while highly vulnerable states such as SIDS may resist losing agenda control. Empirically, the paper finds that UN climate discourse overall is not securitized, P5 states are more likely to securitize climate change, and SIDS are less likely to do so. Summary of Data The text data are speeches by state representatives in the UN General Assembly General Debate, filtered to speech segments discussing climate change. The analysis uses 4,525 climate-related speech segments from 1,987 speeches, with the earliest climate-related speech segment appearing in 1984.
Blaydes et al. (2018)Topic : Art of Rulership Definition Practical guidance on governing, exercising power, and ruling effectively. Classification Rules YES - Describes ruler’s duties, governance practices, or statecraft strategies. NO - Focuses on religion, personal morality, or non-political life. Tricky / borderline - Moral advice tied directly to governing decisions or authority. Label Wording Statecraft, Governance advice, Ruling practices, Political management, Exercise of power / kingship guidance Summary of Paper The paper compares medieval Christian and Islamic political advice texts (“mirrors for princes”) to examine how ideas of governance evolved across regions. Using text-as-data methods, it identifies shared thematic structures and tracks their evolution over time. It finds broad similarities across traditions but key divergences, including declining religious emphasis in Europe and shifting ruler-centered discourse in the Islamic world. Summary of Data The data consist of translated medieval political advice texts addressed to rulers in Christian Europe and the Islamic world. These texts include guidance on governance, morality, religion, and social order, often written by elites for kings, courts, or political audiences.
Bush & Clayton (2023)Topic : taxes Definition Tax-related material costs of climate action. Classification rules Yes -Mentions taxes, fees, prices, or cost increases. No - Mentions non-tax harms, benefits, or general climate concern. Tricky/borderline - Code broad “cost” claims only if tax-like or monetary. Label wording Taxes, Tax burden, Carbon tax / fossil-fuel tax, Fees, penalties, mandatory payments, Higher costs / cost-of-living increases Summary of Paper Bush and Clayton study why the gender gap in climate concern is larger in wealthier countries. They argue that as countries become wealthier, climate mitigation is perceived as more costly and less beneficial, especially among men, who are more likely to associate decarbonization with material and psychological costs. Using cross-national surveys, an original 10-country survey, and focus groups in Peru and the United States, they find that men’s climate concern declines more sharply with economic development and that men in wealthier countries more often emphasize costs of climate action. Summary of Data The relevant text data are open-ended survey responses from citizens in 10 countries in the Americas and Western Europe, collected through Netquest in 2019–2020, asking how acting to stop climate change might personally harm respondents. The “Taxes” category appears in a structural topic model of these open-ended responses; the paper labels Topic 2 as “Taxes,” associated with words such as “increase,” “tax,” and “cost,” and finds it more common in wealthier countries.
Clark & Dolan (2021)Topic : fiscal policy Definition Government taxation, spending, debt, and budgetary policy actions Classification rules Yes - Mentions taxes, spending, deficits, debt, or fiscal measures No - Discusses non-fiscal policies (e.g., monetary, foreign, social) Tricky / borderline - Economic discussion without explicit fiscal instruments or government action Label wording fiscal policy / budget policy, taxation and government spending, deficit, debt, public finance, revenues, expenditures, austerity, government budget management Summary of Paper The paper examines how World Bank policy conditionality reflects the preferences of powerful states, especially the United States. It shows that countries politically aligned with the U.S. receive fewer and less stringent policy conditions. The mechanism is primarily indirect, with World Bank staff designing programs consistent with U.S. preferences rather than explicit bargaining. Summary of Data The data consist of World Bank Development Policy Financing (DPF) loan conditions from 2005–2018, including detailed text and categorical coding of policy requirements. Each observation captures policy conditions (prior actions) imposed on borrowing countries as part of loan agreements.
Feltovich & Giovannoni (2024)Topic : Contains negative campaigning Definition: Criticizes opponent’s behavior, quality, or performance. Yes - Attacks opponent’s behavior, quality, competence, or record. No - Promotes sender without criticizing the opponent. Tricky/borderline - Neutral opponent references without criticism are borderline. Label wording attack; criticism; opponent failure; poor performance; low quality Summary of Paper Feltovich and Giovannoni study how politicians’ track records and campaign messages interact to shape elections and voter welfare. They develop and test a laboratory election model where incumbents and challengers vary in quality, voters observe performance imperfectly, and candidates may send campaign messages before elections. The paper finds that both higher quality variability and campaigning improve voter welfare, while challengers are more likely to use negative campaigning when incumbents perform poorly. Summary of Data The text data are short, free-form campaign announcements written by candidates in a laboratory election experiment after a straw poll and before the final election. Messages were 0–140 characters, written in English, and sent by incumbents and challengers; research assistants coded whether each message was positive, negative, about quality, about past performance, or about future decisions.
Appendix
Table S3 : Text classification across the studies
Figure S1 : Example of coding task for the Arias, 2022 paper.
  
ItemQuestion
Baseline and moderator items
Political knowledgeWhich of the following best describes a political party manifesto or platform? Options: A document describing a party’s policy positions (correct); A court ruling; A private tax document; A voter registration form; Not sure
NewsHow often do you follow news about politics or public policy? Options: Never; Once or a few times a month; A few times a week; Daily
CourseworkHave you ever taken a political science, government, public policy, or international relations course? Options: No; Yes, one; Yes, more than one; Not sure
Demographic items
Appendix
Table S4: Survey Questionnaire
   
PaperTopicText
Arias, 2022Climate securityClimate change is rightfully described as the most urgent threat facing mankind, and, at least for the next several months, it will remain at the top of the global diplomatic and negotiating agenda. But what is the challenge of climate change if not a risk to development, security and peace? What is the threat of climate change, if not a threat to the very notion of human survival and ecological balance?
Blumenau and Lauderdale, 2018FinancePURPOSE: to establish a new instrument for pre-accession, IPA II, in the framework of the reform of the EU external action financial instruments and following on from the unified Instrument for Pre-Accession Assistance, IPA, for the potential candidate countries to accession. PHILOSOPHY AND ACTION PLAN FOR EXTERNAL AID: what happens outside the borders of the EU can and does directly affect the prosperity and security of EU citizens. It is therefore in the interest of the EU to be actively engaged in influencing the world around us, including through the use of financial instruments.
Bush and Clayton, 2023Taxes for climate policyIt would harm me through the increases in costs of energy.
Clark and Dolan, 2021Fiscal policyThe Recipient has demonstrated enhanced predictability of budget execution as reflected by the fact that, during Fiscal Year 2009, at least 85% of Budget Heads expended between 95% and 105% of their total budgetary allocated funding for said year, as maintained to the date of this Agreement.
Feltovich and Giovannoni, 2024Negative campaigningThe other challenger either failed in maths or lied to you.
Appendix
Table S6 : Examples of unambiguous texts
   
PaperTopicText
Arias, 2022Climate securityClimate change must be further mainstreamed into the work of the whole United Nations system, with a view to supporting efforts to help the transition to low-carbon economies consistent with sustainable development, to strengthen countries’ adaptation and resilience in the face of climate change, and to minimize the possible security implications. In the light of diminishing natural resources, environmental degradation, extreme poverty, hunger and diseases, and social unrest, we agree with others that sustainable development has become the defining issue of our time. Our highly globalized and interdependent world means that we share not only the same challenges but a common fate.
Arias, 2022Climate security08-53141 40 being especially vulnerable. the recent hurricanes that left such a trail of destruction across the caribbean once again brought into sharp relief the acute vulnerabilities of small island states, such as the maldives, to global warming and climate change. for the maldives, climate change is not a distant possibility. it is happening now and is a reality that we are experiencing on a daily basis. the continuing degradation of the global environment is not only undermining our development process, but also seriously threatening the very survival of our people and the existence of our tiny country.
Blaydes et al., 2018Art of rulershipthe Church of Christ. [5] We must make the same judgment about the end of the whole multitude as we do of one person.170 If, therefore, the end of a person were some good existing in that one alone, and the ultimate end of the multitude to be governed were similar in that the multitude should acquire such a good and preserve it, and if indeed such an ultimate end were a corporal one, either of one person or of the multitude, then the life and health of that body would be the duty of a physician. If the ultimate end were affluence and riches, a steward should be king of the multitude. If the good of knowing the truth were of the kind which the multitude could attain, the king would have the same duty as a professor. [ 6] It seems that the end of a multitude gathered together is to live according.
Blaydes et al., 2018Art of rulershipAny other person, have revealed to us the true sense of the oracle. Why do we then delay to offer him the crown, whom the Fates have ordained to be our king? The old men immediately quitted the sacred grove, and the chief of them, taking me by the hand, acquainted the people, who waited with impatience for their decision, that I had gained the prize. Scarcely had he done speaking, when a confused noise ran through the whole assembly. Everyone shouted for joy. The whole coast, and neighboring mountains, echoed with these words: “May the son of Ulysses, who resembles Minos, reign over the Cretans.” After waiting a while, I made a sign with my hand, to ask to be heard. In the meantime, Mentor whispered thus in my ear: “Are you going to renounce your country? Will the ambition of reigning make you forget Penelope, who longs for you as her only …”
Clark and Dolan, 2021Fiscal policyThe Recipient has reduced administrative costs arising from (A) the creation of a business and (B) the transfer of urban commercial real property (mutation d’immeubles urbains bâtis et non-bâtis) by amending the Recipient’s Registration Code (Code de l’enregistrement) so as to reduce the registration fees in connection with: (i) the creation of a business; and (2) the transfer of urban commercial real property (mutation d’immeubles urbains bâtis et non-bâtis), each by at least fifty percent (50%).
Appendix
Table S7 : Examples of ambiguous texts
Figure S2 : LLM–expert Krippendorff’s α\alpha by coding difficulty. Points report the mean pairwise Krippendorff’s α\alpha between each model and the individual expert coders, averaged first within papers and then across papers. Error bars show 95% confidence intervals calculated across papers. The red dotted line reports mean pairwise expert–expert Krippendorff’s α\alpha within each difficulty group.
ModelStudy difficultyExperts agreeExperts disagreeDifferencepp -value
Gemma3 (12B)Easy ( >0.8>0.8 )0.0600.1360.0764.47×10−74.47\times 10^{-7}
Gemma3 (12B)Hard ( ≤0.8\leq 0.8 )0.1500.2000.0492.72×10−32.72\times 10^{-3}
Gemma3 (27B)Easy ( >0.8>0.8 )0.0480.1680.1203.39×10−113.39\times 10^{-11}
Gemma3 (27B)Hard ( ≤0.8\leq 0.8 )0.1120.1230.0114.44×10−14.44\times 10^{-1}
Llama3.1 (8B)Easy ( >0.8>0.8 )0.0660.1630.0973.81×10−93.81\times 10^{-9}
Llama3.1 (8B)Hard ( ≤0.8\leq 0.8 )0.1820.2480.0661.94×10−41.94\times 10^{-4}
Appendix
Table S9 : Within-model prompt disagreement by model, expert agreement, and study difficulty. For each model and text, Gini impurity is calculated across the 64 prompt-specific predictions. Higher values indicate greater sensitivity to prompt specification.
Figure S3 : Marginal effects of prompt engineering components and text length on mean LLM–expert agreement (OLS coefficient estimates with paper-clustered 95% confidence intervals).
Figure S4 : Instability of Text Annotations across seven open-source models for each paper. Instability is measured as the proportion of 100 randomly sampled texts that receive more than one classification label across 10 repeated runs under stochastic decoding (temperature = 1.0). The seven open-source models are GPT-OSS (20B), Gemma 3 (12B and 27B), Llama 3.1 (8B), Mistral (7B), and Qwen 2.5 (7B and 14B).
Figure S5 : Instability of Text Annotations for each open-source model across 14 papers. Instability is measured as the proportion of 100 randomly sampled texts that receive more than one classification label across 10 repeated runs under stochastic decoding (temperature = 1.0).
ModelExperts agreeExperts disagreeDifferencepp -value
GPT-OSS (20B)0.1120.4060.293¡0.001
Llama3.1 (8B)0.1310.3080.1770.003
Qwen2.5 (14B)0.0100.0600.0500.002
Mistral (7B)0.1190.1500.0310.436
Gemma3 (12B)0.0210.0330.0120.257
Gemma3 (27B)0.0090.0200.0110.142
Appendix
Table S10 : Annotation Variation for each model under Stochastic Decoding (Temperature = 1) Across 10 Runs, Stratified by Expert Agreement.
Study difficultyExperts agreeExperts disagreeDifferencepp -valueNN agreeNN disagree
Easy (ICR >0.8>0.8 )0.1560.5250.369<2.2×10−16<2.2\times 10^{-16}753147
Hard (ICR ≤0.8\leq 0.8 )0.3230.5590.2374.69×10−134.69\times 10^{-13}252248
Appendix
Table S11 : Cross-model disagreement by expert agreement and study difficulty using entropy. Higher values indicate greater disagreement among the 10 LLMs.
ModelStudy difficultyExperts agreeExperts disagreeDifferencepp -value
Gemma3 (12B)Easy ( >0.8>0.8 )0.1400.3140.1753.85×10−83.85\times 10^{-8}
Gemma3 (12B)Hard ( ≤0.8\leq 0.8 )0.3320.4480.1157.29×10−47.29\times 10^{-4}
Gemma3 (27B)Easy ( >0.8>0.8 )0.1130.3660.2531.24×10−111.24\times 10^{-11}
Gemma3 (27B)Hard ( ≤0.8\leq 0.8 )0.2560.2800.0254.34×10−14.34\times 10^{-1}
Llama3.1 (8B)Easy ( >0.8>0.8 )0.1530.3700.2172.36×10−102.36\times 10^{-10}
Llama3.1 (8B)Hard ( ≤0.8\leq 0.8 )0.3960.5370.1401.06×10−41.06\times 10^{-4}
Appendix
Table S12 : Within-model prompt disagreement by model, expert agreement, and study difficulty. For each model and text, normalized Shannon entropy is calculated across the 64 prompt-specific predictions. Higher values indicate greater sensitivity to prompt specification.
Figure S6 : Agreement with expert coders on the 20-text survey sample, for each LLM, their majority vote, and the crowd workers (in blue). Points are mean pairwise agreement with the three experts; the red dotted line is the mean expert–expert agreement. Levels are lower than in the full corpus because the survey sample oversamples contested texts.
Figure S7 : (Left:) Gini impurity, measure of within-source disagreement; fall in Gini corresponds to H3a (Better codebook, less disagreement). (Right:) agreement of crowd workers with experts and experts with one another; directionally consistent with H3b (Better codebook, closer to experts).
Figure S8 : H4/H5, Preregistered univariate associations between coder characteristics and crowd–expert agreement. Every association is indistinguishable from zero once task fixed effects are added.
 (1) Univariate (preregistered)(2) Joint, task fixed effects
ModeratorEst.SEppEst.SEpp
Panel A: Respondent’s agreement with experts ( N=165N=165 respondents)
H4a: Political knowledge (correct)0.0630.0630.0370.0370.0900.0900.0200.0200.0440.0440.6590.659
H4b: News consumption (high)0.0630.0630.0270.0270.0220.0220.0180.0180.0200.0200.3870.387
H4c: Education (BA or higher)0.0510.0510.0230.0230.0300.0300.0330.0330.0370.0370.3820.382
H4d: Coursework (any)0.0300.0300.0230.0230.1960.196−0.018-0.0180.0260.0260.5030.503
Appendix
Table S18 : Exploratory moderators of crowd coding quality (H4/H5)
Figure S9 : H6, Crowd-worker reliability by LLM sampling stratum and by expert agreement. Points are Krippendorff’s α\alpha over the crowd workers within each group of texts, with bootstrap 95% confidence intervals; the dashed line marks the chance level of zero.
Figure S10 : H2, Krippendorff’s α\alpha between each source and the individual experts (mean pairwise, averaged within papers then across papers), with 95% confidence intervals across papers. The ten LLMs are collapsed into one panel average; the dotted line is the experts’ own mean pairwise α\alpha ; the dashed line marks chance ( α=0\alpha=0 ); crowd workers in blue.
Figure S11 : H3, Crowd–expert agreement (left) and within-source Krippendorff’s α\alpha (right) under the original and improved Schub (2022) codebook.
Figure S12 : Coefficients of β1\beta_{1} (P5) and β2\beta_{2} (SIDS), along with their 95% confidence intervals, based on (i) the proposed ambiguity-aware bounds, (ii) PPI / DSL correction (which corrects the measurement error in LLM annotations by regarding human annotation as ground truth), and (iii) the naive LLM regression (which uses LLM predictions as the outcome without accounting for the underlying ambiguity of the concepts).
More in this topic

LLM-Assisted Annotation

Large language model-assisted annotation uses AI to label, classify, and structure data, often with human review to improve scale and consistency.

See all 197 papers

Explore similar work

CardsList
Sep 22, 2026

Experts Rise Where LLMs Disagree: Using Cross-Model Disagreement to Target Expert Effort in LLM Codebook Revision for Large-Scale Annotation

Large-scale text annotation brings expert insight to millions of documents, often through a codebook that AI annotators follow. Developing a robust codebook, however, takes months. Large language models (LLMs) could speed this process by applying an early codebook to the data, surfacing cases with strong LLM disagreement, and eliciting expert feedback to address them. We examined three ways experts can provide feedback for LLM codebook revision: (i) editing LLM-generated revisions driven by cross-LLM disagreement (Codebook Verifying), (ii) answering questions about LLM disagreements (Question Answering), and (iii) labeling disagreement cases with rationales (Rationale Labeling). Experiments on thousands of tutoring-session transcripts show that Rationale Labeling yielded the highest LLM-labeling accuracy (64.9%) against expert labels, outperforming the expert-revised codebook (57.8%). The best Question Answering setting also outperformed it (60.5%). Our work shows that LLMs can be used to strategically target expert attention, shortening months of codebook revision to days without sacrificing labeling performance.

Zeyu He, Zhuqian Zhou, Kirk Vanacore +2

The Pennsylvania State University, University Park, PA, USA · Cornell University, Ithaca, NY, USA

LLM-Assisted AnnotationHuman-in-the-Loop Annotation
AskPDF
Jul 30, 2026

Agreement Is Not Quality: Blind Expert Verification of Human and LLM Qualitative Coding When Human Consensus Is Not Ground Truth

Evaluations of LLM-assisted qualitative coding almost universally measure model performance as agreement with human coders, a practice that presumes human coding is the standard to approximate. This study provides empirical evidence that the presumption fails in ways agreement metrics cannot detect. Five LLM systems and three trained human coders independently applied a 72-item hierarchical codebook to 2,560 educator messages from a K-12 AI platform. Beyond conventional agreement analysis, an independent domain expert judged 855 pairwise comparisons of code sets blind to source, treating human and machine sources symmetrically. The two evaluation approaches diverge in both directions. Human-LLM agreement (mean Jaccard 0.30) falls well below human-human agreement (0.52), which standard practice would read as inferior LLM coding, yet the blind verifier preferred human and LLM coding at indistinguishable rates (51.5% vs. 48.5%, p = 0.537), and a Bradley-Terry ranking placed two LLMs above two of three human coders. For several substantive codes, human consensus encoded shared bias that the verifier rejected in favor of the LLM interpretation. Agreement-based evaluation is therefore insufficient for automation decisions, and the study demonstrates a transferable verification protocol and a code-level division-of-labor framework.

Alex Liu, Lief Esbenshade, Michael Xiao +4

University of Washington · Colleague AI/ · Colleague AI

LLM-Assisted AnnotationHuman-in-the-Loop Evaluation
AskPDF
Apr 20, 2026

From Fallback to Frontline: When Can LLMs be Superior Annotators of Human Perspectives?

Although large language models (LLMs) are increasingly used as annotators at scale, they are typically treated as a pragmatic fallback rather than a faithful estimator of human perspectives. This work challenges that presumption. By framing perspective-taking as the estimation of a latent group-level judgment, we characterize the conditions under which modern LLMs can outperform human annotators, including in-group humans, when predicting aggregate subgroup opinions on subjective tasks, and show that these conditions are common in practice. This advantage arises from structural properties of LLMs as estimators, including low variance and reduced coupling between representation and processing biases, rather than any claim of lived experience. Our analysis identifies clear regimes where LLMs act as statistically superior frontline estimators, as well as principled limits where human judgment remains essential. These findings reposition LLMs from a cost-saving compromise to a principled tool for estimating collective human perspectives.

Hasan Amin, Harry Yizhou Tian, Xiaoni Duan +3

Purdue University · Washington University in St. Louis

LLM EvaluationLLM-Assisted Annotation
AskPDF