cs.AINov 6, 2026

Definitions, Label Names, or Demonstrations? Auditing What Governs LLM Annotation

Authors: Viraj Mishra, Shubham Gupta, Dhruv Gupta, Yuehong Cassandra Tai, Qunhua Li

Organizations: Department of Computer Science and Engineering, Pennsylvania State University · Department of Statistics, Pennsylvania State University

Abstract

Large language models (LLMs) are increasingly used to annotate social-science texts, but high accuracy does not establish that they apply researcher-supplied category definitions: predictions may instead reflect prior associations with category labels or labeled demonstrations. We introduce a counterfactual audit that reassigns definitions among output labels, distinguishing adherence to supplied definitions from preservation of a dataset's original mapping. Across three political-text datasets, models rarely follow reassigned definitions, and standard fine-tuning strengthens the original mapping without reliably increasing definition adherence. We then study retrieval-augmented annotation. Retrieving the nearest training examples generally outperforms retrieving examples separately within each category despite using fewer demonstrations. On the two tasks with substantial gains, direct kk-nearest-neighbor classification recovers 74--93% of the improvement, indicating that neighborhood labels provide much of the predictive signal. Relabeling fixed demonstrations redirects predictions, whereas aligning demonstrations with reassigned definitions substantially increases adherence to the new mapping. Thus, accuracy-improving interventions need not increase codebook adherence; definitions, category labels, demonstrations, retrieval, and training data should be aligned and validated jointly.

Figures & tables

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Measuring Definition Sensitivity in LLM Annotation of Social Science Concepts

    Nov 9, 2026Bogdan Mamaev, Qizhou Wang, Christopher Leckie +1LLM-Assisted AnnotationPrompt Sensitivity

  2. Auditing LLM Benchmarks with Item Response Theory

    May 28, 2026Sander Land, Daniel M. BikelBenchmark AuditingLLM Auditing