cs.CLNov 3, 2026

Who is the Pronoun Pro? Comparing LLMs and Humans at Number-Disambiguation of 'they'

Authors: Carlos Hartmann

Organizations: University of Zurich

Abstract

This study evaluates whether large language models (LLMs) can reliably replace or complement human research assistants in linguistically complex annotation. In a controlled annotation task, 73 human annotators (local students and crowdworkers with and without a background in linguistics) and more than 40 LLM configurations number-disambiguated 174 authentic uses of 'they' from Reddit, including innovative and context-sensitive cases. The best LLM, Kimi K3, reached 94.5% accuracy, above the human majority vote (86.2%) and the mean of the top half of human annotators (78.5%); only the single best human annotator scored higher (95.7%). LLMs also offer substantial practical advantages in cost and speed. However, qualitative error analysis reveals a crucial complementarity: human reasoning in low-agreement cases exposes inconsistencies in the annotation typology and thereby improves the methodological framework itself. The study contributes (i) a benchmark-style human-LLM comparison for a semantically demanding task, (ii) transparent cost and time evidence, and (iii) a methodological argument against a purely replacement-oriented view of automation. LLMs are viable for large-scale annotation support in linguistics, but their responsible use requires human critical oversight. The dataset with its gold-standard labels, all human and LLM annotations, and the fully reproducible code are openly available as a Quarto-based, public repository.

Figures & tables

Explore similar work

CardsList
  1. Challenges in annotations by humans and LLMs: A case study of evaluative language

    Jul 30, 2026Mirela Imamovic, Aenne Cecilia Kristine Knierim, Khushi Pitroda +1LLM EvaluationLLM-Assisted Annotation

  2. Observational Equivalence of LLM and Human Annotation

    Aug 24, 2026Kentaro Nakamura, Jing Ling Tan, George YeanLLM-Assisted AnnotationAnnotator Disagreement

  3. From Fallback to Frontline: When Can LLMs be Superior Annotators of Human Perspectives?

    Apr 20, 2026Hasan Amin, Harry Yizhou Tian, Xiaoni Duan +3LLM EvaluationLLM-Assisted Annotation