cs.CLNov 16, 2026

Benchmarking Large Language Models for Linguistic Annotation

Authors: Ondřej Tichý, Anna Marklová, Barbora Bulantová, Magdalena Titlbachová, Jiří Milička

Organizations: Faculty of Arts, Charles University · Department of Linguistics Faculty of Arts Charles University

Abstract

Assessing large language models for linguistic annotation requires task-specific evaluation under documented conditions. We present dhAIBench, a Python benchmarking and annotation tool with a web interface for exploring and sharing results. It records prompts, configurations and predictions, enabling comparisons of annotation quality, confidence calibration, repeatability and estimated cost. We demonstrate the framework and the associated methodology through 293 runs of 48 models on nine linguistic annotation tasks spanning morphology, syntax, semantics, discourse-pragmatics, spelling normalization and learner-language correction. Models achieve high agreement with reference annotations on several tasks, including those on non-standardized, historical varieties as well as requiring contextual interpretation. The results support evaluating models against specific linguistic tasks, rather than relying on general rankings. To this end, we make the current and future benchmarks publicly available at dhaibench.silent3.ff.cuni.cz.

Figures & tables

Explore similar work

CardsList
  1. Are LLMs Ready to Replace Bangla Annotators?

    Feb 18, 2026Md. Najib Hasan, Touseef Hasan, Souvika SarkarLLM-Assisted AnnotationLLM Reliability

  2. Challenges in annotations by humans and LLMs: A case study of evaluative language

    Jul 30, 2026Mirela Imamovic, Aenne Cecilia Kristine Knierim, Khushi Pitroda +1LLM EvaluationLLM-Assisted Annotation

  3. Magic Words or Methodical Work? Challenging Conventional Wisdom in LLM-Based Political Text Annotation

    Mar 27, 2026Lorcan McLaren, James Cross, Zuzanna Krakowska +2LLM-Assisted AnnotationLLM Evaluation