cs.AINov 15, 2026

Who Writes the Codebook? An Eight-Step Framework for Auditing LLM-Generated Category Systems

Authors: Thilo Tamme, Anton Hantel, Jakob Scheppers

Organizations: Technical University of Munich (TUM) · Massachusetts Institute of Technology (MIT)

Abstract

Large language models (LLMs) increasingly create the categories through which qualitative evidence is interpreted. Existing coding benchmarks often hold these categories fixed. We propose the Qualitative Variance Audit Framework, an eight-step method that compares codebook creation, repeated application and code distributions under controlled inputs. It combines model-owned and shared-reference codebooks to distinguish variation in the overall pipeline from application differences within a common category system. An application to 668 organisational interviews, 13,166 text units and eight models from five vendors yields 48 codebooks and 270 coding runs. Models create markedly different category structures; mean within-model reliability ranges from .76 to .92, whereas between-model agreement on the shared codebook is about .69. The framework complements LLM-assisted grounded theory by exposing model dependencies before substantive interpretation. It assesses reliability and representational consequences, not interpretive validity or the quality of resulting theory.

Figures & tables

Explore similar work

CardsList
  1. Can Large Language Models Reliably Code Qualitative Humanitarian Data? A Benchmark Study Against Human Expert Adjudication

    Jun 25, 2026Jerome Marston, Tino Kreutzer, Salomé Garnier +3LLM EvaluationHuman-in-the-Loop Annotation

  2. Scaling Qualitative Analysis with Large Language Models: Development and Validation of the Smart Coder Framework for Human-LLM Collaborative Coding

    Oct 23, 2026Cheng Tang, Kun Wang, Yaxuan Yang +4LLM-Assisted AnnotationLLM Prompting