Who Writes the Codebook? An Eight-Step Framework for Auditing LLM-Generated Category Systems
Organizations: Technical University of Munich (TUM) · Massachusetts Institute of Technology (MIT)
Abstract
Large language models (LLMs) increasingly create the categories through which qualitative evidence is interpreted. Existing coding benchmarks often hold these categories fixed. We propose the Qualitative Variance Audit Framework, an eight-step method that compares codebook creation, repeated application and code distributions under controlled inputs. It combines model-owned and shared-reference codebooks to distinguish variation in the overall pipeline from application differences within a common category system. An application to 668 organisational interviews, 13,166 text units and eight models from five vendors yields 48 codebooks and 270 coding runs. Models create markedly different category structures; mean within-model reliability ranges from .76 to .92, whereas between-model agreement on the shared codebook is about .69. The framework complements LLM-assisted grounded theory by exposing model dependencies before substantive interpretation. It assesses reliability and representational consequences, not interpretive validity or the quality of resulting theory.
Figures & tables
| Step | Operation and control | Auditable output |
|---|---|---|
| 1. Fix the audit corpus | Define themes, exclusions and one segmentation rule; freeze unit identifiers. | Corpus inventory and identical coding units. |
| 2. Match elicitation inputs | Select the same interview sample per theme for every model; hold prompt and schema constant. | Sample identifiers and creation specification. |
| 3. Generate codebooks | Obtain labels, definitions, rules, examples and scale declarations; validate structure. | Model-specific instruments and structural diagnostics. |
| 4. Set comparison conditions | Pair each model’s own codebook with one shared reference selected by a declared rule. | Own/shared design and explicit reference dependence. |
| 5. Repeat corpus coding | Apply each condition to all fixed units in separate runs; record settings and failures. | Repeated assignments and execution manifest. |
| 6. Assess representation and agreement | Compare granularity, ordinal share, coverage, within-model reliability and between-model agreement. | A diagnostic profile rather than one quality score. |
| Theme | Language | Interviews | Units |
|---|---|---|---|
| AI Assessment | English | 179 | 4,266 |
| AI Training | English | 95 | 1,821 |
| AI Usage and Literacy | English | 61 | 1,677 |
| Teaming Check-ins | English | 205 | 2,788 |
| Employee Engagement | German | 96 | 2,222 |
| Minority Empowerment | English | 32 | 392 |
| Model | Recorded identifier | Coding route | Sampling and reasoning |
|---|---|---|---|
| Opus 4.8 | claude-opus-4-8 | Subscription CLI | Default; zero reasoning requested, likely active. |
| Sonnet 5 | claude-sonnet-5 | Anthropic API | Default; reasoning disabled; one CLI run. |
| Haiku 4.5 | claude-haiku-4-5-20251001 | Subscription CLI | Default; zero requested, little apparent reasoning. |
| Fable 5 | claude-fable-5 | Anthropic API | Default sampling and reasoning. |
| GPT-4.1 | gpt-4.1 | OpenAI API | Temperature 1 and seed; no reasoning mode. |
| Gemini 2.5 Pro | gemini-2.5-pro | Google API | Temperature 1; reasoning budget 128 tokens. |
| Own-codebook representation | Within-model reliability | |||||
|---|---|---|---|---|---|---|
| Model | Subcodes | Ordinal share | Coverage | Own | Shared | Difference |
| Opus 4.8 | 22.2 | .29–.74 | .741 | .861 | .861 | — |
| Sonnet 5 | 22.0 | .08–.81 | .745 | .861 | .843 | +.018 |
| Haiku 4.5 | 17.5 | .21–.88 | .670 | .804 | .819 | -.015 |
| Fable 5 | 24.8 | .13–.31 | .700 | .917 | .909 | +.008 |
| GPT-4.1 | 15.0 | .00–.53 | .606 | .837 | .838 | -.001 |
| Between-model | Conditional code dispersion | ||||||
|---|---|---|---|---|---|---|---|
| Theme | All 8 | Anthropic | Other | ||||
| AI Assessment | .677 | .705 | .650 | 19 | 16.0 | .942 | .120 |
| AI Training | .707 | .720 | .693 | 21 | 15.4 | .898 | .164 |
| AI Usage and Literacy | .724 | .739 | .711 | 23 | 20.0 | .955 | .094 |
| Teaming Check-ins | .664 | .686 | .653 | 23 | 19.3 | .943 | .099 |
| Employee Engagement | .661 | .681 | .650 | 26 | 22.6 | .958 | .103 |
LLM Auditing
Large language model auditing tests models and deployed systems for bias, safety, privacy, reliability, and provenance using behavioral and internal evidence.
See all 723 papers