Qualitative data can tell software teams what their stakeholders need, yet rigorously analyzing thousands of qualitative responses takes expert time that few teams can afford or justify. Teams therefore often face a false choice: leave the responses unanalyzed, or hand them to a large language model and accept its interpretation and judgment. In this paper, we take a pragmatic approach to AI-assisted qualitative analysis and present TACT (Traceable AI Council for Thematic analysis), a procedure that enables such analysis at scale while keeping teams in control of interpretation and judgment. TACT comes in two variants: TACT-Lite, the core AI council procedure, and TACT-Full, which adds adversaries (models from other model families that critique each model's output) and a decision model that surfaces high-uncertainty code assignments. We evaluated TACT on an independent dataset of 256 Stack Overflow threads qualitatively coded by other researchers. Against their labels, TACT's candidate codebook achieved an F1 of 0.75-0.77 and its final coding an F1 of 0.71-0.72. On a held-out sample, agreement between our own manual coding and TACT's coding was Cohen's κ of 0.79-0.85. TACT is also economical: model calls cost $13.77 (TACT-Lite) to $33.59 (TACT-Full) per 100 threads. Finally, drawing on our experience building and validating TACT, we identify pitfalls in AI-assisted qualitative analysis, describe safeguards to address them, and offer guidance for teams and researchers applying TACT in practice.
Figures & tables
Fig. 1: The tact procedure. After data preparation (Stage 0), Stages 1–3 develop the codebook: AI analysts identify candidate codes, a reconciler merges them, and researchers approve the codebook by coding a blind-pass sample themselves (†). Stages 4–5 apply the codebook to the corpus: AI coders code every unit, a program computes consensus and reliability, and researchers check the result against their own coding of a held-out sample. Models A, B, and C come from three different model families. Solid elements are tact -Lite; dashed elements are tact -Full’s additions, in which adversaries from another model family critique each AI analyst, reconciler, and each AI coder, and a decision model assigns uncertainty scores to code assignments and flags high-uncertainty ones for review.
Role or step ( n )
Inputs
AI analyst (3)
the corpus and the common prompt; no other AI analyst’s output
Discovery adversary F (3)
the corpus, one AI analyst’s output
Provenance check P
the AI analysts’ output and the corpus
Reconciler (1)
candidate codes that passed the provenance check (in tact -Full, with their critique logs); the overlap table
Recon. adversary F (1)
the merge plan, candidate codes and overlap table
Human review R ( ≥2 )
all stage outputs and critique logs, after the blind pass
TABLE I: Roles and steps in tact and their inputs. AI analysts and AI coders: one per model family. Adversaries: another model family. P A program. R Researchers. F tact -Full only.
Study
Used for
Corpus
Stages
Models
Reliability
Human check
To Copilot and Beyond [ 4 ]
themes of AI systems developers want, and their constraints
860 Microsoft developers; 2,580 response sets
1–5
GPT-5.2, Gemini 3.1 Pro, Claude Opus 4.6
α per theme 0.81–0.97, mean 0.94
Two researchers reviewed every theme and negotiated the codebook; two spot-checked rationales
What Makes a Great Co-worker [ 6 ]
checking the authors’ coding; coding rating reasons and work etiquette
α 0.84 (transcripts); median 0.88 (reasons), 0.94 (etiquette)
Two researchers reviewed every theme and negotiated the codebook; two spot-checked rationales
You Shall Not Pass [ 5 ]
classifying responses onto levels of AI autonomy
448 developers; 1,535 task-level records
4–5 (fixed rules)
GPT-5.4, Gemini 3 Flash, Claude Sonnet 4.6
three-rater α 0.93 (two researchers and the council)
Two researchers coded 67 random response sets: human–human κ 0.94; human–council κ 0.95
TABLE II: tact -Lite in the field: three studies that ran all or part of tact -Lite before this paper. Stages: 1 discovery, 2 reconciliation, 3 human review and approval, 4 systematic coding, 5 reliability and consensus.
Codebook
Coding
RQ
Labels
R
P
F1
Labels
R
P
F1
α
Functions
tact -Lite
27
0.70
0.59
0.64
27
0.59
0.68
0.63
0.83
tact -Full
0.74
0.66
0.70
0.59
0.68
0.63
0.94
Purposes
tact -Lite
24
0.88
0.65
0.75
24
0.79
0.82
0.81
0.77
tact -Full
0.96
0.70
0.81
0.83
0.86
0.85
0.83
Benefits
tact -Lite
21
0.86
0.77
0.81
20
0.70
0.76
0.73
0.64
TABLE III: Agreement of tact -Lite and tact -Full with Zhang et al.’s labels [ 7 ] at the codebook and coding stages (RQ: research question; R: recall; P: precision), and the AI coders’ median α .
tact -Lite
tact -Full
Stage
Calls
In
Out
$
Calls
In
Out
$
Discovery
19
3.26
0.64
23.02
31
6.79
1.11
48.69
Reconciliation
5
0.30
0.15
5.14
15
1.12
0.33
12.64
Coding
33
1.59
0.07
7.10
99
4.95
0.34
24.67
Total
57
5.15
0.85
35.26
145
12.86
1.78
86.00
TABLE IV: Model calls, input and output tokens (millions) and cost (USD) of tact -Lite and tact -Full by stage for the 256 threads, at list prices in September 2026. Calls and tokens count generative model calls; tact -Full’s costs also include the decision model’s screening requests.
Stage
Common pitfall
Why it matters
Example safeguard
Discovery
Accepting evidence that is almost right
Edited quotes and inflated support lists pass every agreement check, because AI coders agree on them as readily as on exact ones
Check every identifier and quote against the corpus by program; researchers then judge whether the quotes support the code
Trusting that a model analyzed the whole corpus
A reported count of units analyzed can be correct while the analysis was partial
Require every unit to be cited by a code or labeled as having no code; analyze large corpora in batches
Ignoring order, prompt and batch effects
Code lists depend on the prompt and on the order of units [ 56 ]
Give every AI analyst one versioned prompt and the same three analysis layers
Reconciliation
Merging on similar wording, or averaging opposed positions
Disagreement between AI analysts disappears before reliability is computed
Merge only codes that share units and describe the same thing, with a rationale for each group in the merge plan, which researchers review
Human review
Reviewing the model codes before coding the data independently
Patterns no model proposed stay unseen, and model codes anchor researchers [ 35 , 25 ]
Researchers code a sample before seeing model output, note what the council lacks, and settle codes by negotiated agreement
Coding
Revising the codebook during coding
Early and late batches follow different schemes
Close the codebook before coding, and code new content uncovered for the researchers
TABLE V: Common pitfalls in AI-assisted qualitative analysis, why they matter, and example safeguards, each built into tact .