Qualitative data can tell software teams what their stakeholders need, yet rigorously analyzing thousands of qualitative responses takes expert time that few teams can afford or justify. Teams therefore often face a false choice: leave the responses unanalyzed, or hand them to a large language model and accept its interpretation and judgment. In this paper, we take a pragmatic approach to AI-assisted qualitative analysis and present TACT (Traceable AI Council for Thematic analysis), a procedure that enables such analysis at scale while keeping teams in control of interpretation and judgment. TACT comes in two variants: TACT-Lite, the core AI council procedure, and TACT-Full, which adds adversaries (models from other model families that critique each model's output) and a decision model that surfaces high-uncertainty code assignments. We evaluated TACT on an independent dataset of 256 Stack Overflow threads qualitatively coded by other researchers. Against their labels, TACT's candidate codebook achieved an F1 of 0.75-0.77 and its final coding an F1 of 0.71-0.72. On a held-out sample, agreement between our own manual coding and TACT's coding was Cohen's κ of 0.79-0.85. TACT is also economical: model calls cost $13.77 (TACT-Lite) to $33.59 (TACT-Full) per 100 threads. Finally, drawing on our experience building and validating TACT, we identify pitfalls in AI-assisted qualitative analysis, describe safeguards to address them, and offer guidance for teams and researchers applying TACT in practice.
Figures & tables
Fig. 1: The tact procedure. After data preparation (Stage 0), Stages 1–3 develop the codebook: AI analysts identify candidate codes, a reconciler merges them, and researchers approve the codebook by coding a blind-pass sample themselves (†). Stages 4–5 apply the codebook to the corpus: AI coders code every unit, a program computes consensus and reliability, and researchers check the result against their own coding of a held-out sample. Models A, B, and C come from three different model families. Solid elements are tact -Lite; dashed elements are tact -Full’s additions, in which adversaries from another model family critique each AI analyst, reconciler, and each AI coder, and a decision model assigns uncertainty scores to code assignments and flags high-uncertainty ones for review.
Role or step ( n )
Inputs
AI analyst (3)
the corpus and the common prompt; no other AI analyst’s output
Discovery adversary F (3)
the corpus, one AI analyst’s output
Provenance check P
the AI analysts’ output and the corpus
Reconciler (1)
candidate codes that passed the provenance check (in tact -Full, with their critique logs); the overlap table
Recon. adversary F (1)
the merge plan, candidate codes and overlap table
Human review R ( ≥2 )
all stage outputs and critique logs, after the blind pass
TABLE I: Roles and steps in tact and their inputs. AI analysts and AI coders: one per model family. Adversaries: another model family. P A program. R Researchers. F tact -Full only.
Study
Used for
Corpus
Stages
Models
Reliability
Human check
To Copilot and Beyond [ 4 ]
themes of AI systems developers want, and their constraints
860 Microsoft developers; 2,580 response sets
1–5
GPT-5.2, Gemini 3.1 Pro, Claude Opus 4.6
α per theme 0.81–0.97, mean 0.94
Two researchers reviewed every theme and negotiated the codebook; two spot-checked rationales
What Makes a Great Co-worker [ 6 ]
checking the authors’ coding; coding rating reasons and work etiquette
α 0.84 (transcripts); median 0.88 (reasons), 0.94 (etiquette)
Two researchers reviewed every theme and negotiated the codebook; two spot-checked rationales
You Shall Not Pass [ 5 ]
classifying responses onto levels of AI autonomy
448 developers; 1,535 task-level records
4–5 (fixed rules)
GPT-5.4, Gemini 3 Flash, Claude Sonnet 4.6
three-rater α 0.93 (two researchers and the council)
Two researchers coded 67 random response sets: human–human κ 0.94; human–council κ 0.95
TABLE II: tact -Lite in the field: three studies that ran all or part of tact -Lite before this paper. Stages: 1 discovery, 2 reconciliation, 3 human review and approval, 4 systematic coding, 5 reliability and consensus.
Codebook
Coding
RQ
Labels
R
P
F1
Labels
R
P
F1
α
Functions
tact -Lite
27
0.70
0.59
0.64
27
0.59
0.68
0.63
0.83
tact -Full
0.74
0.66
0.70
0.59
0.68
0.63
0.94
Purposes
tact -Lite
24
0.88
0.65
0.75
24
0.79
0.82
0.81
0.77
tact -Full
0.96
0.70
0.81
0.83
0.86
0.85
0.83
Benefits
tact -Lite
21
0.86
0.77
0.81
20
0.70
0.76
0.73
0.64
TABLE III: Agreement of tact -Lite and tact -Full with Zhang et al.’s labels [ 7 ] at the codebook and coding stages (RQ: research question; R: recall; P: precision), and the AI coders’ median α .
tact -Lite
tact -Full
Stage
Calls
In
Out
$
Calls
In
Out
$
Discovery
19
3.26
0.64
23.02
31
6.79
1.11
48.69
Reconciliation
5
0.30
0.15
5.14
15
1.12
0.33
12.64
Coding
33
1.59
0.07
7.10
99
4.95
0.34
24.67
Total
57
5.15
0.85
35.26
145
12.86
1.78
86.00
TABLE IV: Model calls, input and output tokens (millions) and cost (USD) of tact -Lite and tact -Full by stage for the 256 threads, at list prices in September 2026. Calls and tokens count generative model calls; tact -Full’s costs also include the decision model’s screening requests.
Stage
Common pitfall
Why it matters
Example safeguard
Discovery
Accepting evidence that is almost right
Edited quotes and inflated support lists pass every agreement check, because AI coders agree on them as readily as on exact ones
Check every identifier and quote against the corpus by program; researchers then judge whether the quotes support the code
Trusting that a model analyzed the whole corpus
A reported count of units analyzed can be correct while the analysis was partial
Require every unit to be cited by a code or labeled as having no code; analyze large corpora in batches
Ignoring order, prompt and batch effects
Code lists depend on the prompt and on the order of units [ 56 ]
Give every AI analyst one versioned prompt and the same three analysis layers
Reconciliation
Merging on similar wording, or averaging opposed positions
Disagreement between AI analysts disappears before reliability is computed
Merge only codes that share units and describe the same thing, with a rationale for each group in the merge plan, which researchers review
Human review
Reviewing the model codes before coding the data independently
Patterns no model proposed stay unseen, and model codes anchor researchers [ 35 , 25 ]
Researchers code a sample before seeing model output, note what the council lacks, and settle codes by negotiated agreement
Coding
Revising the codebook during coding
Early and late batches follow different schemes
Close the codebook before coding, and code new content uncovered for the researchers
TABLE V: Common pitfalls in AI-assisted qualitative analysis, why they matter, and example safeguards, each built into tact .
AI-assisted qualitative data analysis (QDA) offers unprecedented opportunities to streamline software engineering (SE) research, yet uncritical use risks compromising analytical rigor and flooding the field with accelerated production of low-quality research. While tactical best practices will naturally evolve over time, SE researchers currently lack strategic guidance to identify and mitigate methodological risks when attempting AI-assisted QDA. Based on our decades of qualitative SE research expertise and experience combined with an understanding of the emerging landscape of AI-assisted QDA, this paper presents a catalog of antipatterns in AI-assisted QDA - a set of assumptions and practices that initially appear advantageous but ultimately undermine analytical rigor and validity. The antipatterns are grouped into three categories reflecting escalating impact: Dangerous Drivers, Operational Missteps, and Analytical Failures. As more SE researchers attempt AI-assisted QDA, these antipatterns will help them identify and avoid common temptations and pitfalls, while reviewers can be equipped with the vocabulary and criteria to call out problematic and failed practice. Ultimately, this catalog of antipatterns can serve as a stepping stone in our responsible methodological evolution toward principled and meaningful human-AI collaboration in qualitative research.
Rashina Hoda, Carolyn Seaman, Victoria Gomes +1
Monash University, Melbourne, Australia · University of Maryland Baltimore County, Baltimore, USA · Virginia Commonwealth University, Richmond, USA
Large Language Models (LLMs) offer new possibilities for scaling qualitative analysis, but existing applications often provide limited methodological transparency regarding how qualitative methods are translated into computational procedures. This paper presents an auditable and privacy-preserving computational operationalization of inductive and latent Thematic Analysis (TA). This paper first derives five design principles from the methodological requirements of TA and the conditions introduced by LLM-based inference: preserving interpretative context, maintaining traceable relationships between empirical material and analytical outputs, representing analytical constructs and reasoning explicitly, constraining LLM inference to interpretative tasks, and enabling privacy-preserving local deployment. Second, it presents a proof-of-concept for a two-phase workflow that operationalizes these principles by combining interpretative LLM inference with deterministic procedural control to generate codes, analytical justifications, themes, and theme descriptions while preserving explicit links to the source material. Third, it proposes an evaluation framework combining structural comparison with human-led TA and independent expert assessment of analytical quality. The evaluation is conducted on semi-structured Danish interview transcripts. and the results shows that the workflow produces code-level outputs with coverage broadly comparable to human annotations and highly rated analytical justifications, while generating a more compressed thematic structure characterized by fewer and broader themes. The findings demonstrate the feasibility of auditable LLM-supported TA through a modular workflow designed to scale to larger datasets, accommodate different LLMs, and support transfer across research domains, with domain adaptation primarily requiring adjustments to the prompting strategy.
Nadia Jul Jeldtoft, Tariq Yousef
Department of Culture and Language 2Department of Mathematics and Computer Science University of Southern Denmark
Large language models are increasingly used for qualitative data analysis, but many workflows obscure how analytic conclusions are produced. We present QualAnalyzer, an open-source Chrome extension for Google Workspace that supports atomistic LLM analysis by processing each data segment independently and preserving the prompt, input, and output for every unit. Through two case studies -- holistic essay scoring and deductive thematic coding of interview transcripts -- we show that this approach creates a legible audit trail and helps researchers investigate systematic differences between LLM and human judgments. We argue that process auditability is essential for making LLM-assisted qualitative research more transparent and methodologically robust.
Max Hao Lu, Ryan Ellegood, Rony Rodriguez-Ramirez +1