Automated Thematic Analysis for Clinical Qualitative Data: Iterative Codebook Refinement with Full Provenance

Seungjun Yi; Joakim Nguyen; Huimin Xu; Terence Lim; Joseph Skrovan; Mehak Beri; Hitakshi Modi; Andrew Well

cs.CLMar 9, 2026

Automated Thematic Analysis for Clinical Qualitative Data: Iterative Codebook Refinement with Full Provenance

Authors: Seungjun Yi, Joakim Nguyen, Huimin Xu, Terence Lim, Joseph Skrovan, Mehak Beri, Hitakshi Modi, Andrew Well, +4 more

Organizations: Department of Biomedical Engineering, University of Texas at Austin, Austin, TX, USA · Department of Computer Science, University of Texas at Austin, Austin, TX, USA · School of Information, University of Texas at Austin, Austin, TX, USA · College of Natural Sciences, University of Texas at Austin, Austin, TX, USA · Graphen, Inc., New York, NY, USA · Department of Pediatrics, University of Texas at Austin, Austin, TX, USA · Department of Cardiac Surgery, Division of Pediatric Cardiac Surgery, Vanderbilt University School of Medicine, Nashville, TN, USA · Pediatric Heart Institute, Monroe Carell Jr. Children’s Hospital at Vanderbilt, Nashville, TN, USA · Department of Population Health, Dell Medical School, University of Texas at Austin, Austin, TX, USA

Abstract

Thematic analysis (TA) is widely used in health research to extract patterns from patient interviews, yet manual TA faces challenges in scalability and reproducibility. LLM-based automation can help, but existing approaches produce codebooks with limited generalizability and lack analytic auditability. We present an automated TA framework combining iterative codebook refinement with full provenance tracking. Evaluated on five corpora spanning clinical interviews, social media, and public transcripts, the framework achieves the highest composite quality score on four of five datasets compared to six baselines. Iterative refinement yields statistically significant improvements on four datasets with large effect sizes, driven by gains in code reusability and distributional consistency while preserving descriptive quality. On two clinical corpora (pediatric cardiology), generated themes align with expert-annotated themes.

Figures & tables

Figure 1: The Traceable Thematic Analysis Framework

	AAOCA 11		SV-CHD 11		Ali		Dreaddit 17		Sheffield
Method	Score	$\Delta$	Score	$\Delta$	Score	$\Delta$	Score	$\Delta$	Score	$\Delta$
OpenCoding 16	0.493		0.450		0.473		0.428		0.516
LLOOM 9	0.476		0.495		0.491		0.458		0.484
GraphRAG 7	0.433		0.391		0.431		0.382		0.467
LightRAG 8	0.415		0.501		0.504		0.368		0.505
Thematic-LM 14	0.485		0.473		0.395		0.375		0.629

Table 1: Comparative performance across methods. Scores report the composite quality metric (mean of Reusability, Fitness, Coverage, Parsimony, and Consistency). LOGOS (Iter-1) and LOGOS (Best) report means over

n{=}5

independent replicates.

\Delta

denotes improvement from iterative refinement. Statistical significance is assessed via paired

t

-tests: ***

p<0.01

Figure 2: Average quality metric scores across five datasets. Error bars indicate $\pm\sigma$ across datasets.

Figure 3: Quality metric trajectories across iterative refinement for all five datasets ( $n{=}5$ replicates each). Solid lines show the mean; shaded bands denote 95% confidence intervals. Metrics shown: Reusability , Fitness , Coverage , Parsimony , Consistency , and Overall (dashed grey; unweighted mean). Iteration 0 is the initial codebook before refinement. Gains emerge within 2–4 iterations and plateau thereafter.

Generated Theme	Closest Human Theme	Sim.
AAOCA (8 generated; mean = .487)
Parental fears & protective instincts	Feeling that my child is safe	.587
Balancing caregiving & prof. responsibilities	Partnership with the care team	.558
Child’s engagement & understanding	Feeling that my child is safe	.515
Emotional & psychological impacts	Coping with stress, anxiety, depression	.504
Communication complexities in health mgmt.	Communicating risks appropriately	.496

Table 2: Comparison of framework-generated themes with human-annotated themes 11 , matched by cosine similarity (all-MiniLM-L6-v2). Rows sorted by descending similarity.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Dataset	Metric	Iter-1	Best	$\boldsymbol{\Delta}$	$\boldsymbol{t}$	$\boldsymbol{p}$	Sig.	$\boldsymbol{d}$ / Effect
AAOCA	Reusability	.119	.140	+.021	$-$ 4.98	.008		2.23 / L
	Fitness	.702	.791	+.089	$-$ 4.00	.016	**	1.79 / L
	Coverage	.600	.760	+.160	$-$ 6.47	.003		2.89 / L
	Parsimony	.569	.608	+.039	$-$ 15.92	$<$ .001		7.12 / L
	Consistency	.362	.362	.000	—	—	—	—
	Overall	.471	.509	+.039	$-$ 6.23	.003		2.78 / L

Appendix

Table 3: Per-metric statistical analysis of iterative refinement (Iter-1 vs. Best). All values are means over

n{=}5

independent replicates.

\Delta

: improvement from refinement.

t

: paired

t

-statistic (df = 4).

d

: Cohen’s

d

(paired). **

p<0.05

; ***

p<0.01

. Effect sizes: S = small (

|d|<0.5

), M = medium (

|d|<0.8

), L = large (

|d|\geq 0.8

Explore similar work

TAMA: A Human-AI Collaborative Thematic Analysis Framework Using Multi-Agent LLMs for Clinical Interviews
Mar 26, 2025Huimin Xu, Seungjun Yi, Terence Lim +9Multi-Agent LLMs Clinical NLP
Department of Biomedical Engineering, University of Texas at Austin, USA · McCombs School of Business, University of Texas at Austin, USA · School of Information, University of Texas at Austin, Austin, TX, USA and Dell Medical School, University of Texas at Austin, Austin, TX, USA
Designing an Auditable LLM-Supported Workflow for Qualitative Thematic Analysis
Aug 31, 2026Nadia Jul Jeldtoft, Tariq YousefLLM-Assisted Annotation
Department of Culture and Language 2Department of Mathematics and Computer Science University of Southern Denmark
CentaurTA Studio: A Self-Improving Human-Agent Collaboration System for Thematic Analysis
Mar 18, 2026Lei Wang, Min Huang, Eduard DragutHuman-AI Collaboration LLM Agent Self-Improvement
Temple University · Independent Researcher

Mar 26, 2025J/K move · Enter open · S save

TAMA: A Human-AI Collaborative Thematic Analysis Framework Using Multi-Agent LLMs for Clinical Interviews

Huimin Xu, Seungjun Yi, Terence Lim, Jiawei Xu+8

Department of Biomedical Engineering, University of Texas at Austin, USA · McCombs School of Business, University of Texas at Austin, USA · School of Information, University of Texas at Austin, Austin, TX, USA and Dell Medical School, University of Texas at Austin, Austin, TX, USA

Multi-Agent LLMs Clinical NLP Human-AI Collaboration

Open paper AskPDF arXiv

Thematic analysis (TA) is a widely used qualitative approach for uncovering latent meanings in unstructured text data. TA provides valuable insights in healthcare but is resource-intensive. Large Language Models (LLMs) have been introduced to perform TA, yet their applications in high-stakes healthcare settings, particularly for qualitative clinical interview analysis, remain limited. Here, we propose TAMA: A Human-AI Collaborative Thematic Analysis framework using Multi-Agent LLMs for clinical interviews. We leverage the scalability and coherence of multi-agent systems through structured conversations between agents and coordinate the expertise of cardiac experts in TA. Using interview transcripts from parents of children with Anomalous Aortic Origin of a Coronary Artery (AAOCA), a rare congenital heart disease, we demonstrate that TAMA outperforms single-agent LLM TA approaches, achieving higher thematic hit rate, coverage, and distinctiveness. TAMA demonstrates strong potential for automated TA in clinical settings by leveraging multi-agent LLM systems with human-in-the-loop integration by enhancing quality while significantly reducing manual workload. The full implementation is publicly available at https://github.com/Charlie-Yi-SJ/TAMA.

Abstract

Explore similar work

TAMA: A Human-AI Collaborative Thematic Analysis Framework Using Multi-Agent LLMs for Clinical Interviews

Designing an Auditable LLM-Supported Workflow for Qualitative Thematic Analysis

CentaurTA Studio: A Self-Improving Human-Agent Collaboration System for Thematic Analysis