CentaurTA Studio: A Self-Improving Human-Agent Collaboration System for Thematic Analysis

Lei Wang; Min Huang; Eduard Dragut

cs.HCMar 18, 2026

CentaurTA Studio: A Self-Improving Human-Agent Collaboration System for Thematic Analysis

Authors: Lei Wang, Min Huang, Eduard Dragut

Organizations: Temple University · Independent Researcher

Abstract

Thematic analysis is difficult to scale: manual workflows are labor-intensive, while fully automated pipelines often lack controllability and transparent evaluation. We present CentaurTA Studio, a web-based system for self-improving human--agent collaboration in open coding and theme construction. The system integrates (1) a two-stage human feedback pipeline separating simulator drafting and expert validation, (2) persistent prompt optimization that distills validated feedback into reusable alignment principles, and (3) rubric-based evaluation with early stopping for process control. Across three domains, CentaurTA achieves the strongest performance in both Open Coding and Theme Construction, reaching up to 92.12% accuracy and consistently outperforming baseline systems. Agreement between the rubric-based LLM judge and human annotators reaches substantial reliability (average $κ= 0.68$ ). Ablation studies show that removing the feedback loop reduces performance from 90% to 81%, while eliminating the Critic or early stopping degrades accuracy or increases interaction cost. The full system reaches peak performance within 10 iterative rounds (about 25 minutes), demonstrating improved efficiency over expert-only refinement.

Figures & tables

Figure 1: The CentaurTA Studio framework integrates an Actor–Critic module, human–agent collaboration, and prompt optimization into a unified iterative loop. Through this closed-loop process, CentaurTA progressively aligns agents behavior with human subjective, knowledge, and preference, enabling iterative self-improving human–agent collaboration for qualitative analysis.

Figure 2: Interface of the Open Coding Lab. The system displays Actor-generated structured codes, independent Critic judgments, two-stage human feedback, and an editable Prompt Optimizer. Auto Run and Auto Critic enable scalable iterative refinement.

Figure 3: Interface of the Theme Aggregation Lab. Themes are constructed from validated codes with explicit definitions and references. Actor–Critic separation, expert-editable feedback, and automated prompt updating support traceable abstraction from codes to themes.

Dataset	# Doc	Lang	# w	# sent	Avg # w
USRS	12	Ch	25,718	471	2,143
ASP	15	Eng	12,678	651	845
Dreaddit	214	Eng	19,358	822	98

Table 1: Data statistics

PID	TA Exp.	Years	Position	HF	RC
P1	Expert	10	Prof.	Y	N
P2	Int.	4	PhD	Y	N
P3	Int.	3	PhD	Y	N
P4	Expert	7	Prof.	N	Y
P5	Int.	5	PhD	N	Y
P6	Int.	4	PhD	N	Y

Table 2: Expert roles and participation. Int. denotes intermediate practitioner; HF indicates human-feedback participation; RC indicates rubric curation.

Figure 4: Rubric-based evaluation (average scores) comparing Open Coding and Theme Construction across three datasets (USRS, ASP, Dreaddit).

Rubric Type	Rubric Acc	Human Acc	Human–LLM $\kappa$
General rules	92	93	0.71
Format / linguistic	94	91	0.68
Domain-specific	84	82	0.64
Avg	90	89	0.68

Table 3: Agreement between rubric-based LLM judge and human annotators.

Exp	Acc (%)	Time (m)	Rnd
CentaurTA	90	25	10
w/o human feedback	81	3	–
w/o simulated (expert-only)	90	42	10
w/o domain expert (sim-only)	85	7	5

Table 4: Feedback separation analysis.

Exp	Acc (%)	Time (m)	Rnd
CentaurTA	90	25	10
w/o prompt opt.	92	180	–
w/o critic agent	87	23	12
w/o rubric early stop	88	75	25

Table 5: Component ablation analysis.

Explore similar work

Automated Thematic Analysis for Clinical Qualitative Data: Iterative Codebook Refinement with Full Provenance
Mar 9, 2026Seungjun Yi, Joakim Nguyen, Huimin Xu +9LLM-Assisted Annotation Iterative Refinement
Department of Biomedical Engineering, University of Texas at Austin, Austin, TX, USA · Department of Computer Science, University of Texas at Austin, Austin, TX, USA · School of Information, University of Texas at Austin, Austin, TX, USA +6
Designing an Auditable LLM-Supported Workflow for Qualitative Thematic Analysis
Aug 31, 2026Nadia Jul Jeldtoft, Tariq YousefLLM-Assisted Annotation
Department of Culture and Language 2Department of Mathematics and Computer Science University of Southern Denmark
TAMA: A Human-AI Collaborative Thematic Analysis Framework Using Multi-Agent LLMs for Clinical Interviews
Mar 26, 2025Huimin Xu, Seungjun Yi, Terence Lim +9Multi-Agent LLMs Clinical NLP
Department of Biomedical Engineering, University of Texas at Austin, USA · McCombs School of Business, University of Texas at Austin, USA · School of Information, University of Texas at Austin, Austin, TX, USA and Dell Medical School, University of Texas at Austin, Austin, TX, USA

Mar 9, 2026J/K move · Enter open · S save

Automated Thematic Analysis for Clinical Qualitative Data: Iterative Codebook Refinement with Full Provenance

Seungjun Yi, Joakim Nguyen, Huimin Xu, Terence Lim+8

Department of Biomedical Engineering, University of Texas at Austin, Austin, TX, USA · Department of Computer Science, University of Texas at Austin, Austin, TX, USA · School of Information, University of Texas at Austin, Austin, TX, USA+6

LLM-Assisted Annotation Iterative Refinement

Open paper AskPDF arXiv

Thematic analysis (TA) is widely used in health research to extract patterns from patient interviews, yet manual TA faces challenges in scalability and reproducibility. LLM-based automation can help, but existing approaches produce codebooks with limited generalizability and lack analytic auditability. We present an automated TA framework combining iterative codebook refinement with full provenance tracking. Evaluated on five corpora spanning clinical interviews, social media, and public transcripts, the framework achieves the highest composite quality score on four of five datasets compared to six baselines. Iterative refinement yields statistically significant improvements on four datasets with large effect sizes, driven by gains in code reusability and distributional consistency while preserving descriptive quality. On two clinical corpora (pediatric cardiology), generated themes align with expert-annotated themes.

Abstract

Explore similar work

Automated Thematic Analysis for Clinical Qualitative Data: Iterative Codebook Refinement with Full Provenance

Designing an Auditable LLM-Supported Workflow for Qualitative Thematic Analysis

TAMA: A Human-AI Collaborative Thematic Analysis Framework Using Multi-Agent LLMs for Clinical Interviews