QuAD: Quantitative Assessment via Distribution-Alignment for Qualitative Data Quality Enabled by a Content-Aware Multi-Agent System

Wenshuo Zhang; Leixian Shen; Qiqi Zhou; Shuchang Xu; Feiyang Hao; Huamin Qu; Haobo Li; Zhao Chen

cs.AIOct 16, 2026

QuAD: Quantitative Assessment via Distribution-Alignment for Qualitative Data Quality Enabled by a Content-Aware Multi-Agent System

Authors: Wenshuo Zhang, Leixian Shen, Qiqi Zhou, Shuchang Xu, Feiyang Hao, Huamin Qu, Haobo Li, Zhao Chen,

Organizations: The Hong Kong University of Science and Technology · Shanghai Jiao Tong University · The Hong Kong University of Science and Technology (Guangzhou)

Abstract

Assessing whether qualitative responses collected from semi-structured questions are ready for a particular analysis goal is a persistent challenge. Manual qualitative analysis is interpretive and labor-intensive, while existing automated methods often lack a dataset-level perspective for deciding whether a response set supports a researcher-selected analytical lens. We introduce QuAD, a framework for quantitative pre-analysis assessment of qualitative response datasets. QuAD compares a reconstructed intrinsic cluster distribution reconstructed from responses with an extrinsic, researcher-defined analytical lens derived from metadata. QuAD is structured into a four-quadrant assessment whose metrics are read jointly, from lens-level relations to within-cluster variation, to provide fitness-for-purpose signals for revealing general patterns and unique insights. Because this assessment relies on lens-conditioned inter-response dissimilarity, we propose a content-aware multi-agent system with two-stage instruction generation and adaptive anchors to improve scoring consistency. We validate QuAD through controlled experiments on synthetic data, including robustness and baseline comparisons, and a real-world social survey case study. The case study shows how QuAD quantitatively assesses whether a selected analysis axis is promising and also surfaces representative or unusual responses for human interpretation.

Figures & tables

Figure 1: Overview of QuAD . QuAD takes a response dataset (A), uses a content-aware multi-agent system with iterative prompt generation (B) and adaptive anchor updating (C) to extract dissimilarity, reconstructs cluster and metadata distributions (D), and performs four-quadrant fitness-for-purpose assessment (E).

Figure 2: The Content-Aware Multi-agent System for Dissimilarity Extraction. It involves two parts: the iterative instruction generation part (Sec. 3.1.2 ) and the adaptive anchor manifold updating part (Sec. 3.1.3 ). The first part efficiently generates context-aware instructions for the second part to perform dissimilarity extraction.

Figure 3: The Four-Quadrant Assessment Framework by research goal (General Patterns/Unique Insights) and analysis level (Distribution/Data Point).

Figure 4: User study results: (a) cognitive load (higher = more load); (b)–(c) task satisfaction (7 = very satisfied).

	Small-Scale (Pairwise)				Large-Scale (Scalar Projection)
	Baseline		Ours		Baseline		Ours
	Spear.	Pears.	Spear.	Pears.	Spear.	Pears.	Spear.	Pears.
Qwen_turbo	-0.083	-0.035	0.708	0.681	0.687	0.713	0.747	0.775
Qwen_plus	0.228	0.292	0.725	0.718	0.735	0.753	0.719	0.741
Qwen_max	-0.036	-0.013	0.725	0.728	0.705	0.726	0.789	0.816
GLM	0.599	0.621	0.830	0.836	0.731	0.747	0.797	0.819

Table 1: Quality of instructions generated by baseline vs ours, measured by Spearman and Pearson correlation. Best indicates the higher value between Baseline and Ours for each metric (within the same setting).

Figure 5: Result of the distribution shift experiment.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Small-scale (pairwise) setting
	Full		No Example		Fixed Example		No Novelty Check		No Pruning		Heuristic Quality
Model	Spear.	Pears.	Spear.	Pears.	Spear.	Pears.	Spear.	Pears.	Spear.	Pears.	Spear.	Pears.
Qwen_turbo	0.708	0.681	0.644	0.656	0.659	0.663	0.604	0.625	0.620	0.604	0.606	0.640
Qwen_plus	0.725	0.718	0.674	0.659	0.760	0.757	0.700	0.683	0.652	0.667	0.643	0.636
Qwen_max	0.725	0.728	0.633	0.621	0.672	0.663	0.683	0.695	0.691	0.697	0.701	0.709
GLM	0.830	0.836	0.770	0.765	0.822	0.829	0.826	0.839	0.831	0.842	0.729	0.732

Appendix

Table 2: Ablation results of the adaptive anchor updating algorithm. Bold and underlined represents the top-1 result and bold represents the top-2 result (within each block).

Quantity	Role	How it is determined
$C$	Number of analytical groups	Defined by metadata and research question
$K$	Number of response clusters	Set to $C$ so each group has one matched cluster
Scale	LLM score range	Fixed by scoring rubric, e.g., $[0,\text{Scale}]$
$\sigma$	Typical dissimilarity unit	Derived from $\text{Scale}/C$
$\gamma$	DC normalization	Derived from scoring scale and group count
$k_{\max}$	Anchor capacity	Set by context-window and cost budget

Appendix

Table 3: Origin and role of main quantities in QuAD .

Method	Small Spear.	Small Pears.	Large Spear.	Large Pears.
Qwen3-Embed (cosine)	0.595	0.602	0.614	0.625
Qwen3-Embed (euclid)	0.595	0.593	0.614	0.619
OpenAI-3-small (cosine)	0.594	0.602	0.698	0.717
OpenAI-3-small (euclid)	0.594	0.594	0.698	0.709
Llama-Embed-8B (cosine)	0.621	0.629	0.683	0.700
Llama-Embed-8B (euclid)	0.621	0.606	0.683	0.693

Appendix

Table 4: Comparison with embedding-based dissimilarity extraction. Correlations are computed against synthetic ground-truth dissimilarities and averaged across four domains.

Generator	Evaluator	Small Spear.	Small Pears.	Large Spear.	Large Pears.
DeepSeek-v4-Flash	Gemini-3.1-Flash-Lite	0.733	0.754	0.842	0.855
Gemini-3.1-Flash-Lite	DeepSeek-v4-Flash	0.727	0.730	0.878	0.890

Appendix

Table 5: Cross-family generator–evaluator robustness check. Communities, candidates, and target scores are held fixed while only response text is regenerated.

Method	Ens.	ARI	NMI	Purity
K-medoids	No	0.639	0.710	0.830
K-medoids	Yes	0.631	0.698	0.831
Agglomerative	No	0.592	0.679	0.786
Agglomerative	Yes	0.592	0.679	0.786
Spectral	No	0.607	0.686	0.818
Spectral	Yes	0.608	0.686	0.818

Appendix

Table 6: Fixed-

K

clustering sensitivity and ensemble ablation.

Source	Dist. Spear.	GMR	DDR	DC	DP
qwen-max	0.80	$0.839\pm 0.050$	$0.231\pm 0.026$	$0.359\pm 0.032$	$0.898\pm 0.043$
deepseek-v3	0.94	$0.956\pm 0.008$	$0.321\pm 0.018$	$0.304\pm 0.019$	$0.968\pm 0.007$
Cross-model	0.87	–	–	–	–

Appendix

Table 7: Repeated-run and cross-model reliability. Metric entries are mean

\pm

SD over three runs.

	Pairwise			Scalar projection
Metric	$N=10$	$N=15$	$N=30$	$N=30$	$N=122$
Total tokens	55,108	115,345	449,765	13,391	56,727
Input tokens	46,781	98,231	384,725	11,088	47,707
Output tokens	8,327	17,114	65,040	2,303	9,020
Estimated cost (USD)	$0.0057	$0.0118	$0.0453	$0.0015	$0.0060

Appendix

Table 8: Measured API usage for pairwise and scalar inference on the food-choice task.

Model	Pairwise Spear./Pears.	Scalar Spear./Pears.
Qwen-turbo	0.708/0.681	0.747/0.775
Qwen-plus	0.725/0.718	0.719/0.741
Qwen-max	0.725/0.728	0.789/0.816

Appendix

Table 9: Correlation across Qwen model tiers.

Analysis	Quantity	Result
Grade–label association ( $N=122$ )	Cramér’s $V$	0.130
Same-label pairs ( $n=3{,}433$ )	Mean / median distance	14.61 / 10.00
Different-label pairs ( $n=3{,}588$ )	Mean / median distance	30.38 / 30.00
Pair-distance contrast	Mean gap	15.77

Appendix

Table 10: Association and pair-level validation against human-coded labels.

Method	Mismatch AUC	Top-3 pur.	Top-5 pur.	Fine-code AUC
Random	0.500	0.489	0.489	0.500
TF–IDF	0.551	0.650	0.619	0.555
Qwen3-Embed (cosine)	0.604	0.748	0.733	0.587
Direct LLM †	0.613	–	–	0.602
QuAD	0.749	0.742	0.694	0.629

Appendix

Table 11: Real-label comparison of response-distance and retrieval baselines (

N=119

). Fine-code AUC uses the dataset’s 13-category theme code ( eating_changes_coded1 ), which is not the scoring criterion of the QuAD instruction. † Direct LLM yields a partition rather than a distance, so neighbour purity is undefined; values are means over three shuffled-order runs (AUC SD

=0.077

). All entries are descriptive point estimates. The upper block does not see the research question; the lower block does.

Lens	GMR	DDR	DC	DP	Reading (Tab. 17 )
Grade level	0.39	0.02	0.90	0.43	Weak lens; themes cut across groups
Eating-habit change	0.53	0.52	0.72	0.75	Lens explains response structure

Appendix

Table 12: QuAD assessment of the same responses under a weak and a strong analytical lens. Grade-level values are those reported in Sec. 4.2 (all 122 responses). The eating-habit-change lens uses the 119 responses with a non-unclear code, since unclear is not a substantive group. Eating-habit-change values are means over 20 reconstruction runs (SD

0.06

for GMR and DP,

0

for DDR and DC).

$p$	0.0	0.2	0.4	0.6	0.8	1.0
DDR	0.52	0.39	0.28	0.13	0.07	0.02
DP	0.76	0.70	0.66	0.61	0.61	0.64
KW $\varepsilon^{2}$	0.45	0.27	0.15	0.08	0.02	0.01

Appendix

Table 13: Graded lens informativeness: a fraction

p

of eating-habit-change labels is randomly reassigned (

N=119

; means over 20 seeds). KW

\varepsilon^{2}

is the Kruskal–Wallis effect size of QuAD scores across the noisy groups.

Measure	Random	Qwen3-Embed	Direct LLM	QuAD
Representative hit rate	0.66	0.80	0.63	0.81
Unusual hit rate	0.34	0.29	0.47	0.82
Partition ARI (coarse / fine)	0.00 / 0.00	0.01 / 0.00	0.11 / 0.08	0.30 / 0.11
Grade level judged “strong”	–	–	3/3 runs	No (joint profile)
Top-3 overlap across reorderings (Jaccard)	–	–	0.12	1.00 §
Partition agreement across reorderings (ARI)	–	–	0.37	–

Appendix

Table 14: Direct LLM versus QuAD on the eight metadata lenses (means over lenses; LLM values also averaged over three shuffled-order runs). Triage hit rates use the human code as a proxy. Partition agreement is ARI with the coarse / fine human codes.

			Standard tests on QuAD scores				DDR (separation)
Lens	$N$ / $C$	$V^{\ast}$	$\chi^{2}$	KW	$\varepsilon^{2}$	PERM	value	$p$
Grade level	122/4	0.11	0.755	0.271	0.01	0.519	0.02	0.381
Gender	122/2	0.10	0.932	0.297	0.00	0.537	0.04	0.112
Living situation	119/3	0.17	0.701	0.706	0.00	0.781	0.00	1.000
Current diet	119/3	0.25	0.214	0.031	0.04	0.023	0.03	0.164
Self-rated health	122/3	0.20	0.181	0.080	0.03	0.056	0.03	0.100

Appendix

Table 15: Convergent-validity check of DDR, the group-separation component of the QuAD profile, against standard tests on the same case-study scores.

V^{\ast}

: association of the lens with the human code (unclear excluded).

\chi^{2}

uses the three-way rubric code; KW is Kruskal–Wallis on scores; PERM is PERMANOVA on

|S(d_{i})-S(d_{j})|

. DDR follows Sec. 3.2 . Entries are

p

-values unless noted;

p<0.05

is in bold.

Figure 6: Behavior changes across different grade levels.

Statement	E1	E2	E3	Median	Range
Correctness	4	6	7	6	4–7
Interpretability	6	7	6	6	6–7
Usefulness for coding	6	7	6	6	6–7

Appendix

Table 16: Step-2 report ratings by the three evaluators (seven-point agreement scales; higher is better).

Signals	Reading	Next step
High GMR, DP, DDR	The lens explains response structure	Compare groups; read representative responses first
Low GMR, DP, DDR; high DC	The lens weakly explains structure; themes cut across groups	Revise the lens or code cross-group themes; read unusual responses first
High GMR, high DC	Groups are recovered with wide internal spread	Code sub-themes within groups

Appendix

Table 17: Reading the joint QuAD profile at an analysis checkpoint.

ID	Group	Participant ID	Method Order	Task 1	Task 2	Time
1	Finance	Participant 1.1	A $\to$ B	Q1 (Method A)	Q2 (Method B)	1h
2	Finance	Participant 1.2	A $\to$ B	Q2 (Method A)	Q3 (Method B)	1h
3	Finance	Participant 1.3	A $\to$ B	Q3 (Method A)	Q1 (Method B)	1h
4	Finance	Participant 1.4	B $\to$ A	Q4 (Method B)	Q5 (Method A)	1h
5	Finance	Participant 1.5	B $\to$ A	Q5 (Method B)	Q4 (Method A)	1h
6	HCI	Participant 2.1	A $\to$ B	Q6 (Method A)	Q7 (Method B)	1h

Appendix

Table 18: Overview of User Study Schedule. We recruited 20 postgraduate students (11M/9F, aged 21-31) from diverse domains. Participants self-rated their domain experience (

M=3.50,SD=0.83

) and familiarity with instruction generation (

M=3.15,SD=1.27

) on a 5-point Likert scale. The system implements the adaptation phase of Sec. 3.1.2 . The baseline condition utilized the same UI but deactivated the agent-assisted features, requiring users to compose instructions manually. Participants were tasked with generating instructions for dissimilarity extraction for two distinct questions, using both ours and the baseline. Each session was scheduled for one hour, including about 45 minutes of tasks, and was compensated with 8 USD. Participants agreed to share their data for this research.

Dimension	Baseline Mean	Ours Mean	Mean Difference	t-statistic	p-value	Significant
1	3.30	6.35	3.05	6.45	$<$ 0.00001	Yes
2	3.35	5.85	2.50	6.69	$<$ 0.00001	Yes
3	3.00	6.10	3.10	5.66	0.00002	Yes
4	2.95	6.45	3.50	8.46	$<$ 0.00001	Yes
5	2.85	6.35	3.50	8.74	$<$ 0.00001	Yes
6	2.90	6.00	3.10	8.24	$<$ 0.00001	Yes

Appendix

Table 19: Statistical Comparison Between Baseline and QuAD

P ID	G	Age	Edu	Field	Prof.	Exp.	P/C/S
1.1	1	23	2	Finance Data Analysis	4	3	4/4/4
1.2	1	24	3	Economics	4	7	3/4/4
1.3	1	23	1	Finance	3	4	2/4/5
1.4	2	22	2	Finance	5	2	5/5/5
1.5	2	23	2	Finance	4	4	3/4/5
2.1	1	21	1	Computer Science, HCI & VIS	4	3	5/5/4

Appendix

Table 20: Participant Demographic Information. Ratings are on a 1–5 Likert scale. We report Prompt Familiarity / ChatGPT Usage / System Willingness as P/C/S.

Figure 7: In the first stage, users select a question for which they want to generate a prompt. This serves as the entry point into the system.

Figure 8: In the second stage, users manually input a prompt into the Prompt Zone and click Modify to initiate the dissimilarity extraction process. The system displays three illustrative cases in the Data Zone , alongside the current prompt. Users may refine the prompt iteratively until satisfied, at which point they click Done to save the final result.

Figure 9: Upon entering the system, users select a question for which they wish to generate a prompt. This step mirrors the baseline system’s question selection process.

Figure 10: Upon accessing the system, users are presented with an auto-generated initial prompt and dissimilarity extraction results in the Data Zone . Users can provide modifications in the Prompt Zone , triggering the system to automatically adjust the prompt. Once users are satisfied, they can click the Done button to save the final result. Alternatively, they can directly edit the prompt in the Prompt Zone for manual adjustments.

Automated Evaluation

Automated evaluation develops metrics, benchmarks, and AI judges to assess model outputs and system behavior at scale.

See all 396 papers

Affording Process Auditability with QualAnalyzer: An Atomistic LLM Analysis Tool for Qualitative Research
Apr 4, 2026Max Hao Lu, Ryan Ellegood, Rony Rodriguez-Ramirez +1LLM-Assisted Annotation LLM Auditing
Harvard University / United States
What Makes a Good Response? An Empirical Analysis of Quality in Qualitative Interviews
Apr 6, 2026Jonathan Ivey, Anjalie Field, Ziang XiaoAutomated Evaluation
Johns Hopkins University
A Framework for Deductive Semantic Content Analysis at Scale in Science Education Using Text Embeddings
Aug 27, 2025Jonas Timmann Mjaaland, Markus Fleten Kreutzer, Halvor Tyseng +5Educational Technology Text Embedding Evaluation
Center for Interdisciplinary Education, University of Oslo, NO-0316 Oslo, Norway · Laboratory of Atomic and Solid State Physics, Cornell University, Ithaca, NY 14853, USA · Department of Physics, California State University Fullerton, Fullerton, CA 92831, USA +1

Apr 4, 2026J/K move · Enter open · S save

Affording Process Auditability with QualAnalyzer: An Atomistic LLM Analysis Tool for Qualitative Research

Max Hao Lu, Ryan Ellegood, Rony Rodriguez-Ramirez, Sophia Blumert

Harvard University / United States

LLM-Assisted Annotation LLM Auditing

Open paper AskPDF arXiv

Large language models are increasingly used for qualitative data analysis, but many workflows obscure how analytic conclusions are produced. We present QualAnalyzer, an open-source Chrome extension for Google Workspace that supports atomistic LLM analysis by processing each data segment independently and preserving the prompt, input, and output for every unit. Through two case studies -- holistic essay scoring and deductive thematic coding of interview transcripts -- we show that this approach creates a legible audit trail and helps researchers investigate systematic differences between LLM and human judgments. We argue that process auditability is essential for making LLM-assisted qualitative research more transparent and methodologically robust.

Abstract

Automated Evaluation

Explore similar work

Affording Process Auditability with QualAnalyzer: An Atomistic LLM Analysis Tool for Qualitative Research

What Makes a Good Response? An Empirical Analysis of Quality in Qualitative Interviews

A Framework for Deductive Semantic Content Analysis at Scale in Science Education Using Text Embeddings