Structure vs. Chain-of-Thought: Evaluating LLM Criteria Extraction for Depression Severity

Xinkai Chen

cs.CLSep 30, 2026

Structure vs. Chain-of-Thought: Evaluating LLM Criteria Extraction for Depression Severity

Authors: Xinkai Chen

Organizations: Independent Researcher

Abstract

A large language model (LLM) can rate depression severity directly from a social media post or mark which clinical criteria the post shows and let code turn the count into a label. The latter is easier to audit because a clinician can check each marked criterion. We compare these approaches on two Reddit corpora using three LLMs (from 9B to frontier scale) and two questionnaires (PHQ-9, BDI-II), and measure agreement with quadratic weighted kappa. For the two frontier models, criteria extraction scores above chain-of-thought on one corpus only when its decision thresholds are fitted on labeled data. Neither model's gain is significant, with or without recalibrating chain-of-thought on the same labels. With thresholds fixed a priori from PHQ-9's criteria, extraction shows no gain on either corpus, even where models mark over two criteria per post. The 9B model behaves differently on a corpus from depression communities. It labels most posts severe, whether prompted directly or with chain-of-thought, while the a priori rule beats both without labels. After chain-of-thought is recalibrated on the same labels, no significant gap remains, consistent with a calibration effect. Yet higher ordinal agreement does not ensure better detection of severe cases. PHQ-9 criteria extraction misses most severe posts, and moving from direct prompting to chain-of-thought and then to extraction increases misses in nearly all comparisons. On the primary corpus, a relabeled stress dataset, a model using that dataset's own features, including word counts from the text, is not significantly different from frontier criteria extraction under the a priori rule.

Figures & tables

	DepSeverity	DepSign
lowest class	513 minimum	184 not dep.
middle class(es)	58 mild / 79 mod.	472 mod.
highest class	56 severe	50 severe
majority acc.	0.727	0.669
majority $\kappa_{w}$	0.000	0.000
median words	80	104

Table I: Test splits. Both corpora are evaluated at

n{=}706

DepSeverity (4 classes, 56 severe )
Model	Cond.	$\kappa_{w}$	MAE	Acc	F1 M	sev $\downarrow$
Qwen3.5-9B	C1	0.288	0.948	0.445	0.335	29/56
	C2	0.340	0.877	0.456	0.327	34/56
	C3	0.489 / 0.322	0.465	0.728	0.367	43/56
	C4	0.495 /0.470/0.057	0.517	0.643	0.420	40/56
DeepSeek	C1	0.219	1.280	0.329	0.251	16/56

Table II: Test-set results,

n{=}706

per corpus.

\kappa_{w}

= quadratic weighted kappa; F1 M = macro-F1; sev

\downarrow

= fraction of severe posts given a lower label (

1-{}

recall on that class). C3 rows show fitted / a priori (DSM-5) thresholds, C4 rows fitted / DSM-5 / BDI-II bands; other columns are fitted. Bold: highest

\kappa_{w}

per model and corpus, fewest missed severe posts per corpus.

Fig. 1: Paired bootstrap

\Delta\kappa_{w}

on the same posts (4000 resamples); solid markers exclude zero at 95%, faded hollow ones do not. C2

-

C1 uses no thresholds. C4 is in Table II .

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Corpus	Model	Comparison	$\Delta\kappa_{w}$ [95% CI]
DepSeverity	Qwen3.5-9B	C2 $-$ C1	+0.052 [+0.009, +0.098]	C
DepSeverity	Qwen3.5-9B	C3 fitted $-$ C2	+0.149 [+0.072, +0.223]	C,H
DepSeverity	Qwen3.5-9B	C3 a priori $-$ C2	$-$ 0.018 [ $-$ 0.089, +0.054]	C
DepSeverity	Qwen3.5-9B	C4 fitted $-$ C2	+0.155 [+0.082, +0.227]
DepSeverity	Qwen3.5-9B	C4 a priori $-$ C2	+0.130 [+0.050, +0.208]
DepSeverity	Qwen3.5-9B	C3 fitted $-$ C1	+0.201 [+0.127, +0.276]

Appendix

Table III: All paired comparisons (

\Delta\kappa_{w}

, 95% paired bootstrap interval, 4000 resamples). Rows marked C are the confirmatory family; H = survives Holm correction at

\alpha=0.05

. C4 uses the DSM-5 rule for a priori .

Corpus	Model	Cond.	fitted	a priori
DepSeverity	Qwen3.5-9B	C3	[0.5, 1, 1.5]	[0.5, 2.5, 4.5]
DepSeverity	Qwen3.5-9B	C4	[0.5, 1.5, 3.5]	[0.5, 2.5, 4.5]
DepSeverity	DeepSeek	C3	[0.5, 1.5, 2.5]	[0.5, 2.5, 4.5]
DepSeverity	DeepSeek	C4	[0.5, 1.5, 3.5]	[0.5, 2.5, 4.5]
DepSeverity	Claude	C3	[0.5, 1.5, 2.5]	[0.5, 2.5, 4.5]
DepSeverity	Claude	C4	[0.5, 2.5, 5.5]	[0.5, 2.5, 4.5]

Appendix

Table IV: Thresholds on the count of present criteria (C3: 0–9, C4: 0–21). Fitted thresholds maximize

\kappa_{w}

on the 600-post fitting split.

Corpus	Model		raw	+cal	map	C3 fitted $-$ +cal
DepSeverity	Qwen3.5-9B	C1	0.288	0.320	0012	+0.169 [+0.090, +0.246]
DepSeverity	Qwen3.5-9B	C2	0.340	0.379	0013	+0.109 [+0.027, +0.189]
DepSeverity	DeepSeek	C1	0.219	0.330	0002	+0.196 [+0.113, +0.277]
DepSeverity	DeepSeek	C2	0.504	0.504	0123	+0.022 [ $-$ 0.039, +0.086]
DepSeverity	Claude	C1	0.299	0.352	0013	+0.147 [+0.067, +0.226]
DepSeverity	Claude	C2	0.462	0.462	0123	+0.037 [ $-$ 0.029, +0.103]

Appendix

Table V: C1 and C2 given the same labels as C3: a monotone relabeling fitted on the 600-post fitting split. The map lists, for each predicted label from lowest to highest, the label it becomes ( 0123 leaves labels unchanged). The last column is C3 fitted minus the recalibrated condition.

Corpus	Model	Comparison	$\Delta\kappa_{w}$ [95% CI]
DepSeverity	Qwen3.5-9B	C3 a priori $-$ C2	$-$ 0.058 [ $-$ 0.136, +0.023]
DepSeverity	Qwen3.5-9B	C4 fitted $-$ C2	+0.116 [+0.024, +0.202]
DepSeverity	Qwen3.5-9B	C4 a priori $-$ C2	+0.091 [ $-$ 0.001, +0.181]
DepSeverity	Qwen3.5-9B	C3 a priori $-$ C1	+0.002 [ $-$ 0.072, +0.079]
DepSeverity	Qwen3.5-9B	C4 fitted $-$ C1	+0.176 [+0.085, +0.259]
DepSeverity	Qwen3.5-9B	C4 a priori $-$ C1	+0.150 [+0.060, +0.235]

Appendix

Table VI: The a priori rules and C4 against C1 and C2. Every C1 and C2 prediction here is recalibrated on the 600-post fitting split with the monotone relabeling of Table V .

\Delta\kappa_{w}

with 95% paired bootstrap interval; exploratory; bold intervals exclude zero.

Model	Cond.	minimum	mild	moderate	severe
Qwen3.5-9B	C1	0.46	0.29	0.43	0.48
	C2	0.50	0.26	0.38	0.39
	C3 fitted	0.92	0.00	0.35	0.23
	C3 a priori	0.92	0.36	0.03	0.00
	C4 fitted	0.77	0.29	0.35	0.29
	C4 a priori	0.77	0.53	0.14	0.12

Appendix

Table VII: Per-class recall on DepSeverity test (

n

per class: 513, 58, 79, 56).

Model	Cond.	not dep.	moderate	severe
Qwen3.5-9B	C1	0.22	0.24	0.74
	C2	0.13	0.16	0.80
	C3 fitted	0.40	0.85	0.04
	C3 a priori	0.40	0.78	0.06
	C4 fitted	0.38	0.70	0.24
	C4 a priori	0.38	0.52	0.42

Appendix

Table VIII: Per-class recall on DepSign test (

n

per class: 184, 472, 50).

Corpus	Model	C1	C2	C3	C4
DepSeverity	Qwen3.5-9B	179 / 3	212 / 519	535 / 186	767 / 459
DepSeverity	DeepSeek	162 / 2	194 / 363	499 / 152	712 / 371
DepSeverity	Claude	238 / 5	282 / 545	737 / 227	1021 / 541
DepSign	Qwen3.5-9B	276 / 3	309 / 492	633 / 200	865 / 476
DepSign	DeepSeek	262 / 2	294 / 429	600 / 171	813 / 404
DepSign	Claude	369 / 4	413 / 614	871 / 251	1155 / 579

Appendix

Table IX: Mean tokens per test call, input / output, as reported by each provider.

Variant	pres./post	absent	fitted	a pr.	vs C2, fitted	vs C2, a pr.
original	0.45	0.30%	0.526	0.423	+0.022 [ $-$ 0.039, +0.086]	$-$ 0.081 [ $-$ 0.147, $-$ 0.010]
permissive	0.72	0.25%	0.507	0.425	+0.002 [ $-$ 0.059, +0.067]	$-$ 0.079 [ $-$ 0.145, $-$ 0.012]
symmetric evidence	0.47	0.35%	0.555	0.433	+0.051 [ $-$ 0.011, +0.116]	$-$ 0.071 [ $-$ 0.138, +0.000]

Appendix

Table X: C3 prompt variants, DeepSeek-V4.1-Flash on DepSeverity (same nine criteria and JSON schema; wording only). Last two columns:

\Delta\kappa_{w}

against C2.

Explore similar work

Fine-tuning LLMs for Passive Depression Severity Estimation from AI Mental Health Dialogue
Jun 16, 2026Olivier Tieleman, Ziyi Zhu, Ting Su +3Depression Well-Being
"Mirror" Large Language Model Evaluations of Depression are Criterion Contaminated
Date pendingTong Li, Rasiq Hussain, Mehak Gupta +1Depression Large Language Model Evaluation
Division of Computational & Data Sciences (DCDS), Washington University in St. Louis, St. Louis, MO · Department of Computer Science, SMU · Department of Psychological & Brain Sciences, Washington University in St. Louis
Depression Risk Assessment in Social Media via Large Language Models
Apr 21, 2026Giorgia Gulino, Manuel PetrucciDepression Social Media
1Guglielmo Marconi University, Department of Human Sciences, Rome, Italy

Jun 16, 2026, cs.CLJ/K move · Enter open · S save

Fine-tuning LLMs for Passive Depression Severity Estimation from AI Mental Health Dialogue

Olivier Tieleman, Ziyi Zhu, Ting Su, Samuel J. Bell+2

Depression Well-Being Severity Large Language Model Fine-Tuning Biases

Open paper AskPDF arXiv

Depression is the leading cause of disability worldwide, and early detection of symptom change is essential for timely intervention. Validated instruments such as the Patient Health Questionnaire-9 (PHQ-9) support symptom monitoring at scale, but real-world completion rates are low, introducing response bias and systematic missingness. Passive approaches that infer severity from routinely generated data could close this gap. We address this by predicting PHQ-9 total scores directly from transcripts of conversations between users and an AI mental health application, requiring only conversation text and no additional clinical data. We fine-tune a Qwen3.5-27B backbone with a regression head, augment 3,111 ground-truth labels with pseudolabels generated by a reasoning model (Claude Opus) and iteratively trained intermediate models, for a combined dataset of 6,283 users. On a held-out test set of 842 users, our best model achieves MAE = 2.6, RMSE = 4.0, Pearson r = 0.80, and AUC = 0.91 at the PHQ-9 >= 10 clinical threshold. We also find AUC > 0.87 at every severity threshold from PHQ-9 >= 3 to PHQ-9 >= 24, demonstrating that the model captures depression severity across the full clinical spectrum. This work opens the door to passive, continuous symptom monitoring in AI mental health platforms, without requiring users to complete self-report measures.

Abstract

Explore similar work

Fine-tuning LLMs for Passive Depression Severity Estimation from AI Mental Health Dialogue

"Mirror" Large Language Model Evaluations of Depression are Criterion Contaminated

Depression Risk Assessment in Social Media via Large Language Models