AudiPsy: Evidence-Grounded Multimodal Reasoning over Spoken Counseling Conversations for Depression Assessment

Chuanbo Zhu; Run Luo; Weibo Zhao; Yue Li; Kun Qian; Qiang Qu; Yike Guo; Minghuan Tan

cs.CLOct 20, 2026

AudiPsy: Evidence-Grounded Multimodal Reasoning over Spoken Counseling Conversations for Depression Assessment

Authors: Chuanbo Zhu, Run Luo, Weibo Zhao, Yue Li, Kun Qian, Qiang Qu, Yike Guo, Minghuan Tan,

Organizations: The Hong Kong University of Science and Technology · National University of Singapore · University of the Chinese Academy of Sciences · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences ·

Abstract

Depression assessment from counseling speech requires integrating textual and acoustic psychological evidence across turns, yet existing end-to-end classification approaches lack explicit multimodal evidence interpretation and context-aware reasoning. We propose AudiPsy, an evidence-grounded omni-LLM for depression assessment over spoken counseling conversations. AudiPsy employs an Observation-Assessment-Summary (OAS) paradigm to integrate dialogue transcripts, textual and acoustic emotions, and textual-acoustic emotion incongruence into structured reasoning. OAS incrementally organizes multimodal observations, performs contextualized assessment, and maintains an evolving dialogue summary for subsequent turns. To support this framework, we construct AudiPsy-Counsel, a synthetic spoken counseling conversation corpus with multimodal context augmentation, and develop a unified three-stage curriculum for depression-oriented reasoning. Experiments on DAIC-WOZ demonstrate superior depression assessment performance over specialized depression detection methods and general-purpose speech/omni-LLMs, along with improved reasoning quality, while supplementary evaluation on EATD-Corpus supports AudiPsy's applicability to naturalistic conversations.

Figures & tables

Figure 1: Overview of the AudiPsy, from structured multimodal reasoning to data augmentation and model training.

Figure 2: Topic distribution of AudiPsy-Counsel.

Method	MCC	F1	Recall	Precision	Specificity
BiLSTM + GRU ∗	0.11	0.53	0.56	0.56	-
WavLM Layer 10 Wu et al. (2023)	-	0.72	-	-	-
Llama2 + Acoustic Landmarks Zhang et al. (2024b)	-	0.72	-	-	-
SpeechT-RAG Zhang et al. (2025)	-	0.69	-	-	-
Kimi-Audio-7B-Instruct Ding et al. (2025)	0.00	0.00	0.00	0.00	0.00
MOSS-Audio-8B-Instruct Yang et al. (2026)	0.36	0.62	0.75	0.53	0.62

Table 1: Depression assessment performance on the DAIC-WOZ dataset in terms of MCC, F1, recall, precision, and specificity.

Panel of Figure 3 — Figure 3: Human evaluation results on the DAIC-WOZ dataset. (a) Relative scores of each method across four evaluation dimensions (EG, CP, MMI, and CEI), calculated as deviations from the mean score across methods. (b) Win–Tie–Loss percentages of AudiPsy against four baseline methods across the same evaluation dimensions.

Method	MCC	F1
BiLSTM + GRU ∗	0.19	0.32
MOSS-Audio-8B-Thinking	0.27	0.37
Gemini 2.5 Flash	0.13	0.29
Gemini 2.5 Pro	0.23	0.33
Qwen2.5-Omni-7B	0.11	0.21
Qwen3-Omni-30B-A3B-Instruct	0.20	0.33

Table 2: Depression assessment performance on EATD-Corpus in terms of MCC and F1.

Model	Positive				Neutral				Negative
Model	Happy	Angry	Neutral	Sad	Happy	Angry	Neutral	Sad	Happy	Angry	Neutral	Sad
Gemini 2.5 Pro	0.58	0.40	0.64	0.62	0.47	0.75	0.61	0.43	0.57	0.83	0.47	0.80
Qwen2.5-Omni-7B	0.45	0.18	0.64	0.33	0.00	0.33	0.45	0.17	0.00	0.80	0.36	0.56
AudiPsy (Ours)	0.39 -.06	0.18 –	0.44 -.20	0.67 +.34	0.64 +.64	0.33 –	0.67 +.22	0.64 +.47	0.63 +.63	0.89 +.09	0.67 +.31	0.84 +.28
AudiPsy w/o $\mathcal{L}_{\mathrm{S1}}$	0.44 -.01	0.00 -.18	0.53 -.11	0.00 -.33	0.56 +.56	0.33 –	0.56 +.11	0.29 +.12	0.18 +.18	0.67 -.13	0.32 -.04	0.62 +.06

Table 3: Per-class F1 scores across transcript polarity groups. Smaller values below AudiPsy and its ablation indicate absolute F1 changes relative to the Qwen2.5-Omni-7B backbone; green and red denote improvements and degradations, respectively.

Figure 4: Occurrence rates of evidence types employed by AudiPsy and its ablation without Emotion Incongruence predicting “Yes (Depressed)” or “No (Non-depressed)”.

Case 1 (EATD-Corpus)	Case 2 (DAIC-WOZ)
Regretful reflection + persistent low VAD	Family concerns + sad speech
$\rightarrow$ Prior EI suggests masked distress	$\rightarrow$ Prior EI, but context-consistent sadness
$\rightarrow$ Accumulated depressive evidence	$\rightarrow$ Limited cumulative depressive evidence
$\Rightarrow$ Depression assessment: Yes	$\Rightarrow$ Depression assessment: No

Table 4: Illustrative reasoning chains showing contextualized interpretation of affective evidence.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Field	Attribute	Value
Observation	Dialogue Transcript	Text dialogue content by counselor and seeker
	Textual Emotion	Natural-language description
	Text Polarity	Positive / Neutral / Negative
	Acoustic Emotion	Natural-language description
	Acoustic Emotion Category	Anger / Happy / Neutral / Sad / Surprise / Disgust / Fear
	VAD	Very Low / Low / Medium / High / Very High

Appendix

Table 5: Attributes and corresponding value specifications for each field in the OAS reasoning paradigm.

Statistic	Value
Total Dialogues	11,818
Total Utterances	70,908
Hours of Speech	163.75
CounselChat / CpsyCounR (Dialogue)	5,550 / 6,268
English / Chinese (Dialogue)	5,909 / 5,909
Male / Female (Dialogue)	39,018 / 31,890

Appendix

Table 6: Statistics of AudiPsy-Counsel.

Turn	Speaker	Utterance	Acoustic Emotion	EI
1	Seeker	People always think success keeps you safe, but lately I’ve felt exhausted and emotionally numb despite all my achievements.	indifferent, slightly amused	✓
1	Counselor	It’s more common than you think. Even those who seem successful on the outside may struggle with depression, especially when emotions have to be hidden under a strong exterior.	understanding, gentle	–
2	Seeker	I really don’t find joy in my daily life anymore, and it’s hard to share that with people who see me as someone who “has it all.”	downcast, hesitant	$\times$
2	Counselor	That pressure to always maintain your image can be exhausting. It can actually make it more difficult to express negative feelings or ask for help.	warm, emphasizing	–
3	Seeker	So maybe I should talk to close friends or professionals instead of bottling it up?	tentative, seeking reassurance	$\times$
3	Counselor	Absolutely. Reaching out is an important step. Taking time for personal relaxation and lowering your guard around trusted people can really help with emotional release.	encouraging, supportive	–

Appendix

Table 7: An example instance from AudiPsy-Counsel (Topic: Depression & Self-harm). The first seeker utterance is intentionally annotated as emotional incongruence (EI), where the expressed vocal emotion is inconsistent with the semantic content. Metadata: Source = CPsyCounR, Language = English, Topic = Depression & Self-harm, Gender = Male.

Speech Emotion	Neg.	Neu.	Pos.	Total
Angry	45	45	45	135
Disgust	2	0	0	2
Fear	2	26	2	30
Happy	90	90	90	270
Neutral	90	90	89	269
Sad	90	90	90	270

Appendix

Table 8: Distribution of the filtered IEMOCAP training set across acoustic emotion and text polarity categories.

Speech Emotion	Neg.	Neu.	Pos.	Total
Neutral	56	56	56	168
Happy	52	52	52	156
Sad	147	147	147	441
Angry	58	58	58	174
Fearful	23	27	11	61
Disgusted	21	7	1	29

Appendix

Table 9: Emotion Distribution of the SynEI dataset

Hyperparameter	Stage-1	Stage-2	Stage-3
Objective	SFT	SFT	ORPO
Epochs	3	3	1
Learning rate	$2{\times}10^{-5}$	$5{\times}10^{-5}$	$5{\times}10^{-6}$
Per-device batch size	8	4	4
Gradient accumulation	2	2	2
Max sequence length	2048	3072	3072

Appendix

Table 10: Stage-specific training hyperparameters.

Depression Detection

Depression detection uses language, behavior, speech, and biological signals to identify depressive symptoms and estimate their severity.

See all 77 papers

EmoTrack: Clinical-Semantic Modeling for Text-Based Depression Severity Estimation
May 21, 2026Zhaomin Wu, Jiayi Li, Bingsheng HeMental Health Counseling Depression Detection
Department of Computer Science National University of Singapore
Dynamic Summary Generation for Interpretable Multimodal Depression Detection
Apr 13, 2026Shiyu Teng, Jiaqing Liu, Hao Sun +6Depression Detection Depression Severity Estimation
College of Information Science and Engineering, Ritsumeikan University, Shiga, Japan · College of Computer Science and Technology, Zhejiang University, Hangzhou, China · Faculty of Engineering, University of the Ryukyus, Japan
Dep-LLM: Training-Free Depression Diagnosis via Evidence-Guided Structured Multi-factor with Reliable LLM Reasoning
Jun 9, 2026Yiqing Lyu, Xianbing Zhao, Buzhou Tang +1LLM Inference Depression Detection
School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen, Guangdong, China · School of Artificial Intelligence and Computer Science, Jiangnan University, Wuxi, China, Harbin Institute of Technology, Shenzhen, Guangdong, China, and Guangdong Provincial Key Laboratory of Intelligent Information Processing · School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen, Guangdong, China, Pengcheng Laboratory, Guangdong Provincial Key Laboratory of Intelligent Information Processing +1

May 21, 2026J/K move · Enter open · S save

EmoTrack: Clinical-Semantic Modeling for Text-Based Depression Severity Estimation

Zhaomin Wu, Jiayi Li, Bingsheng He

Department of Computer Science National University of Singapore

Mental Health Counseling Depression Detection Mental Health Clinical Outcome Prediction Depression Severity Estimation Clinical NLP

Open paper AskPDF arXiv

Text-based counseling provides a valuable source of information for assessing depression severity. We study prediction of the total score on the eight-item Patient Health Questionnaire (PHQ-8), a self-report measure of depression severity, from counseling transcripts. Clinical-based methods rely mainly on large language model (LLM) inference to obtain structured session-level assessments, but these assessments provide limited information about which utterances support each score. Training-based methods train predictors directly on sentences or their semantic embeddings and preserve local conversational detail, but must learn clinical structure from limited labeled transcripts. Integrating a small set of clinical feature scores with a long sequence of high-dimensional utterance representations is challenging, as simple fusion can overemphasize dialogue evidence and fail to link clinical features to supporting utterances. Combining structured clinical assessments with sentence-level semantics, we propose EmoTrack, which jointly encodes clinical feature scores and utterance representations to predict the total PHQ-8 score. For longitudinal assessment across successive sessions, the model can also incorporate a compressed representation of the preceding session as optional memory. On the real-world DAIC-WOZ benchmark, EmoTrack reduces mean absolute error from 2.8234 to 2.4708 relative to the strongest evaluated baseline. Across five settings covering single-session estimation, cross-dataset transfer without adaptation, and longitudinal assessment, it reduces normalized aggregate error by approximately 6.1% relative to the strongest baseline.

Abstract

Depression Detection

Explore similar work

EmoTrack: Clinical-Semantic Modeling for Text-Based Depression Severity Estimation

Dynamic Summary Generation for Interpretable Multimodal Depression Detection

Dep-LLM: Training-Free Depression Diagnosis via Evidence-Guided Structured Multi-factor with Reliable LLM Reasoning