From Sparse Representations to Behavioral Insights for Multimodal Depression Assessment

Guimin Hu; Zihao Song; Jiachen Luo; Jiayuan Xie; Ruichu Cai

cs.CLOct 8, 2026

From Sparse Representations to Behavioral Insights for Multimodal Depression Assessment

Authors: Guimin Hu, Zihao Song, Jiachen Luo, Jiayuan Xie, Ruichu Cai

Organizations: Guangdong University of Technology · Queen Mary University of London · The Hong Kong Polytechnic University

Abstract

Multimodal depression assessment offers a promising approach to analyzing behavioral patterns associated with depression. However, existing methods often rely on dense and opaque multimodal representations, making it difficult to interpret the behavioral patterns underlying their predictions. In this work, we introduce BehavDep, a sparse factor-based framework that decomposes multimodal behavioral representations into sparse latent factors and associates them with behaviorally meaningful concepts through a semantic bridge. To address the mismatch between user-level annotations and heterogeneous video-level behaviors, BehavDep further learns video-level depression tendency scores under weak supervision and aggregates information across multiple observations for user-level assessment. Extensive experiments demonstrate that BehavDep achieves the best overall assessment performance while revealing complementary modality contributions, heterogeneous behavioral patterns across observations, and prediction responses to concept-level editing. These results show that BehavDep provides a structured and interpretable approach to analyzing multimodal behavioral representations for depression assessment.

Figures & tables

Method	Acc.	F1	Pre.	Rec.
DepDetector cat	54.4	45.0	44.8	34.1
DepDetector han	74.7	74.5	69.1	81.1
TAMFN cat	64.4	57.2	59.8	54.7
TAMFN han	71.2	70.7	65.7	76.7
BiLSTM cat	73.6	71.1	71.5	72.2
BiLSTM han	76.4	75.4	72.6	79.7

Table 1: Performance comparison on the MUD3 dataset. Baseline results are adopted from Wang et al. (2025) . Results are reported as mean

\pm

standard deviation over five independent runs.

Method	Normal		Depressed
Method	Pre.	F1	Pre.	F1
ConvLSTM	83.1	35.1	30.1	45.2
PerceiverIO	74.3	81.5	50.0	36.3
AFABNet	77.4	83.6	62.2	45.8
Qwen2-Audio-Instruct	79.1	79.1	50.0	51.3
Mental-Perceiver	78.4	86.8	83.6	50.4

Table 2: Performance comparison on the DAIC-WOZ dataset. Baseline results are reproduced from Qin et al. (2025) .

Figure 2: Temporal activation analysis of clinically grounded concepts for a depressed user. The heatmap illustrates the dynamic activation of depression-related behavioral concepts across video segments.

Panel of Figure 3 — Figure 3: Video-level prediction trajectory. Dashed line denotes the threshold for depressive symptoms.

Panel of Figure 4 — Figure 4: Representative clinical prediction reports generated by BehavDep.

Method	Acc.	F1	Pre.	Rec.
BehavDep	83.3	81.9	80.6	83.3
w/o SAE	77.2	76.5	71.4	78.1
w/o Audio	80.3	79.4	73.1	80.7
w/o Video	78.6	78.1	72.8	78.5
w/o Concept	77.3	76.4	71.9	78.5
w/o Audio Concept	78.7	77.6	72.1	78.9

Table 3: Ablation study on different components of BehavDep.

Edited Concepts	Modality	Total Impact	$\Delta$ F1 (%)	Label Flip
Slow speech ( $c_{1}$ )	A	0.65	-0.11	Depressive $\rightarrow$ Depressive
[0.5pt/2pt] Slow speech ( $c_{1}$ ) Long pauses ( $c_{2}$ )	A	1.14	-0.25	Depressive $\rightarrow$ Depressive
[0.5pt/2pt] Flat facial affect ( $c_{6}$ )	V	0.72	-0.27	Depressive $\rightarrow$ Depressive
[0.5pt/2pt] Flat facial affect ( $c_{6}$ ) Reduced facial movement ( $c_{7}$ )	V	1.36	-1.84	Depressive $\rightarrow$ Normal
[0.5pt/2pt] Slow speech ( $c_{1}$ ), Long pauses ( $c_{2}$ ) Flat facial affect ( $c_{6}$ ), Reduced facial movement ( $c_{7}$ ) Low eye contact ( $c_{8}$ )	A,V	3.53	-3.35	Depressive $\rightarrow$ Normal
[0.5pt/2pt] Long pauses ( $c_{2}$ ), Flat facial affect ( $c_{6}$ ) Reduced facial movement ( $c_{7}$ ), Low eye contact ( $c_{8}$ ) Psychomotor retardation ( $c_{11}$ ), Emotional flattening ( $c_{12}$ )	A,V,S	3.81	-4.58	Depressive $\rightarrow$ Normal

Table 4: Counterfactual editing effects of clinically meaningful behavioral concepts. A, V, and S denote the audio, visual, and cross-modal concept categories, respectively.

Panel of Figure 5 — Figure 5: (a) Contribution analysis of different modalities and (b) acoustic feature importance

Figure 6: User-level case study involving three participants: User A is non-depressed, while Users B and C are diagnosed with depression.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Clinical Concept	Clinical Basis	Observable Manifestation
Low eye contact	Reduced social engagement Buyukdura et al. (2011)	Reduced gaze toward the interlocutor
Flat facial affect	Diminished emotional expression Buyukdura et al. (2011)	Limited facial and emotional expression
Long pauses	Psychomotor slowing Bennabi et al. (2013)	Prolonged pauses or delayed responses
Monotone speech	Diminished emotional expression Bennabi et al. (2013)	Reduced pitch and prosodic variation
Soft speaking	Reduced energy Almaghrabi et al. (2023)	Low vocal intensity and speaking volume
Slow speech	Psychomotor slowing Buyukdura et al. (2011)	Reduced speech rate

Appendix

Table 5: Behavioral concepts, their literature bases, and corresponding observable manifestations used in BehavDep.

Figure 8: Distribution of Depression Evidence.

Panel of Figure 9 — Figure 9: Temporal prediction trajectory of BehavDep.

Panel of Figure 10 — Figure 10: Representative behavioral insight report generated by BehavDep, illustrating how sparse multimodal behavioral factors are translated into interpretable evidence for depression assessment.

Explore similar work

EviDep: Uncertainty-Aware Multimodal Depression Estimation via Disentangled Evidential Learning
Apr 17, 2026Fangyuan Liu, Sirui Zhao, Yangsong Zhang +5Disentangled Representation Learning Evidential Deep Learning
School of Computer Science and Technology, University of Science and Technology of China, Hefei, China · aSchool of Computer Science and Technology, University of Science and Technology of China, Hefei, China · bSchool of Computer Science and Technology, Laboratory for Brain Science and Artificial Intelligence, Southwest University of Science and Technology, Mianyang, China +1
Machine Learning for Depression Screening and Intervention: an Original Circadian Rhythm Score-based Methodology
Jul 6, 2026Bin Wang, Shuo Lian, Yuanyuan Hou +5Conditional Average Treatment Effect Estimation Depression Detection
Faculty of Information Science and Engineering Ocean University of China Qingdao, China · Department of Rehabilitation Medicine The Affiliated Hospital of Qingdao University Qingdao, China · School of Intelligent Medicine Chengdu University of Traditional Chinese Medicine Chengdu, China +1
Expresso-AI: Explainable Video-Based Deep Learning Models for Depression Diagnosis
Jun 24, 2026Felipe Moreno, Sharifa Alghowinem, Hae Won Park +1Depression Detection Neural Network Interpretability
Media Lab MIT Cambridge, USA

Apr 17, 2026, cs.LGJ/K move · Enter open · S save

EviDep: Uncertainty-Aware Multimodal Depression Estimation via Disentangled Evidential Learning

Fangyuan Liu, Sirui Zhao, Yangsong Zhang, Jinyang Huang+4

School of Computer Science and Technology, University of Science and Technology of China, Hefei, China · aSchool of Computer Science and Technology, University of Science and Technology of China, Hefei, China · bSchool of Computer Science and Technology, Laboratory for Brain Science and Artificial Intelligence, Southwest University of Science and Technology, Mianyang, China+1

Disentangled Representation Learning Evidential Deep Learning Depression Detection Depression Severity Estimation Multimodal Fusion Epistemic Uncertainty

Open paper AskPDF arXiv

Audio--visual recordings provide complementary cues for estimating depression severity, but their informativeness varies across time and modalities. Point predictions alone do not express the uncertainty associated with these estimates. We present EviDep, a multimodal evidential regression framework that integrates multi-scale temporal modeling and shared--private representation learning for uncertainty-aware depression estimation. Frequency-aware Feature Extraction decomposes behavioral feature sequences into multiple frequency bands and refines them with scale-specific experts. Disentangled Evidential Learning encourages the disentanglement of cross-modal shared and modality-specific information in the refined features. Multi-branch Evidential Regression maps the resulting shared and private representations to three Normal-Inverse-Gamma (NIG) outputs and uses evidence-weighted aggregation to estimate depression severity and quantify aleatoric and epistemic uncertainty. Experiments on AVEC 2013, AVEC 2014, DAIC-WOZ, and E-DAIC show competitive prediction accuracy, with ablation studies supporting the contributions of frequency-aware refinement and shared--private disentanglement. Further analyses show that estimated epistemic uncertainty helps identify higher-error predictions, while both uncertainty estimates generally increase under controlled feature degradation.

Abstract

Explore similar work

EviDep: Uncertainty-Aware Multimodal Depression Estimation via Disentangled Evidential Learning

Machine Learning for Depression Screening and Intervention: an Original Circadian Rhythm Score-based Methodology

Expresso-AI: Explainable Video-Based Deep Learning Models for Depression Diagnosis