Learning to Attend to Depression-Related Patterns: An Adaptive Cross-Modal Gating Network for Depression Detection

Hangbin Yu; Yudong Yang; Rongfeng Su; Nan Yan; Lan Wang

cs.SDApr 11, 2026

Learning to Attend to Depression-Related Patterns: An Adaptive Cross-Modal Gating Network for Depression Detection

Authors: Hangbin Yu, Yudong Yang, Rongfeng Su, Nan Yan, Lan Wang

Organizations: Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, China · University of Chinese Academy of Sciences · Key Laboratory of Biomedical Imaging Science and System, Chinese Academy of Sciences, China

Abstract

Automatic depression detection using speech signals with acoustic and textual modalities is a promising approach for early diagnosis. Depression-related patterns exhibit sparsity in speech: diagnostically relevant features occur in specific segments rather than being uniformly distributed. However, most existing methods treat all frames equally, assuming depression-related information is uniformly distributed and thus overlooking this sparsity. To address this issue, we proposes a depression detection network based on Adaptive Cross-Modal Gating (ACMG) that adaptively reassigns frame-level weights across both modalities, enabling selective attention to depression-related segments. Experimental results show that the depression detection system with ACMG outperforms baselines without it. Visualization analyses further confirm that ACMG automatically attends to clinically meaningful patterns, including low-energy acoustic segments and textual segments containing negative sentiments.

Figures & tables

Figure 1: Overall framework of the proposed Adaptive Cross-Modal Gating (ACMG) network, (A) and (B) illustrate the details of Multi-Head Attention and ACMG modules, respectively.

Comparison with Existing Methods
Method	Acc.	F1	Prec.	Recall
Al Hanai et al. (2018) [ 3 ] *	72.43	72.46	72.25	72.67
Gómez-Zaragoza et al. (2025) [ 28 ] *	75.37	75.55	76.03	75.07
RoBERTa as Pre-trained Language Model
Transformer(RoBERTa) [ 20 ]	75.74	76.17	75.64	76.70
+ ACMG w Unimodal	77.57	77.33	77.07	77.59

Table 1: Performance comparison on PDCD2025 using official 5-fold cross-validation.

Method	Audio	Text	F1	Prec.	Recall
Al Hanai et al. (2018) [ 3 ]	$\checkmark$	$\times$	50.00	71.00	38.00
Al Hanai et al. (2018) [ 3 ]	$\times$	$\checkmark$	59.00	71.00	50.00
Gómez-Zaragoza et al. (2025) [ 28 ]	$\checkmark$	$\checkmark$	64.30	-	-
Wang et al. (2022) [ 29 ]	$\checkmark$	$\checkmark$	68.34	-	-
Transformer(Qwen)	$\checkmark$	$\checkmark$	64.38	64.77	64.13
+ ACMG w Cross-modal	$\checkmark$	$\checkmark$	69.39	71.79	68.48

Table 2: Experimental Results on the DAIC-WOZ Benchmark using Official Development Partition.

Panel of Figure 2 — Figure 2: Visualization of gating weights in the ACMG module. The top curves and bottom heatmap illustrate the adaptive feature selection across acoustic and textual modalities, respectively. The blue shaded region denotes the low-energy region in the acoustic signal, while the red boxes highlight negative words in the transcript.

Explore similar work

Multimodal Domain Generalization for Depression Detection: An Attention-Based BiLSTM Network with Domain-Adversarial Training
Jul 24, 2026Ali Tabaraei, Federico Simonetta, Stavros NtalampirasDepression Detection Domain Generalization
Department of Computer Science, University of Milan, Milan, Italy · Computer Science Department, Gran Sasso Science Institute (GSSI), L’Aquila, Italy
Clinically Inspired Symptom-Guided Depression Detection from Emotion-Aware Speech Representations
Feb 17, 2026Chaithra Nerella, Chiranjeevi YarraDepression Severity Estimation Clinical Speech Processing
Speech Processing Lab, LTRC IIIT Hyderabad Hyderabad, India
Can Conversational Temporal Dynamics Improve Depression Detection in Dyads? A Preliminary Investigation in Multi-Modality Perspectives
Jul 4, 2026Hanie Kang, Huang-Cheng Chou, Sudarsana Reddy Kadiri +1Depression Detection Multimodal Fusion
1Thomas Lord Department of Computer Science, University of Southern California, USA · 2Signal Analysis and Interpretation Laboratory (SAIL), University of Southern California, USA

Jul 24, 2026J/K move · Enter open · S save

Multimodal Domain Generalization for Depression Detection: An Attention-Based BiLSTM Network with Domain-Adversarial Training

Ali Tabaraei, Federico Simonetta, Stavros Ntalampiras

Department of Computer Science, University of Milan, Milan, Italy · Computer Science Department, Gran Sasso Science Institute (GSSI), L’Aquila, Italy

Depression Detection Domain Generalization

Open paper AskPDF arXiv

Automatic depression detection with deep learning has shown promise but often suffers from limited generalization due to domain shift arising from inter-speaker variability. To address this critical issue, we present the first patient-independent multimodal depression detection framework that incorporates domain generalization (DG), jointly leveraging both acoustic and textual modalities. The proposed model integrates bidirectional Long Short-Term Memory (BiLSTM) with intra- and cross-modal attention mechanisms, accompanied by segment-level fusion for decision-making. Generalization is further enhanced by applying a gradient reversal layer inspired by Domain-Adversarial Training of Neural Networks (DANN), which promotes domain-invariant representations by adversarially limiting the model's ability to identify individual speakers, effectively reducing patient-specific bias. Conducting experiments on the Androids-Corpus dataset with a 5-fold cross-validation (CV) protocol, various pairings of audio and text feature extractors were evaluated over different segment durations, determining MelSpec and ItalianBERT as the optimal baseline at a 30-second segment duration. The addition of DG to this baseline yields a 2.5% increase in accuracy and 3.3% in F1-score, achieving 93.2% accuracy, 93.2% precision, 96.2% recall, and 94.2% F1-score, surpassing all existing benchmarks. Extensive ablation studies assess the impact of multimodal fusion, deep architectural choices, and DG, highlighting their combined contribution to robust and generalizable depression detection.

Abstract

Explore similar work

Multimodal Domain Generalization for Depression Detection: An Attention-Based BiLSTM Network with Domain-Adversarial Training

Clinically Inspired Symptom-Guided Depression Detection from Emotion-Aware Speech Representations

Can Conversational Temporal Dynamics Improve Depression Detection in Dyads? A Preliminary Investigation in Multi-Modality Perspectives