Leakage-aware calibrated decision-level fusion for mental-health text classification
Authors: Wadii Boulila, Giuseppe Varone, Bilel Benjdira, Maha Driss
Organizations: Robotics and Internet-of-Things Laboratory, Prince Sultan University, Riyadh 11586, Saudi Arabia · Department of Psychiatry, Harvard Medical School, Boston, MA 02115, United States · College of Engineering and Advanced Computing, Alfaisal University, Riyadh 11533, Saudi Arabia
Decision-level fusion benefits from complementary errors, calibrated probabilities, and leakage-aware evaluation. We evaluated a heterogeneous transformer fusion framework on 262,276 proxy-labeled Reddit records covering anxiety, depression, schizophrenia, and suicidal ideation. In a stratified random split, MinHash-LSH identified 29.1% of test records as lexical near-duplicates of training data. Augmentation-lineage auditing found 50.2% lineage overlap with training, including 30.0% of records retained after lexical filtering. We therefore used lineage-disjoint partitions. Twelve encoders, comprising three seeds each of Bio-ClinicalBERT, MentalBERT, MentalRoBERTa, and RoBERTa-base, underwent cross-fitted validation for calibration and ensemble selection. A class-specific Complementarity Index (CI), defined as the Jaccard distance between members' correctly classified validation sets, was assessed against independent fusion gain. CI correlated with gain for depression (r=0.50) and suicidal ideation (r=0.68), but not anxiety; results for schizophrenia were mixed. The selected MentalBERT, MentalRoBERTa, and RoBERTa-base triad achieved macro-F1 0.844 [0.823, 0.863], MCC 0.779, and ECE 0.0033 after post-fusion recalibration, improving macro-F1 by 0.021 [0.007, 0.037] over the validation-selected best single encoder. Integrated-Gradients attribution diversity was 2.3 times greater in fusion-rescued cases. Removing divergence-driving words destroyed 58% of rescues, versus 12% under random removal. These findings show that lineage auditing detects dependencies missed by lexical filtering, while class-specific complementarity and post-fusion calibration reveal ensemble behavior obscured by aggregate accuracy.
Figures & tables
Study
Multi-arch
Calibr.
Leakage
CI
XAI
Primary focus
ensemble
reported
audit
metric
link
Alsentzer et al. [ 3 ]
–
–
–
–
–
Clinical-notes BERT embeddings
Ji et al. [ 18 ]
–
–
–
–
–
MentalBERT/MentalRoBERTa release
Tavchioski et al. [ 40 ]
partial
–
–
–
–
Reddit depression ensembles
Harrigian et al. [ 13 ]
–
–
partial
–
–
Critique of data practices
Chancellor & De Choudhury [ 6 ]
–
–
–
–
–
Methodological critique
Table 1: Comparison of this work with representative prior studies on transformer-based mental-health text classification.
Figure 1: Overview of the leakage-aware, cross-fitted calibration and complementarity-guided decision-level fusion framework.
Figure 2: lineage-linked records in the stratified random test split, by class. The three augmented classes are almost entirely linked to training posts through augmentation lineage; the suicidal-ideation class, which was not augmented, is unaffected.
Figure 3: Per-class F1 of the twelve fine-tuned encoders on the lineage-disjoint test set; error bars are 95% bootstrap intervals. Depression is the hardest of the three large classes; schizophrenia has 75 test posts and correspondingly wide intervals.
Figure 4: Complementarity Index versus realized fusion gain. (a) CI computed on the validation set against the class-wise F1 gain of the uniform-mean pair over the better member’s class-wise F1 on the test set (Section 3.4 ), for all 66 pairs and 4 classes, with per-class least-squares fits. (b) Within-class Pearson correlation with 95% encoder-level bootstrap intervals (encoders resampled with replacement, each pair weighted by its member multiplicities). The association is positive for depression and suicidal ideation, absent for anxiety, and unmeasurable for the 75 -record schizophrenia class.
Figure 5: One-versus-rest ROC curves of the three headline members and of the calibrated triad on the lineage-disjoint test set.
Figure 6: Row-normalized confusion matrices of the three headline members and of the calibrated triad. The dominant residual confusion is depression → suicidal ideation ( 17 – 22% of depression posts for the members, 19% for the triad); the triad attains the highest recall on schizophrenia ( 0.80 versus 0.76 – 0.77 ).
Figure 7: Reliability diagrams of the headline triad before (left) and after (right) the post-hoc temperature, 15 equal-width bins. The gap between confidence and accuracy is reduced without any change in the predicted labels.
Figure 8: (a) Word-level attribution diversity on all-correct versus fusion-rescued cases, for all member pairs and for same-target pairs only. (b) Fraction of decisions preserved after removing five words by rule; rescued decisions depend on the divergent words.
The classification of mental health is challenging for a variety of reasons. For one, there is overlap between the mental health issues. In addition, the signs of mental health issues depend on the context of the situation, making classification difficult. Although fine-tuning transformers has improved the performance for mental health classification, standard cross-entropy training tends to create entangled feature spaces and fails to utilize all the information the transformers contain. We present a new framework that focuses on representations to improve mental health classification. This is done using two methods. First, layer-attentive residual aggregation which works on residual connections to to weigh and fuse representations from all transformer layers while maintaining high-level semantics. Second, supervised contrastive feature learning uses temperature-scaled supervised contrastive learning with progressive weighting to increase the geometric margin between confusable mental health problems and decrease class overlap by restructuring the feature space. With a score of 74.36%, the proposed method is the best performing on the SWMH benchmark and outperforms models that are domain-specialized, such as MentalBERT and MentalRoBERTa by margins of (3.25% - 2.2%) and 2.41 recall points over the highest achieving model. These findings show that domain-adaptive pretraining for mental health text classification can be surpassed by carefully designed representation geometry and layer-aware residual integration, which also provide enhanced interpretability through learnt layer importance.
Menna Elgabry, Ali Hamdi, Khaled Shaban
Department of Computer Science, Modern Sciences and Arts University, Cairo, Egypt. · Computer Science and Engineering Department, College of Engineering, Qatar University, Doha, Qatar.
Social media text data are often used to train Machine Learning (ML) models to identify users exhibiting high-risk mental health behaviors. However, sharing this sensitive data poses privacy risks and limits the growth of benchmark datasets. We comprehensively evaluate whether privacy-preserving ML techniques can enable safer data sharing while preserving performance. Specifically, we apply federated learning (FL) and Differentially Private FL for two widely-studied mental health prediction tasks: depression detection on X (Twitter) and suicide crisis detection on Reddit. We simulate realistic data-sharing scenarios by treating each user as a client in a non-IID setting, evaluating across different client fractions, aggregation strategies, and privacy budgets. While FL achieves comparable performance to centralized training (centralized F1 = 85.63; best FL model F1 = 83.16) on depression identification, we find that Differentially Private FL has a large performance-privacy trade-off (up to F1 = 27.01 drop) even with low levels of noise (epsilon = 50). This is due to the distortion of highly informative yet sparse mental health linguistic markers related to mental health, like health topics and emotion words. This research empirically demonstrates the potential and limitations of current privacy preservation techniques for mental health inference tasks.
Nuredin Ali Abdelkadir, Anjali Ratnam, Zeerak Talat +1
Person-level psychological assessment requires aggregating meaning across many messages from the same individual, a task that document-level training objectives were not explicitly designed for. We present a systematic, empirical comparison between architecturally matched traditional (a) base-transformers and (b) document-tuned-transformers (further contrastively fine-tuned at the document-level, sometimes referred to as "sentence transformers") under otherwise identical conditions. Comparing layer-wise and overall performance across two longitudinal mental health and psychological datasets, we find document-tuned models demonstrated a consistent improvement over base representations (increase in Pearson r of 13.4%, p=.015). Robustness analyses revealed document-tuned models remained more accurate under perturbations to word deletion, synonym replacement, typo injection, and back translation. Further, hedged language (e.g., usually') was more characteristic of outcomes in document-tuned embeddings while abundance (e.g., lot') was more characteristic of base-transformers, suggesting document-tuned models may better capture uncertainty. These results suggest representation choice impacts mental health prediction, document-tuned models often being more adept.
Aaron Marker, Oscar Kjell, Vasudha Varadarajan +1
Vanderbilt University · Lund University · Carnegie Mellon University