Leakage-aware calibrated decision-level fusion for mental-health text classification
Organizations: Robotics and Internet-of-Things Laboratory, Prince Sultan University, Riyadh 11586, Saudi Arabia · Department of Psychiatry, Harvard Medical School, Boston, MA 02115, United States · College of Engineering and Advanced Computing, Alfaisal University, Riyadh 11533, Saudi Arabia
Abstract
Decision-level fusion benefits from complementary errors, calibrated probabilities, and leakage-aware evaluation. We evaluated a heterogeneous transformer fusion framework on 262,276 proxy-labeled Reddit records covering anxiety, depression, schizophrenia, and suicidal ideation. In a stratified random split, MinHash-LSH identified 29.1% of test records as lexical near-duplicates of training data. Augmentation-lineage auditing found 50.2% lineage overlap with training, including 30.0% of records retained after lexical filtering. We therefore used lineage-disjoint partitions. Twelve encoders, comprising three seeds each of Bio-ClinicalBERT, MentalBERT, MentalRoBERTa, and RoBERTa-base, underwent cross-fitted validation for calibration and ensemble selection. A class-specific Complementarity Index (CI), defined as the Jaccard distance between members' correctly classified validation sets, was assessed against independent fusion gain. CI correlated with gain for depression (r=0.50) and suicidal ideation (r=0.68), but not anxiety; results for schizophrenia were mixed. The selected MentalBERT, MentalRoBERTa, and RoBERTa-base triad achieved macro-F1 0.844 [0.823, 0.863], MCC 0.779, and ECE 0.0033 after post-fusion recalibration, improving macro-F1 by 0.021 [0.007, 0.037] over the validation-selected best single encoder. Integrated-Gradients attribution diversity was 2.3 times greater in fusion-rescued cases. Removing divergence-driving words destroyed 58% of rescues, versus 12% under random removal. These findings show that lineage auditing detects dependencies missed by lexical filtering, while class-specific complementarity and post-fusion calibration reveal ensemble behavior obscured by aggregate accuracy.
Figures & tables
| Study | Multi-arch | Calibr. | Leakage | CI | XAI | Primary focus |
|---|---|---|---|---|---|---|
| ensemble | reported | audit | metric | link | ||
| Alsentzer et al. [ 3 ] | – | – | – | – | – | Clinical-notes BERT embeddings |
| Ji et al. [ 18 ] | – | – | – | – | – | MentalBERT/MentalRoBERTa release |
| Tavchioski et al. [ 40 ] | partial | – | – | – | – | Reddit depression ensembles |
| Harrigian et al. [ 13 ] | – | – | partial | – | – | Critique of data practices |
| Chancellor & De Choudhury [ 6 ] | – | – | – | – | – | Methodological critique |
Classification
Classification maps observations to discrete labels, including familiar, unknown, hierarchical, or multiple categories.
See all 451 papers