We propose AttentionMixer, a unified deep learning framework for multimodal detection of brain edema that combines structural head CT (HCT) with routine clinical metadata. While HCT provides rich spatial information, clinical variables such as age, laboratory values, and scan timing capture complementary context that might be ignored or naively concatenated. AttentionMixer is designed to fuse these heterogeneous sources in a principled and efficient manner. HCT volumes are first encoded using a self-supervised Vision Transformer Autoencoder (ViT-AE++), without requiring large labeled datasets. Clinical metadata are mapped into the same feature space and used as keys and values in a cross-attention module, where HCT-derived feature vector serves as queries. This cross-attention fusion allows the network to dynamically modulate imaging features based on patient-specific context and provides an interpretable mechanism for multimodal integration. A lightweight MLP-Mixer then refines the fused representation before final classification, enabling global dependency modeling with substantially reduced parameter overhead. Missing or incomplete metadata are handled via a learnable embedding, promoting robustness to real-world clinical data quality. We evaluate AttentionMixer on a curated brain HCT cohort with expert edema annotations using five-fold cross-validation. Compared with strong HCT-only, metadata-only, and prior multimodal baselines, AttentionMixer achieves superior performance (accuracy 87.32%, precision 92.10%, F1-score 85.37%, AUC 94.14%). Ablation studies confirm the benefit of both cross-attention and MLP-Mixer refinement, and permutation-based metadata importance analysis highlights clinically meaningful variables driving predictions. These results demonstrate that structured, interpretable multimodal fusion can substantially improve edema detection in clinical practice.
Figures & tables
Figure 1: Overall diagram of AttentionMixer. HCT volumes are encoded by a self-supervised ViT-based encoder, while tabular metadata are embedded into the same latent space. A cross-attention module uses HCT features as queries and metadata as keys/values to produce a fused representation, which is further refined by an MLP-Mixer before classification.
Model
Modality
Accuracy (%)
Precision (%)
F1-Score (%)
AUC (%)
3D CNN Tran et al. (2015)
HCT only
78.68
86.69
76.49
84.00
DenseNet Huang et al. (2017)
HCT only
77.07
81.05
72.21
79.95
ViT Vaswani et al. (2017)
HCT only
70.24
61.34
55.92
73.35
MLP-Mixer Tolstikhin et al. (2021)
HCT only
74.63
74.79
71.67
76.57
MLP
Metadata only
84.88
89.90
82.77
92.31
HyperFusion Duenias et al. (2025)
HCT + metadata
66.66
73.33
72.72
86.66
Table 1: Performance comparison of edema classification models.
Fusion Method
Attention Type
Accuracy (%)
Precision (%)
F1-Score (%)
AUC (%)
AttentionMixer (w/o CA)
None
86.83
87.92
85.55
92.66
AttentionMixer (w/o MLP-Mixer)
Cross-attention only
85.85
92.66
84.04
93.35
Early Fusion
None
85.37
84.45
83.95
90.00
Only MLP-Mixer
None
70.00
87.32
67.31
65.00
Table 2: Ablation study: impact of fusion strategies on edema classification. CA denotes cross-attention.
Figure 2: Average ROC curve of AttentionMixer across 5-fold cross-validation. The mean AUC is 94%.
Figure 3: Top-10 metadata variables ranked by permutation feature importance. Importance is measured as the mean decrease in AUC ( ΔAUC ) across five cross-validation folds when each feature is independently permuted. Higher values indicate stronger contribution of the metadata feature to the multimodal edema classification model.
Figure 4: Distribution of predicted probabilities for edema and non-edema cases.
Department of Computer Science, Hong Kong Baptist University · Division of Artificial Intelligence, Lingnan University · School of Nursing, The Hong Kong Polytechnic University