A Multimodal Deep Learning Framework for Edema Classification Using HCT and Clinical Data

Aram Ansary Ogholbake; Hannah Choi; Spencer Brandenburg; Alyssa Antuna; Zahraa Al-Sharshahi; Makayla Cox; Haseeb Ahmed; Jacqueline Frank

cs.CVMar 20, 2026

A Multimodal Deep Learning Framework for Edema Classification Using HCT and Clinical Data

Authors: Aram Ansary Ogholbake, Hannah Choi, Spencer Brandenburg, Alyssa Antuna, Zahraa Al-Sharshahi, Makayla Cox, Haseeb Ahmed, Jacqueline Frank, +5 more

Abstract

We propose AttentionMixer, a unified deep learning framework for multimodal detection of brain edema that combines structural head CT (HCT) with routine clinical metadata. While HCT provides rich spatial information, clinical variables such as age, laboratory values, and scan timing capture complementary context that might be ignored or naively concatenated. AttentionMixer is designed to fuse these heterogeneous sources in a principled and efficient manner. HCT volumes are first encoded using a self-supervised Vision Transformer Autoencoder (ViT-AE++), without requiring large labeled datasets. Clinical metadata are mapped into the same feature space and used as keys and values in a cross-attention module, where HCT-derived feature vector serves as queries. This cross-attention fusion allows the network to dynamically modulate imaging features based on patient-specific context and provides an interpretable mechanism for multimodal integration. A lightweight MLP-Mixer then refines the fused representation before final classification, enabling global dependency modeling with substantially reduced parameter overhead. Missing or incomplete metadata are handled via a learnable embedding, promoting robustness to real-world clinical data quality. We evaluate AttentionMixer on a curated brain HCT cohort with expert edema annotations using five-fold cross-validation. Compared with strong HCT-only, metadata-only, and prior multimodal baselines, AttentionMixer achieves superior performance (accuracy 87.32%, precision 92.10%, F1-score 85.37%, AUC 94.14%). Ablation studies confirm the benefit of both cross-attention and MLP-Mixer refinement, and permutation-based metadata importance analysis highlights clinically meaningful variables driving predictions. These results demonstrate that structured, interpretable multimodal fusion can substantially improve edema detection in clinical practice.

Figures & tables

Figure 1: Overall diagram of AttentionMixer. HCT volumes are encoded by a self-supervised ViT-based encoder, while tabular metadata are embedded into the same latent space. A cross-attention module uses HCT features as queries and metadata as keys/values to produce a fused representation, which is further refined by an MLP-Mixer before classification.

Model	Modality	Accuracy (%)	Precision (%)	F1-Score (%)	AUC (%)
3D CNN Tran et al. (2015)	HCT only	78.68	86.69	76.49	84.00
DenseNet Huang et al. (2017)	HCT only	77.07	81.05	72.21	79.95
ViT Vaswani et al. (2017)	HCT only	70.24	61.34	55.92	73.35
MLP-Mixer Tolstikhin et al. (2021)	HCT only	74.63	74.79	71.67	76.57
MLP	Metadata only	84.88	89.90	82.77	92.31
HyperFusion Duenias et al. (2025)	HCT + metadata	66.66	73.33	72.72	86.66

Table 1: Performance comparison of edema classification models.

Fusion Method	Attention Type	Accuracy (%)	Precision (%)	F1-Score (%)	AUC (%)
AttentionMixer (w/o CA)	None	86.83	87.92	85.55	92.66
AttentionMixer (w/o MLP-Mixer)	Cross-attention only	85.85	92.66	84.04	93.35
Early Fusion	None	85.37	84.45	83.95	90.00
Only MLP-Mixer	None	70.00	87.32	67.31	65.00

Table 2: Ablation study: impact of fusion strategies on edema classification. CA denotes cross-attention.

Figure 2: Average ROC curve of AttentionMixer across 5-fold cross-validation. The mean AUC is 94%.

Figure 3: Top-10 metadata variables ranked by permutation feature importance. Importance is measured as the mean decrease in AUC (

\Delta\text{AUC}

) across five cross-validation folds when each feature is independently permuted. Higher values indicate stronger contribution of the metadata feature to the multimodal edema classification model.

Figure 4: Distribution of predicted probabilities for edema and non-edema cases.

Explore similar work

Jun 13, 2026

Fusion is not one-size-fits-all: Cross-Modal Representation Alignment for Time-to-Event Modeling

Accurate time-to-event (TTE) prediction from multimodal clinical data remains challenging due to modality imbalance and distribution shift. We introduce a foundation model-driven framework for cross-modal representation alignment between CT imaging and longitudinal EHR data, designed to generalize across tasks and institutions. CT and EHR modalities are encoded independently using domain-specific foundation models and aligned in a shared latent space through four principled fusion strategies: late fusion, contrastive alignment, cross-attention, and co-attention. We evaluate two clinically distinct TTE tasks: pulmonary embolism (PE) mortality and cardiovascular disease (CVD) outcomes, on large-scale multi-institutional cohorts (PE: N=3,099 train; 1,098 internal; 435 external; CVD: N=2,951 train; 837 internal; 682 external). Fusion consistently improves concordance index by 1.5-5.4% over unimodal baselines when modalities contribute comparably. Overall, contrastive multimodal fusion, particularly with CLMBR representations, provided the most consistent and statistically robust improvements, especially for PE mortality prediction. For MACE, cross-attention (one-hot) achieved the highest internal performance and image-guided co-attention achieved the best external performance. We therefore introduce a generalizable foundation model-based cross-modal alignment framework and provide the first systematic analysis of fusion behavior under modality imbalance in TTE prediction. Our results establish task-aware multimodal alignment as a necessary design principle for robust generalization and scalable clinical deployment.

Zhemin Zhang, Weijie Chen, David Le +7

Arizona State University · Mayo Clinic

Event Time Prediction Multimodal Robustness Cross-Modal Alignment Clinical Outcome Prediction Multimodal Survival Analysis Multimodal Fusion

Ask PDF

Jun 9, 2026

Multimodal Brain Tumour Classification Using Feature Fusion

Clinicians diagnose brain tumors by synthesizing patient symptoms, medical history, and quantitative imaging data from modalities such as MRI and CT scans into a unified clinical judgement. However, most deep learning models rely on MRI/CT images alone, failing to replicate the clinicians multimodal reasoning. We explore a two-branch multimodal network combining raw MRI scans with 91 extracted radiomic features (intensity, texture, shape, and boundary descriptors) to classify brain tumors into glioma, meningioma, pituitary, and no-tumor. A pre-trained CNN backbone encodes the image stream, whereas a dedicated MLP encodes the radiomic stream. Both streams are fused via concatenation, gated, or bidirectional cross-modal attention strategies. Across nine experimental runs on a balanced 7,200 image dataset, all multimodal configurations outperform unimodal baselines with gated fusion achieving the best accuracy of 96.13%.

Wajih ul Islam, Muhammad Yaqoob, Javed Ali Khan +1

School of Physics, Engineering and Computer Science, University of Hertfordshire, UK

Multimodal Classification Medical Image Classification Oncology Brain Tumor Classification Multimodal Fusion

Ask PDF

Feb 27, 2026

When Does Multimodal Learning Help in Healthcare? A Benchmark on EHR and Chest X-Ray Fusion

Machine learning holds promise for advancing clinical decision support, yet it remains unclear when multimodal learning truly helps in practice, particularly under modality missingness and fairness constraints. In this work, we conduct a systematic benchmark of multimodal fusion between Electronic Health Records (EHR) and chest X-rays (CXR) on standardized cohorts from MIMIC-IV and MIMIC-CXR, aiming to answer four fundamental questions: when multimodal fusion improves clinical prediction, how different fusion strategies compare, how robust existing methods are to missing modalities, and whether multimodal models achieve algorithmic fairness. Our study reveals several key insights. Multimodal fusion improves performance when modalities are complete, with gains concentrating in diseases that require complementary information from both EHR and CXR. While cross-modal learning mechanisms capture clinically meaningful dependencies beyond simple concatenation, the rich temporal structure of EHR introduces strong modality imbalance that architectural complexity alone cannot overcome. Under realistic missingness, multimodal benefits rapidly degrade unless models are explicitly designed to handle incomplete inputs. Moreover, multimodal fusion does not inherently improve fairness, with subgroup disparities mainly arising from unequal sensitivity across demographic groups. To support reproducible and extensible evaluation, we further release a flexible benchmarking toolkit that enables plug-and-play integration of new models and datasets. Together, this work provides actionable guidance on when multimodal learning helps, when it fails, and why, laying the foundation for developing clinically deployable multimodal systems that are both effective and reliable. The open-source toolkit can be found at https://github.com/jakeykj/CareBench.

Kejing Yin, Haizhou Xu, Wenfang Yao +5

Department of Computer Science, Hong Kong Baptist University · Division of Artificial Intelligence, Lingnan University · School of Nursing, The Hong Kong Polytechnic University

Multimodal Fusion Multimodal Model Evaluation Missing-Modality Learning Electronic Health Records Algorithmic Fairness

Ask PDF