Effective and Robust Multimodal Medical Image Analysis

Joy Dhar; Nayyar Zaidi; Maryam Haghighat

cs.CVFeb 17, 2026

Effective and Robust Multimodal Medical Image Analysis

Authors: Joy Dhar, Nayyar Zaidi, Maryam Haghighat

Organizations: Indian Institute of Technology Ropar, Rupnagar, Punjab, India · Deakin University, Melbourne, Victoria, Australia · Queensland University of Technology, Brisbane, QLD, Australia

Abstract

Multimodal Fusion Learning (MFL), leveraging disparate data from various imaging modalities (e.g., MRI, CT, SPECT), has shown great potential for addressing medical problems such as skin cancer and brain tumor prediction. However, existing MFL methods face three key limitations: a) they often specialize in specific modalities, and overlook effective shared complementary information across diverse modalities, hence limiting their generalizability for multi-disease analysis; b) they rely on computationally expensive models, restricting their applicability in resource-limited settings; and c) they lack robustness against adversarial attacks, compromising reliability in medical AI applications. To address these limitations, we propose a novel Multi-Attention Integration Learning (MAIL) network, incorporating two key components: a) an efficient residual learning attention block for capturing refined modality-specific multi-scale patterns and b) an efficient multimodal cross-attention module for learning enriched complementary shared representations across diverse modalities. Furthermore, to ensure adversarial robustness, we extend MAIL network to design Robust-MAIL by incorporating random projection filters and modulated attention noise. Extensive evaluations on 20 public datasets show that both MAIL and Robust-MAIL outperform existing methods, achieving performance gains of up to 9.34% while reducing computational costs by up to 78.3%. These results highlight the superiority of our approaches, ensuring more reliable predictions than top competitors. Code: https://github.com/misti1203/MAIL-Robust-MAIL.

Figures & tables

Figure 1. (Left) Cost–performance trade-off: comparison of our proposed MAIL framework vs. computationally-intensive fusion models (e.g., DRIFA-Net ( Dhar et al., 2024b ) , MuMu ( Islam and Iqbal, 2022 ) ). (Right) Adversarial robustness: comparison of proposed Robust-MAIL vs. SOTA methods under a PGD attack.

Figure 2. (A–B) Attention-alignment paradigms for MFL : cascaded attention (e.g., DRIFA-Net ( Dhar et al., 2024b ) , MuMu ( Islam and Iqbal, 2022 ) ) vs. our parallel fusion attention ( EMCAM in MAIL ). The parallel design reduces information loss during shared-representation learning unlike cascaded pipelines.

Figure 3. Architecture of MAIL network (comprising of (A) MSTL and (B) TMTL phases). Key components with-in MSTL phase are (C) ERLA and (F) EMCAM blocks. ERLA is based on (D) EMILA module, comprising of (G) MSGDC block inspired from EMCAD’s MSDC ( Rahman et al., 2024 ) and (E) Channel Attention ( CA ). EMCAM details are in Figure 4 .

Figure 4. Components of EMCAM : (A) MFIFA module captures multi-frequency multimodal global contexts, while (B) EMSCA module refines multimodal spatial representations.

Figure 5. (A-B) Overview of Robust-MAIL , where MAIL integrates EMCAM , incorporating RPF and MAN to form the RPAN module. RPAN consists of key components: RPF[A] / RPF[I] (sampled Gaussian matrices), EMCAM[A] / EMCAM[I] , and RPAN[A] / RPAN[I] , where [A] and [I] denote attack and inference phases. (C–D) Illustration of MAN utilizing learnable feature-layer noise, where noise is modulated through learnable weights combined with random noise to enhance robustness. (E) Integration of RPF into the MSGDC from ERLA and EMCAM .

Datasets $\rightarrow$	D1: Nickparvar			D2: IQ-OTHNCCD			D3: Tuberculosis			D4: BCCD			D5: HAM10000			D6: SIPaKMeD			D7: CRC			D8: CNMC-2019			D9: KVASIR			D10: CBIS-DDSM			Cost Analysis
Models $\downarrow$	ACC	F1	AUC	ACC	F1	AUC	ACC	F1	AUC	ACC	F1	AUC	ACC	F1	AUC	ACC	F1	AUC	ACC	F1	AUC	ACC	F1	AUC	ACC	F1	AUC	ACC	F1	AUC	Params	Flops
DDA-Net	96.8	96.6	97.1	98.1	97.4	98.1	98.6	97.7	98.8	98.2	98.2	98.6	92.2	91.8	92.6	92.5	92.8	91.9	93.4	92.7	93.6	92.6	91.8	93.1	90.6	90.2	91.1	92.5	91.8	91.5	12.1	1.12
NAT	95.5	95.5	95.6	97.5	97.3	97.2	98.4	96.7	98.2	98.1	97.7	98	93.1	92.6	93.3	91.1	91.1	91.5	96.8	96.7	96.8	94.1	93.7	93.4	89.3	87.6	90.1	92.0	91.7	91.9	20	1.1
POTTER	95.1	94.7	94.7	97.2	96.5	97.2	97.8	97.2	98.2	96.8	96.4	96.1	91.3	91.1	91.8	92.3	92.1	92.3	95.8	95.2	96.2	93.8	93.5	92.7	91.5	90.7	90.9	91.3	91.1	90.8	12	0.95
MFMSA	98.3	98.0	98.4	99.5	99.3	99.5	98.9	97.4	98.6	98.3	98.5	98.8	97.9	97.3	97.9	94.7	94.4	95.3	97.3	97.3	97.6	96.2	96.0	96.3	90.9	90.5	90.7	94.6	94.5	94.6	26.9	1.4
Gloria	98.1	97.5	97.9	98.5	98.3	98.5	98.2	97.5	98.9	98.1	97.6	98.4	93.7	93.7	94.5	94.2	94.2	94.2	96.2	95.5	95.9	93.7	93.5	93.2	89.6	89.1	90.2	92.5	91.2	91.9	30.8	1.54

Table 1. Performance comparison of MAIL with existing methods ( M1–M10 ) on D1–D10 datasets for classification. Bold and red indicate the best and second-best results, respectively. MAIL employs ResNet18 and ResNet50 as base networks, denoted as MAIL-RN18 and MAIL-RN50 .

Datasets $\rightarrow$	D11: PathMNIST		D12: PneumoniaMNIST		D13: RetinaMNIST		D14: BreastMNIST		D15: TissueMNIST		D16: OrganAMNIST			D17: KVASIR		D18: ISIC 2018		D19: BraTs 2020		D20: LiTs		Params	Flops
Models $\downarrow$	ACC	AUC	ACC	AUC	ACC	AUC	ACC	AUC	ACC	AUC	ACC	AUC	Models $\downarrow$	DICE	mIOU	DICE	mIOU	DICE	mIOU	DICE	mIOU	Params	Flops
DDA-Net	92.25	99.84	90.25	97.25	67.62	78.46	86.35	90.18	68.90	94.54	95.75	99.80	UNet	80.55	72.60	87.32	80.24	81.79	76.54	88.80	84.64	34.5	65.5
NAT	91.80	99.78	89.95	97.12	66.58	77.60	86.24	90.10	68.42	94.39	95.70	99.80	UNet++	84.37	77.48	87.30	80.25	82.55	76.37	89.45	85.08	9.2	34.7
POTTER	91.45	99.50	89.89	97.74	63.54	74.26	87.15	90.90	67.98	93.29	96.06	99.85	AttnUNet	83.89	77.15	87.80	80.54	82.78	76.48	86.72	84.98	34.9	66.6
MFMSA	92.28	99.70	91.49	98.15	69.10	78.84	88.90	93.37	72.45	95.29	96.12	99.90	PolypPVT	91.67	85.90	90.42	83.85	86.38	79.50	93.83	88.20	25.1	5.3
Gloria	91.94	99.78	91.25	97.88	66.89	77.55	87.52	91.07	69.88	94.72	95.68	99.80	TransUNet	86.45	81.37	87.28	81.20	84.90	78.24	92.12	87.85	105	38.5

Table 2. Performance comparison of MAIL with existing methods ( M1–M10 ) on D11–D16 (classification) and ( M11–M21 ) on D17–D20 (segmentation). Bold indicates the best result. MAIL employs MobileNetV2, ResNet18, and SegNet as base networks, denoted as MAIL-M2 , MAIL-RN18 , and MAIL-Seg .

Datasets $\rightarrow$	D5: HAM10000					D3: Tuberculosis CXR					D10: CBIS-DDSM					D11: PathMNIST					D16: OrganAMNIST					D8: CNMC-2019
Datasets $\rightarrow$	PGD	BIM	MIM	AA	Square	PGD	BIM	MIM	AA	Square	PGD	BIM	MIM	AA	Square	PGD	BIM	MIM	AA	Square	PGD	BIM	MIM	AA	Square	PGD	BIM	MIM	AA	Square
PNI	51.8	55.2	53.5	60.9	63.1	74.9	76.4	74.9	80.6	81.8	47.6	49.4	47.3	59.2	56.2	39.3	40.6	39.6	49.8	48.5	39.8	38.1	37.4	49.4	46.8	70.5	72.4	70.2	77.4	77.6
DBN	48.3	50.9	48.3	57.6	59.4	72.5	74.6	73.8	78.1	79.6	40.3	45.9	40.3	58.4	56.2	38.4	39.2	38.1	47.8	43.7	38.7	39.5	38.0	45.6	45.3	69.1	73.6	70.4	74.5	72.9
Learn2Perturb	53.0	54.6	51.8	61.4	59.4	77.1	79.3	77.1	82.5	81.9	45.8	43.2	45.8	61.1	62.3	40.7	41.8	40.7	49.9	50.1	39.4	44.7	39.4	48.2	47.8	74.5	77.8	76.3	79.6	79.0
RPF	59.5	62.1	60.3	68.9	69.4	80.8	83.6	81.3	85.1	85.5	51.0	49.3	48.0	67.2	63.7	42.5	43.9	41.2	54.7	48.6	41.8	40.6	38.8	52.9	51.2	81.9	81.4	79.1	85.1	84.3
CAP	60.8	63.2	56.4	69.2	72.4	86.2	87.5	84.2	90.7	90.3	52.3	52.2	50.1	68.1	64.5	43.4	44.7	43.3	55.8	50.2	40.7	42.1	41.2	51.9	50.6	82.3	82.8	79.5	86.2	85.1

Table 3. Comparison of Robust-MAIL with SOTA defenses ( M22–M27 ) in accuracy on D5 , D3 , D10 , D11 , D16 , and D8 datasets. White-box (PGD, BIM , MIM ) and black-box ( AutoAttack (AA) , Square ) attacks are used for evaluation. Bold indicates the best result.

Dataset

ERLA

MFIFA

EMSCA

Acc

Dataset

ERLA

MFIFA

EMSCA

Acc

Params

\times

\times

\times

94.9

94.7

\times

\times

\times

90.1

88.9

23.4

\times

\times

\checkmark

96.5

\times

\times

✓

91.9

91.4

23.7

\times

\checkmark

\times

96.9

96.8

\times

\checkmark

\times

92.3

23.9

\times

\checkmark

\checkmark

98.9

98.8

\times

\checkmark

\checkmark

95.8

95.6

24.2

\checkmark

\times

\times

95.9

95.6

\checkmark

\times

\times

91.5

91.2

9.83

\checkmark

\times

\checkmark

97.7

\checkmark

\times

\checkmark

93.6

93.4

10.2

Table 4. Performance comparison of ERLA , MFIFA , and EMSCA modules in MAIL on benchmark datasets D5 and D6 .

EMILA

MFIFA

EMSCA

Robust-MAIL

MSGDC

CMI

Acc

Params

RPF@E

RPF

MAN

Acc

\times

\times

\times

\times

\times

\times

94.9

90.1

23.4

\times

\times

\times

2.58

\checkmark

\times

\checkmark

\times

\checkmark

\times

96.49

92.15

9.85

\checkmark

\times

\times

18.37

25.42

\times

\checkmark

\times

\checkmark

\times

\checkmark

97.78

94.79

24.2

\times

\times

\checkmark

48.27

73.39

\checkmark

\times

\times

\checkmark

\times

\checkmark

97.42

93.90

10.4

\times

\checkmark

\times

46.95

70.44

\checkmark

\checkmark

\times

\checkmark

\times

\checkmark

98.48

95.52

10.8

\times

\checkmark

\checkmark

61.18

82.56

Table 5. Evaluation of EMILA components (MSGDC and CA), MFIFA components (CO and DF), and EMSCA components (MSGDC and CMI) (ref. section 4.3 ) on D5 and D6 datasets. For Robust-MAIL , we assess the RPAN components: RPF and MAN , along with utilizing RPF@E on the D5 and D3 datasets. "+" and "*" denote evaluations for MAIL and Robust-MAIL .

Dataset	PFA	CAT	Acc	F1	Dataset	PFA	CAT	Acc	F1	Params
D5	$\times$	$\checkmark$	99.1	99.0	D6	$\times$	$\checkmark$	95.9	95.9	11.7
D5	$\checkmark$	$\times$	99.4	99.4	D6	$\checkmark$	$\times$	96.3	96.3	11.7

Table 6. Performance comparison of cascaded attention ( CAT ) and parallel fusion attention ( PFA ) within our MAIL on D5 and D6 datasets.

Figure 7. Visual representation of the important regions highlighted by our proposed MAIL networks i.e., MAIL-RN18 and MAIL-RN50 and ten SOTA methods using the GRAD-CAM technique on two benchmark datasets D5 and D1 .

Dataset	Modality	Task	Classes / Labels	#Samples	Split	Remarks
Nickparvar MRI ( Nickparvar, 2021 )	Brain MRI (T1–weighted)	Classification	Glioma (1621), Meningioma (1645), No Tumor (2000), Pituitary (1757)	7023	80%/10%/10%	Aggregated from Figshare, SARTAJ, BrH35 datasets.
IQ-OTHNCCD ( Alyasriy and Muayed, 2020 )	Chest CT	Classification	Normal, Benign, Malignant	1098	80%/10%/10%	Lung nodule diagnosis.
Tuberculosis CXR ( Rahman et al., 2020 )	Chest X-ray	Classification	Normal vs Tuberculosis	4200	80%/10%/10%	Binary TB screening.
BCCD ( Shenggan, 2018 )	Microscopy (blood smear)	Classification	Eosinophil, Lymphocyte, Monocyte, Neutrophil	9957	80%/10%/10%	Leukocyte subtype recognition.
HAM10000 ( Tschandl et al., 2018 )	Dermoscopy	Classification	7 skin lesion categories	10015	80%/10%/10%	$>$ 50% histopathology-confirmed labels.
SIPaKMeD ( Plissiti et al., 2018 )	Cytology	Classification	5 epithelial cell types	4049	80%/10%/10%	Pap smear cell classification of cervical cancer.

Table 7. Summary of datasets used in our study.

Explore similar work

HyPCA-Net: Advancing Multimodal Fusion in Medical Image Analysis
Feb 18, 2026J. Dhar, M. K. Pandey, D. Chakladar +4Multimodal Fusion Multimodal Learning
Indian Institute of Technology Ropar, India · RoentGen Health, India · Lulea University of Technology, Sweden +4
Advancing Multimodal Fusion on Heterogeneous Medical Data with Hybrid Geometry Attention
Jul 21, 2026Joy Dhar, Manish Kumar Pandey, Nayyar Zaidi +4Cross-Modal Representation Learning Cross-Modal Attention
Indian Institute of Technology Ropar Rupnagar, Punjab, India · RoentGen Health New Delhi, Delhi, India · Deakin University Melbourne, Victoria, Australia +3
Context-driven Missing-Modality Learning for Robust Medical Diagnosis with Image-Tabular Data
May 25, 2026Tianling Liu, Lequan Yu, Tong Han +1Missing-Modality Learning Multimodal Robustness
College of Intelligence and Computing, Tianjin University, Tianjin 300350, China. · Department of Statistics and Actuarial Science, School of Computing and Data Science, The University of Hong Kong, Hong Kong. · Department of Radiology, Tianjin Huanhu Hospital, Tianjin 300350, China. +2

Feb 18, 2026J/K move · Enter open · S save

HyPCA-Net: Advancing Multimodal Fusion in Medical Image Analysis

J. Dhar, M. K. Pandey, D. Chakladar, M. Haghighat+3

Indian Institute of Technology Ropar, India · RoentGen Health, India · Lulea University of Technology, Sweden+4

Multimodal Fusion Multimodal Learning Efficient Multimodal Inference Cross-Modal Attention

Open paper AskPDF arXiv

Multimodal fusion frameworks, which integrate diverse medical imaging modalities (e.g., MRI, CT), have shown great potential in applications such as skin cancer detection, dementia diagnosis, and brain tumor prediction. However, existing multimodal fusion methods face significant challenges. First, they often rely on computationally expensive models, limiting their applicability in low-resource environments. Second, they often employ cascaded attention modules, which potentially increase risk of information loss during inter-module transitions and hinder their capacity to effectively capture robust shared representations across modalities. This restricts their generalization in multi-disease analysis tasks. To address these limitations, we propose a Hybrid Parallel-Fusion Cascaded Attention Network (HyPCA-Net), composed of two core novel blocks: (a) a computationally efficient residual adaptive learning attention block for capturing refined modality-specific representations, and (b) a dual-view cascaded attention block aimed at learning robust shared representations across diverse modalities. Extensive experiments on ten publicly available datasets exhibit that HyPCA-Net significantly outperforms existing leading methods, with improvements of up to 5.2% in performance and reductions of up to 73.1% in computational cost. Code: https://github.com/misti1203/HyPCA-Net.

Abstract

Explore similar work

HyPCA-Net: Advancing Multimodal Fusion in Medical Image Analysis

Advancing Multimodal Fusion on Heterogeneous Medical Data with Hybrid Geometry Attention

Context-driven Missing-Modality Learning for Robust Medical Diagnosis with Image-Tabular Data