HyPCA-Net: Advancing Multimodal Fusion in Medical Image Analysis

J. Dhar; M. K. Pandey; D. Chakladar; M. Haghighat; A. Alavi; S. Mistry; N. Zaidi

cs.CVFeb 18, 2026

HyPCA-Net: Advancing Multimodal Fusion in Medical Image Analysis

Authors: J. Dhar, M. K. Pandey, D. Chakladar, M. Haghighat, A. Alavi, S. Mistry, N. Zaidi

Organizations: Indian Institute of Technology Ropar, India · RoentGen Health, India · Lulea University of Technology, Sweden · QUT, Australia · RMIT University, Australia · Curtin University, Australia · Deakin University, Australia

Abstract

Multimodal fusion frameworks, which integrate diverse medical imaging modalities (e.g., MRI, CT), have shown great potential in applications such as skin cancer detection, dementia diagnosis, and brain tumor prediction. However, existing multimodal fusion methods face significant challenges. First, they often rely on computationally expensive models, limiting their applicability in low-resource environments. Second, they often employ cascaded attention modules, which potentially increase risk of information loss during inter-module transitions and hinder their capacity to effectively capture robust shared representations across modalities. This restricts their generalization in multi-disease analysis tasks. To address these limitations, we propose a Hybrid Parallel-Fusion Cascaded Attention Network (HyPCA-Net), composed of two core novel blocks: (a) a computationally efficient residual adaptive learning attention block for capturing refined modality-specific representations, and (b) a dual-view cascaded attention block aimed at learning robust shared representations across diverse modalities. Extensive experiments on ten publicly available datasets exhibit that HyPCA-Net significantly outperforms existing leading methods, with improvements of up to 5.2% in performance and reductions of up to 73.1% in computational cost. Code: https://github.com/misti1203/HyPCA-Net.

Figures & tables

Figure 2: Overview of HyPCA-Net framework composing of two phases: (A) RMIL , which learns robust shared representations $\{x_{i}^{S}\}_{i=1}^{m}$ ; and (B) MML , which performs multi-disease classification. Within RMIL phase stands our novel (C) HyPCA block comprises of RALA block (refining unimodal features $\{x^{\prime}_{i}\}_{i=1}^{m}$ ) and (D) DVCA block (capturing dual-domain multimodal information)

Abbreviation	Full Term	Appears in
HyPCA - Net	Hybrid Parallel-fusion Cascaded Attention Network	Sec. 1 , 3
HyPCA	HyPCA block (RALA + DVCA)	Sec. 3
RMIL	Robust Multimodal Information Learning	Sec. 3.1
RALA	Residual Adaptive Learning Attention	Sec. 3.1.1
SCALA	Spatial–Channel convolution Adaptive Learning Attention	Sec. 3.1.1
MSHC	Multi-scale Spatial Heterogeneous Convolution	Sec. 3.1.1

Table 1: List of abbreviations based on modules and components.

Figure 3: (A) Overview of RALA block (first main component of HyPCA ), which is composed of SCALA blocks (B), which in turn is made out of MSHC block (C) and SCPFA block (D). The SCPFA block is composed of SHIA (E) and CHIA (F) components.

Figure 4: Overview of Hy-SFA (key component of DVCA block) comprising (A) TFSI and (B) FDCA blocks. FDCA includes Heterogeneous Channel Attention ( HCA ).

Figure 5: (A) Overview of the MMMUA block (key component of DVCA block). It consists of MCBI (B), FCIF (C), and SMIF (D) blocks. The MCBI block integrates the outputs of FCIF and SMIF blocks. FCIF employs Hierarchical Channel Fusion ( HCF ) mechanism.

Datasets $\rightarrow$	Backbone	D1: Nickparvar			D2: IQ-OTHNCCD			D3: Tuberculosis			D4: CNMC-2019			D5: HAM10000			D6: SIPaKMeD			D7: CRC			D8: CBIS-DDSM			Overall
Models $\downarrow$	Backbone	ACC	F1	AUC	ACC	F1	AUC	ACC	F1	AUC	ACC	F1	AUC	ACC	F1	AUC	ACC	F1	AUC	ACC	F1	AUC	ACC	F1	AUC	$\#$ P	$\#$ F
DDA-Net	ResNet18	96.8	96.6	97.2	98.1	97.5	98.2	95.8	95.1	95.4	92.2	91.8	92.1	92.2	92.0	92.6	92.5	92.8	91.9	92.7	92.7	93.1	92.5	91.8	91.5	12.1	1.12
MSCAM	PVT2-B2	97.6	97.6	97.9	99.5	99.5	99.5	97.8	96.9	97.6	95.3	95.1	95.3	97.6	97.2	97.6	94.2	94.0	94.9	96.4	96.2	96.6	94.1	93.9	94.2	26.9	$\underline{1.3}$
MFMSA	ResNet50	97.9	97.7	98.0	99.5	99.3	99.5	98.1	97.3	98.3	95.6	95.2	95.9	97.9	97.4	97.9	94.8	94.4	95.3	96.7	96.5	96.6	94.6	94.5	94.6	26.9	1.4
Gloria	ResNet50	98.1	97.6	97.9	98.5	98.4	98.5	96.6	96.0	96.9	93.3	93.3	93.4	93.8	93.8	94.5	94.2	94.2	94.2	95.9	95.6	95.7	92.5	91.2	91.9	30.8	1.54
MTTU-Net	ResNet50	97.9	97.9	98.0	99.5	99.2	99.5	97.3	96.6	97.6	94.3	93.9	94.1	97.4	96.5	97.2	91.9	92.3	92.6	96.9	96.8	97.0	94.1	93.3	94.5	38.1	6.8

Table 2: Performance comparison of HyPCA-Net with SOTA methods ( M1-M8 ) on

8

datasets ( D1-D8 ) for classification. Bold/underlined values indicate the best/second-best results, respectively. We report no. of parameters (in millions) and GFLOPS for each method.

Model	D9: COVID-19		D10: ISIC		$\#$ P	$\#$ F
Model	DSC	IoU	DSC	IoU	$\#$ P	$\#$ F
UNet	47.7	38.6	87.3	80.2	34.5	65.5
PolypPVT	81.7	74.4	90.4	83.9	25.1	5.3
MTTU-Net	82.2	75.7	89.2	82.6	71.6	20.9
MADGNet	83.9	76.9	90.2	83.8	31	14.2
PVT-CASCADE	84.1	77.3	90.4	84.0	34.1	7.6

Table 3: Performance comparison of HyPCA-Net-Seg ( SegNet backbone) and HyPCA-Net-EMCAD ( EMCAD backbone) with SOTA methods on D9 – D10 datasets

Integrated modules

D5: HAM10000

D6: SIPaKMeD

Integrated components within module

D5: HAM10000

D6: SIPaKMeD

RALA

Hy-SFA

MMMUA

\#

MSHC

CHIA

SHIA

FCIF

SMIF

MCBI

TFSI

FDCA

Acc

\#

\#

\times

\times

\times

93.2

89.3

23.4

\times

\times

\times

\times

\times

\times

\times

\times

93.2

93.1

89.5

89.3

23.4

1.16

\times

\times

\checkmark

97.1

93.0

25.2

\checkmark

\times

\times

\checkmark

\times

\times

\checkmark

\times

96.8

96.6

92.9

92.8

11.1

1.8

\times

\checkmark

\times

96.7

92.8

28.1

\checkmark

\checkmark

\times

\checkmark

\times

\times

\checkmark

\times

97.6

93.9

93.7

12.4

1.9

\times

\checkmark

\checkmark

98.6

94.9

30.5

\times

\times

\checkmark

\checkmark

\times

\times

\checkmark

\times

97.3

97.2

93.6

93.4

28.6

2.84

✓

\times

\times

96.1

92.3

10.1

\checkmark

\checkmark

\checkmark

\checkmark

\times

\times

\checkmark

\times

97.9

97.7

94.3

94.2

12.7

1.92

Table 4: Ablation of HyPCA-Net18 modules (left) and components (right), highlighting their contributions to performance gains on D5 – D6 datasets.

Integrated modules

D9: COVID-19

D10: ISIC

Integrated components within module

D9: COVID-19

D10: ISIC

RALA

Hy-SFA

MMMUA

DSC

IoU

DSC

IoU

MSHC

CHIA

SHIA

FCIF

SMIF

MCBI

TFSI

FDCA

DSC

IoU

DSC

IoU

\#

\#

\times

\times

\times

82.8

75.5

85.9

79.1

27.2

\times

\times

\times

\times

\times

\times

\times

\times

82.8

75.5

85.9

79.1

27.2

4.32

\times

\times

\checkmark

86.2

78.8

89.1

81.9

29.2

\checkmark

\times

\times

\checkmark

\times

\times

\checkmark

\times

85.5

78.2

87.4

81.2

15.1

6.27

\times

\checkmark

\times

85.5

77.9

88.4

81.1

32.1

\checkmark

\checkmark

\times

\checkmark

\times

\times

\checkmark

\times

86.5

79.1

88.9

82.5

16.2

6.58

\times

\checkmark

\checkmark

88.5

80.9

90.7

84.1

34.4

\times

\times

\checkmark

\checkmark

\times

\times

\checkmark

\times

86.2

78.8

88.5

82.1

32.4

9.46

\checkmark

\times

\times

84.7

77.4

87.9

80.8

13.9

\checkmark

\checkmark

\checkmark

\checkmark

\times

\times

\checkmark

\times

86.8

79.5

89.3

82.8

16.5

6.64

Table 5: Ablation of HyPCA-Net-EMCAD modules (left) and components (right), highlighting their contributions to performance gains on D9 – D10 datasets.

Figure 6: Visual representation of the important regions highlighted by our proposed HyPCA-Net framework and ten other SOTA methods using the GRAD-CAM technique on two benchmark datasets D5 and D1 .

HA	CA	D5	D6	$\#$ P	$\#$ F
$\checkmark$	$\times$	99.3	95.7	14.47	2.25
$\times$	$\checkmark$	98.9	95.2	14.47	2.25

Table 6: Performance comparison (F1-score) of cascaded ( CA ) vs. hybrid ( HA ) attention in CHIA and SHIA on D5 and D6 , keeping other components fixed.

Explore similar work

Effective and Robust Multimodal Medical Image Analysis
Feb 17, 2026Joy Dhar, Nayyar Zaidi, Maryam HaghighatMedical Image Analysis Multimodal Fusion
Indian Institute of Technology Ropar, Rupnagar, Punjab, India · Deakin University, Melbourne, Victoria, Australia · Queensland University of Technology, Brisbane, QLD, Australia
Advancing Multimodal Fusion on Heterogeneous Medical Data with Hybrid Geometry Attention
Jul 21, 2026Joy Dhar, Manish Kumar Pandey, Nayyar Zaidi +4Cross-Modal Representation Learning Cross-Modal Attention
Indian Institute of Technology Ropar Rupnagar, Punjab, India · RoentGen Health New Delhi, Delhi, India · Deakin University Melbourne, Victoria, Australia +3
When Fusion Fails: Corruption-Aware Rebalanced Fusion for Multi-Modal Medical Image Segmentation
Sep 9, 2026Yuchen Pei, Xiaoyu Hu, Yixiong Zou +5Cross-Modality Medical Image Segmentation Multi-Source Image Fusion
Central China Normal University Wuhan, China · Huazhong University of Science and Technology Wuhan, China · Guangzhou University of Chinese Medicine Guangzhou, China +2

Feb 17, 2026J/K move · Enter open · S save

Effective and Robust Multimodal Medical Image Analysis

Joy Dhar, Nayyar Zaidi, Maryam Haghighat

Indian Institute of Technology Ropar, Rupnagar, Punjab, India · Deakin University, Melbourne, Victoria, Australia · Queensland University of Technology, Brisbane, QLD, Australia

Medical Image Analysis Multimodal Fusion Efficient Multimodal Inference Multimodal Robustness Cross-Attention

Open paper AskPDF arXiv

Multimodal Fusion Learning (MFL), leveraging disparate data from various imaging modalities (e.g., MRI, CT, SPECT), has shown great potential for addressing medical problems such as skin cancer and brain tumor prediction. However, existing MFL methods face three key limitations: a) they often specialize in specific modalities, and overlook effective shared complementary information across diverse modalities, hence limiting their generalizability for multi-disease analysis; b) they rely on computationally expensive models, restricting their applicability in resource-limited settings; and c) they lack robustness against adversarial attacks, compromising reliability in medical AI applications. To address these limitations, we propose a novel Multi-Attention Integration Learning (MAIL) network, incorporating two key components: a) an efficient residual learning attention block for capturing refined modality-specific multi-scale patterns and b) an efficient multimodal cross-attention module for learning enriched complementary shared representations across diverse modalities. Furthermore, to ensure adversarial robustness, we extend MAIL network to design Robust-MAIL by incorporating random projection filters and modulated attention noise. Extensive evaluations on 20 public datasets show that both MAIL and Robust-MAIL outperform existing methods, achieving performance gains of up to 9.34% while reducing computational costs by up to 78.3%. These results highlight the superiority of our approaches, ensuring more reliable predictions than top competitors. Code: https://github.com/misti1203/MAIL-Robust-MAIL.

Abstract

Explore similar work

Effective and Robust Multimodal Medical Image Analysis

Advancing Multimodal Fusion on Heterogeneous Medical Data with Hybrid Geometry Attention

When Fusion Fails: Corruption-Aware Rebalanced Fusion for Multi-Modal Medical Image Segmentation