CFCML: A Coarse-to-Fine Crossmodal Learning Framework For Disease Diagnosis Using Multimodal Images and Tabular Data

Tianling Liu; Hongying Liu; Fanhua Shang; Lequan Yu; Tong Han; Liang Wan

cs.CVMar 20, 2026

CFCML: A Coarse-to-Fine Crossmodal Learning Framework For Disease Diagnosis Using Multimodal Images and Tabular Data

Authors: Tianling Liu, Hongying Liu, Fanhua Shang, Lequan Yu, Tong Han, Liang Wan

Organizations: College of Intelligence and Computing, Tianjin University, Tianjin 300350, China. · Medical School of Tianjin University, Tianjin 300072, China. · Peng Cheng Lab, Shenzhen 518055, China. · Department of Statistics and Actuarial Science, School of Computing and Data Science, The University of Hong Kong, Hong Kong. ·

Abstract

In clinical practice, crossmodal information including medical images and tabular data is essential for disease diagnosis. There exists a significant modality gap between these data types, which obstructs advancements in crossmodal diagnostic accuracy. Most existing crossmodal learning (CML) methods primarily focus on exploring relationships among high-level encoder outputs, leading to the neglect of local information in images. Additionally, these methods often overlook the extraction of task-relevant information. In this paper, we propose a novel coarse-to-fine crossmodal learning (CFCML) framework to progressively reduce the modality gap between multimodal images and tabular data, by thoroughly exploring inter-modal relationships. At the coarse stage, we explore the relationships between multi-granularity features from various image encoder stages and tabular information, facilitating a preliminary reduction of the modality gap. At the fine stage, we generate unimodal and crossmodal prototypes that incorporate class-aware information, and establish hierarchical anchor-based relationship mining (HRM) strategy to further diminish the modality gap and extract discriminative crossmodal information. This strategy utilize modality samples, unimodal prototypes, and crossmodal prototypes as anchors to develop contrastive learning approaches, effectively enhancing inter-class disparity while reducing intra-class disparity from multiple perspectives. Experimental results indicate that our method outperforms the state-of-the-art (SOTA) methods, achieving improvements of 1.53% and 0.91% in AUC metrics on the MEN and Derm7pt datasets, respectively. The code is available at https://github.com/IsDling/CFCML.

Figures & tables

Figure 1: Overview of proposed CFCML framework. For clarity, the figure illustrates a scenario integrating a single image with tabular data. Initially, we extract multi-granularity features from various stages of the image encoder and obtain features from the tabular data using the pretrained CLIP text encoder. Subsequently, the MG-CIE module explore interactions between these multi-granularity features and tabular features to obtain the enhanced unimodal features while preliminary reducing the modality gap. Finally, the CCRM strategy employs class information as a bridge to further narrow the modality gap and extract discriminative features from the crossmodal features. The resulting class-aware crossmodal features are then concatenated to facilitate the final prediction. Specifically, in the CCRM Strategy section, different colors represent samples from distinct classes, while different shapes denote samples from various modalities. The symbol $\oplus$ indicates the concatenation operation.

Figure 2: Architecture of the proposed CIE Module.

Figure 3: Illustration of the CCRM strategy across three modalities and three class conditions. CCRM encompasses sample-anchor-based, unimodal-anchor-based, and crossmodal-anchor-based CL strategies. Specifically, in each strategy, we select an anchor sample and identify its positive samples (connected by the red dashed line) and negative samples (connected by the black dashed line). The objective is to minimize the distance between the anchor and positive samples while maximizing the distance between the anchor and negative samples.

Figure 4: Display of multimodal images from the adopted datasets.

Attributes	Templates
management	The {} for the patient is {}
sex	The {} of patient is {}
age	The {} of patient is {}
tumor area	The {} in the brain is {}.
edema area
tumor location

Table 1: The templates for different attributes. In each template, the first {} represents the attribute, and the second {} represents the attribute value.

Methods	Metrics							Complexity
Methods	ACC	ACC_G1	ACC_G2inv	ACC_G2ninv	weighted-F1	macro-F1	AUC	Param	GFLOPs
ETMC Han et al. (2022b)	88.66 $\pm$ 4.74*	93.99 $\pm$ 7.22*	66.94 $\pm$ 8.68*	63.73 $\pm$ 15.01*	89.02 $\pm$ 3.40*	75.39 $\pm$ 3.54*	90.19 $\pm$ 0.98*	10.50	18.51
DMD Li et al. (2023)	92.21 $\pm$ 0.51*	96.46 $\pm$ 1.42*	91.49 $\pm$ 7.70	60.72 $\pm$ 9.45*	92.13 $\pm$ 0.25*	81.34 $\pm$ 1.15*	96.25 $\pm$ 0.73*	171.66	18.74
MVCNet Guo et al. (2024)	93.84 $\pm$ 1.33*	97.53 $\pm$ 2.72	86.72 $\pm$ 7.35*	71.11 $\pm$ 6.74*	93.81 $\pm$ 1.12*	85.84 $\pm$ 2.50*	96.83 $\pm$ 1.05*	10.67	18.51
CCML Liu et al. (2024)	91.33 $\pm$ 1.06*	96.92 $\pm$ 1.63	81.22 $\pm$ 13.45*	56.19 $\pm$ 14.02*	90.91 $\pm$ 0.92*	79.50 $\pm$ 3.00*	95.41 $\pm$ 1.36*	10.36	18.51
GLoMo Zhuang et al. (2024)	94.35 $\pm$ 0.95*	96.62 $\pm$ 0.69	86.56 $\pm$ 3.41*	82.62 $\pm$ 5.82*	94.50 $\pm$ 0.91*	87.40 $\pm$ 2.70*	97.04 $\pm$ 1.19*	12.85	18.60

Table 2: The comparison results on the MEN dataset (mean

\pm

standard deviation). The best results for each metric are highlighted in bold . * indicates that our proposed method achieves statistically significant improvements over the other compared methods (p-value¡0.05). The complexity, including the number of parameters (M) and GFLOPs (G), is displayed on the right. The term MRIM+ denotes the integration of the proposed MG-CIE module into the comparative method MRIM, while Proposed- indicates the replacement of the MG-CIE module with the single-granularity CIE at the final stage of the encoder (SG-CIE).

Methods	Metrics							Complexity
Methods	SEN	SPE	ACC	G_Mean	Ba_ACC	AUPRC	AUC	Param	GFLOPs
ETMC Han et al. (2022b)	77.56 $\pm$ 8.42	73.98 $\pm$ 5.54*	75.11 $\pm$ 2.90*	75.57 $\pm$ 3.02*	75.77 $\pm$ 3.22*	52.13 $\pm$ 3.44*	83.06 $\pm$ 1.67*	28.69	44.66
DMD Li et al. (2023)	69.97 $\pm$ 5.80*	88.89 $\pm$ 2.16	82.92 $\pm$ 0.36*	78.79 $\pm$ 2.38*	79.43 $\pm$ 1.82*	61.56 $\pm$ 0.99*	89.61 $\pm$ 1.06*	184.49	44.76
MVCNet Guo et al. (2024)	73.27 $\pm$ 3.57*	86.45 $\pm$ 1.90*	82.29 $\pm$ 0.65*	79.56 $\pm$ 1.24*	79.86 $\pm$ 1.07*	60.77 $\pm$ 1.12*	88.88 $\pm$ 0.50*	28.89	44.66
CCML Liu et al. (2024)	69.97 $\pm$ 2.06*	83.71 $\pm$ 2.16*	79.38 $\pm$ 1.90*	76.53 $\pm$ 1.84*	76.84 $\pm$ 1.84*	56.05 $\pm$ 2.80*	86.63 $\pm$ 2.41*	28.58	44.66
GLoMo Zhuang et al. (2024)	79.87 $\pm$ 3.48	78.69 $\pm$ 8.12*	79.06 $\pm$ 4.81*	79.15 $\pm$ 3.17*	79.28 $\pm$ 3.03*	57.48 $\pm$ 5.61*	88.77 $\pm$ 1.85*	30.67	44.69

Table 3: The comparison results on the Derm7pt dataset.

Proposed		ACC	ACC _G1	ACC _G2inv	ACC _G2ninv	weighted -F1	macro -F1	AUC
MG-RIE	CCRM	ACC	ACC _G1	ACC _G2inv	ACC _G2ninv	weighted -F1	macro -F1	AUC
$\times$	$\times$	91.32	95.84	82.06	64.13	91.37	80.95	95.77
✓	$\times$	92.84	96.00	84.73	74.44	92.99	84.15	95.90
$\times$	✓	91.70	94.61	86.48	73.49	92.04	82.42	96.02
✓	✓	95.61	96.77	93.24	88.57	95.76	91.13	98.57

Table 4: The ablation study on the MEN dataset.

Proposed		SEN	SPE	ACC	G_Mean	Ba_ACC	AUPRC	AUC
MG-CIE	CCRM	SEN	SPE	ACC	G_Mean	Ba_ACC	AUPRC	AUC
$\times$	$\times$	75.58	83.56	81.04	79.36	79.57	59.38	87.52
✓	$\times$	74.92	87.52	83.54	80.91	81.22	63.01	89.51
$\times$	✓	73.27	88.28	83.54	80.32	80.77	62.89	88.96
✓	✓	75.91	89.49	85.21	82.40	82.70	66.07	90.52

Table 5: The ablation study on the Derm7pt dataset.

Figure 5: Display of t-SNE visualization results for proposed and comparison methods. In each sub-figure, different colors represent samples belonging to distinct classes, while different shapes denote samples from various modalities. The samples exhibiting different shapes but sharing the same color are more compact, while those of varying colors are more dispersed, suggesting that the model is more effective in reducing modality gap and extracting richer class-aware information.

Figure 6: Display of the ablation study results about various predefined token numbers on both adopted datasets.

Figure 7: Activation map visualization results of the proposed and comparison methods for three cases in the MEN dataset using Grad-CAM. Original FLAIR-C images are displayed on the right, with the lesion areas highlighted by red boxes. In the Grad-CAM images, darker red regions signify the areas with the highest contribution to the prediction.

Figure 8: Display of MDA visualization results for proposed and comparison methods. In each sub-figure, different colors represent samples belonging to distinct classes.

Figure 9: The comparison results of the AUC metric for CLIP embedding and MLP embedding of tabular data on the MEN and Derm7pt datasets.

Figure 10: The comparison results for different token mapping operations, including

Conv1D

and

Perceiver

, on the MEN (left part) and Derm7pt (right part) datasets.

Figure 11: The ablation results for the AUC metric of SG-CIE at various stages on the MEN dataset. The notation “SG-CIE@i” denotes SG-CIE at the

i

th stage of the encoders.

Cross-Modal Learning

Cross-modal learning connects information across different data types, enabling models to align, transfer, and reason across modalities.

See all 144 papers

May 25, 2026

Context-driven Missing-Modality Learning for Robust Medical Diagnosis with Image-Tabular Data

While multimodal data integrating diverse imaging and clinical tabular records is crucial for accurate medical diagnosis, the arbitrary absence of specific modalities is prevalent in clinical practice, severely degrading the performance of multimodal models. Existing methods either discard missing modalities, leading to information loss, or struggle to synthesize them without capturing complex inter-modal dependencies. To address these limitations, we propose a novel Context-driven Missing-Modality Learning (CMML) framework, which sequentially performs modality synthesis and semantic alignment to achieve robust diagnosis under arbitrary missing conditions. Specifically, we design a Cascade Residual Transformer-based Autoencoder (CRTA) that leverages learnable context tokens acting as dataset-level semantic prior to capture inter-modal dependencies and synthesize key missing representations. These representations are further enriched by modality-specific memory banks. To resolve the discrepancy between original available and synthesized representations, we transform the learned context tokens into instance-adaptive semantic references by infusing multimodal representations from the CRTA's outputs. This reference guides the alignment of heterogeneous modality representations into a unified space, where class-aware contrastive refinement is finally applied to explore discriminative diagnostic cues. Extensive evaluations on skin lesion (Derm7pt), ocular disease (ODIR), and meningioma (MEN) datasets demonstrate that CMML significantly outperforms state-of-the-art (SOTA) methods, yielding AVG AUC improvements of 1.26%, 0.97%, and 1.32%, respectively.

Tianling Liu, Lequan Yu, Tong Han +1

College of Intelligence and Computing, Tianjin University, Tianjin 300350, China. · Department of Statistics and Actuarial Science, School of Computing and Data Science, The University of Hong Kong, Hong Kong. · Department of Radiology, Tianjin Huanhu Hospital, Tianjin 300350, China. +2

Missing-Modality Learning Multimodal Robustness Medical Diagnosis Cross-Modal Alignment

Ask PDF

Aug 10, 2026

UniMod: Enhancing Multi-Modal Medical Diagnosis through Cross-Modality and Within-Modality Alignment

Multi-modal learning combining medical images and clinical text is promising for disease diagnosis. However, standard multi-modal training leads to shortcut learning: models exploit the easier modality (e.g., diagnostic cues in text) while neglecting harder-to-learn features (e.g., subtle visual patterns). We propose UniMod, a framework that mitigates shortcut learning by requiring each modality to predict the diagnosis on its own. It supervises image-only, text-only, and multi-modal classification simultaneously, so each modality must extract diagnostic features. We add cross-modality alignment for knowledge transfer and within-modality supervised contrastive alignment over same-diagnosis patients. On Harvard-Glaucoma, UniMod reaches 0.850 AUC, outperforming OGM-GE and Gradient Blending by 1.6-1.8%; on CheXpert Plus, it reaches 0.966 AUC, surpassing them by over 5%. UniMod also extends to 5-class multi-label diagnosis without architectural change, improving mean AUC by 0.097 over CGGM.

Zijian Gu, Weikai Lin, Shuang Zhou +2

University of Central Florida · Orlando, FL, United States · University of Rochester +5

Medical Diagnosis Cross-Modal Alignment Multimodal Classification Shortcut Learning

Ask PDF

Feb 17, 2026

Effective and Robust Multimodal Medical Image Analysis

Multimodal Fusion Learning (MFL), leveraging disparate data from various imaging modalities (e.g., MRI, CT, SPECT), has shown great potential for addressing medical problems such as skin cancer and brain tumor prediction. However, existing MFL methods face three key limitations: a) they often specialize in specific modalities, and overlook effective shared complementary information across diverse modalities, hence limiting their generalizability for multi-disease analysis; b) they rely on computationally expensive models, restricting their applicability in resource-limited settings; and c) they lack robustness against adversarial attacks, compromising reliability in medical AI applications. To address these limitations, we propose a novel Multi-Attention Integration Learning (MAIL) network, incorporating two key components: a) an efficient residual learning attention block for capturing refined modality-specific multi-scale patterns and b) an efficient multimodal cross-attention module for learning enriched complementary shared representations across diverse modalities. Furthermore, to ensure adversarial robustness, we extend MAIL network to design Robust-MAIL by incorporating random projection filters and modulated attention noise. Extensive evaluations on 20 public datasets show that both MAIL and Robust-MAIL outperform existing methods, achieving performance gains of up to 9.34% while reducing computational costs by up to 78.3%. These results highlight the superiority of our approaches, ensuring more reliable predictions than top competitors. Code: https://github.com/misti1203/MAIL-Robust-MAIL.

Joy Dhar, Nayyar Zaidi, Maryam Haghighat

Indian Institute of Technology Ropar, Rupnagar, Punjab, India · Deakin University, Melbourne, Victoria, Australia · Queensland University of Technology, Brisbane, QLD, Australia

Medical Image Analysis Multimodal Fusion Efficient Multimodal Inference Multimodal Robustness Cross-Attention

Ask PDF

Abstract

Cross-Modal Learning

Explore similar work

Context-driven Missing-Modality Learning for Robust Medical Diagnosis with Image-Tabular Data

UniMod: Enhancing Multi-Modal Medical Diagnosis through Cross-Modality and Within-Modality Alignment

Effective and Robust Multimodal Medical Image Analysis