CFCML: A Coarse-to-Fine Crossmodal Learning Framework For Disease Diagnosis Using Multimodal Images and Tabular Data

Tianling Liu; Hongying Liu; Fanhua Shang; Lequan Yu; Tong Han; Liang Wan

cs.CVMar 20, 2026

CFCML: A Coarse-to-Fine Crossmodal Learning Framework For Disease Diagnosis Using Multimodal Images and Tabular Data

Authors: Tianling Liu, Hongying Liu, Fanhua Shang, Lequan Yu, Tong Han, Liang Wan

Organizations: College of Intelligence and Computing, Tianjin University, Tianjin 300350, China. · Medical School of Tianjin University, Tianjin 300072, China. · Peng Cheng Lab, Shenzhen 518055, China. · Department of Statistics and Actuarial Science, School of Computing and Data Science, The University of Hong Kong, Hong Kong. ·

Abstract

In clinical practice, crossmodal information including medical images and tabular data is essential for disease diagnosis. There exists a significant modality gap between these data types, which obstructs advancements in crossmodal diagnostic accuracy. Most existing crossmodal learning (CML) methods primarily focus on exploring relationships among high-level encoder outputs, leading to the neglect of local information in images. Additionally, these methods often overlook the extraction of task-relevant information. In this paper, we propose a novel coarse-to-fine crossmodal learning (CFCML) framework to progressively reduce the modality gap between multimodal images and tabular data, by thoroughly exploring inter-modal relationships. At the coarse stage, we explore the relationships between multi-granularity features from various image encoder stages and tabular information, facilitating a preliminary reduction of the modality gap. At the fine stage, we generate unimodal and crossmodal prototypes that incorporate class-aware information, and establish hierarchical anchor-based relationship mining (HRM) strategy to further diminish the modality gap and extract discriminative crossmodal information. This strategy utilize modality samples, unimodal prototypes, and crossmodal prototypes as anchors to develop contrastive learning approaches, effectively enhancing inter-class disparity while reducing intra-class disparity from multiple perspectives. Experimental results indicate that our method outperforms the state-of-the-art (SOTA) methods, achieving improvements of 1.53% and 0.91% in AUC metrics on the MEN and Derm7pt datasets, respectively. The code is available at https://github.com/IsDling/CFCML.

Figures & tables

Figure 1: Overview of proposed CFCML framework. For clarity, the figure illustrates a scenario integrating a single image with tabular data. Initially, we extract multi-granularity features from various stages of the image encoder and obtain features from the tabular data using the pretrained CLIP text encoder. Subsequently, the MG-CIE module explore interactions between these multi-granularity features and tabular features to obtain the enhanced unimodal features while preliminary reducing the modality gap. Finally, the CCRM strategy employs class information as a bridge to further narrow the modality gap and extract discriminative features from the crossmodal features. The resulting class-aware crossmodal features are then concatenated to facilitate the final prediction. Specifically, in the CCRM Strategy section, different colors represent samples from distinct classes, while different shapes denote samples from various modalities. The symbol $\oplus$ indicates the concatenation operation.

Figure 2: Architecture of the proposed CIE Module.

Figure 3: Illustration of the CCRM strategy across three modalities and three class conditions. CCRM encompasses sample-anchor-based, unimodal-anchor-based, and crossmodal-anchor-based CL strategies. Specifically, in each strategy, we select an anchor sample and identify its positive samples (connected by the red dashed line) and negative samples (connected by the black dashed line). The objective is to minimize the distance between the anchor and positive samples while maximizing the distance between the anchor and negative samples.

Figure 4: Display of multimodal images from the adopted datasets.

Attributes	Templates
management	The {} for the patient is {}
sex	The {} of patient is {}
age	The {} of patient is {}
tumor area	The {} in the brain is {}.
edema area
tumor location

Table 1: The templates for different attributes. In each template, the first {} represents the attribute, and the second {} represents the attribute value.

Methods	Metrics							Complexity
Methods	ACC	ACC_G1	ACC_G2inv	ACC_G2ninv	weighted-F1	macro-F1	AUC	Param	GFLOPs
ETMC Han et al. (2022b)	88.66 $\pm$ 4.74*	93.99 $\pm$ 7.22*	66.94 $\pm$ 8.68*	63.73 $\pm$ 15.01*	89.02 $\pm$ 3.40*	75.39 $\pm$ 3.54*	90.19 $\pm$ 0.98*	10.50	18.51
DMD Li et al. (2023)	92.21 $\pm$ 0.51*	96.46 $\pm$ 1.42*	91.49 $\pm$ 7.70	60.72 $\pm$ 9.45*	92.13 $\pm$ 0.25*	81.34 $\pm$ 1.15*	96.25 $\pm$ 0.73*	171.66	18.74
MVCNet Guo et al. (2024)	93.84 $\pm$ 1.33*	97.53 $\pm$ 2.72	86.72 $\pm$ 7.35*	71.11 $\pm$ 6.74*	93.81 $\pm$ 1.12*	85.84 $\pm$ 2.50*	96.83 $\pm$ 1.05*	10.67	18.51
CCML Liu et al. (2024)	91.33 $\pm$ 1.06*	96.92 $\pm$ 1.63	81.22 $\pm$ 13.45*	56.19 $\pm$ 14.02*	90.91 $\pm$ 0.92*	79.50 $\pm$ 3.00*	95.41 $\pm$ 1.36*	10.36	18.51
GLoMo Zhuang et al. (2024)	94.35 $\pm$ 0.95*	96.62 $\pm$ 0.69	86.56 $\pm$ 3.41*	82.62 $\pm$ 5.82*	94.50 $\pm$ 0.91*	87.40 $\pm$ 2.70*	97.04 $\pm$ 1.19*	12.85	18.60

Table 2: The comparison results on the MEN dataset (mean

\pm

standard deviation). The best results for each metric are highlighted in bold . * indicates that our proposed method achieves statistically significant improvements over the other compared methods (p-value¡0.05). The complexity, including the number of parameters (M) and GFLOPs (G), is displayed on the right. The term MRIM+ denotes the integration of the proposed MG-CIE module into the comparative method MRIM, while Proposed- indicates the replacement of the MG-CIE module with the single-granularity CIE at the final stage of the encoder (SG-CIE).

Methods	Metrics							Complexity
Methods	SEN	SPE	ACC	G_Mean	Ba_ACC	AUPRC	AUC	Param	GFLOPs
ETMC Han et al. (2022b)	77.56 $\pm$ 8.42	73.98 $\pm$ 5.54*	75.11 $\pm$ 2.90*	75.57 $\pm$ 3.02*	75.77 $\pm$ 3.22*	52.13 $\pm$ 3.44*	83.06 $\pm$ 1.67*	28.69	44.66
DMD Li et al. (2023)	69.97 $\pm$ 5.80*	88.89 $\pm$ 2.16	82.92 $\pm$ 0.36*	78.79 $\pm$ 2.38*	79.43 $\pm$ 1.82*	61.56 $\pm$ 0.99*	89.61 $\pm$ 1.06*	184.49	44.76
MVCNet Guo et al. (2024)	73.27 $\pm$ 3.57*	86.45 $\pm$ 1.90*	82.29 $\pm$ 0.65*	79.56 $\pm$ 1.24*	79.86 $\pm$ 1.07*	60.77 $\pm$ 1.12*	88.88 $\pm$ 0.50*	28.89	44.66
CCML Liu et al. (2024)	69.97 $\pm$ 2.06*	83.71 $\pm$ 2.16*	79.38 $\pm$ 1.90*	76.53 $\pm$ 1.84*	76.84 $\pm$ 1.84*	56.05 $\pm$ 2.80*	86.63 $\pm$ 2.41*	28.58	44.66
GLoMo Zhuang et al. (2024)	79.87 $\pm$ 3.48	78.69 $\pm$ 8.12*	79.06 $\pm$ 4.81*	79.15 $\pm$ 3.17*	79.28 $\pm$ 3.03*	57.48 $\pm$ 5.61*	88.77 $\pm$ 1.85*	30.67	44.69

Table 3: The comparison results on the Derm7pt dataset.

Proposed		ACC	ACC _G1	ACC _G2inv	ACC _G2ninv	weighted -F1	macro -F1	AUC
MG-RIE	CCRM	ACC	ACC _G1	ACC _G2inv	ACC _G2ninv	weighted -F1	macro -F1	AUC
$\times$	$\times$	91.32	95.84	82.06	64.13	91.37	80.95	95.77
✓	$\times$	92.84	96.00	84.73	74.44	92.99	84.15	95.90
$\times$	✓	91.70	94.61	86.48	73.49	92.04	82.42	96.02
✓	✓	95.61	96.77	93.24	88.57	95.76	91.13	98.57

Table 4: The ablation study on the MEN dataset.

Proposed		SEN	SPE	ACC	G_Mean	Ba_ACC	AUPRC	AUC
MG-CIE	CCRM	SEN	SPE	ACC	G_Mean	Ba_ACC	AUPRC	AUC
$\times$	$\times$	75.58	83.56	81.04	79.36	79.57	59.38	87.52
✓	$\times$	74.92	87.52	83.54	80.91	81.22	63.01	89.51
$\times$	✓	73.27	88.28	83.54	80.32	80.77	62.89	88.96
✓	✓	75.91	89.49	85.21	82.40	82.70	66.07	90.52

Table 5: The ablation study on the Derm7pt dataset.

Figure 5: Display of t-SNE visualization results for proposed and comparison methods. In each sub-figure, different colors represent samples belonging to distinct classes, while different shapes denote samples from various modalities. The samples exhibiting different shapes but sharing the same color are more compact, while those of varying colors are more dispersed, suggesting that the model is more effective in reducing modality gap and extracting richer class-aware information.

Figure 6: Display of the ablation study results about various predefined token numbers on both adopted datasets.

Figure 7: Activation map visualization results of the proposed and comparison methods for three cases in the MEN dataset using Grad-CAM. Original FLAIR-C images are displayed on the right, with the lesion areas highlighted by red boxes. In the Grad-CAM images, darker red regions signify the areas with the highest contribution to the prediction.

Figure 8: Display of MDA visualization results for proposed and comparison methods. In each sub-figure, different colors represent samples belonging to distinct classes.

Figure 9: The comparison results of the AUC metric for CLIP embedding and MLP embedding of tabular data on the MEN and Derm7pt datasets.

Figure 10: The comparison results for different token mapping operations, including

Conv1D

and

Perceiver

, on the MEN (left part) and Derm7pt (right part) datasets.

Figure 11: The ablation results for the AUC metric of SG-CIE at various stages on the MEN dataset. The notation “SG-CIE@i” denotes SG-CIE at the

i

th stage of the encoders.

Cross-Modal Learning

Cross-modal learning connects information across different data types, enabling models to align, transfer, and reason across modalities.

See all 144 papers

Context-driven Missing-Modality Learning for Robust Medical Diagnosis with Image-Tabular Data
May 25, 2026Tianling Liu, Lequan Yu, Tong Han +1Missing-Modality Learning Multimodal Robustness
College of Intelligence and Computing, Tianjin University, Tianjin 300350, China. · Department of Statistics and Actuarial Science, School of Computing and Data Science, The University of Hong Kong, Hong Kong. · Department of Radiology, Tianjin Huanhu Hospital, Tianjin 300350, China. +2
UniMod: Enhancing Multi-Modal Medical Diagnosis through Cross-Modality and Within-Modality Alignment
Aug 10, 2026Zijian Gu, Weikai Lin, Shuang Zhou +2Medical Diagnosis Cross-Modal Alignment
University of Central Florida · Orlando, FL, United States · University of Rochester +5
Effective and Robust Multimodal Medical Image Analysis
Feb 17, 2026Joy Dhar, Nayyar Zaidi, Maryam HaghighatMedical Image Analysis Multimodal Fusion
Indian Institute of Technology Ropar, Rupnagar, Punjab, India · Deakin University, Melbourne, Victoria, Australia · Queensland University of Technology, Brisbane, QLD, Australia

May 25, 2026J/K move · Enter open · S save

Context-driven Missing-Modality Learning for Robust Medical Diagnosis with Image-Tabular Data

Tianling Liu, Lequan Yu, Tong Han, Liang Wan

College of Intelligence and Computing, Tianjin University, Tianjin 300350, China. · Department of Statistics and Actuarial Science, School of Computing and Data Science, The University of Hong Kong, Hong Kong. · Department of Radiology, Tianjin Huanhu Hospital, Tianjin 300350, China.+2

Missing-Modality Learning Multimodal Robustness Medical Diagnosis Cross-Modal Alignment

Open paper AskPDF arXiv

While multimodal data integrating diverse imaging and clinical tabular records is crucial for accurate medical diagnosis, the arbitrary absence of specific modalities is prevalent in clinical practice, severely degrading the performance of multimodal models. Existing methods either discard missing modalities, leading to information loss, or struggle to synthesize them without capturing complex inter-modal dependencies. To address these limitations, we propose a novel Context-driven Missing-Modality Learning (CMML) framework, which sequentially performs modality synthesis and semantic alignment to achieve robust diagnosis under arbitrary missing conditions. Specifically, we design a Cascade Residual Transformer-based Autoencoder (CRTA) that leverages learnable context tokens acting as dataset-level semantic prior to capture inter-modal dependencies and synthesize key missing representations. These representations are further enriched by modality-specific memory banks. To resolve the discrepancy between original available and synthesized representations, we transform the learned context tokens into instance-adaptive semantic references by infusing multimodal representations from the CRTA's outputs. This reference guides the alignment of heterogeneous modality representations into a unified space, where class-aware contrastive refinement is finally applied to explore discriminative diagnostic cues. Extensive evaluations on skin lesion (Derm7pt), ocular disease (ODIR), and meningioma (MEN) datasets demonstrate that CMML significantly outperforms state-of-the-art (SOTA) methods, yielding AVG AUC improvements of 1.26%, 0.97%, and 1.32%, respectively.

Abstract

Cross-Modal Learning

Explore similar work

Context-driven Missing-Modality Learning for Robust Medical Diagnosis with Image-Tabular Data

UniMod: Enhancing Multi-Modal Medical Diagnosis through Cross-Modality and Within-Modality Alignment

Effective and Robust Multimodal Medical Image Analysis