Integrating Unimodal and Vision-Language Representations in Latent Space for Multi-Label Chest X-Ray Classification

Quang-Huy Tran; Duc-Tuan Ngo; Minh-Khoi Nguyen-Bui; Dang-Khoa Bui; Thanh-Trong Tran; Tuan-Khoi Nguyen; Hoang-Anh Ngo

cs.CVAug 31, 2026

Integrating Unimodal and Vision-Language Representations in Latent Space for Multi-Label Chest X-Ray Classification

Authors: Quang-Huy Tran, Duc-Tuan Ngo, Minh-Khoi Nguyen-Bui, Dang-Khoa Bui, Thanh-Trong Tran, Tuan-Khoi Nguyen, Hoang-Anh Ngo

Organizations: Faculty of Computer Science and Engineering, Ho Chi Minh City University of Technology, 268 Ly Thuong Kiet Street, Dien Hong Ward, Ho Chi Minh City, Vietnam · Vietnam National University Ho Chi Minh City, Dong Hoa Ward, Ho Chi Minh City, Vietnam · Applied AI & Data Science Division, AK Technologies Company Limited, Ho Chi Minh City, Vietnam

Abstract

Multi-label chest X-ray classification has attracted considerable attention in recent years, with the effective use of visual representations and clinical semantic knowledge playing an important role. This study proposes a framework that combines unimodal representations from RAD-DINO with vision--language representations from BioViL-T for the classification of 14 labels in the MIMIC-CXR-JPG dataset. The RAD-DINO and BioViL-T embeddings and their combined representation are refined separately in latent space before being normalized and fused across the three branches. In addition to improving classification performance, the study aims to clarify the role of each embedding source and the degree to which they complement one another. Experiments show that RAD-DINO outperforms BioViL-T when used independently, whereas early fusion further improves the results, indicating that the two embedding sources contain complementary information. The best-performing model achieves a mean AUROC of 0.840 and an mAP of 0.467. Ablation analysis shows that hybrid fusion provides consistent and statistically significant improvements over early fusion when each embedding source is refined in latent space, suggesting that fusion effectiveness depends on the quality of the representation supplied by each branch. However, the study has only been evaluated internally on MIMIC-CXR-JPG; its generalizability to data from other healthcare institutions therefore remains to be validated. The source code is available at: https://anonymous.4open.science/r/mimic-report-C210/.

Figures & tables

Figure 1: Overview of the latent hybrid fusion architecture.

Configuration	AUROC (95% CI)	$\boldsymbol{\Delta}$ AUROC	mAP (95% CI)
DenseNet121 [ 20 ]	0.773	–	–
RAD-DINO only	0.827 [0.825, 0.828]	0.053	0.443 [0.439, 0.448]
BioViL-T only	0.805 [0.804, 0.807]	0.032	0.408 [0.404, 0.411]
Early fusion	0.837 [0.835, 0.838]	0.064	0.464 [0.459, 0.469]
Hybrid fusion	0.840 [0.839, 0.842]	0.067	0.467 [0.462, 0.471]

Table 1: Comparison with the DenseNet121 results reported by Singh [ 20 ] .

Model	Reference model	$\boldsymbol{\Delta}$ AUROC	$\boldsymbol{p}$	Sig.	$\boldsymbol{\Delta}$ mAP	$\boldsymbol{p}$	Sig.
Early fusion	BioViL-T only	0.031	$3.61\times 10^{-9}$	$\checkmark$	0.056	$7.51\times 10^{-8}$	$\checkmark$
Early fusion	RAD-DINO only	0.010	$1.01\times 10^{-6}$	$\checkmark$	0.020	$2.54\times 10^{-6}$	$\checkmark$
Hybrid fusion	Early fusion	0.003	$6.30\times 10^{-5}$	$\checkmark$	0.003	$3.28\times 10^{-4}$	$\checkmark$

Table 2: Statistical tests across experiments.

Comparison on 13 labels (excluding No Finding)
Model	AUROC $\uparrow$	mAP $\uparrow$	Macro F1 $\uparrow$
MCX-Net 13 [ 21 ]	$0.816$	–	$0.451$
Ours 13	0.837	–	0.464
Comparison on 14 labels
DenseNet121 [ 20 ]	$0.773$	–	–
MBRANet [ 12 ]	$0.805$	–	–

Table 3: Comparison of the proposed method with other models on MIMIC-CXR.

Figure 2: ROC curves of latent hybrid fusion for the 14 labels on the test set. Mean AUROC ranges from $0.742$ for Enlarged Cardiomediastinum to $0.928$ for Support Devices .

ID	Configuration	AUROC (95% CI)	Comparison	$\boldsymbol{\Delta}$ AUROC	$\boldsymbol{p}$	Sig.
Latent-refinement block
A1	Embedding only	0.8356 [0.8340, 0.8372]
A2	A1 + Autoencoder	0.8366 [0.8346, 0.8386]	A2 $-$ A1	$+0.0010$	$2.74\times 10^{-2}$	$\times$
A3	A2 + noise	0.8369 [0.8354, 0.8385]	A3 $-$ A1	$+0.0013$	$1.64\times 10^{-4}$	✓
Prediction head (Latent-refinement gain under different heads)
A4	MLP head	0.8368 [0.8352, 0.8384]	A4 $-$ A1	$+0.0012$	$1.68\times 10^{-2}$	$\times$

Table 4: Ablation over the latent-refinement block, the prediction head, and the fusion stage. Each value is the mean macro AUROC and 95% confidence interval across five seeds;

p

is the two-sided paired

t

-test p-value across seeds.

Tier	Labels	$\boldsymbol{\Delta}$ AUROC	Excl. 0	$\boldsymbol{\Delta}$ AP	Excl. 0
Common ( $>15\%$ )	8	$+0.0022$	8/8	$+0.0038$	8/8
Mid ( $5$ – $15\%$ )	3	$+0.0036$	3/3	$+0.0024$	2/3
Rare ( $<5\%$ )	3	$+0.0067$	3/3	$+0.0012$	0/3

Table 5: Improvement of hybrid fusion over early fusion by label-prevalence tier (exploratory).

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Label	Positive studies	Prevalence
Atelectasis	$52{,}818$	$25.017\%$
Cardiomegaly	$47{,}579$	$22.535\%$
Consolidation	$14{,}190$	$6.721\%$
Edema	$37{,}934$	$17.967\%$
Enlarged Cardiomediastinum	$15{,}497$	$7.340\%$
Fracture	$4{,}590$	$2.174\%$

Appendix

Table 1: Label distribution in the aligned cohort.

Item	Value
Aligned studies	$211{,}130$
Disease labels	$14$
RAD-DINO embedding dimension	$768$
BioViL-T embedding dimension	$128$
Random seeds	$42,43,44,45,46$
Number of independent runs	$5$

Appendix

Table 2: Cohort and reproducibility settings.

Split	Ratio	Studies
Train	$70\%$	$147{,}791$
Validation	$10\%$	$21{,}113$
Test	$20\%$	$42{,}226$
Total	$100\%$	$211{,}130$

Appendix

Table 3: Train, validation, and test set sizes shared across all five random seeds.

Parameter	Setting
Model dimensions
Predictor hidden dimension	$512$
Latent dimension	$256$
Feed-forward dimension	$1024$
Training schedule
Batch size	$1024$

Appendix

Table 4: Training and model-selection hyperparameters.

Configuration	Params	FLOPs	Latency
	(M)	(M/study)	(ms/study)
RAD-DINO only	$7.1$	$13.1$	$0.057$
BioViL-T only	$6.1$	$11.8$	$0.057$
Early fusion	$7.3$	$13.4$	$0.057$
Hybrid fusion	$20.5$	$38.3$	$0.170$

Appendix

Table 5: Inference and training cost of the fusion stage, measured on a single NVIDIA A100 GPU.

Figure 1: Per-label performance of latent hybrid fusion. Each cell shows the mean $\pm$ standard deviation across five seeds.

Label	MBRANet [ 12 ]	Ours	$\boldsymbol{\Delta}$
Atelectasis	$0.779$	0.828	$+0.049$
Cardiomegaly	0.825	$0.820$	$-0.005$
Consolidation	$0.788$	0.797	$+0.009$
Edema	$0.833$	0.893	$+0.060$
Cardio.	0.823	$0.742$	$-0.081$
Fracture	$0.703$	0.773	$+0.070$

Appendix

Table 6: Per-label AUROC comparison between MBRANet and the proposed method on MIMIC-CXR.

Comparison	Metric	$\boldsymbol{\Delta}$	CI	Rel. gain (%)	Headroom (%)
Early $-$ BioViL-T	AUROC	0.0314	[0.0301, 0.0327]	3.90	16.15
	mAP	0.0560	[0.0544, 0.0579]	13.75	–
	Macro F1 e	0.0451	[0.0437, 0.0464]	10.33	–
Early $-$ RAD-DINO	AUROC	0.0104	[0.0097, 0.0111]	1.25	5.98
	mAP	0.0203	[0.0193, 0.0215]	4.59	–
	Macro F1 e	0.0122	[0.0111, 0.0133]	2.59	–

Appendix

Table 7: Patient-level clustered bootstrap of paired metric differences (

B=2{,}000

subject-level resamples; within each replicate the paired difference is computed per seed on an independently drawn subject resample and averaged across the five seeds). Confirmatory comparisons report simultaneous

99.17\%

confidence intervals; Macro F1 is exploratory and reports

95\%

intervals.

Comparison	Uncorrected		Bonferroni		Median $\boldsymbol{p}$
	$n$	%	$n$	%
Early $-$ BioViL-T	$70$	$100.0$	$69$	$98.6$	$1.7\text{\times}{10}^{-24}$
Early $-$ RAD-DINO	$61$	$87.1$	$52$	$74.3$	$2.7\text{\times}{10}^{-11}$
Hybrid $-$ Early	$59$	$84.3$	$39$	$55.7$	$1.2\text{\times}{10}^{-3}$

Appendix

Table 8: Per-label DeLong tests for the three confirmatory comparisons, computed on the one-study-per-patient subset of the test set.

Label	RAD-DINO latent	BioViL-T latent	Early fusion
Atelectasis	$0.182\pm 0.038$	$0.168\pm 0.038$	$0.650\pm 0.049$
Cardiomegaly	$0.196\pm 0.059$	$0.252\pm 0.038$	$0.552\pm 0.063$
Consolidation	$0.168\pm 0.080$	$0.280\pm 0.049$	$0.552\pm 0.063$
Edema	$0.224\pm 0.059$	$0.196\pm 0.031$	$0.580\pm 0.049$
Cardio.	$0.252\pm 0.080$	$0.196\pm 0.059$	$0.552\pm 0.038$
Fracture	$0.210\pm 0.111$	$0.112\pm 0.063$	$0.678\pm 0.063$

Appendix

Table 9: Label-wise late-fusion weights for each label. Each value is reported as the mean

\pm

standard deviation across five seeds.

Label	Pearson (score)	Spearman (score)	Pearson (error)
Atelectasis	$0.867$	$0.874$	$0.972$
Cardiomegaly	$0.873$	$0.875$	$0.971$
Consolidation	$0.835$	$0.842$	$0.948$
Edema	$0.893$	$0.891$	$0.948$
Enlarged Cardio.	$0.846$	$0.837$	$0.965$
Fracture	0.466	0.456	0.817

Appendix

Table 10: Between-branch correlation of predicted scores and errors for the RAD-DINO and BioViL-T branches.

Label	Both $\checkmark$	Only R	Only B	Both $\times$	Disagr.	Acc. R	Acc. B	Oracle
Atelectasis	$0.658$	$0.073$	$0.065$	$0.204$	$0.138$	$0.731$	$0.723$	$0.796$
Cardiomegaly	$0.655$	$0.076$	$0.077$	$0.192$	$0.153$	$0.731$	$0.732$	$0.808$
Consolidation	$0.785$	$0.069$	$0.052$	$0.095$	$0.120$	$0.853$	$0.836$	$0.905$
Edema	$0.788$	$0.051$	$0.043$	$0.118$	$0.094$	$0.839$	$0.831$	$0.882$
Enlarged Cardio.	$0.664$	$0.101$	$0.087$	$0.148$	$0.188$	$0.765$	$0.751$	$0.852$
Fracture	$0.824$	$0.109$	$0.034$	$0.033$	$0.143$	$0.933$	$0.858$	$0.967$

Appendix

Table 11: Decision-level agreement between the RAD-DINO and BioViL-T branches.

Configuration	Metric	Mean [95% CI]	$\boldsymbol{\Delta}$ vs. full [95% CI]
Full (three branches)	mAUROC	0.8404 [0.8388, 0.8419]	–
	mAP	0.4666 [0.4619, 0.4713]	–
w/o RAD-DINO latent	mAUROC	0.8367 [0.8352, 0.8381]	$-0.0037$ [ $-0.0039$ , $-0.0035$ ]
	mAP	0.4604 [0.4558, 0.4650]	$-0.0062$ [ $-0.0065$ , $-0.0058$ ]
w/o BioViL-T latent	mAUROC	0.8382 [0.8365, 0.8399]	$-0.0022$ [ $-0.0025$ , $-0.0018$ ]
	mAP	0.4643 [0.4595, 0.4692]	$-0.0023$ [ $-0.0029$ , $-0.0016$ ]

Appendix

Table 12: Leave-one-branch-out analysis of the main configuration.

Configuration	ECE	Brier score
RAD-DINO only	$0.0125\pm 0.0006$	$0.0858\pm 0.0002$
BioViL-T only	$0.0125\pm 0.0005$	$0.0887\pm 0.0002$
Early fusion	$0.0139\pm 0.0000$	$0.0836\pm 0.0002$
Hybrid fusion	$0.0140\pm 0.0002$	$0.0832\pm 0.0002$

Appendix

Table 13: Calibration after per-label Platt scaling fitted on the validation set (mean

\pm

standard deviation across five seeds).

Aggregation scheme	Weights (F/L)	mAUROC [95% CI]	mAP [95% CI]
Frontal only	–	0.8394 [0.8381, 0.8406]	0.4661 [0.4615, 0.4707]
Frontal + lateral	0.5/0.5	0.8398 [0.8381, 0.8414]	0.4652 [0.4605, 0.4700]
	0.6/0.4	0.8404 [0.8388, 0.8419]	0.4666 [0.4619, 0.4713]
	0.7/0.3	0.8401 [0.8383, 0.8410]	0.4663 [0.4609, 0.4707]

Appendix

Table 14: Sensitivity of the main configuration to the frontal–lateral aggregation scheme.

$\boldsymbol{\sigma_{\max}}$	mAUROC	mAP
0.00	$0.8368\pm 0.0011$	$0.4636\pm 0.0039$
0.20	$0.8368\pm 0.0011$	$0.4638\pm 0.0039$
0.65	$0.8369\pm 0.0013$	$0.4638\pm 0.0041$
1.00	$0.8369\pm 0.0013$	$0.4638\pm 0.0040$
1.25	$0.8369\pm 0.0013$	$0.4638\pm 0.0039$
1.50	$0.8370\pm 0.0012$	$0.4638\pm 0.0039$

Appendix

Table 15: Sensitivity of the early-fusion branch to the noise ceiling

\sigma_{\max}

Latent transformation, $\widehat{z}=g(z,0)$
Quantity	Mean $\pm$ SD
$\cos(z,\widehat{z})$	0.8510 $\pm$ 0.0093
$\lVert\widehat{z}-z\rVert_{2}$	5.4684 $\pm$ 0.1642
Relative $L_{2}$ deviation	0.5465 $\pm$ 0.0151
$\lVert z\rVert_{2}$	10.0288 $\pm$ 0.1116
Autoencoder reconstruction

Appendix

Table 16: Geometry of the latent transformation at

s=0

and reconstruction quality of the autoencoder, on the early-fusion branch. Values are mean

\pm

standard deviation across five seeds.

Configuration	Mean	$\boldsymbol{\Delta}$	$\boldsymbol{p}$
Macro AUROC
Applied, $\widehat{z}=g(z,0)$	0.8369	–	–
Bypassed, $\widehat{z}=z$	0.8356	$-0.0013$	$6.10\times 10^{-4}$
Objective removed	0.8369	$-0.0000$	$8.34\times 10^{-1}$
mAP
Applied, $\widehat{z}=g(z,0)$	0.4638	–	–

Appendix

Table 17: Inference-time role of the latent denoiser on the early-fusion branch

Representation	Fusion rule	macro AUROC	mAP
Refined	Uniform ( $1/3$ )	$0.8404\pm 0.0013$	$0.4666\pm 0.0038$
Refined	Learned label-wise	$0.8398\pm 0.0011$	$0.4674\pm 0.0039$
Raw	Uniform ( $1/3$ )	$0.8362\pm 0.0012$	$0.4570\pm 0.0036$
Raw	Learned label-wise	$0.8364\pm 0.0010$	$0.4621\pm 0.0033$
Effect		$\Delta$ AUROC ( $p$ )	$\Delta$ mAP ( $p$ )
Representation $\mid$ uniform		$+0.0042$ ( $1.73\times 10^{-5}$ ) ✓	$+0.0096$ ( $1.46\times 10^{-6}$ ) ✓

Appendix

Table 18: Two-factor decomposition of the A8

-

A9 contrast: representation quality crossed with the late-fusion rule. Cell values are mean

\pm

standard deviation across five seeds.

Chest X-Ray Classification

Chest X-ray classification uses machine learning to detect and categorize diseases and findings in radiographs for screening, triage, and clinical decision support.

See all 100 papers

CheXFuse: Guided Differential Joint Attention Fusion for Robust Multimodal Chest X-Ray Classification in Data-Scarce Medical Settings
Oct 19, 2026Muntaqim Ahmed Raju, Priyanka Siddappa, Md Shifat Haider Al Amin +2Multimodal Fusion Chest X-Ray Classification
University of Massachusetts Lowell, USA · NYC Health & Hospitals/Jacobi-NCB, USA
Med-AR: Autoregressive Vision-Language Pretraining for Long-Tailed Chest X-Ray Classification and Uncertainty-Aware Evaluation
Sep 24, 2026Janhavi Prabhu, Sahil, Akshay V +3Long-Tailed Classification Chest X-Ray Classification
Qure.ai, Bangalore, India
Improving Imbalanced Multi-Label Chest X-Ray Diagnosis via CBAM-Enhanced CNN Backbones
May 4, 2026Duy Nguyen Huu, Duy Hoang Khuong, Ngu Huynh Cong VietClass-Imbalanced Learning Chest X-Ray Classification
Department of Artificial Intelligence FPT University Ho Chi Minh City, Vietnam · Department of Computing Fundamental FPT University Ho Chi Minh City, Vietnam

Oct 19, 2026J/K move · Enter open · S save

CheXFuse: Guided Differential Joint Attention Fusion for Robust Multimodal Chest X-Ray Classification in Data-Scarce Medical Settings

Muntaqim Ahmed Raju, Priyanka Siddappa, Md Shifat Haider Al Amin, Ruizhe Ma+1

University of Massachusetts Lowell, USA · NYC Health & Hospitals/Jacobi-NCB, USA

Multimodal Fusion Chest X-Ray Classification Multimodal Robustness

Open paper AskPDF arXiv

A central challenge in multimodal learning is how to fuse modalities that differ substantially in predictive strength. When one modality provides a much stronger representation than another, indiscriminate cross-modal interaction can introduce weak, redundant, or unstable signals, limiting or even degrading performance relative to the strongest unimodal representation. To address this problem, we propose CheXFuse, a multimodal fusion framework based on Guided Differential Joint Attention (GuDJA). CheXFuse combines image and text token representations within a joint multimodal sequence, where learned token guidance emphasizes informative representations and differential attention reduces less useful attention responses. We evaluate CheXFuse on the Indiana University (IU) Chest X-ray dataset, using binary (normal vs. abnormal) classification as the downstream task. Experiments are conducted under both original-report and masked-report settings, with the latter serving as a controlled stress test under reduced textual informativeness. CheXFuse with GuDJA achieves the strongest mean overall performance across the evaluated baselines and ablations, reaching 95.73% accuracy, 95.79% F1-score, 96.54% recall, and 95.28% sensitivity at 95% specificity (Sens@95Spec) with original reports, and 95.58%, 95.60%, 95.60%, and 94.34%, respectively, under masked reports. These results suggest that GuDJA provides a promising fusion strategy under asymmetric or partially reduced modality informativeness.

Abstract

Chest X-Ray Classification

Explore similar work

CheXFuse: Guided Differential Joint Attention Fusion for Robust Multimodal Chest X-Ray Classification in Data-Scarce Medical Settings

Med-AR: Autoregressive Vision-Language Pretraining for Long-Tailed Chest X-Ray Classification and Uncertainty-Aware Evaluation

Improving Imbalanced Multi-Label Chest X-Ray Diagnosis via CBAM-Enhanced CNN Backbones