Revisiting Integration of Image and Metadata for DICOM Series Classification: Cross-Attention and Dictionary Learning

Tuan Truong; Melanie Dohmen; Sara Lorio; Matthias Lenga

eess.IVFeb 27, 2026

Revisiting Integration of Image and Metadata for DICOM Series Classification: Cross-Attention and Dictionary Learning

Authors: Tuan Truong, Melanie Dohmen, Sara Lorio, Matthias Lenga

Organizations: Bayer AG, Berlin, Germany

Abstract

Automated identification of DICOM image series is essential for large-scale medical image analysis, quality control, protocol harmonization, and reliable downstream processing. However, DICOM series classification remains challenging due to heterogeneous slice content, variable series length, and entirely missing, incomplete or inconsistent DICOM metadata. We propose an end-to-end multimodal framework for DICOM series classification that jointly models image content and acquisition metadata while explicitly accounting for all these challenges. (i) Images and metadata are encoded with modality-aware modules and fused using a bi-directional cross-modal attention mechanism. (ii) Metadata is processed by a sparse, missingness-aware encoder based on learnable feature dictionaries and value-conditioned modulation. By design, the approach does not require any form of imputation. (iii) Variability in series length and image data dimensions is handled via a 2.5D visual encoder and attention operating on equidistantly sampled slices. We evaluate the proposed approach on the publicly available Duke Liver MRI dataset and a large multi-institutional in-house cohort, assessing both in-domain performance and out-of-domain generalization. Across all evaluation settings, the proposed method consistently outperforms relevant image only, metadata-only and multimodal 2D/3D baselines. The results demonstrate that explicitly modeling metadata sparsity and cross-modal interactions improves robustness for DICOM series classification.

Figures & tables

Figure 1: Proposed method: pixel data of $S$ DICOM slices is embedded in visual feature pathway. DICOM metadata is embedded by the Sparse Metadata Encoder. Bi-directional cross-modal attention contextualizes all image and metadata embeddings. Final integration to a series-level representation is done by learnable pooling.

Exp.	Modality	Image Enc.	MD Enc.	Imputer	Fusion
(1)	Image	2D DenseNet121	N/A	N/A	N/A
(2)	Image [ 14 ]	3D ResNet18	N/A	N/A	N/A
(3)	Metadata	N/A	XGBoost	No	N/A
(4)	Joint	2.5D DenseNet121	MLP (dense)	Zero	Concat
(5)	Joint	2.5D DenseNet121	MLP (dense)	Learned	Concat
(6)	Joint [ 8 ]	2D DenseNet121	RF	No	No

Table 1: Baselines for in-domain evaluation on the Duke dataset.

Figure 2: In-domain evaluation: five-fold cross-validation per-class F1 scores (%) on the Duke Liver MRI dataset.

Exp.	Precision (%)	Recall (%)	F1 (%)
(1)	85.67 $\pm$ 1.22	85.37 $\pm$ 1.19	85.09 $\pm$ 1.31
(2)	88.77 $\pm$ 1.80	88.39 $\pm$ 1.89	88.33 $\pm$ 1.92
(3)	75.14 $\pm$ 2.39	76.66 $\pm$ 2.20	74.71 $\pm$ 2.34
(4)	93.83 $\pm$ 1.71	93.62 $\pm$ 1.76	93.51 $\pm$ 1.89
(5)	93.62 $\pm$ 3.02	93.37 $\pm$ 3.27	93.21 $\pm$ 3.48
(6)	87.81 $\pm$ 0.83	87.20 $\pm$ 1.16	87.01 $\pm$ 0.97

Table 2: In-domain evaluation on Duke liver MRI sequence classification task. Baselines are (1) 2D Image-only, (2) 3D Image-only [ 14 ] , (3) Metadata-only (XGBoost), (4) Joint: Concat + fixed imputation (zeros), (5) Joint: Concat + learned imputation (MLP), (6) Two-stage [ 8 ] . Our method with

S=10

Task	Label	In-house test (in-domain)	Duke (out-of-domain)
Sequence type	T1	97.77	92.27
	T2	99.64	97.33
	DWI	99.88	96.61
	ADC	99.77	100.00
	DIXON_IN	96.72	96.46
	DIXON_OPP	97.56	74.07

Table 3: Out-of-domain evaluation: Model trained on in-house cohort training split. Per-label F1 (%) on the in-house holdout and the external Duke dataset.

$S=1$	$S=3$	$S=5$	$S=10$	$S=20$
85.09 $\pm$ 1.79	95.46 $\pm$ 0.69	96.17 $\pm$ 1.00	96.66 $\pm$ 1.03	95.88 $\pm$ 0.82

Table 4: Ablation on the number of input slices

S

. Weighted F1 scores (%) on Duke (five-fold cross-validation) for

S\in\{1,3,5,10,20\}

with all other settings fixed.

Explore similar work

PRIMA: Pre-training with Risk-integrated Image-Metadata Alignment for Medical Diagnosis via LLM
Feb 26, 2026Yiqing Wang, Chunming He, Ming-Chen Lu +4Cross-Modal Alignment Multimodal Pretraining
Department of Biomedical Engineering, Duke University, Durham, NC, USA · Department of Ophthalmology and Visual Sciences, University of Michigan, Ann Arbor, MI, USA
DINO-Med: A Unified Patch-Based Adaptation Framework for Multi-Modal Medical Image Analysis Applied to Liver Fibrosis Staging
Sep 10, 2026Boya Wang, Ruizhe Li, Chao Chen +1Medical Imaging Foundation Models Multimodal Classification
School of Computer Science, University of Nottingham, UK · Nottingham Biomedical Research Centre (BRC), School of Medicine, University of Nottingham, UK
MultiMedVision: Multi-Modal Medical Vision Framework
May 9, 2026Frank Li, Bardia Khosravi, Mohammadreza Chavoshi +5Medical Imaging Foundation Models Vision Transformer
Emory University · Yale University

Feb 26, 2026, cs.CVJ/K move · Enter open · S save

PRIMA: Pre-training with Risk-integrated Image-Metadata Alignment for Medical Diagnosis via LLM

Yiqing Wang, Chunming He, Ming-Chen Lu, Mercy Pawar+3

Department of Biomedical Engineering, Duke University, Durham, NC, USA · Department of Ophthalmology and Visual Sciences, University of Michigan, Ann Arbor, MI, USA

Cross-Modal Alignment Multimodal Pretraining Medical Image Classification Medical VLMs

Open paper AskPDF arXiv

Medical diagnosis requires the effective synthesis of visual manifestations and clinical metadata. However, existing methods often treat metadata as isolated tags, failing to exploit the rich semantic knowledge embedded in clinical descriptions. We propose PRIMA (Pre-training with Risk-integrated Image-Metadata Alignment), a framework that integrates domain-specific knowledge into multi-modal representation learning. We first curate an expert corpus of risk--disease correlations via Retrieval-Augmented Generation (RAG) to refine Clinical ModernBERT, embedding diagnostic priors into the text encoder. To bridge the modality gap, we introduce a dual-encoder pre-training strategy utilizing DINOv3 and our refined Clinical ModernBERT, optimized by a suite of four complementary loss functions. These losses are designed to capture multi-granular semantic alignment and handle the ambiguity of clinical correlations through soft labels. Finally, we leverage Qwen3 to fuse these aligned features for precise disease classification. Extensive experiments demonstrate that PRIMA effectively harmonizes pixel-level features with abstract clinical expertise, consistently outperforming other state-of-the-art methods. Notably, our framework achieves strong performance without requiring massive data collection or exhaustive computational resources. Our code will is available at https://github.com/yqwang01/PRIMA.

Abstract

Explore similar work

PRIMA: Pre-training with Risk-integrated Image-Metadata Alignment for Medical Diagnosis via LLM

DINO-Med: A Unified Patch-Based Adaptation Framework for Multi-Modal Medical Image Analysis Applied to Liver Fibrosis Staging

MultiMedVision: Multi-Modal Medical Vision Framework