Multimodal classification of Radiation-Induced Contrast Enhancements and tumor recurrence using deep learning
Organizations: Deutsches Krebsforschungszentrum (DKFZ), Division of Medical Image Computing, Heidelberg, Germany · Medical Faculty, Heidelberg University, Heidelberg, Germany · Pattern Analysis and Learning Group, Department of Radiation Oncology, Heidelberg University Hospital, Heidelberg, Germany · Research Campus M2OLIE, Mannheim, Germany ·
Abstract
The differentiation between tumor recurrence and radiation-induced contrast enhancements in post-treatment glioblastoma patients remains a major clinical challenge. Existing approaches rely on clinically sparsely available diffusion MRI or do not consider radiation maps, which are gaining increasing interest in the tumor board for this differentiation. We introduce RICE-NET, a multimodal 3D deep learning model that integrates longitudinal MRI data with radiotherapy dose distributions for automated lesion classification using conventional T1-weighted MRI data. Using a cohort of 92 patients, the model achieved an F1 score of 0.92 on an independent test set. During extensive ablation experiments, we quantified the contribution of each timepoint and modality and showed that reliable classification largely depends on the radiation map. Occlusion-based interpretability analyses further confirmed the model's focus on clinically relevant regions. These findings highlight the potential of multimodal deep learning to enhance diagnostic accuracy and support clinical decision-making in neuro-oncology.
Figures & tables
Multimodal Classification
Multimodal classification combines images, text, audio, signals, metadata, or other inputs to assign labels more accurately and reliably.
See all 199 papers