DeepLearning (DL)-based Computer-Aided Diagnostic (CAD) systems have shown great promise in supporting clinicians, but their performance heavily depends on the availability of large, well-annotated datasets. In medical lesion detection, the scarcity of comprehensive, stan dardized annotations and the heterogeneity of existing datasets limit the development of robust foundation models. In this work, we introduce a novel approach to building a multi-domain foundation model for le sion detection by integrating diverse medical image datasets annotated for both anatomical structures and abnormalities. We first propose a student-teacher training strategy to effectively combine datasets with heterogeneous label spaces, mitigating catastrophic forgetting, and im proving feature learning. In addition, building on our previously pro posed example-based EM-DETR framework, we adapt it to jointly learn from multiple domain-specific datasets, enabling fine-grained, example driven detection across modalities. Our model achieves state-of-the-art (SOTA) results, with image-level AUC scores exceeding previous SOTA by 1% point on four key findings in CXRs. We achieve a leading mAP50 in CXRs and mammography, surpassing EM-DETR trained on a single dataset by 2-3% points. This approach paves the way for scalable foundation models in medical imaging that leverage heterogeneous data while maintaining robust and generalizable performance.
Figures & tables
Finding
Positive Training Samples
Total Test images
Positive Test Samples
Consolidation
7266 (4.01%)
333
141 (42.34%)
Atelectasis
128 (0.07%)
333
83 (24.93%)
Pleural Effusion
5181 (2.86%)
331
94 (28.34%)
Pneumothorax
4212 (2.32%)
412
170 (51.21%)
Table 1: Distribution of positive findings in training and test sets of dataset B . The training dataset consists of a total 16,953 images with at-least one abnormality. Images may have multiple findings, hence % represent the proportion of total findings.
Figure 1: Overview of the student-teacher training process to create a foundation model with GDINO [ 14 ] . A GDINO model is trained on each dataset with all available classes and used to generate weak annotations on remaining sets. A final model is trained in phase II with both gold-standard and generated annotations. The model outputs both predicted bounding boxes and classification scores for each box.
Figure 2: Overview of EM-DETR [ 1 ] detection. An exemplar feature embedding is generated for each class and interleaved with the text embedding. The detection encoder/decoder perform cross-attention between the image and interleaved embeddings, thereby training the decoder to search for classes based on their correlated text phrases as well as class-specific features.
Figure 3: Overview of training the EM-DETR based foundation model, called EM-FM. In the first stage, all available radiology datasets are used to pretrain the detection encoder/decoder pair. The decoder is trained to detect medical features that correlate to specific text embeddings. In the second stage, the weights are refined for class-specific detection using a small subset of foreground vs. background annotations. The background annotations are generated based on data availability [ 1 ] .
Finding
RPS ε
Baseline †
AAFM †
EM-FM †
Consolidation
0.932
0.926
0.943 *
0.921
Atelectasis
0.949
0.959
0.974 *
0.932
Pleural effusion
0.993
0.994
0.993
0.993
Pneumothorax
0.975
0.978
0.978
0.968
Average
0.962
0.959
0.972 *
0.954
Table 2: Comparison of the image-level AUC classification scores, where ∗ shows statistical significance of p<0.05 when AAFM and EM-FM are compared to the Reference Product Solution (RPS) [ 6 ] . ε indicates a ResNet-101 backbone, while † indicates a Swin-L architecture. All results have a standard deviation ϵ<±0.005 .
Findings in CXRs
Baseline †
AAFM †
EM-DETR †
EM-FM †
Consolidation
0.450
0.386*
0.539*
0.596 *
Atelectasis
0.363
0. 366
0.346
0.363
Pleural effusion
0.244
0.240
0.715*
0.756 *
Pneumothorax
0.419
0.444*
0.508*
0.556 *
Average
0.369
0.359
0.527
0.567 *
Findings in Mammography
Table 3: Comparison of the average detection mAP50 results between the baseline, AAFM, EM-DETR and EM-FM. † indicates a Swin-L image backbone and ∗ shows statistical significance with p<0.05 . All results have a standard deviation ϵ<±0.1 .
Finding
RPS ε
Baseline †
AAFM †
EM-FM †
Consolidation
0.900
0.840
0.740
0.891
Atelectasis
0.950
0.970
0.940
0.896
Pleural Effusion
0.997
0.987
0.987
0.987
Pneumothorax
0.970
0.945
0.973
0.935
Average
0.954
0.936
0.910
0.927
Table 4: Comparison of average sensitivity at 0.5 FP between RPS [ 6 ] , baseline, AAFM and EM-FM. ε indicates a ResNet-101 backbone, while † indicates a Swin-L architecture. All results have a standard deviation ϵ<±0.01 .
Figure 4: Qualitative examples of bounding box predictions for pleural effusion (first four) and pneumothorax (next four) in CXR images, followed by mass predictions in mammograms. Groundtruth in yellow and predictions in green with their prediction scores are also shown.
Figure 5: Comparison of per-epoch pneumothorax mAP50 values for the baseline and EM-FM methods. The EM-FM model demonstrates faster and more stable convergence compared to the baseline and EM-DETR.
Fraunhofer Institute for Digital Medicine MEVIS · Institute for Experimental Molecular Imaging, RWTH Aachen University, Aachen, Germany · Institute of Diagnostic and Interventional Radiology, Medizinische Hochschule Hannover, Hannover, Germany +11
Digital Technology and Innovation, Siemens Healthineers, Princeton, NJ USA · Department of Computed Tomography, Siemens Healthineers, Forchheim, Germany · Department of Radiology, University of Maryland Medical Center, Baltimore, MD USA +4