Medical object detection suffers when a single detector is trained on mixed medical modalities (e.g., CXR, CT, MRI) due to heterogeneous statistics and disjoint representation spaces. To address this challenge, we turn to representation alignment, an approach that has proven effective for bringing features from different sources into a shared space. Specifically, we target the representations of DETR-style object queries and propose a simple, detector-agnostic framework to align them with modality context. First, we define modality tokens: compact, text-derived embeddings encoding imaging modality that are lightweight and require no extra annotations. We integrate the modality tokens into the detection process via Multimodality Context Attention (MoCA), mixing object-query representations via self-attention to propagate modality context within the query set. This preserves DETR-style architectures and adds negligible latency while injecting modality cues into object queries. We further introduce QueryREPA, a short pretraining stage that aligns query representations to their modality tokens using a task-specific contrastive objective with modality-balanced batches. Together, MoCA and QueryREPA produce modality-aware, class-faithful queries that transfer effectively to downstream training. Across diverse modalities trained altogether, the proposed approach consistently improves AP with minimal overhead and no architectural modifications, offering a practical path toward robust multimodality medical object detection.
Figures & tables
Figure 1: (a) Contrastive Representation Alignment of Object Queries (QueryREPA): During a pretraining stage, each image in a modality-balanced batch is associated with a text-derived modality token . For each image, we mean-pool the decoder’s object-query embeddings into a single query representation, project it into the modality-token space, and optimize a contrastive alignment loss between the projected queries and the modality tokens, yielding a modality-aware query manifold. (b) Multimodality Context Attention (MoCA): For a given image, a modality token is selected and concatenated with the set of object queries to form an augmented query set. Information fusion occurs in the self-attention layer of the decoder, where each query attends to the modality token to attain modality-specific context.
Figure 2: Comparison of text/modality integration in DETR-style detectors. (a) DINO (baseline): image-only encoder–decoder with query selection. (b) Grounding DINO: language-guided query selection and cross-attention in the decoder (object queries attend to a sequence of text tokens). (c) Ours (DINO+MoCA): append compact modality tokens to the object queries and fuse by self-attention in the decoder; the tokens act as semantic anchors that refine queries in a modality-aware way with minimal overhead.
Method
AP
AP 50
AP 75
AP s
AP m
AP l
Deformable DETR [ 46 ]
35.0
54.4
37.8
17.4
27.6
38.2
Sparse R-CNN [ 33 ]
35.9
54.4
38.8
18.4
29.0
38.0
DiffusionDet [ 5 ]
38.2
60.3
40.1
22.4
30.6
41.6
GLIP [ 22 ]
37.4
57.5
39.9
20.8
30.5
42.2
GLIP-A [ 22 ]
37.7
58.3
40.1
20.9
31.0
41.4
GLIP-B [ 22 ]
37.1
57.4
39.4
22.4
29.7
40.8
Table 1: Quantitative comparison with state-of-the-art methods in medical object detection. Applying our method (MoCA + QueryREPA) on the DINO [ 42 ] detector with E as CLIP, in both the modality-only ( S=D ) and class-conditioned ( S=D×C ) configurations, outperforms all other detection methods. Bold = best, underline = second best.
Token
Partition
AP
AP 50
AP 75
AP s
AP m
AP l
Baseline (no token)
—
37.7
58.6
37.8
19.9
29.8
41.7
Capacity controls
Learnable
global
37.6
58.1
40.6
21.4
29.6
41.7
Random
modality
37.9
58.1
40.8
19.9
30.3
41.8
Correctness control
Shuffled
modality
38.3
59.0
40.9
20.7
30.3
42.3
Table 2: Results for alternative token designs. Bold = best, underline = second best.
Figure 3: QueryREPA incorporates modality-aware semantics into the query space. UMAP embeddings of the mean query representation qˉ(l) , computed per image and colored by imaging modality. Before QueryREPA (left) the representations are highly entangled across modalities, whereas after QueryREPA (right) they separate into disentangled clusters.
Figure 4: Qualitative Comparison. Comparison results between various state-of-the-art detection methods and the proposed method are shown above. Our method effectively leverages modality context to significantly enhance anomaly localization (highlighted in red), compared to baseline results (highlighted in blue). Ground truth bounding boxes are highlighted in green. For cases where the bounding boxes are small, insets show an enlarged view of the highlighted yellow region.
Deployment
Total
CXR
MRI
Colonoscopy
Pathology
CT
Detector
Method
Models
Params (M)
AP
AP 50
AP
AP 50
AP
AP 50
AP
AP 50
AP
AP 50
AP
AP 50
Def. DETR
Baseline (Specialist)
5
200.50
36.2
55.4
14.9
30.9
77.4
96.7
69.0
84.4
47.3
74.1
31.0
63.6
Baseline (Unified)
1
40.11
35.0
54.4
15.1
30.9
75.8
96.2
61.3
77.3
46.5
74.1
30.5
62.3
+ Ours
1
40.89
37.2
60.2
18.2
39.9
76.1
96.7
66.7
84.6
45.4
76.0
31.3
61.6
DINO
Baseline (Specialist)
5
237.70
39.1
59.1
17.2
34.6
80.4
97.2
69.9
84.3
52.9
81.9
36.5
70.6
Baseline (Unified)
1
47.55
37.7
58.6
17.8
36.1
78.0
96.6
65.1
81.3
50.5
77.8
33.0
66.6
Table 3: Comparison of modality-specific expert detectors and unified multimodality detectors. Specialist totals assume each image is sent to the expert for its own modality with no routing errors, and predictions are pooled over the full 27-category taxonomy. Bold = best per column within each detector.
Table 8
QueryREPA
MoCA
AP
AP 50
AP 75
AP s
AP m
AP l
37.7
58.6
37.8
19.9
29.8
41.7
✓
38.1
58.4
40.9
21.0
29.8
42.3
✓
40.8
65.1
43.4
25.1
33.5
45.2
✓
✓
41.1
65.5
43.4
25.4
34.6
44.9
Table 6: Ablation of components using DINO [ 42 ] as the base model and CLIP [ 31 ] as the encoder E . Checkmarks denote modules enabled. Bold = best per column.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Table 10
Layer l
AP
AP 50
AP 75
AP s
AP m
AP l
Layer 2
41.0
65.8
43.5
24.0
33.3
45.0
Layer 3
41.0
65.6
43.3
27.0
32.8
45.1
Layer 4
41.0
65.8
43.1
25.3
33.3
45.3
Layer 5
41.1
65.5
43.4
25.4
34.6
44.9
Appendix
Table 8: Ablation of decoder layer l used for alignment in QueryREPA. Our method is robust, producing optimal results regardless of decoder layer l .
Method
AP
AP 50
AP 75
AP s
AP m
AP l
DiffusionDet
38.2
60.3
40.1
22.4
30.6
41.6
DiffusionDet + Ours ( E : CLIP)
40.2
65.4
41.6
26.0
34.5
42.6
Appendix
Table 9: Quantitative comparison of applying our method (MoCA + QueryREPA) on the DiffusionDet [ 5 ] model. Bold = best per column.
QueryREPA
MoCA
AP
AP 50
3.6
10.0
✓
4.1
11.3
✓
7.4
20.7
✓
✓
8.0
22.2
Appendix
Table 10: Zero-shot transfer on ChestX-Det. Models trained on our mixed corpus are evaluated without any fine-tuning. Bold = best per column.
Figure 6: UMAP embedding of modality tokens for varying E .
Figure 7: Attention heatmaps at decoder layer 5 (each query weighted by predicted-class confidence). (a) Input Image with ground-truth boxes; (b) Baseline; (c) Ours with the correct modality token, where attention concentrates on the target structures; (d) Swapped token, an unrelated token ( “COVID-19 in lung CT” ), where the attention scatters. This indicates that the modality token causally shifts the attention pattern.
Total
CXR
MRI
Colonoscopy
Pathology
CT
Method
E
LQRA
AP
AP 50
AP
AP 50
AP
AP 50
AP
AP 50
AP
AP 50
AP
AP 50
Deformable DETR
✗
✗
35.0
54.4
15.1
30.9
75.8
95.9
61.3
77.3
46.5
74.1
30.5
62.3
CLIP
✗
37.3
60.2
18.0
39.1
75.3
96.2
64.6
83.7
49.3
79.1
31.5
62.5
✓
37.2
60.2
18.2
39.9
76.1
96.7
66.7
84.6
45.4
76.0
31.3
61.6
BiomedCLIP
✗
36.7
60.2
17.5
39.1
74.5
96.7
65.1
85.4
47.7
77.6
29.9
61.8
✓
36.7
59.6
17.7
38.9
75.7
96.5
64.2
83.4
47.1
76.8
30.1
61.0
Appendix
Table 11: Quantitative comparison leveraging various modality token encoders E as CLIP [ 31 ] , BiomedCLIP [ 43 ] , and PubMedCLIP [ 10 ] under the same training setup. Our method shows improved results regardless of E used, proving its robustness and wide applicability. Bold indicates best results per column.
Stage
Time
Peak Memory
Baseline training
∼ 25h
14.8 GB
QueryREPA training
∼ 29h
8.1 GB
Detection training (w/ MoCA)
∼ 27h
14.8 GB
Appendix
Table 12: Training cost on a single RTX 4090.
Figure 8: Qualitative comparisons on chest X-ray images. Blue boxes indicate predictions from baseline detectors (Sparse R-CNN, GLIP, DiffusionDet, Grounding DINO, DINO), Green boxes denote ground-truth annotations, and Red boxes show predictions from our method.
Figure 9: Qualitative comparisons on MRI images. Blue boxes indicate predictions from baseline detectors (Sparse R-CNN, GLIP, DiffusionDet, Grounding DINO, DINO), Green boxes denote ground-truth annotations, and Red boxes show predictions from our method.
Figure 10: Qualitative comparisons on Colonoscopy images. Blue boxes indicate predictions from baseline detectors (Sparse R-CNN, GLIP, DiffusionDet, Grounding DINO, DINO), Green boxes denote ground-truth annotations, and Red boxes show predictions from our method.
Figure 11: Qualitative comparisons on Pathology images. Blue boxes indicate predictions from baseline detectors (Sparse R-CNN, GLIP, DiffusionDet, Grounding DINO, DINO), Green boxes denote ground-truth annotations, and Red boxes show predictions from our method.
Figure 12: Qualitative comparisons on CT images. Blue boxes indicate predictions from baseline detectors (Sparse R-CNN, GLIP, DiffusionDet, Grounding DINO, DINO), Green boxes denote ground-truth annotations, and Red boxes show predictions from our method.
In multimodal learning, CLIP has emerged as the de-facto approach for mapping different modalities into a shared latent space by bringing semantically similar representations closer while pushing apart dissimilar ones. However, CLIP-based contrastive losses exhibit unintended behaviors that negatively impact true semantic alignment, leading to sparse and fragmented latent spaces. This phenomenon, known as the modality gap, has been partially mitigated for standard text and image pairs but remains unknown and unresolved in more complex multimodal settings, such as the medical domain. In this work, we study this phenomenon in the latter case, revealing that the modality gap is present also in medical alignment, and we propose a modality-agnostic framework that closes this gap, ensuring that semantically related representations are more aligned, regardless of their source modality. Our method enhances alignment between radiology images and clinical text, improving cross-modal retrieval and image captioning.
Multi-modal medical imaging enables comprehensive diagnostics, yet current foundation models process 2D (e.g. X-ray) and 3D (e.g. CT) data with separate, dimensionality-specific architectures. We present MultiMedVision, a unified framework for joint 2D/3D representation learning built on a Sparse Vision Transformer. Our model uses 3D Rotary Positional Embeddings and variable-length sequence packing to process mixed-modality batches natively within a shared latent space, without modality-specific adapters or treating 3D volumes as 2D slice sequences. Trained with a self-supervised objective on chest X-rays (MIMIC-CXR) and CT scans (CT-RATE), and using a single shared encoder with 5x less data, MultiMedVision achieves competitive performance on both 2D benchmarks (Macro AUROC 0.82 on MIMIC, 0.84 on CheXpert) and 3D tasks (0.85 on CT-RATE). Analysis of the learned representations reveals coexisting modality-specific and shared feature subspaces, demonstrating that unified cross-dimensional representation learning is feasible without sacrificing modality-specific performance.
Frank Li, Bardia Khosravi, Mohammadreza Chavoshi +5
Vision-language models (VLMs) show strong potential for complex diagnostic tasks in medical imaging. However, applying VLMs to multi-organ medical imaging introduces two principal challenges: (1) modality-specific vision-language alignment and (2) cross-modal feature fusion. In this work, we propose MedMAP, a Medical Modality-Aware Pretraining framework that enhances vision-language representation learning in 3D MRI. MedMAP comprises a modality-aware vision-language alignment stage and a fine-tuning stage for multi-organ abnormality detection. During the pre-training stage, the modality-aware encoders implicitly capture the joint modality distribution and improve alignment between visual and textual representations. We then fine-tune the pre-trained vision encoders (while keeping the text encoder frozen) for downstream tasks. To this end, we curated MedMoM-MRI3D, comprising 7,392 3D MRI volume-report pairs spanning twelve MRI modalities and nine abnormalities tailored for various 3D medical analysis tasks. Extensive experiments on MedMoM-MRI3D demonstrate that MedMAP significantly outperforms existing VLMs in 3D MRI-based multi-organ abnormality detection. Our code is available at https://github.com/RomantiDr/MedMAP.
Haowen Zhu, Ning Yin, Xiaogen Zhou
School of Computer Science and Engineering, Southeast University, China · Department of Medical Imaging, Suzhou Traditional Chinese Medicine Hospital, China · School of Electronic, Electrical Engineering and Physics, Fujian University of Technology