Align Your Query: Representation Alignment for Multimodality Medical Object Detection

Ara Seo; Bryan Sangwoo Kim; Hyungjin Chung; Jong Chul Ye

cs.CVOct 3, 2025

Align Your Query: Representation Alignment for Multimodality Medical Object Detection

Authors: Ara Seo, Bryan Sangwoo Kim, Hyungjin Chung, Jong Chul Ye

Organizations: KAIST AI · EverEx

Abstract

Medical object detection suffers when a single detector is trained on mixed medical modalities (e.g., CXR, CT, MRI) due to heterogeneous statistics and disjoint representation spaces. To address this challenge, we turn to representation alignment, an approach that has proven effective for bringing features from different sources into a shared space. Specifically, we target the representations of DETR-style object queries and propose a simple, detector-agnostic framework to align them with modality context. First, we define modality tokens: compact, text-derived embeddings encoding imaging modality that are lightweight and require no extra annotations. We integrate the modality tokens into the detection process via Multimodality Context Attention (MoCA), mixing object-query representations via self-attention to propagate modality context within the query set. This preserves DETR-style architectures and adds negligible latency while injecting modality cues into object queries. We further introduce QueryREPA, a short pretraining stage that aligns query representations to their modality tokens using a task-specific contrastive objective with modality-balanced batches. Together, MoCA and QueryREPA produce modality-aware, class-faithful queries that transfer effectively to downstream training. Across diverse modalities trained altogether, the proposed approach consistently improves AP with minimal overhead and no architectural modifications, offering a practical path toward robust multimodality medical object detection.

Figures & tables

Figure 1: (a) Contrastive Representation Alignment of Object Queries (QueryREPA): During a pretraining stage, each image in a modality-balanced batch is associated with a text-derived modality token . For each image, we mean-pool the decoder’s object-query embeddings into a single query representation, project it into the modality-token space, and optimize a contrastive alignment loss between the projected queries and the modality tokens, yielding a modality-aware query manifold. (b) Multimodality Context Attention (MoCA): For a given image, a modality token is selected and concatenated with the set of object queries to form an augmented query set. Information fusion occurs in the self-attention layer of the decoder, where each query attends to the modality token to attain modality-specific context.

Figure 2: Comparison of text/modality integration in DETR-style detectors. (a) DINO (baseline): image-only encoder–decoder with query selection. (b) Grounding DINO: language-guided query selection and cross-attention in the decoder (object queries attend to a sequence of text tokens). (c) Ours (DINO+MoCA): append compact modality tokens to the object queries and fuse by self-attention in the decoder; the tokens act as semantic anchors that refine queries in a modality-aware way with minimal overhead.

Method	AP	AP 50	AP 75	AP s	AP m	AP l
Deformable DETR [ 46 ]	35.0	54.4	37.8	17.4	27.6	38.2
Sparse R-CNN [ 33 ]	35.9	54.4	38.8	18.4	29.0	38.0
DiffusionDet [ 5 ]	38.2	60.3	40.1	22.4	30.6	41.6
GLIP [ 22 ]	37.4	57.5	39.9	20.8	30.5	42.2
GLIP-A [ 22 ]	37.7	58.3	40.1	20.9	31.0	41.4
GLIP-B [ 22 ]	37.1	57.4	39.4	22.4	29.7	40.8

Table 1: Quantitative comparison with state-of-the-art methods in medical object detection. Applying our method (MoCA + QueryREPA) on the DINO [ 42 ] detector with

{\mathcal{E}}

as CLIP, in both the modality-only (

{\mathcal{S}}={\mathcal{D}}

) and class-conditioned (

{\mathcal{S}}={\mathcal{D}}\times{\mathcal{C}}

) configurations, outperforms all other detection methods. Bold = best, underline = second best.

Token	Partition	AP	AP 50	AP 75	AP s	AP m	AP l
Baseline (no token)	—	37.7	58.6	37.8	19.9	29.8	41.7
Capacity controls
Learnable	global	37.6	58.1	40.6	21.4	29.6	41.7
Random	modality	37.9	58.1	40.8	19.9	30.3	41.8
Correctness control
Shuffled	modality	38.3	59.0	40.9	20.7	30.3	42.3

Table 2: Results for alternative token designs. Bold = best, underline = second best.

Figure 3: QueryREPA incorporates modality-aware semantics into the query space. UMAP embeddings of the mean query representation $\bar{q}^{(l)}$ , computed per image and colored by imaging modality. Before QueryREPA (left) the representations are highly entangled across modalities, whereas after QueryREPA (right) they separate into disentangled clusters.

Figure 4: Qualitative Comparison. Comparison results between various state-of-the-art detection methods and the proposed method are shown above. Our method effectively leverages modality context to significantly enhance anomaly localization (highlighted in red), compared to baseline results (highlighted in blue). Ground truth bounding boxes are highlighted in green. For cases where the bounding boxes are small, insets show an enlarged view of the highlighted yellow region.

		Deployment		Total		CXR		MRI		Colonoscopy		Pathology		CT
Detector	Method	Models	Params (M)	AP	AP 50	AP	AP 50	AP	AP 50	AP	AP 50	AP	AP 50	AP	AP 50
Def. DETR	Baseline (Specialist)	5	200.50	36.2	55.4	14.9	30.9	77.4	96.7	69.0	84.4	47.3	74.1	31.0	63.6
	Baseline (Unified)	1	40.11	35.0	54.4	15.1	30.9	75.8	96.2	61.3	77.3	46.5	74.1	30.5	62.3
	+ Ours	1	40.89	37.2	60.2	18.2	39.9	76.1	96.7	66.7	84.6	45.4	76.0	31.3	61.6
DINO	Baseline (Specialist)	5	237.70	39.1	59.1	17.2	34.6	80.4	97.2	69.9	84.3	52.9	81.9	36.5	70.6
	Baseline (Unified)	1	47.55	37.7	58.6	17.8	36.1	78.0	96.6	65.1	81.3	50.5	77.8	33.0	66.6

Table 3: Comparison of modality-specific expert detectors and unified multimodality detectors. Specialist totals assume each image is sent to the expert for its own modality with no routing errors, and predictions are pooled over the full 27-category taxonomy. Bold = best per column within each detector.

Table 8

QueryREPA	MoCA	AP	AP 50	AP 75	AP s	AP m	AP l
		37.7	58.6	37.8	19.9	29.8	41.7
✓		38.1	58.4	40.9	21.0	29.8	42.3
	✓	40.8	65.1	43.4	25.1	33.5	45.2
✓	✓	41.1	65.5	43.4	25.4	34.6	44.9

Table 6: Ablation of components using DINO [ 42 ] as the base model and CLIP [ 31 ] as the encoder

{\mathcal{E}}

. Checkmarks denote modules enabled. Bold = best per column.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Table 10

Layer $l$	AP	AP 50	AP 75	AP s	AP m	AP l
Layer 2	41.0	65.8	43.5	24.0	33.3	45.0
Layer 3	41.0	65.6	43.3	27.0	32.8	45.1
Layer 4	41.0	65.8	43.1	25.3	33.3	45.3
Layer 5	41.1	65.5	43.4	25.4	34.6	44.9

Appendix

Table 8: Ablation of decoder layer

l

used for alignment in QueryREPA. Our method is robust, producing optimal results regardless of decoder layer

l

Method	AP	AP 50	AP 75	AP s	AP m	AP l
DiffusionDet	38.2	60.3	40.1	22.4	30.6	41.6
DiffusionDet + Ours ( ${\mathcal{E}}$ : CLIP)	40.2	65.4	41.6	26.0	34.5	42.6

Appendix

Table 9: Quantitative comparison of applying our method (MoCA + QueryREPA) on the DiffusionDet [ 5 ] model. Bold = best per column.

QueryREPA	MoCA	AP	AP 50
		3.6	10.0
✓		4.1	11.3
	✓	7.4	20.7
✓	✓	8.0	22.2

Appendix

Table 10: Zero-shot transfer on ChestX-Det. Models trained on our mixed corpus are evaluated without any fine-tuning. Bold = best per column.

Figure 6: UMAP embedding of modality tokens for varying ${\mathcal{E}}$ .

Figure 7: Attention heatmaps at decoder layer 5 (each query weighted by predicted-class confidence). (a) Input Image with ground-truth boxes; (b) Baseline; (c) Ours with the correct modality token, where attention concentrates on the target structures; (d) Swapped token, an unrelated token ( “COVID-19 in lung CT” ), where the attention scatters. This indicates that the modality token causally shifts the attention pattern.

			Total		CXR		MRI		Colonoscopy		Pathology		CT
Method	${\mathcal{E}}$	${\mathcal{L}}_{\text{QRA}}$	AP	AP 50	AP	AP 50	AP	AP 50	AP	AP 50	AP	AP 50	AP	AP 50
Deformable DETR	✗	✗	35.0	54.4	15.1	30.9	75.8	95.9	61.3	77.3	46.5	74.1	30.5	62.3
	CLIP	✗	37.3	60.2	18.0	39.1	75.3	96.2	64.6	83.7	49.3	79.1	31.5	62.5
	CLIP	✓	37.2	60.2	18.2	39.9	76.1	96.7	66.7	84.6	45.4	76.0	31.3	61.6
	BiomedCLIP	✗	36.7	60.2	17.5	39.1	74.5	96.7	65.1	85.4	47.7	77.6	29.9	61.8
	BiomedCLIP	✓	36.7	59.6	17.7	38.9	75.7	96.5	64.2	83.4	47.1	76.8	30.1	61.0

Appendix

Table 11: Quantitative comparison leveraging various modality token encoders

{\mathcal{E}}

as CLIP [ 31 ] , BiomedCLIP [ 43 ] , and PubMedCLIP [ 10 ] under the same training setup. Our method shows improved results regardless of

{\mathcal{E}}

used, proving its robustness and wide applicability. Bold indicates best results per column.

Stage	Time	Peak Memory
Baseline training	$\sim$ 25h	14.8 GB
QueryREPA training	$\sim$ 29h	8.1 GB
Detection training (w/ MoCA)	$\sim$ 27h	14.8 GB

Appendix

Table 12: Training cost on a single RTX 4090.

Figure 8: Qualitative comparisons on chest X-ray images. Blue boxes indicate predictions from baseline detectors (Sparse R-CNN, GLIP, DiffusionDet, Grounding DINO, DINO), Green boxes denote ground-truth annotations, and Red boxes show predictions from our method.

Figure 9: Qualitative comparisons on MRI images. Blue boxes indicate predictions from baseline detectors (Sparse R-CNN, GLIP, DiffusionDet, Grounding DINO, DINO), Green boxes denote ground-truth annotations, and Red boxes show predictions from our method.

Figure 10: Qualitative comparisons on Colonoscopy images. Blue boxes indicate predictions from baseline detectors (Sparse R-CNN, GLIP, DiffusionDet, Grounding DINO, DINO), Green boxes denote ground-truth annotations, and Red boxes show predictions from our method.

Figure 11: Qualitative comparisons on Pathology images. Blue boxes indicate predictions from baseline detectors (Sparse R-CNN, GLIP, DiffusionDet, Grounding DINO, DINO), Green boxes denote ground-truth annotations, and Red boxes show predictions from our method.

Figure 12: Qualitative comparisons on CT images. Blue boxes indicate predictions from baseline detectors (Sparse R-CNN, GLIP, DiffusionDet, Grounding DINO, DINO), Green boxes denote ground-truth annotations, and Red boxes show predictions from our method.

Medical Image Object Detection

See all 14 papers

Closing the gap in multimodal medical representation alignment
Feb 23, 2026Eleonora Grassucci, Giordano Cicchetti, Danilo ComminielloModality Gap in VLMs Vision-Language Alignment
Dept. of Information Engineering, Electronics, and Telecommunications, Sapienza University of Rome, Italy
MultiMedVision: Multi-Modal Medical Vision Framework
May 9, 2026Frank Li, Bardia Khosravi, Mohammadreza Chavoshi +5Medical Imaging Foundation Models Vision Transformer
Emory University · Yale University
3D Modality-Aware Pre-training for Vision-Language Model in MRI Multi-organ Abnormality Detection
Feb 27, 2026Haowen Zhu, Ning Yin, Xiaogen ZhouMedical VLMs Vision-Language Pretraining
School of Computer Science and Engineering, Southeast University, China · Department of Medical Imaging, Suzhou Traditional Chinese Medicine Hospital, China · School of Electronic, Electrical Engineering and Physics, Fujian University of Technology

Feb 23, 2026J/K move · Enter open · S save

Closing the gap in multimodal medical representation alignment

Eleonora Grassucci, Giordano Cicchetti, Danilo Comminiello

Dept. of Information Engineering, Electronics, and Telecommunications, Sapienza University of Rome, Italy

Modality Gap in VLMs Vision-Language Alignment Medical VLMs Multimodal Contrastive Learning Medical Image Representation Learning Cross-Modal Retrieval

Open paper AskPDF arXiv

In multimodal learning, CLIP has emerged as the de-facto approach for mapping different modalities into a shared latent space by bringing semantically similar representations closer while pushing apart dissimilar ones. However, CLIP-based contrastive losses exhibit unintended behaviors that negatively impact true semantic alignment, leading to sparse and fragmented latent spaces. This phenomenon, known as the modality gap, has been partially mitigated for standard text and image pairs but remains unknown and unresolved in more complex multimodal settings, such as the medical domain. In this work, we study this phenomenon in the latter case, revealing that the modality gap is present also in medical alignment, and we propose a modality-agnostic framework that closes this gap, ensuring that semantically related representations are more aligned, regardless of their source modality. Our method enhances alignment between radiology images and clinical text, improving cross-modal retrieval and image captioning.

Abstract

Medical Image Object Detection

Explore similar work

Closing the gap in multimodal medical representation alignment

MultiMedVision: Multi-Modal Medical Vision Framework

3D Modality-Aware Pre-training for Vision-Language Model in MRI Multi-organ Abnormality Detection