SLQ: Bridging Modalities via Shared Latent Queries for Retrieval with Frozen MLLMs

Haoran Lou; Ziyan Liu; Chunxiao Fan; Yuexin Wu; Yue Ming; Hao Wu; Kai Zuo; Yibo Chen

cs.CVApr 15, 2026

SLQ: Bridging Modalities via Shared Latent Queries for Retrieval with Frozen MLLMs

Authors: Haoran Lou, Ziyan Liu, Chunxiao Fan, Yuexin Wu, Yue Ming, Hao Wu, Kai Zuo, Yibo Chen,

Organizations: School of Electronic Engineering, Beijing University of Posts and Telecommunications, China · Xiaohongshu Inc., China

Abstract

Multimodal Large Language Models (MLLMs) possess intrinsic reasoning and world-knowledge capabilities, yet adapting them for dense retrieval remains challenging. Existing approaches rely on invasive parameter updates, such as full fine-tuning and LoRA, which may disrupt the pre-trained semantic space and impair the structured knowledge essential for reasoning. To address this, we propose SLQ, a parameter-efficient tuning framework that adapts MLLMs for retrieval while keeping the backbone entirely frozen. SLQ introduces a small set of Shared Latent Queries that are appended to both text and image tokens, leveraging the model's native causal attention to aggregate multimodal context into a unified embedding space. Furthermore, to better evaluate retrieval beyond superficial pattern matching, we construct KARR-Bench, a benchmark designed for knowledge-aware reasoning retrieval. Extensive experiments show that SLQ outperforms full fine-tuning and LoRA on COCO and Flickr30K, while achieving competitive performance on MMEB and yielding substantial gains on KARR-Bench, validating that preserving the pre-trained representations via non-invasive adaptation is an effective strategy for MLLM-based retrieval. The code is available under: https://github.com/CnFaker/SLQ.

Figures & tables

Figure 1: Parameter Efficiency vs. Retrieval Performance. We compare the number of trainable parameters against the retrieval performance, calculated as the average Recall@5 of COCO imageto-text and text-to-image retrieval. Our method achieves superior accuracy while tuning only thousands of parameters. This demonstrates an orders-of-magnitude efficiency gain compared to LoRA, which requires updating millions of parameters.

Panel of Figure 2 — Figure 2: Diagnostic pilot study. We compare the zero-shot retrieval performance of the last token baseline against query token method on the InternVL3-1B backbone. The retrieval score based on cosine similarity is reported as the metric. The results suggest that the query token method better aggregates global context, enabling implicit reasoning for retrieval.

Panel of Figure 3 — Figure 3: Overview of KARR-Bench. (a) Comparison between standard explicit captions and our knowledge reasoning captions. (b) The comprehensive distribution of categories in KARR-Bench.

Figure 4: Overview of the SLQ framework. SLQ bridges the modality gap using a set of Shared Latent Queries that interact with the frozen MLLM via causal attention. The term “Shared” indicates that the same set of queries is appended to both image and text tokens, projecting them into a unified embedding space. During training, the MLLM backbone remains frozen, and only the queries are optimized via the contrastive objective to align vision and language representations.

Dual-Encoder
Model	Flickr30K (1K test set)						COCO (5K test set)
	Image $\rightarrow$ Text			Text $\rightarrow$ Image			Image $\rightarrow$ Text			Text $\rightarrow$ Image
	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10
CLIP ViT-B ( Radford et al., 2021 )	77.8	95.0	98.2	58.8	83.3	89.8	51.0	74.9	83.5	30.5	56.0	66.8
CLIP ViT-L ( Radford et al., 2021 )	87.2	98.3	99.4	67.3	89.0	93.3	58.1	81.0	87.8	37.0	61.6	71.5
BLIP ViT-L ( Li et al., 2022 )	75.5	95.1	97.7	70.0	91.2	95.2	63.5	86.5	92.5	48.4	74.4	83.2

Table 1: Performance comparison on Flickr30K and COCO retrieval benchmarks. The best results are highlighted in bold and the second-best are underlined .

Backbone	Dim.	Method	Param.	GPU Hour	Flickr30K		COCO		MMMU	RealWQA	OCRBench
Backbone	Dim.	Method	Param.	GPU Hour	IR	TR	IR	TR	MMMU	RealWQA	OCRBench
InternVL3-1B	896	Full FT	0.6B	18.0	90.8	94.8	72.4	84.2	40.7	54.3	758
		LoRA	4.4M	11.4	90.4	95.7	72.9	84.4	42.1	56.8	785
		SLQ	36k	3.9	92.9	97.8	74.3	84.9	43.4	58.2	790
Qwen3VL-2B	2048	Full FT	1.7B	42.5	91.7	97.0	73.2	82.1	51.7	59.4	824
		LoRA	8.7M	23.8	92.1	97.2	74.8	82.7	53.1	62.8	832

Table 3: Comparison of tuning strategies. We report Recall@5 for retrieve and assess the retention of the MLLM’s inherent vision-language understanding capability on MMMU, RealWorldQA, and OCRBench. Training cost is measured in GPU hours on COCO for 5 epochs using H800 GPUs.

Figure 5: Performance comparison on KARR-Bench. Left to Right: Results on InternVL3-1B, Qwen3VL-2B, Qwen3VL-4B, and InternVL3-8B. Our method (shown in Orange ) consistently outperforms both invasive tuning baselines. Specifically, on the strongest InternVL3-8B backbone, our method achieves significant gains over LoRA and Full FT. These results indicate that preserving a frozen backbone while SLQ is effective for knowledge-aware reasoning retrieval.

Figure 6: Visualization of the unified representation space using PCA. Red points represent Image embeddings, and Blue points represent Text embeddings. Full FT and LoRA exhibit a noticeably broader spatial spread. In contrast, SLQ maintains a much more compact distribution , resulting in a smaller centroid distance gap and demonstrating superior cross-modal alignment.

Method	$\\|\text{gap}\\|$ ( $\downarrow$ )	Alignment ( $\downarrow$ )	Uniformity ( $\downarrow$ )
Method	$\\|\text{gap}\\|$ ( $\downarrow$ )	Alignment ( $\downarrow$ )	Text	Image
Full FT	0.193	0.182	-1.15	-1.09
LoRA	0.176	0.155	-1.30	-1.26
SLQ (Ours)	0.165	0.148	-1.45	-1.42

Table 4: Analysis of representation geometry. Lower (

\downarrow

) Alignment and

\|\text{gap}\|

indicate better matching. Lower Uniformity indicates evenly distributed spaces.

Figure 7: Ablation on the number of queries. The left axis shows the performance on COCO while the right axis shows on KARR-Bench.

Category	Design	I $\to$ T	T $\to$ I
PEFT Strategy	LoRA	84.4	72.9
	Prompt Tuning ( Khattak et al., 2023 )	82.2	69.7
	Adapter Tuning ( Sung et al., 2022 )	83.2	71.0
Extraction Design	Head (Linear)	78.7	67.4
	Head (TF Block)	78.2	66.7
	Separate Queries	83.2	72.5

Table 5: All ablations are conducted using InternVL3-1B and report Recall@5 on COCO.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Optimizer	AdamW
LR Schedule	Cosine Decay
Warmup Ratio	0.03
DeepSpeed	ZeRO-2
LoRA Rank	8
LoRA Alpha	16

Appendix

Table 6: Global optimization hyperparameters. These settings were applied consistently across all experiments to ensure a fair comparison.

Backbone	Dim	Method	Bsz / GPU	Total GPUs	Global Bsz	LR	Train Time	GPU Hours
InternVL3-8B	3584	Full FT	16	32 $\times$ H800	512	1e-5	12:37:02	403.8
		LoRA	16	32 $\times$ H800	512	1e-5	4:03:49	130.1
		SLQ	32	16 $\times$ H800	512	5e-4	2:25:53	38.9
Qwen3VL-4B	2560	Full FT	32	32 $\times$ H800	1024	1e-5	3:00:22	96.2
		LoRA	32	32 $\times$ H800	1024	1e-5	1:31:08	48.6
		SLQ	64	16 $\times$ H800	1024	5e-4	0:50:37	13.5

Appendix

Table 7: Hyperparameter settings and training costs for different backbones.

Method	Image $\rightarrow$ Text	Text $\rightarrow$ Image
VLM2VEC-7B ( Jiang et al., 2024b )	49.9	50.1
SLQ-1B (Ours)	44.9	44.1
SLQ-2B (Ours)	47.3	47.9
SLQ-4B (Ours)	51.8	51.6
SLQ-8B (Ours)	58.7	55.7

Appendix

Table 8: SOTA comparison on KARR-Bench (Recall@5).

Method	Rank	Image $\rightarrow$ Text	Text $\rightarrow$ Image
LoRA	4	84.5	72.5
LoRA	8	84.4	72.9
LoRA	16	83.9	72.2
LoRA	32	82.1	71.6
SLQ (Ours)	—	84.9	74.3

Appendix

Table 9: Ablation on LoRA ranks on COCO (Recall@5), using InternVL-1B.

Method	FashionIQ (Avg R@50)	CIRR (R@5)
E5-V-7B* ( Jiang et al., 2024a )	30.8	33.5
InternVL3-1B (Full FT)	32.7	53.2
InternVL3-1B (LoRA)	33.1	51.4
InternVL3-1B (SLQ)	35.0	57.8
InternVL3-8B (Full FT)	37.6	55.7
InternVL3-8B (LoRA)	39.3	56.8

Appendix

Table 10: Zero-shot composed image retrieval performance on FashionIQ (Average R@50) and CIRR (R@5). * indicates E5-V trained with image–text pairs.

Method	image retrieval						text (render as image) retrieval
	I2I-Flickr30K			I2I-COCO			I2I-Flickr30K			I2I-COCO
	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10
E5-V-7B ( Jiang et al., 2024a )	67.8	89.2	93.6	41.2	66.7	76.2	79.5	95.2	97.8	51.6	76.8	84.9
InternVL3-1B (Full FT)	61.2	84.3	89.7	39.1	65.4	75.8	73.5	91.3	96.0	49.7	77.3	84.2
InternVL3-1B (LoRA)	60.4	84.6	90.2	39.0	66.0	76.5	73.3	91.8	96.3	49.7	77.0	85.3
InternVL3-1B (SLQ)	64.8	87.3	92.2	41.2	67.7	77.9	76.9	95.0	97.2	54.1	80.0	87.7

Appendix

Table 11: Zero-shot image-image retrieval performance on I2I-Flickr30K and I2I-COCO.

Figure 8: Linguistic and Semantic Comparison between COCO and KARR-Bench. (a) The cosine similarity distribution demonstrates that KARR-Bench Captions maintain semantic relevance to the visual content despite abstraction. (b) KARR-Bench exhibits significantly higher Syntactic Complexity (MDD). (c) The Cognitive Load (Flesch-Kincaid Grade Level) doubles from COCO to KARR-Bench, indicating a shift to advanced reading comprehension. (d) The drastic drop in Entity Density (Proper Nouns) confirms the removal of explicit naming cues.

Figure 9: Word Cloud Comparison. The vocabulary shifts from observational primitives in COCO (a) to abstract, functional, and relational terms in KARR-Bench (b), reflecting the requirement for implicit reasoning.

Panel of Table 12 — Table 12: Stage 1 prompt for KARR-Bench construction.

Panel of Table 13 — Table 13: Stage 2 prompt for KARR-Bench construction.

Figure 10: Qualitative comparison of Text-to-Image Retrieval on KARR-Bench.

Figure 11: Qualitative comparison of Image-to-Text Retrieval on KARR-Bench.

Multimodal IR

Multimodal information retrieval finds relevant images, videos, documents, audio, or other media using queries that may combine different modalities.

See all 122 papers

RETLLM: Training and Data-Free MLLMs for Multimodal Information Retrieval
Feb 25, 2026Dawei Su, Dongsheng WangMultimodal IR Multimodal Reranking
College of Computer Science and Software Engineering, Shenzhen University, Shenzhen, China
Think When Needed: Adaptive Reasoning-Driven Multimodal Embeddings with a Dual-LoRA Architecture
May 14, 2026Longxiang Zhang, Weilong Dai, Guanghao Zhang +2Efficient Multimodal Inference Multimodal Embedding
Alibaba Group
TRACE: Task-Adaptive Reasoning and Representation Learning for Universal Multimodal Retrieval
Mar 3, 2026Xiangzhao Hao, Shijie Wang, Tianyu Yang +3Multimodal IR Multimodal Embedding
Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences

Feb 25, 2026J/K move · Enter open · S save

RETLLM: Training and Data-Free MLLMs for Multimodal Information Retrieval

Dawei Su, Dongsheng Wang

College of Computer Science and Software Engineering, Shenzhen University, Shenzhen, China

Multimodal IR Multimodal Reranking Multimodal Large Language Models

Open paper AskPDF arXiv

Multimodal information retrieval (MMIR) has gained attention for its flexibility in handling text, images, or mixed queries and candidates. Recent breakthroughs in multimodal large language models (MLLMs) boost MMIR performance by incorporating MLLM knowledge under the contrastive finetuning framework. However, they suffer from pre-training inconsistency and require large datasets. In this work, we introduce a novel framework, RetLLM, designed to query MLLMs for MMIR in a training- and data-free manner. Specifically, we formulate MMIR as a similarity score generation task and prompt MLLMs to directly predict retrieval scores in a coarse-then-fine pipeline. At the coarse stage, a top-k filtering strategy builds a small yet high-quality candidate pool for each query, enabling MLLMs to focus on semantically relevant candidates. Subsequently, the retrieval score is predicted by feeding both the query and candidate into MLLMs at the fine stage. Importantly, we propose a visual enhancement module during reasoning to help MLLMs re-pick forgotten visuals, improving retrieval. Extensive experiments on MMIR benchmarks show that RetLLM outperforms fine-tuned models. Ablation studies further verify each component. Our work demonstrates that MLLMs can achieve strong MMIR performance without any training, highlighting their inherent multimodal reasoning ability in a simple, scalable framework. We release our code at: https://github.com/alivecat05/RETLLM

Abstract

Multimodal IR

Explore similar work

RETLLM: Training and Data-Free MLLMs for Multimodal Information Retrieval

Think When Needed: Adaptive Reasoning-Driven Multimodal Embeddings with a Dual-LoRA Architecture

TRACE: Task-Adaptive Reasoning and Representation Learning for Universal Multimodal Retrieval