Multimodal information retrieval (MMIR) has gained attention for its flexibility in handling text, images, or mixed queries and candidates. Recent breakthroughs in multimodal large language models (MLLMs) boost MMIR performance by incorporating MLLM knowledge under the contrastive finetuning framework. However, they suffer from pre-training inconsistency and require large datasets. In this work, we introduce a novel framework, RetLLM, designed to query MLLMs for MMIR in a training- and data-free manner. Specifically, we formulate MMIR as a similarity score generation task and prompt MLLMs to directly predict retrieval scores in a coarse-then-fine pipeline. At the coarse stage, a top-k filtering strategy builds a small yet high-quality candidate pool for each query, enabling MLLMs to focus on semantically relevant candidates. Subsequently, the retrieval score is predicted by feeding both the query and candidate into MLLMs at the fine stage. Importantly, we propose a visual enhancement module during reasoning to help MLLMs re-pick forgotten visuals, improving retrieval. Extensive experiments on MMIR benchmarks show that RetLLM outperforms fine-tuned models. Ablation studies further verify each component. Our work demonstrates that MLLMs can achieve strong MMIR performance without any training, highlighting their inherent multimodal reasoning ability in a simple, scalable framework. We release our code at: https://github.com/alivecat05/RETLLM
Figures & tables
Figure 1: Overview of the RetLLM framework, which integrates Top- k filtering, vision enhancement, and entropy-based selection for effective multimodal retrieval
Short Caption Retrieval
Long Caption Retrieval
Compositional Retrieval
Method
Flickr30K
COCO
ShareGPT4V
Urban1K
Replace
Swap
Add
qi→ct
qt→ci
qi→ct
qt→ci
qi→ct
qt→ci
qi→ct
qt→ci
CLIP(ViT-L)
87.2
67.3
58.1
37.0
81.8
84.0
47.0
47.0
79.5
62.7
74.9
EVA-CLIP
93.9
78.8
68.8
51.1
93.1
81.2
80.4
77.8
85.9
70.3
86.7
E5V
88.7
79.5
62.0
52.0
85.1
82.1
88.9
83.2
86.3
67.6
66.9
VLM2Vec
90.6
76.0
66.6
46.8
89.8
86.9
91.3
82.4
89.5
64.8
94.2
Table 1: Results of zero-shot text-image retrieval on short caption datasets (Flickr30K and MS-COCO), long caption datasets (ShareGPT4V and Urban1K) and compositional benchmark (SugarCrepe). The best results are shown in bold.
Models
#Parameters
Per Meta-Task Score
Average Score
Classification
VQA
Retrieval
Grounding
IND
OOD
Overall
# of Datasets →
10
10
12
4
20
16
36
CLIP(ViT-L)
0.4B
42.8
9.1
53.0
51.8
37.1
38.7
39.2
OpenCLIP(ViT-L)
0.4B
41.5
6.9
44.6
53.5
32.8
36.0
36.6
SigLIP(So/14)
0.9B
40.3
8.4
31.6
59.5
32.3
38.0
35.0
CLIP(ViT-BigG/14)
2.5B
52.3
14.0
50.5
60.3
38.9
45.8
44.3
Table 2: Results on the MMEB benchmark. IND represents the in-distribution dataset, and OOD represents the out-of-distribution dataset. The reported scores are the average Precision@1 over the corresponding datasets on zero-shot manner. The best results are marked in bold.
Components
Flickr30k
COCO
qi→ct
qt→ci
qi→ct
qt→ci
ALL
94.5
81.8
69.2
52.1
entropy only
94.0
81.2
68.7
50.8
enhancement only
94.0
80.7
68.8
51.5
MLLM only
93.6
80.2
66.9
50.3
Table 3: Ablation study of visual enhancement and entropy-based selection on Flickr30k and COCO.
CLIP-version
ShareGPT4V
Urban1k
qi→ct
qt→ci
qi→ct
qt→ci
CLIP-ViT-B
94.8
88.1
84.0
71.8
CLIP-ViT-L
97.6
94.2
88.9
78.6
Long-CLIP-L
96.6
95.1
95.2
95.8
Table 4: Performance comparison using different CLIP versions with fixed Qwen2.5-VL.
Figure 2: Ablation studies on the impact of top-k values on retrieval performance and inference efficiency.
MLLMs
ShareGPT4V
Urban1k
qi→ct
qt→ci
qi→ct
qt→ci
phi3.5v
86.5
72.3
78.9
73.5
Qwen2-VL
93.8
94.1
87.2
78.2
Qwen2.5-VL
97.6
94.2
88.9
78.6
Table 5: Performance comparison using different MLLMs with fixed CLIP-ViT-L.
Multimodal Large Language Models (MLLMs) possess intrinsic reasoning and world-knowledge capabilities, yet adapting them for dense retrieval remains challenging. Existing approaches rely on invasive parameter updates, such as full fine-tuning and LoRA, which may disrupt the pre-trained semantic space and impair the structured knowledge essential for reasoning. To address this, we propose SLQ, a parameter-efficient tuning framework that adapts MLLMs for retrieval while keeping the backbone entirely frozen. SLQ introduces a small set of Shared Latent Queries that are appended to both text and image tokens, leveraging the model's native causal attention to aggregate multimodal context into a unified embedding space. Furthermore, to better evaluate retrieval beyond superficial pattern matching, we construct KARR-Bench, a benchmark designed for knowledge-aware reasoning retrieval. Extensive experiments show that SLQ outperforms full fine-tuning and LoRA on COCO and Flickr30K, while achieving competitive performance on MMEB and yielding substantial gains on KARR-Bench, validating that preserving the pre-trained representations via non-invasive adaptation is an effective strategy for MLLM-based retrieval. The code is available under: https://github.com/CnFaker/SLQ.
Haoran Lou, Ziyan Liu, Chunxiao Fan +6
School of Electronic Engineering, Beijing University of Posts and Telecommunications, China · Xiaohongshu Inc., China
Multimodal Large Language Models (MLLMs) have demonstrated strong cross-modal reasoning capabilities, yet their potential for vision-only tasks remains underexplored. We investigate MLLMs as training-free similarity estimators for instance-level image-to-image retrieval. Our approach prompts the model with paired images and converts next-token probabilities into similarity scores, enabling zero-shot re-ranking within large-scale retrieval pipelines. This design avoids specialized architectures and fine-tuning, leveraging the rich visual discrimination learned during multimodal pre-training. We address scalability by combining MLLMs with memory-efficient indexing and top-k candidate re-ranking. Experiments across diverse benchmarks show that MLLMs outperform task-specific re-rankers outside their native domains and exhibit superior robustness to clutter, occlusion, and small objects. Despite strong results, we identify failure modes under severe appearance changes, highlighting opportunities for future research. Our findings position MLLMs as a promising alternative for open-world large-scale image retrieval.
Bahey Tharwat, Giorgos Kordopatis-Zilos, Pavel Suma +2
Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE · VRG, FEE, Czech Technical University in Prague, Prague, Czech Republic
Multimodal Large Language Models (MLLMs) have recently been applied to universal multimodal retrieval, where Chain-of-Thought (CoT) reasoning improves candidate reranking. However, existing approaches remain largely language-driven, relying on static visual encodings and lacking the ability to actively verify fine-grained visual evidence, which often leads to speculative reasoning in visually ambiguous cases. We propose V-Retrver, an evidence-driven retrieval framework that reformulates multimodal retrieval as an agentic reasoning process grounded in visual inspection. V-Retrver enables an MLLM to selectively acquire visual evidence during reasoning via external visual tools, performing a multimodal interleaved reasoning process that alternates between hypothesis generation and targeted visual verification.To train such an evidence-gathering retrieval agent, we adopt a curriculum-based learning strategy combining supervised reasoning activation, rejection-based refinement, and reinforcement learning with an evidence-aligned objective. Experiments across multiple multimodal retrieval benchmarks demonstrate consistent improvements in retrieval accuracy (with 23.0% improvements on average), perception-driven reasoning reliability, and generalization.
Dongyang Chen, Chaoyang Wang, Dezhao Su +6
Tsinghua University · University of Central Florida · Fudan University +4