Multimodal information retrieval (MMIR) has gained attention for its flexibility in handling text, images, or mixed queries and candidates. Recent breakthroughs in multimodal large language models (MLLMs) boost MMIR performance by incorporating MLLM knowledge under the contrastive finetuning framework. However, they suffer from pre-training inconsistency and require large datasets. In this work, we introduce a novel framework, RetLLM, designed to query MLLMs for MMIR in a training- and data-free manner. Specifically, we formulate MMIR as a similarity score generation task and prompt MLLMs to directly predict retrieval scores in a coarse-then-fine pipeline. At the coarse stage, a top-k filtering strategy builds a small yet high-quality candidate pool for each query, enabling MLLMs to focus on semantically relevant candidates. Subsequently, the retrieval score is predicted by feeding both the query and candidate into MLLMs at the fine stage. Importantly, we propose a visual enhancement module during reasoning to help MLLMs re-pick forgotten visuals, improving retrieval. Extensive experiments on MMIR benchmarks show that RetLLM outperforms fine-tuned models. Ablation studies further verify each component. Our work demonstrates that MLLMs can achieve strong MMIR performance without any training, highlighting their inherent multimodal reasoning ability in a simple, scalable framework. We release our code at: https://github.com/alivecat05/RETLLM
Figures & tables
Figure 1: Overview of the RetLLM framework, which integrates Top- k filtering, vision enhancement, and entropy-based selection for effective multimodal retrieval
Short Caption Retrieval
Long Caption Retrieval
Compositional Retrieval
Method
Flickr30K
COCO
ShareGPT4V
Urban1K
Replace
Swap
Add
qi→ct
qt→ci
qi→ct
qt→ci
qi→ct
qt→ci
qi→ct
qt→ci
CLIP(ViT-L)
87.2
67.3
58.1
37.0
81.8
84.0
47.0
47.0
79.5
62.7
74.9
EVA-CLIP
93.9
78.8
68.8
51.1
93.1
81.2
80.4
77.8
85.9
70.3
86.7
E5V
88.7
79.5
62.0
52.0
85.1
82.1
88.9
83.2
86.3
67.6
66.9
VLM2Vec
90.6
76.0
66.6
46.8
89.8
86.9
91.3
82.4
89.5
64.8
94.2
Table 1: Results of zero-shot text-image retrieval on short caption datasets (Flickr30K and MS-COCO), long caption datasets (ShareGPT4V and Urban1K) and compositional benchmark (SugarCrepe). The best results are shown in bold.
Models
#Parameters
Per Meta-Task Score
Average Score
Classification
VQA
Retrieval
Grounding
IND
OOD
Overall
# of Datasets →
10
10
12
4
20
16
36
CLIP(ViT-L)
0.4B
42.8
9.1
53.0
51.8
37.1
38.7
39.2
OpenCLIP(ViT-L)
0.4B
41.5
6.9
44.6
53.5
32.8
36.0
36.6
SigLIP(So/14)
0.9B
40.3
8.4
31.6
59.5
32.3
38.0
35.0
CLIP(ViT-BigG/14)
2.5B
52.3
14.0
50.5
60.3
38.9
45.8
44.3
Table 2: Results on the MMEB benchmark. IND represents the in-distribution dataset, and OOD represents the out-of-distribution dataset. The reported scores are the average Precision@1 over the corresponding datasets on zero-shot manner. The best results are marked in bold.
Components
Flickr30k
COCO
qi→ct
qt→ci
qi→ct
qt→ci
ALL
94.5
81.8
69.2
52.1
entropy only
94.0
81.2
68.7
50.8
enhancement only
94.0
80.7
68.8
51.5
MLLM only
93.6
80.2
66.9
50.3
Table 3: Ablation study of visual enhancement and entropy-based selection on Flickr30k and COCO.
CLIP-version
ShareGPT4V
Urban1k
qi→ct
qt→ci
qi→ct
qt→ci
CLIP-ViT-B
94.8
88.1
84.0
71.8
CLIP-ViT-L
97.6
94.2
88.9
78.6
Long-CLIP-L
96.6
95.1
95.2
95.8
Table 4: Performance comparison using different CLIP versions with fixed Qwen2.5-VL.
Figure 2: Ablation studies on the impact of top-k values on retrieval performance and inference efficiency.
MLLMs
ShareGPT4V
Urban1k
qi→ct
qt→ci
qi→ct
qt→ci
phi3.5v
86.5
72.3
78.9
73.5
Qwen2-VL
93.8
94.1
87.2
78.2
Qwen2.5-VL
97.6
94.2
88.9
78.6
Table 5: Performance comparison using different MLLMs with fixed CLIP-ViT-L.