Indexing Multimodal Language Models for Large-scale Image Retrieval

Bahey Tharwat; Giorgos Kordopatis-Zilos; Pavel Suma; Ian Reid; Giorgos Tolias

cs.CVApr 14, 2026

Indexing Multimodal Language Models for Large-scale Image Retrieval

Authors: Bahey Tharwat, Giorgos Kordopatis-Zilos, Pavel Suma, Ian Reid, Giorgos Tolias

Organizations: Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE · VRG, FEE, Czech Technical University in Prague, Prague, Czech Republic

Abstract

Multimodal Large Language Models (MLLMs) have demonstrated strong cross-modal reasoning capabilities, yet their potential for vision-only tasks remains underexplored. We investigate MLLMs as training-free similarity estimators for instance-level image-to-image retrieval. Our approach prompts the model with paired images and converts next-token probabilities into similarity scores, enabling zero-shot re-ranking within large-scale retrieval pipelines. This design avoids specialized architectures and fine-tuning, leveraging the rich visual discrimination learned during multimodal pre-training. We address scalability by combining MLLMs with memory-efficient indexing and top- $k$ candidate re-ranking. Experiments across diverse benchmarks show that MLLMs outperform task-specific re-rankers outside their native domains and exhibit superior robustness to clutter, occlusion, and small objects. Despite strong results, we identify failure modes under severe appearance changes, highlighting opportunities for future research. Our findings position MLLMs as a promising alternative for open-world large-scale image retrieval.

Figures & tables

Panel of Figure 1 — Figure 1: Performance vs . re-ranking time. MLLM-based re-rankers have higher per-image inference time than methods trained specifically for instance-level image retrieval with re-ranking (AMES), but under a fixed query-time budget, they achieve better retrieval performance, already with as few as 20 re-ranked images, indicated by the colored numbers. All reported methods have roughly the same memory footprint.

Figure 2: Overview of the proposed MLLM-based re-ranking approach for instance-level image retrieval. A query image, a database image, and a task-specific textual prompt are jointly used as input for similarity estimation. A vision encoder extracts visual tokens from both images ( $X_{q}$ , $X_{db}$ ), which are concatenated with the textual prompt tokens ( $T$ ) and the end-of-sentence (EOS) token to form the multimodal input to the MLLM. Scalable retrieval is achieved by applying compression and indexing schemes, e.g . Product Quantization (PQ), to the database image tokens, enabling memory-efficient storage. The MLLM outputs the updated EOS token representation, which is the joint representation of all inputs and is then compared via dot product with the classifier vectors $\mathbf{z}_{\texttt{1}}$ and $\mathbf{z}_{\texttt{0}}$ , corresponding to vocabulary tokens “ 1 ” and “ 0 ”, respectively. The final similarity score ( $s_{r}$ ) is given by the softmax ( $\sigma$ ) probability assigned to token “ 1 ”.

Panel of Figure 3 — Figure 3: Performance vs . memory. Memory footprint per image is approximated assuming a fixed resolution, where the longer image side is set to the target value and the shorter side is scaled to 3/4 of it, reflecting a typical aspect ratio. Five models for re-ranking, operating on top of global similarities obtained from the PE model with linear adaptation. All indexing strategies are applied on the Qwen model with 560px image resolution. Text indicates: subspace dimension for PQ ( e.g . PQ 16 has 16 subspace dimensions), pixels for different resolutions ( e.g . 560px), and number of tokens ( e.g . 70) for the rest of the indexing methods. AMES used with descriptor sets of increasing cardinalities. LamRA and Qwen3R are shown both with their default prompts and also with our prompt, which performs better.

Panel of Figure 4 — Figure 4: Performance comparison across different amounts of object area coverage and background clutter. Positives across all queries are jointly ranked based on coverage or clutter and split into 4 equal-sized groups. mAP@1k averaged over the queries in each group. Comparison between Qwen with and without compression (Qwen-C), AMES, and PE.

Panel of Figure 5 — Figure 5: Robustness analysis of MLLMs under controlled transformations. Similarity scores for positive image pairs are shown across ten types of visual perturbations: contrast, brightness, rotation, scale, background scaling, blur, tiling, noise, clutter, and occlusion. 50 queries from ILIAS are used to generate positive pairs. Dashed lines indicate the average hard-negative similarity of the 5th percentile of the negatives in the re-ranking list, acting as a baseline. The crossing between the solid and dashed lines indicates the robustness point. Comparison between Qwen without compression, AMES, and PE. The horizontal axis represents the strength of each transformation.

PE [ 7 ] with linear adaptation [ 26 ]
Method	ILIAS	INSTRE	$\mathcal{R}$ OP	Prod1M
No re-rank	33.4	92.6	60.7	66.5
AMES [ 53 ]	41.9	91.9	79.5	66.8
LamRA [ 34 ]	47.8	95.3	67.5	72.9
Qwen [ 6 ]	41.0	94.6	63.0	72.3
LamRA	53.3	96.6	66.5	73.1

Table 1: Performance (mAP@1k) comparison on multiple datasets using two different global descriptors. We present four instance-level to show the different behavior between training-free Qwen and LaMRA, which is fine-tuned but for more generic retrieval tasks. AMES is reported using 400 local descriptors (102kB), which achieves the best performance on average. Gray indicates the use of our task-specific prompt; otherwise, a generic prompt is used.

Panel of Figure 6 — Figure 6: Qualitative examples where one method benefits the most compared to another. We compare global (PE) and AMES vs . Qwen by showing pairs of query and positive image. $\rightarrow$ indicates the number of negative images ranked before the positive for two models, and it goes from the weaker to the stronger model for each pair.

Panel of Figure 8 — Figure A : Impact of compression for different resolutions. mAP@1k of Qwen on ILIAS for seven PQ compressions ( i.e . PQ {1, 4, 8, 16, 32, 64, 128} from right to left) and four image resolutions ( i.e . {392px, 560px, 756px, 1008px} in different colors).

Panel of Figure 9 — Figure B : Impact of compression and resolution on a better MLLM. mAP@1k comparison of Qwen3R and Qwen3 on ILIAS. Compression is applied via PQ with 560px image resolution.

Panel of Figure 10 — Figure C : Performance comparison per category. mAP@1k averaged over objects in the same mid-level taxonomy category of ILIAS, grouped by their primary-level category size, with sorting within each group by Qwen performance. Comparison between Qwen with and without compression (Qwen-C), AMES, LamRA, and PE.

Panel of Figure 11 — Figure D : Performance comparison per domain. mAP@1k averaged over objects in the same coarser taxonomy level of ILIAS, sorted by Qwen performance. Comparison between Qwen without and with compression (Qwen-C), AMES, LamRA, and PE.

Model	392px	560px	756px
No-rerank	33.4	33.4	33.4
Qwen-3B	19.7	24.3	25.3
Qwen-7B	48.5	53.2	54.4
Qwen-32B	48.1	51.3	52.0
Qwen-72B	54.2	54.5	OOM

Table A : Impact of MLLM size. mAP@1k of four Qwen variants with three image resolutions on ILIAS. No re-ranking is provided for comparison. OOM stands for out-of-memory.

Panel of Figure 13 — Figure F : Impact of re-ranking. mAP@1k of three models on ILIAS for increasing shortlist sizes, up to 5k images per query.

Panel of Figure 14 — Figure G : Latency vs . prompt length. The plot shows the latency of the LLM as a function of the number of prompt tokens. The central line represents the mean latency, while the upper and lower bounds indicate the variance.

Figure H : Prompts for similarity estimation. We use one generic and three task-specific prompts to evaluate the benchmarked MLLMs. We use object prompt for ILIAS, INSTRE, and Product1M, and the landmark prompt for

\mathcal{R}

OP+1M.

prompt	ILIAS	INSTRE	ROP	Prod1M
generic	42.0	94.6	63.0	72.3
object	53.3	96.4	65.7	74.5
landmark	40.0	90.7	68.1	66.9

Table B : Prompt Sensitivity. Retrieval performance (mAP) across datasets for different prompt types.

Model	Qwen3R	Qwen3R-C
default	60.2	58.8
object	62.0	61.2

Table C : Qwen3R Prompting. Retrieval performance (mAP) for Qwen3R and Qwen3R-C with the original generic default prompt, provided in the original repository, and with our task-specific object prompt.

Panel of Figure 18 — Figure I : More qualitative examples where one method benefits the most compared to another. We compare global (PE) and AMES vs . Qwen by showing pairs of query and positive image. $\rightarrow$ indicates the number of negative images ranked before the positive for two models, and it goes from the weaker to the stronger model for each pair.

Panel of Figure 19 — Figure J : Examples of transformed images for robustness analysis, including variations in contrast, brightness, rotation, downscaling, background scaling, blur, tiling, noise, clutter, and occlusion. Each row shows the gradual increase in transformation strength from left to right. These transformations are applied to create positive query–target pairs for controlled robustness evaluation.

Image Retrieval

Image retrieval finds and ranks images from visual, textual, or combined queries across large and specialized collections.

See all 74 papers

RETLLM: Training and Data-Free MLLMs for Multimodal Information Retrieval
Feb 25, 2026Dawei Su, Dongsheng WangMultimodal IR Multimodal Reranking
College of Computer Science and Software Engineering, Shenzhen University, Shenzhen, China
FiRE: Enhancing MLLMs with Fine-Grained Context Learning for Complex Image Retrieval
Jul 30, 2026Bohan Hou, Haoqiang Lin, Xuemeng Song +4Fine-Grained Image Retrieval Multimodal Large Language Models
Shandong University Qingdao, Shandong, China · City University of Hong Kong Hong kong, China · Harbin Institute of Technology (Shenzhen) Shenzhen, Guangdong, China +2
VisLens: Single-Pass Interpretable Visual Search for Multimodal LLMs
Aug 31, 2026Jingyi He, Sanghwan Kim, Zeynep AkataEfficient Multimodal Inference Multimodal Large Language Models
Technical University of Munich · Helmholtz Munich · Munich Center for Machine Learning

Feb 25, 2026J/K move · Enter open · S save

RETLLM: Training and Data-Free MLLMs for Multimodal Information Retrieval

Dawei Su, Dongsheng Wang

College of Computer Science and Software Engineering, Shenzhen University, Shenzhen, China

Multimodal IR Multimodal Reranking Multimodal Large Language Models

Open paper AskPDF arXiv

Multimodal information retrieval (MMIR) has gained attention for its flexibility in handling text, images, or mixed queries and candidates. Recent breakthroughs in multimodal large language models (MLLMs) boost MMIR performance by incorporating MLLM knowledge under the contrastive finetuning framework. However, they suffer from pre-training inconsistency and require large datasets. In this work, we introduce a novel framework, RetLLM, designed to query MLLMs for MMIR in a training- and data-free manner. Specifically, we formulate MMIR as a similarity score generation task and prompt MLLMs to directly predict retrieval scores in a coarse-then-fine pipeline. At the coarse stage, a top-k filtering strategy builds a small yet high-quality candidate pool for each query, enabling MLLMs to focus on semantically relevant candidates. Subsequently, the retrieval score is predicted by feeding both the query and candidate into MLLMs at the fine stage. Importantly, we propose a visual enhancement module during reasoning to help MLLMs re-pick forgotten visuals, improving retrieval. Extensive experiments on MMIR benchmarks show that RetLLM outperforms fine-tuned models. Ablation studies further verify each component. Our work demonstrates that MLLMs can achieve strong MMIR performance without any training, highlighting their inherent multimodal reasoning ability in a simple, scalable framework. We release our code at: https://github.com/alivecat05/RETLLM

Abstract

Image Retrieval

Explore similar work

RETLLM: Training and Data-Free MLLMs for Multimodal Information Retrieval

FiRE: Enhancing MLLMs with Fine-Grained Context Learning for Complex Image Retrieval

VisLens: Single-Pass Interpretable Visual Search for Multimodal LLMs