Indexing Multimodal Language Models for Large-scale Image Retrieval
Authors: Bahey Tharwat, Giorgos Kordopatis-Zilos, Pavel Suma, Ian Reid, Giorgos Tolias
Organizations: Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE · VRG, FEE, Czech Technical University in Prague, Prague, Czech Republic
Multimodal Large Language Models (MLLMs) have demonstrated strong cross-modal reasoning capabilities, yet their potential for vision-only tasks remains underexplored. We investigate MLLMs as training-free similarity estimators for instance-level image-to-image retrieval. Our approach prompts the model with paired images and converts next-token probabilities into similarity scores, enabling zero-shot re-ranking within large-scale retrieval pipelines. This design avoids specialized architectures and fine-tuning, leveraging the rich visual discrimination learned during multimodal pre-training. We address scalability by combining MLLMs with memory-efficient indexing and top-k candidate re-ranking. Experiments across diverse benchmarks show that MLLMs outperform task-specific re-rankers outside their native domains and exhibit superior robustness to clutter, occlusion, and small objects. Despite strong results, we identify failure modes under severe appearance changes, highlighting opportunities for future research. Our findings position MLLMs as a promising alternative for open-world large-scale image retrieval.
Figures & tables
Figure 1: Performance vs . re-ranking time. MLLM-based re-rankers have higher per-image inference time than methods trained specifically for instance-level image retrieval with re-ranking (AMES), but under a fixed query-time budget, they achieve better retrieval performance, already with as few as 20 re-ranked images, indicated by the colored numbers. All reported methods have roughly the same memory footprint.
Figure 2: Overview of the proposed MLLM-based re-ranking approach for instance-level image retrieval. A query image, a database image, and a task-specific textual prompt are jointly used as input for similarity estimation. A vision encoder extracts visual tokens from both images ( Xq , Xdb ), which are concatenated with the textual prompt tokens ( T ) and the end-of-sentence (EOS) token to form the multimodal input to the MLLM. Scalable retrieval is achieved by applying compression and indexing schemes, e.g . Product Quantization (PQ), to the database image tokens, enabling memory-efficient storage. The MLLM outputs the updated EOS token representation, which is the joint representation of all inputs and is then compared via dot product with the classifier vectors z1 and z0 , corresponding to vocabulary tokens “ 1 ” and “ 0 ”, respectively. The final similarity score ( sr ) is given by the softmax ( σ ) probability assigned to token “ 1 ”.
Figure 3: Performance vs . memory. Memory footprint per image is approximated assuming a fixed resolution, where the longer image side is set to the target value and the shorter side is scaled to 3/4 of it, reflecting a typical aspect ratio. Five models for re-ranking, operating on top of global similarities obtained from the PE model with linear adaptation. All indexing strategies are applied on the Qwen model with 560px image resolution. Text indicates: subspace dimension for PQ ( e.g . PQ 16 has 16 subspace dimensions), pixels for different resolutions ( e.g . 560px), and number of tokens ( e.g . 70) for the rest of the indexing methods. AMES used with descriptor sets of increasing cardinalities. LamRA and Qwen3R are shown both with their default prompts and also with our prompt, which performs better.
Figure 4: Performance comparison across different amounts of object area coverage and background clutter. Positives across all queries are jointly ranked based on coverage or clutter and split into 4 equal-sized groups. mAP@1k averaged over the queries in each group. Comparison between Qwen with and without compression (Qwen-C), AMES, and PE.
Figure 5: Robustness analysis of MLLMs under controlled transformations. Similarity scores for positive image pairs are shown across ten types of visual perturbations: contrast, brightness, rotation, scale, background scaling, blur, tiling, noise, clutter, and occlusion. 50 queries from ILIAS are used to generate positive pairs. Dashed lines indicate the average hard-negative similarity of the 5th percentile of the negatives in the re-ranking list, acting as a baseline. The crossing between the solid and dashed lines indicates the robustness point. Comparison between Qwen without compression, AMES, and PE. The horizontal axis represents the strength of each transformation.
Method
ILIAS
INSTRE
R OP
Prod1M
PE [ 7 ] with linear adaptation [ 26 ]
No re-rank
33.4
92.6
60.7
66.5
AMES [ 53 ]
41.9
91.9
79.5
66.8
LamRA [ 34 ]
47.8
95.3
67.5
72.9
Qwen [ 6 ]
41.0
94.6
63.0
72.3
LamRA
53.3
96.6
66.5
73.1
Table 1: Performance (mAP@1k) comparison on multiple datasets using two different global descriptors. We present four instance-level to show the different behavior between training-free Qwen and LaMRA, which is fine-tuned but for more generic retrieval tasks. AMES is reported using 400 local descriptors (102kB), which achieves the best performance on average. Gray indicates the use of our task-specific prompt; otherwise, a generic prompt is used.
Figure 6: Qualitative examples where one method benefits the most compared to another. We compare global (PE) and AMES vs . Qwen by showing pairs of query and positive image. → indicates the number of negative images ranked before the positive for two models, and it goes from the weaker to the stronger model for each pair.
Figure A : Impact of compression for different resolutions. mAP@1k of Qwen on ILIAS for seven PQ compressions ( i.e . PQ {1, 4, 8, 16, 32, 64, 128} from right to left) and four image resolutions ( i.e . {392px, 560px, 756px, 1008px} in different colors).
Figure B : Impact of compression and resolution on a better MLLM. mAP@1k comparison of Qwen3R and Qwen3 on ILIAS. Compression is applied via PQ with 560px image resolution.
Figure C : Performance comparison per category. mAP@1k averaged over objects in the same mid-level taxonomy category of ILIAS, grouped by their primary-level category size, with sorting within each group by Qwen performance. Comparison between Qwen with and without compression (Qwen-C), AMES, LamRA, and PE.
Figure D : Performance comparison per domain. mAP@1k averaged over objects in the same coarser taxonomy level of ILIAS, sorted by Qwen performance. Comparison between Qwen without and with compression (Qwen-C), AMES, LamRA, and PE.
Model
392px
560px
756px
No-rerank
33.4
33.4
33.4
Qwen-3B
19.7
24.3
25.3
Qwen-7B
48.5
53.2
54.4
Qwen-32B
48.1
51.3
52.0
Qwen-72B
54.2
54.5
OOM
Table A : Impact of MLLM size. mAP@1k of four Qwen variants with three image resolutions on ILIAS. No re-ranking is provided for comparison. OOM stands for out-of-memory.
Figure F : Impact of re-ranking. mAP@1k of three models on ILIAS for increasing shortlist sizes, up to 5k images per query.
Figure G : Latency vs . prompt length. The plot shows the latency of the LLM as a function of the number of prompt tokens. The central line represents the mean latency, while the upper and lower bounds indicate the variance.
Figure H : Prompts for similarity estimation. We use one generic and three task-specific prompts to evaluate the benchmarked MLLMs. We use object prompt for ILIAS, INSTRE, and Product1M, and the landmark prompt for R OP+1M.
prompt
ILIAS
INSTRE
ROP
Prod1M
generic
42.0
94.6
63.0
72.3
object
53.3
96.4
65.7
74.5
landmark
40.0
90.7
68.1
66.9
Table B : Prompt Sensitivity. Retrieval performance (mAP) across datasets for different prompt types.
Model
Qwen3R
Qwen3R-C
default
60.2
58.8
object
62.0
61.2
Table C : Qwen3R Prompting. Retrieval performance (mAP) for Qwen3R and Qwen3R-C with the original generic default prompt, provided in the original repository, and with our task-specific object prompt.
Figure I : More qualitative examples where one method benefits the most compared to another. We compare global (PE) and AMES vs . Qwen by showing pairs of query and positive image. → indicates the number of negative images ranked before the positive for two models, and it goes from the weaker to the stronger model for each pair.
Figure J : Examples of transformed images for robustness analysis, including variations in contrast, brightness, rotation, downscaling, background scaling, blur, tiling, noise, clutter, and occlusion. Each row shows the gradual increase in transformation strength from left to right. These transformations are applied to create positive query–target pairs for controlled robustness evaluation.
Multimodal information retrieval (MMIR) has gained attention for its flexibility in handling text, images, or mixed queries and candidates. Recent breakthroughs in multimodal large language models (MLLMs) boost MMIR performance by incorporating MLLM knowledge under the contrastive finetuning framework. However, they suffer from pre-training inconsistency and require large datasets. In this work, we introduce a novel framework, RetLLM, designed to query MLLMs for MMIR in a training- and data-free manner. Specifically, we formulate MMIR as a similarity score generation task and prompt MLLMs to directly predict retrieval scores in a coarse-then-fine pipeline. At the coarse stage, a top-k filtering strategy builds a small yet high-quality candidate pool for each query, enabling MLLMs to focus on semantically relevant candidates. Subsequently, the retrieval score is predicted by feeding both the query and candidate into MLLMs at the fine stage. Importantly, we propose a visual enhancement module during reasoning to help MLLMs re-pick forgotten visuals, improving retrieval. Extensive experiments on MMIR benchmarks show that RetLLM outperforms fine-tuned models. Ablation studies further verify each component. Our work demonstrates that MLLMs can achieve strong MMIR performance without any training, highlighting their inherent multimodal reasoning ability in a simple, scalable framework. We release our code at: https://github.com/alivecat05/RETLLM
Dawei Su, Dongsheng Wang
College of Computer Science and Software Engineering, Shenzhen University, Shenzhen, China
Due to their strong generalizable multimodal processing and reasoning capabilities, Multimodal Large Language Models (MLLMs) have demonstrated significant potential as universal image retrievers, effectively addressing diverse real-world image retrieval tasks. Nevertheless, pioneering studies, while promising, overlook the potential of fine-grained context modeling and disentangled fine-tuning objectives in enhancing MLLMs' retrieval performance, particularly for complex tasks such as long-text-to-image retrieval, visual dialog retrieval, and composed image retrieval (CIR). Therefore, in this work, we propose an automated fine-grained multimodal quintuple dataset construction pipeline and a novel two-stage fine-grained multimodal fine-tuning strategy. The dataset generation pipeline produces a comprehensive CIR dataset with fine-grained image captions and modification text, facilitating fine-grained context modeling. Beyond the previously entangled fine-tuning paradigm, our approach separates the fine-tuning process into two distinct stages: (1) fine-grained context reasoning-oriented fine-tuning and (2) fine-grained retrieval-oriented fine-tuning. These stages aim to sequentially enhance the model's context understanding and query-target alignment capabilities, thereby improving retrieval performance. Extensive experiments across five datasets encompassing diverse and complex image retrieval tasks demonstrate the remarkable superiority of our method over existing approaches in zero-shot retrieval settings, even with a more lightweight MLLM backbone compared to those methods.
Bohan Hou, Haoqiang Lin, Xuemeng Song +4
Shandong University Qingdao, Shandong, China · City University of Hong Kong Hong kong, China · Harbin Institute of Technology (Shenzhen) Shenzhen, Guangdong, China +2
Multimodal large language models (MLLMs) struggle with fine-grained Visual Search, the task of locating small or rare objects in high-resolution images. Existing remedies fall into two families: (1) Training-free methods based on attention or confidence scores are accurate but slow, since they require multiple MLLM queries per example. (2) Reinforcement Learning (RL) trained tool-use models are faster at inference but opaque, since their tool calls remain uncontrollable and hard to interpret. To overcome this, we propose VisLens (Visual Focus via Logit Lens), a Visual Search method built on the logit lens, which decodes the semantics held in a hidden state by projecting it through the LLM head. VisLens further uses a lightweight tuned-lens that maps early hidden states into the final hidden state space, so visual tokens can be read out from early layers. These tokens are matched to target words in the query to generate a crop of the relevant region, which is fed back in alongside the original image to produce the final answer. The whole process, from decoding to the final answer, completes in a single forward pass without repeated queries. VisLens matches or exceeds prior baselines while delivering a substantial latency advantage, running 8.5--9.9× faster than Thyme and up to 22.2× faster than training-free multi-pass search methods.
Jingyi He, Sanghwan Kim, Zeynep Akata
Technical University of Munich · Helmholtz Munich · Munich Center for Machine Learning