BRIDGE: Multimodal-to-Text Retrieval via Reinforcement-Learned Query Alignment

Mohamed Darwish Mounis; Mohamed Mahmoud; Shaimaa Sedek; Mahmoud Abdalla; Mahmoud SalahEldin Kasem; Abdelrahman Abdallah; Hyun-Soo Kang

cs.IRApr 8, 2026

BRIDGE: Multimodal-to-Text Retrieval via Reinforcement-Learned Query Alignment

Authors: Mohamed Darwish Mounis, Mohamed Mahmoud, Shaimaa Sedek, Mahmoud Abdalla, Mahmoud SalahEldin Kasem, Abdelrahman Abdallah, Hyun-Soo Kang

Organizations: High institute for computer & information systems · Chungbuk National University · Assiut University · University of Innsbruck

Abstract

Multimodal retrieval systems struggle to resolve image-text queries against text-only corpora: the best vision-language encoder achieves only 27.6 nDCG@10 on MM-BRIGHT, underperforming strong text-only retrievers. We argue the bottleneck is not the retriever but the query -- raw multimodal queries entangle visual descriptions, conversational noise, and retrieval intent in ways that systematically degrade embedding similarity. We present BRIDGE, a two-component system that resolves this mismatch without multimodal encoders. FORGE (Focused Retrieval Query Generator) is a query alignment model trained via reinforcement learning, which distills noisy multimodal queries into compact, retrieval-optimized search strings. LENS (Language-Enhanced Neural Search) is a reasoning-enhanced dense retriever fine-tuned on reasoning-intensive retrieval data to handle the intent-rich queries FORGE produces. Evaluated on MM-BRIGHT (2,803 queries, 29 domains), BRIDGE achieves 29.7 nDCG@10, surpassing all multimodal encoder baselines including Nomic-Vision (27.6). When FORGE is applied as a plug-and-play aligner on top of Nomic-Vision, the combined system reaches 33.3 nDCG@10 -- exceeding the best text-only retriever (32.2) -- demonstrating that query alignment is the key bottleneck in multimodal-to-text retrieval. https://github.com/mm-bright/multimodal-reasoning-retrieval

Figures & tables

Figure 1: An example of BRIDGE in action. The raw multimodal query mixes conversational context and visual content, producing a noisy embedding that fails to retrieve the correct document. FORGE distills the query into a compact, intent-focused search string that LENS successfully resolves against the text corpus.

Figure 2: Overview of the BRIDGE framework. Given a multimodal query $(q_{t},q_{v})$ , GPT-4o first generates a dense image caption $\delta(q_{v})$ . FORGE then receives the concatenated input $(q_{t},\delta(q_{v}))$ and distills it into a compact, retrieval-optimized query $\hat{q}$ via reinforcement-learned alignment. LENS encodes $\hat{q}$ and retrieves the final ranked documents from the text corpus. No multimodal encoder is used at retrieval time.

STEM & Life Sciences
DOMAIN	BGE-VL	CLIP	GME-2B	GME-7B	JINA-CLIP	NOMIC	SIGLIP	BRIDGE
Acad	4.2	4.8	16.2	27.6	22.3	22.6	3.6	27.7
Bio	5.7	14.8	22.9	15.2	20.5	26.9	11.9	41.1
Chem	10.8	9.6	27.2	21.9	30.6	30.6	11.6	34.3
Phys	6.8	6.1	13.3	14.0	14.4	17.2	7.3	20.7
Math	13.1	17.9	16.4	9.3	27.0	34.0	15.3	31.9

Table 1: Per-domain nDCG@10 on MM-BRIGHT (multimodal-to-text track) across all 29 domains, grouped by thematic category. BRIDGE achieves the highest score in 22 out of 29 domains. Bold denotes the best score per row.

Figure 3: Effect of FORGE LLM backbone on BRIDGE performance (nDCG@10 averaged over 29 MM-BRIGHT domains). Llama-3.2-11B achieves the strongest result (31.0), outperforming both larger models (Qwen2.5-72B: 29.3) and the GPT-4o zero-shot baseline (29.6). Our default FORGE backbone (Qwen2.5-7B, 29.7 ) offers the best performance-to-cost tradeoff.

Configuration	nDCG@10
LENS only (raw $q_{t}$ , no image)	25.4
LENS + Caption (raw $q_{t}+\delta(q_{v})$ )	27.8
LENS + FORGE-SFT (supervised rewriter)	28.5
LENS + FORGE-RL (full BRIDGE)	29.7

Table 2: Component ablation on MM-BRIGHT. Each row adds one component on top of the previous. Default BRIDGE configuration in bold .

Base Retriever	Base	+FORGE	$\boldsymbol{\Delta}$
BM25	8.5	15.8	+7.3
GME-Qwen2-VL-7B	22.0	28.3	+6.3
Nomic-Vision	27.6	33.3	+5.7
LENS (ours)	25.4	29.7	+4.3

Table 3: FORGE applied to different base retrievers on MM-BRIGHT.

\Delta

= absolute nDCG@10 gain from FORGE alignment.

Query Method	nDCG@10
Raw $q_{t}$ (no rewrite)	25.4
HyDE [ 13 ]	11.2
Query2Doc [ 41 ]	14.5
FORGE-RL (ours)	29.7

Table 4: Comparison of query rewriting strategies on MM-BRIGHT, all using LENS as the base retriever. FORGE-RL denotes our full RL-trained aligner.

Multimodal IR

Multimodal information retrieval finds relevant images, videos, documents, audio, or other media using queries that may combine different modalities.

See all 128 papers

Beyond Global Similarity: Towards Fine-Grained, Multi-Condition Multimodal Retrieval
Mar 1, 2026Xuan Lu, Kangle Li, Haohang Huang +3Multimodal IR Cross-Modal Retrieval
Shanghai Jiao Tong University · Institute of Digital Twin, Eastern Institute of Technology, Ningbo · Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative +1
Multimodal Representation Alignment for Cross-modal Information Retrieval
Jun 10, 2025Fan Xu, Luis A. LeivaCross-Modal Learning Multimodal IR
aDepartment of Computer Science, University of Luxembourg, 6, avenue de la Fonte, Esch-sur-Alzette, L-4364, Luxembourg
UniCA: Bi-directional Cross-Attention with Positive Similarity Loss for Robust Multi-Modal Retrieval
Jun 3, 2026Yini Huang, Wenlong ZhangMultimodal IR Cross-Modal Alignment
The Hong Kong University of Science and Technology (Guangzhou), Guangdong, China · Southern Medical University, Guangdong, China

Mar 1, 2026J/K move · Enter open · S save

Beyond Global Similarity: Towards Fine-Grained, Multi-Condition Multimodal Retrieval

Xuan Lu, Kangle Li, Haohang Huang, Rui Meng+2

Shanghai Jiao Tong University · Institute of Digital Twin, Eastern Institute of Technology, Ningbo · Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative+1

Multimodal IR Cross-Modal Retrieval Multimodal Model Evaluation Multimodal Reranking

Open paper AskPDF arXiv

Recent advances in multimodal large language models (MLLMs) have substantially expanded the capabilities of multimodal retrieval, enabling systems to align and retrieve information across visual and textual modalities. Yet, existing benchmarks largely focus on coarse-grained or single-condition alignment, overlooking real-world scenarios where user queries specify multiple interdependent constraints across modalities. To bridge this gap, we introduce MCMR (Multi-Conditional Multimodal Retrieval): a large-scale benchmark designed to evaluate fine-grained, multi-condition cross-modal retrieval under natural-language queries. MCMR spans five product domains: upper and bottom clothing, jewelry, shoes, and furniture. It also preserves rich long-form metadata essential for compositional matching. Each query integrates complementary visual and textual attributes, requiring models to jointly satisfy all specified conditions for relevance. We benchmark a diverse suite of MLLM-based multimodal retrievers and vision-language rerankers to assess their condition-aware reasoning abilities. Experimental results reveal: (i) distinct modality asymmetries across models; (ii) visual cues dominate early-rank precision, while textual metadata stabilizes long-tail ordering; and (iii) MLLM-based pointwise rerankers markedly improve fine-grained matching by explicitly verifying query-candidate consistency. Overall, MCMR establishes a challenging and diagnostic benchmark for advancing multimodal retrieval toward compositional, constraint-aware, and interpretable understanding. Our code and dataset is available at https://github.com/EIT-NLP/MCMR

Abstract

Multimodal IR

Explore similar work

Beyond Global Similarity: Towards Fine-Grained, Multi-Condition Multimodal Retrieval

Multimodal Representation Alignment for Cross-modal Information Retrieval

UniCA: Bi-directional Cross-Attention with Positive Similarity Loss for Robust Multi-Modal Retrieval