Multimodal large language models (MLLMs) have shown strong visual reasoning abilities, but knowledge-intensive visual question answering often requires external textual evidence beyond the image and the model's parametric knowledge. Existing multimodal RAG systems commonly rely on Top-K retrieval or reranking, which may return redundant passages and provide limited control over whether an answer update is sufficiently supported by the retrieved evidence. We propose CLIMB, a training-free inference-time framework for multimodal RAG. CLIMB first constructs a compact complementary evidence pool using an MMR-style objective that balances query relevance and passage-level redundancy. It then performs confidence-controlled refinement within this fixed pool: an R/E/C critic scores passages by relevance, evidence specificity, and cross-modal alignment, while an evidence-grounded confidence estimator accepts an updated answer only when the estimated confidence increases. This design provides a simple stopping criterion and reduces unnecessary refinement without modifying the underlying retriever or MLLM. Experiments on Encyclopedic-VQA and InfoSeek show that CLIMB consistently improves over retrieval-augmented multimodal baselines. Ablations further indicate that complementary pooling, critic-based scoring, and iterative confidence-controlled refinement each contribute to the final performance.
Figures & tables
Figure 1: Comparison of multimodal RAG approaches. (a) Standard methods Cocchi et al. (2025) ; Caffagni et al. (2024b) ; Yan and Xie (2024) rely on Top-K similarity selection, which often produces redundant evidence with limited coverage and lacks explicit quality control for answer generation. (b) Our CLIMB framework addresses these limitations through two key innovations: diversity-aware evidence construction that balances relevance and complementarity to build a non-redundant passage pool, and confidence-controlled decoding that uses multi-faceted scoring and monotonic confidence constraints to ensure reliable answer updates.
Figure 2: Overview of CLIMB . Given an image and a question, CLIMB retrieve a small candidate set from the knowledge base and build a complementary evidence pool via similarity–diversity selection. An R/E/C critic scores passages, and a frozen MLLM generates answers from the top- M evidence while a confidence estimator and gap diagnosis drive an iterative refinement loop, accepting updates only when confidence increases.
Method
E-VQA
InfoSeek
Full
34.8
39.1
w/o Facet Diagnosis
33.7
38.7
w/o Complementary Evidence Pool
32.4
38.1
w/o Critic
26.9
33.4
w/o Iteration
25.8
33.0
Table 2: Ablation study results on InfoSeek and E-VQA.
Confidence bin
Accuracy
U<0.70
32.1
0.70≤U<0.80
34.3
0.80≤U<0.90
38.7
U≥0.90
40.2
Table 3: Confidence-binned accuracy on InfoSeek under the default CLIMB setting.
Figure 3: Sensitivity scores for hyperparameters USTAR , K , λ , measured as the accuracy range across different parameter settings.
Figure 4: Hyperparameter sensitivity of CLIMB . (a) Accuracy across different maximum iteration Tmax . (b) Aggregated sensitivity scores across key hyperparameters, indicating that the framework is stable within a broad range of settings.
Figure 5: Question–Image pairs from the InfoSeek, shown with answers produced by GPT-5.2 and CLIMB .
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Step
Evidence focus
Diagnosis / action
Model answer
Failure signal
Question: “What is the closest parent taxonomy of this animal?” Gold answer: Loriinae .
Initial generation
Evidence mentions Loriini as a tribe and emphasizes the broader parrot family Psittaculidae .
Generate an answer from the initial selected subset.
Psittaculidae
The answer is related and specific, but it is a broader ancestor rather than the closest parent.
Facet diagnosis
Current evidence supports taxonomy but does not clearly resolve the nearest parent.
Diagnose the missing facet as “taxonomic parentage”.
–
The diagnosis is reasonable, so the error is not simply a failed facet proposal.
Pool-internal refresh
Additional passages contain adjacent taxonomy levels: family Psittaculidae , subfamily Loriinae , and tribe Loriini .
Refresh evidence inside the fixed pool and regenerate.
Psittaculidae
Evidence is more complete and internally consistent, but still permits the wrong granularity.
Acceptance
The refined answer appears supported by concrete taxonomy terms.
Accept the update because estimated confidence increases.
Final: Psittaculidae .
Confidence can increase despite incorrectness because specificity and consistency do not guarantee closest-parent resolution.
Despite the strong capabilities of multimodal large language models (MLLMs), their parametric knowledge remains incomplete and difficult to update, motivating multimodal retrieval-augmented generation (RAG) to ground responses in external text and images. However, existing benchmarks face two major limitations: (i) they typically emphasize single-hop retrieval or reasoning over a small set of provided contexts rather than open-domain evidence discovery; and (ii) they provide fragmented coverage of cross-modal reasoning paths, leaving complex multi-hop and multi-image reasoning underexplored. In this paper, we introduce CrossModalQA, an open-domain benchmark for evaluating multimodal retrieval and reasoning over heterogeneous corpora. CrossModalQA contains 1,863 question-answer pairs constructed from 4,987 Wikipedia articles and 4,431 Wikimedia Commons images. It covers five complementary reasoning paths: vision-to-text, text-to-vision, vision-to-text-to-vision, multi-image intersection, and image-set reasoning. Every question requires retrieving and composing distributed textual and visual evidence, with an average reasoning depth of 3.50 hops. We construct the benchmark through multimodal knowledge graph-guided subgraph sampling and apply rule-based consistency checking and LLM verification to ensure multimodal dependence and traceable evidence. Extensive experiments demonstrate that existing multimodal RAG systems struggle to recover complete evidence chains and can underperform closed-book models when incomplete retrieval introduces distracting context. Further analysis reveals that complete cross-modal retrieval contributes more to answer accuracy than generator scaling, while multi-image retrieval and reasoning remain the primary bottlenecks limiting end-to-end performance.
Jiacheng Cai, Zijin Hong, Zheng Yuan +3
The Hong Kong Polytechnic University · Jilin University
Multimodal Large Language Models (MLLMs) have recently been applied to universal multimodal retrieval, where Chain-of-Thought (CoT) reasoning improves candidate reranking. However, existing approaches remain largely language-driven, relying on static visual encodings and lacking the ability to actively verify fine-grained visual evidence, which often leads to speculative reasoning in visually ambiguous cases. We propose V-Retrver, an evidence-driven retrieval framework that reformulates multimodal retrieval as an agentic reasoning process grounded in visual inspection. V-Retrver enables an MLLM to selectively acquire visual evidence during reasoning via external visual tools, performing a multimodal interleaved reasoning process that alternates between hypothesis generation and targeted visual verification.To train such an evidence-gathering retrieval agent, we adopt a curriculum-based learning strategy combining supervised reasoning activation, rejection-based refinement, and reinforcement learning with an evidence-aligned objective. Experiments across multiple multimodal retrieval benchmarks demonstrate consistent improvements in retrieval accuracy (with 23.0% improvements on average), perception-driven reasoning reliability, and generalization.
Dongyang Chen, Chaoyang Wang, Dezhao Su +6
Tsinghua University · University of Central Florida · Fudan University +4
Visual evidence selection is a critical component of multimodal retrieval-augmented generation (RAG), yet existing methods typically rely on semantic relevance or surface-level similarity, which are often misaligned with the actual utility of visual evidence for downstream reasoning. We reformulate multimodal evidence selection from an information-theoretic perspective by defining evidence utility as the information gain induced on a model's output distribution. To overcome the intractability of answer-space optimization, we introduce a latent notion of evidence helpfulness and theoretically show that, under mild assumptions, ranking evidence by information gain on this latent variable is equivalent to answer-space utility. We further propose a training-free, surrogate-accelerated framework that efficiently estimates evidence utility using lightweight multimodal models. Experiments on MRAG-Bench and Visual-RAG across multiple model families demonstrate that our method consistently outperforms state-of-the-art RAG baselines while achieving substantial reductions in computational cost.
Weiqing Luo, Zongye Hu, Xiao Wang +3
♦Arizona State University · ♥Texas A&M University · ♠Morgan Stanley