CLIMB: Confidence-Guided Complementary Evidence for Multimodal Retrieval-Augmented Generation

Hang Gao; Wujiang Xu; Zhixing Zhang; Kai Mei; Jingyi Yang; Dimitris N. Metaxas

cs.CLOct 2, 2026

CLIMB: Confidence-Guided Complementary Evidence for Multimodal Retrieval-Augmented Generation

Authors: Hang Gao, Wujiang Xu, Zhixing Zhang, Kai Mei, Jingyi Yang, Dimitris N. Metaxas

Organizations: Rutgers University · Google

Abstract

Multimodal large language models (MLLMs) have shown strong visual reasoning abilities, but knowledge-intensive visual question answering often requires external textual evidence beyond the image and the model's parametric knowledge. Existing multimodal RAG systems commonly rely on Top- $K$ retrieval or reranking, which may return redundant passages and provide limited control over whether an answer update is sufficiently supported by the retrieved evidence. We propose CLIMB, a training-free inference-time framework for multimodal RAG. CLIMB first constructs a compact complementary evidence pool using an MMR-style objective that balances query relevance and passage-level redundancy. It then performs confidence-controlled refinement within this fixed pool: an R/E/C critic scores passages by relevance, evidence specificity, and cross-modal alignment, while an evidence-grounded confidence estimator accepts an updated answer only when the estimated confidence increases. This design provides a simple stopping criterion and reduces unnecessary refinement without modifying the underlying retriever or MLLM. Experiments on Encyclopedic-VQA and InfoSeek show that CLIMB consistently improves over retrieval-augmented multimodal baselines. Ablations further indicate that complementary pooling, critic-based scoring, and iterative confidence-controlled refinement each contribute to the final performance.

Figures & tables

Figure 1: Comparison of multimodal RAG approaches. (a) Standard methods Cocchi et al. (2025) ; Caffagni et al. (2024b) ; Yan and Xie (2024) rely on Top-K similarity selection, which often produces redundant evidence with limited coverage and lacks explicit quality control for answer generation. (b) Our CLIMB framework addresses these limitations through two key innovations: diversity-aware evidence construction that balances relevance and complementarity to build a non-redundant passage pool, and confidence-controlled decoding that uses multi-faceted scoring and monotonic confidence constraints to ensure reliable answer updates.

Figure 2: Overview of CLIMB . Given an image and a question, CLIMB retrieve a small candidate set from the knowledge base and build a complementary evidence pool via similarity–diversity selection. An R/E/C critic scores passages, and a frozen MLLM generates answers from the top- $M$ evidence while a confidence estimator and gap diagnosis drive an iterative refinement loop, accepting updates only when confidence increases.

Method	E-VQA	InfoSeek
Full	34.8	39.1
w/o Facet Diagnosis	33.7	38.7
w/o Complementary Evidence Pool	32.4	38.1
w/o Critic	26.9	33.4
w/o Iteration	25.8	33.0

Table 2: Ablation study results on InfoSeek and E-VQA.

Confidence bin	Accuracy
$U<0.70$	32.1
$0.70\leq U<0.80$	34.3
$0.80\leq U<0.90$	38.7
$U\geq 0.90$	40.2

Table 3: Confidence-binned accuracy on InfoSeek under the default CLIMB setting.

Figure 3: Sensitivity scores for hyperparameters $U_{STAR}$ , $K$ , $\lambda$ , measured as the accuracy range across different parameter settings.

Panel of Figure 4 — Figure 4: Hyperparameter sensitivity of CLIMB . (a) Accuracy across different maximum iteration $T_{\max}$ . (b) Aggregated sensitivity scores across key hyperparameters, indicating that the framework is stable within a broad range of settings.

Figure 5: Question–Image pairs from the InfoSeek, shown with answers produced by GPT-5.2 and CLIMB .

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Step	Evidence focus	Diagnosis / action	Model answer	Failure signal
Question: “What is the closest parent taxonomy of this animal?” Gold answer: Loriinae .
Initial generation	Evidence mentions Loriini as a tribe and emphasizes the broader parrot family Psittaculidae .	Generate an answer from the initial selected subset.	Psittaculidae	The answer is related and specific, but it is a broader ancestor rather than the closest parent.
Facet diagnosis	Current evidence supports taxonomy but does not clearly resolve the nearest parent.	Diagnose the missing facet as “taxonomic parentage”.	–	The diagnosis is reasonable, so the error is not simply a failed facet proposal.
Pool-internal refresh	Additional passages contain adjacent taxonomy levels: family Psittaculidae , subfamily Loriinae , and tribe Loriini .	Refresh evidence inside the fixed pool and regenerate.	Psittaculidae	Evidence is more complete and internally consistent, but still permits the wrong granularity.
Acceptance	The refined answer appears supported by concrete taxonomy terms.	Accept the update because estimated confidence increases.	Final: Psittaculidae .	Confidence can increase despite incorrectness because specificity and consistency do not guarantee closest-parent resolution.

Appendix

Table 4: A failure analysis from InfoSeek.

Multimodal RAG

Multimodal retrieval-augmented generation grounds AI answers in retrieved text, images, video, audio, and structured visual evidence.

See all 158 papers

CrossModalQA: A Cross-modal and Multi-hop Benchmark for Multimodal Retrieval-augmented Generation
Aug 31, 2026Jiacheng Cai, Zijin Hong, Zheng Yuan +3Multimodal RAG Multimodal QA
The Hong Kong Polytechnic University · Jilin University
V-Retrver: Evidence-Driven Agentic Reasoning for Universal Multimodal Retrieval
Feb 5, 2026Dongyang Chen, Chaoyang Wang, Dezhao Su +6Multimodal IR Multimodal Grounding
Tsinghua University · University of Central Florida · Fudan University +4
Utility-Oriented Visual Evidence Selection for Multimodal Retrieval-Augmented Generation
May 13, 2026Weiqing Luo, Zongye Hu, Xiao Wang +3Multimodal RAG Evidence Selection
♦Arizona State University · ♥Texas A&M University · ♠Morgan Stanley

Aug 31, 2026J/K move · Enter open · S save

CrossModalQA: A Cross-modal and Multi-hop Benchmark for Multimodal Retrieval-augmented Generation

Jiacheng Cai, Zijin Hong, Zheng Yuan, Huachi Zhou+2

The Hong Kong Polytechnic University · Jilin University

Multimodal RAG Multimodal QA Cross-Modal Retrieval

Open paper AskPDF arXiv

Despite the strong capabilities of multimodal large language models (MLLMs), their parametric knowledge remains incomplete and difficult to update, motivating multimodal retrieval-augmented generation (RAG) to ground responses in external text and images. However, existing benchmarks face two major limitations: (i) they typically emphasize single-hop retrieval or reasoning over a small set of provided contexts rather than open-domain evidence discovery; and (ii) they provide fragmented coverage of cross-modal reasoning paths, leaving complex multi-hop and multi-image reasoning underexplored. In this paper, we introduce CrossModalQA, an open-domain benchmark for evaluating multimodal retrieval and reasoning over heterogeneous corpora. CrossModalQA contains 1,863 question-answer pairs constructed from 4,987 Wikipedia articles and 4,431 Wikimedia Commons images. It covers five complementary reasoning paths: vision-to-text, text-to-vision, vision-to-text-to-vision, multi-image intersection, and image-set reasoning. Every question requires retrieving and composing distributed textual and visual evidence, with an average reasoning depth of 3.50 hops. We construct the benchmark through multimodal knowledge graph-guided subgraph sampling and apply rule-based consistency checking and LLM verification to ensure multimodal dependence and traceable evidence. Extensive experiments demonstrate that existing multimodal RAG systems struggle to recover complete evidence chains and can underperform closed-book models when incomplete retrieval introduces distracting context. Further analysis reveals that complete cross-modal retrieval contributes more to answer accuracy than generator scaling, while multi-image retrieval and reasoning remain the primary bottlenecks limiting end-to-end performance.

Abstract

Multimodal RAG

Explore similar work

CrossModalQA: A Cross-modal and Multi-hop Benchmark for Multimodal Retrieval-augmented Generation

V-Retrver: Evidence-Driven Agentic Reasoning for Universal Multimodal Retrieval

Utility-Oriented Visual Evidence Selection for Multimodal Retrieval-Augmented Generation