Existing Multimodal Knowledge-Based Visual Question Answering (MKB-VQA) benchmarks suffer from "visual shortcuts", as the query image typically matches the primary subject entity of the target document. We demonstrate that models can exploit these shortcuts, achieving comparable results using visual cues alone. To address this, we introduce Relational Entity Text-Image kNowledge Augmented (RETINA) benchmark, automatically constructed using an LLM-driven pipeline, consisting of 120k training and 2k human-curated test set. RETINA contains queries referencing secondary subjects (i.e. related entities) and pairs them with images of these related entities, removing the visual shortcut. When evaluated on RETINA existing models show significantly degraded performance, confirming their reliance on the shortcut. Furthermore, we propose Multi-Image MultImodal Retriever (MIMIR), which enriches document embeddings by augmenting images of multiple related entities, effectively handling RETINA, unlike prior work that uses only a single image per document. Our experiments validate the limitations of existing benchmarks and demonstrate the effectiveness of RETINA and MIMIR. Our project is available at: Project Page.
Figures & tables
Figure 1: Comparison of Previous MKB-VQA Benchmarks and RETINA. (a) In previous benchmarks, the query image corresponds to the same visual entity ( red ) as (c) the shared GT document, creating a visual shortcut. (b) In RETINA, the query uses a different visual entity ( green ), breaking the visual shortcut.
Benchmark
#Train
#Test
Answer Type
Query Entity
Shortcut-Free
Auto Construction
KVQA Sanket et al. (2019)
165.7K
18.3K
Entity
GT Document Image
✗
✗
ViQuAE Lerner et al. (2022)
1.2K
1.2K
Span
GT Document Image
✗
✗
OVEN Hu et al. (2023)
6M
24.9K
Entity
GT Document Image
✗
✓
InfoSeek Chen et al. (2023)
676K
4.7K
Entity/Span
GT Document Image
✗
✓
EVQA Mensink et al. (2023)
167K
3.7K
Span/Free
GT Document Image
✗
✓
RETINA (Ours)
120K
2K
Entity
Related Entity Image
✓
✓
Table 1: Comparison with Existing MKB-VQA Benchmarks. RETINA is the only benchmark in which the query image depicts a related entity rather than the main entity of the target document, eliminating the visual shortcut.
Figure 2: Preliminary Experiment on Existing Benchmarks. Recall@ k on benchmarks such as Infoseek Chen et al. (2023) and EVQA Mensink et al. (2023) , which exhibit visual shortcuts Deng et al. (2025) . Models fine-tuned with image-only ( blue ) queries versus image+text ( green ) queries.
Figure 3: RETINA Generation Pipeline. (a) constructs one-hop neighborhood graphs by extracting named entities ( gray box) related to the answer entity (white box) and their relations using an LLM (relations are omitted); (b) samples a query entity ( red ) and qualifying entity ( teal ) to form a target subgraph with the answer entity ( green ); and (c) feeds the target subgraph into an LLM to generate a textual query and collect a corresponding image from M2KR Images Lin et al. (2024b) . The query is then paraphrased to minimize lexical overlap with the document.
Figure 4: Overview of MIMIR Document Encoder Architecture. (a) given a document, related named entities are identified with an LLM Bai et al. (2023) , and corresponding images are collected from the KB; (b) textual, global image, and patch-level features are extracted, with patch features attending to textual features through cross-attention to yield multimodal features; and (c) entity token embeddings are incorporated into the textual features prior to cross-attention for richer contextualization; (d) the final document embedding jointly integrates textual, global, and multimodal features projected into the same embedding space.
InfoSeek
EVQA
RETINA
01pt. 01pt. 01pt.
Seen
Unseen
01pt. 01pt. Method
Document Modality
Recall@5
Acc
Recall@5
BEM
Recall@5
BEM
Recall@5
BEM
PreFLMR Lin et al. (2024b)
text
40.7
33.5
63.9
57.0
23.6
19.9
11.1
13.6
UniIR Wei et al. (2024)
text + single-image
34.5
26.4
22.0
25.7
13.8
14.6
4.7
11.2
ReT Caffagni et al. (2025)
text + single-image
50.1
40.4
46.3
43.7
15.1
15.2
6.8
11.4
MuKA Deng et al. (2025)
text + single-image
51.7
41.0
64.9
60.7
20.8
18.7
8.6
12.2
Table 2: Evaluating Robustness Against Visual Shortcuts . Retrieval and QA performance are reported. All models are jointly trained on EVQA and RETINA. The GT Document row represents an upper bound using gold knowledge. MIMIR consistently outperforms prior methods in both settings with and without visual shortcuts, showing the benefit of leveraging multi-image context over single-image or text-only approaches.
InfoSeek
EVQA
RETINA
Train
Method
Seen
Unseen
R@5
R@5
R@5
R@5
InfoSeek
PreFLMR
43.3
62.4
18.5
7.9
MuKA
54.0
63.8
17.4
7.7
MIMIR
53.9
64.3
35.2
32.7
EVQA
PreFLMR
41.2
64.4
19.9
9.0
Table 3: Cross-dataset Transfer. Rows indicate the training source, columns the evaluation target.
Figure 5: Image-only vs. Multimodal retrieval. On EVQA (left), image-only MuKA ( blue ) nearly matches the multimodal model ( green ), revealing the visual shortcut. On RETINA (right), MIMIR shows a substantial gap, indicating that multimodal processing is required.
Figure 6: Qualitative Comparison of Retrieval on RETINA Benchmark. Results for (a) MuKA Deng et al. (2025) and (b) MIMIR. GT documents are in green ; hard distractors containing the query image entity are in red ; query entity in GT documents are highlighted in green . While MuKA is biased toward visual shortcuts and often retrieves documents containing images similar to the query, MIMIR leverages related images (Fraxinus americana) and (Nunavut), which are augmented to the document embeddings.
InfoSeek
EVQA
RETINA
Method
Size-matched aug.
Seen
Unseen
PreFLMR
–
40.7
63.9
23.6
11.1
PreFLMR
+ repr.
41.0
64.2
24.2
12.4
PreFLMR
+ related passages
27.3
43.1
27.2
14.7
MuKA
–
51.7
64.9
20.8
8.6
MuKA
+ repr.
52.0
64.8
21.2
8.9
Table 4: Capacity-Controlled Comparison (R@5). Each control enlarges the per-document representation to MIMIR’s index size without adding related-entity visual content.
InfoSeek
EVQA
RETINA
Method
Seen
Unseen
Baseline (MuKA Deng et al. (2025) )
51.7
64.9
20.8
8.6
+ MI
51.5
65.1
38.1
33.3
+ MMF
51.6
65.3
41.1
34.5
+ ETE (Full MIMIR)
53.3
65.8
43.2
36.9
Table 5: MIMIR Component Ablation. Cumulative impact of components on retrieval performance: baseline, +MI (Multiple Images), +MMF (Multimodal Features), and +ETE (Entity Token Embedding).
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Example of a Valid Question Filtered Out due to Evaluation Constraints. While both the question and its answer are factually valid, multiple duck species may be as closely related to the mandarin duck (green) as the wood duck (red) is. This ambiguity complicates strict automated evaluation, which is why such samples are filtered from the test set.
Value
Number of Main Entities
19,408
Number of Extracted Relations
66,244
Avg. Train Question Length (tokens)
26.4
Avg. Train Answer Length (tokens)
5.8
Avg. Test Question Length (tokens)
28.0
Avg. Test Answer Length (tokens)
5.5
Appendix
Table 6: Summary of Dataset Statistics.
Method
Document Modality
InfoSeek
EVQA
RETINA (Seen)
RETINA (Unseen)
PreFLMR Lin et al. (2024b)
text
40.7
63.9
23.6
11.1
UniIR Wei et al. (2024)
text + single-image
34.5
22.0
13.8
4.7
ReT Caffagni et al. (2025)
text + single-image
50.1
46.3
15.1
6.8
MuKA Deng et al. (2025)
text + single-image
51.7
64.9
20.8
8.6
MIMIR (Ours)
text + multi-image
53.3 ( ±0.5 )
65.8 ( ±0.4 )
43.2 ( ±0.5 )
36.9 ( ±0.6 )
Appendix
Table 7: Standard deviation analysis . Average retrieval performance and standard deviations of MIMIR are reported.
Model
Retrieval
Seen
Unseen
VILA-13B Lin et al. (2024a)
✗
11.2
10.1
✓
32.7
31.6
InternVL3-8B Zhu et al. (2025)
✗
18.7
17.6
✓
41.3
39.2
Qwen3-VL-8B Yang et al. (2025)
✗
20.3
19.4
✓
45.7
43.9
Appendix
Table 8: MIMIR with Diverse VLM Backbones. QA performance (BEM Bulian et al. (2022) ) with and without retrieval.
(a) Image selection strategy (Recall@5)
Strategy
K=1
K=2
K=3
K=4
Random
22.5
27.8
35.9
39.2
Frequent
24.7
28.4
38.4
41.3
(b) Noise robustness (Recall@5)
Noise rate
0%
10%
20%
30%
MIMIR
43.2
42.8
40.9
36.8
Appendix
Table 9: Image Selection and Noise Robustness on RETINA. (a) Both selection strategies improve consistently with K , with frequency-based selection slightly ahead. (b) MIMIR degrades gracefully under noisy related-entity images, remaining stable up to 20%. Noise replaces the given fraction of related-entity images with images of unrelated entities while keeping the number of images per document, the architecture, and training fixed, so it varies only the correctness of the entity association.
InfoSeek
EVQA
RETINA
Method
Seen
Unseen
RAG-Anything Guo et al. (2025)
34.0
27.7
21.3
19.2
mKG-RAG Yuan et al. (2025)
45.7
35.8
25.8
22.8
mKG-RAG + MuKA
48.4
37.2
28.2
24.9
mKG-RAG + MIMIR
47.9
38.6
32.6
29.3
MIMIR
53.3
65.8
43.2
36.9
Appendix
Table 10: Comparison with Multimodal KG-RAG Methods (R@5) on the same KB under a shared top-5 retrieval protocol.
Figure 8: Robustness to Visual-similar Distractors. MIMIR ( green ) and MuKA ( red ) curves show recall for distractor (dotted) and GT (solid) documents. Distractors are non-GT documents whose main entity matches the query image’s entity.
Figure 9: Qualitative Examples of RETINA Training Set before Human Curation.
Figure 10: Entity Extraction Prompt.
Figure 11: Question Generation Prompt.
Figure 12: Paraphrasing Prompt.
Figure 13: VILA Training Prompt.
Figure 14: Retriever Instruction Prompts.
Figure 15: Human Curation User Interface An image, document title, a question, and an answer are presented for curation, along with guidelines for identifying common issues.
Knowledge-based visual question answering combines image understanding with external or learned knowledge to answer questions requiring facts beyond the image.
We present M3-VQA, a novel knowledge-based Visual Question Answering (VQA) benchmark, to enhance the evaluation of multimodal large language models (MLLMs) in fine-grained multimodal entity understanding and complex multi-hop reasoning. Unlike existing VQA datasets that focus on coarse-grained categories and simple reasoning over single entities, M3-VQA introduces diverse multi-entity questions involving multiple distinct entities from both visual and textual sources. It requires models to perform both sequential and parallel multi-hop reasoning across multiple documents, supported by traceable, detailed evidence and a curated multimodal knowledge base. We evaluate 16 leading MLLMs under three settings: without external knowledge, with gold evidence, and with retrieval-augmented input. The poor results reveal significant challenges for MLLMs in knowledge acquisition and reasoning. Models perform poorly without external information but improve markedly when provided with precise evidence. Furthermore, reasoning-aware agentic retrieval surpasses heuristic methods, highlighting the importance of structured reasoning for complex multimodal understanding. M3-VQA presents a more challenging evaluation for advancing the multimodal reasoning capabilities of MLLMs. Our code and dataset are available at https://github.com/CASIA-IVA-Lab/M3VQA.
Jiatong Ma, Longteng Guo, Yuchen Liu +4
Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences
Knowledge-based Visual Question Answering (KB-VQA) requires Multimodal Large Language Models (MLLMs) to identify and combine fine-grained visual cues with retrieved textual evidence. However, retrieval often introduces noisy and partially relevant content, while images contain distracting visual regions, causing pretrained MLLMs to overlook the evidence that actually supports the answer. To address this, we introduce Look Twice (LoT), a training-free inference-time framework that turns the model's own internal attention into an explicit multimodal evidence-selection mechanism. LoT first leverages the model's internal attention patterns to identify query-relevant image regions and textual sentences, filters attention sinks and distracting content, and reformulates the input to explicitly highlight the selected evidence before answer generation. The method requires no parameter updates, auxiliary models, or architectural modifications. Across four KB-VQA benchmarks and ten off-the-shelf MLLMs ranging from 2B to 38B parameters, LoT improves every evaluated backbone, with average gains of up to +12.5 accuracy points. It also provides further gains when combined with established context-refinement strategies, yielding additional improvements over already refined inputs. These results establish LoT as a general and effective mechanism for enabling pretrained MLLMs to exploit available multimodal evidence more accurately. Source code is publicly available at https://aimagelab.github.io/LoT/.
Knowledge-Based Visual Question Answering (KB-VQA) relies on retrieving external information to answer queries involving long-tail entities. However, existing retrieval pipelines predominantly employ CLIP-style dual encoders, which prioritize surface-level visual similarity over entity-level semantic alignment. This paradigm often fails when semantically identical concepts exhibit large visual variations or when distinct entities appear visually similar. To address this, we propose KBMR, the first MLLM-based embedding retriever tailored for KB-VQA. Leveraging the robust autoregressive capabilities of MLLMs, KBMR maps images into a semantic space that better preserves concept identity. To tackle the challenge of noisy supervision in Wikipedia-scale retrieval, we introduce an MLLM-based semantic discriminator that generates continuous entity-consistency weights. These weights guide a novel continuous semantic distillation objective, enabling effective hard negative sampling and soft supervision beyond rigid binary labels. Extensive experiments demonstrate that KBMR significantly outperforms CLIP baselines, yielding up to a 14.7% improvement in retrieval Recall@1 and a 9.4% gain in end-to-end VQA accuracy. Code is available at https://github.com/realHarryX/KBMR.
Hangrui Xu, Zhengxian Wu, Yunyao Yu +6
Shenzhen International Graduate School, Tsinghua University Shenzhen, China · University of Arizona Tucson, USA