Existing Multimodal Knowledge-Based Visual Question Answering (MKB-VQA) benchmarks suffer from "visual shortcuts", as the query image typically matches the primary subject entity of the target document. We demonstrate that models can exploit these shortcuts, achieving comparable results using visual cues alone. To address this, we introduce Relational Entity Text-Image kNowledge Augmented (RETINA) benchmark, automatically constructed using an LLM-driven pipeline, consisting of 120k training and 2k human-curated test set. RETINA contains queries referencing secondary subjects (i.e. related entities) and pairs them with images of these related entities, removing the visual shortcut. When evaluated on RETINA existing models show significantly degraded performance, confirming their reliance on the shortcut. Furthermore, we propose Multi-Image MultImodal Retriever (MIMIR), which enriches document embeddings by augmenting images of multiple related entities, effectively handling RETINA, unlike prior work that uses only a single image per document. Our experiments validate the limitations of existing benchmarks and demonstrate the effectiveness of RETINA and MIMIR. Our project is available at: Project Page.
Figures & tables
Figure 1: Comparison of Previous MKB-VQA Benchmarks and RETINA. (a) In previous benchmarks, the query image corresponds to the same visual entity ( red ) as (c) the shared GT document, creating a visual shortcut. (b) In RETINA, the query uses a different visual entity ( green ), breaking the visual shortcut.
Benchmark
#Train
#Test
Answer Type
Query Entity
Shortcut-Free
Auto Construction
KVQA Sanket et al. (2019)
165.7K
18.3K
Entity
GT Document Image
✗
✗
ViQuAE Lerner et al. (2022)
1.2K
1.2K
Span
GT Document Image
✗
✗
OVEN Hu et al. (2023)
6M
24.9K
Entity
GT Document Image
✗
✓
InfoSeek Chen et al. (2023)
676K
4.7K
Entity/Span
GT Document Image
✗
✓
EVQA Mensink et al. (2023)
167K
3.7K
Span/Free
GT Document Image
✗
✓
RETINA (Ours)
120K
2K
Entity
Related Entity Image
✓
✓
Table 1: Comparison with Existing MKB-VQA Benchmarks. RETINA is the only benchmark in which the query image depicts a related entity rather than the main entity of the target document, eliminating the visual shortcut.
Figure 2: Preliminary Experiment on Existing Benchmarks. Recall@ k on benchmarks such as Infoseek Chen et al. (2023) and EVQA Mensink et al. (2023) , which exhibit visual shortcuts Deng et al. (2025) . Models fine-tuned with image-only ( blue ) queries versus image+text ( green ) queries.
Figure 3: RETINA Generation Pipeline. (a) constructs one-hop neighborhood graphs by extracting named entities ( gray box) related to the answer entity (white box) and their relations using an LLM (relations are omitted); (b) samples a query entity ( red ) and qualifying entity ( teal ) to form a target subgraph with the answer entity ( green ); and (c) feeds the target subgraph into an LLM to generate a textual query and collect a corresponding image from M2KR Images Lin et al. (2024b) . The query is then paraphrased to minimize lexical overlap with the document.
Figure 4: Overview of MIMIR Document Encoder Architecture. (a) given a document, related named entities are identified with an LLM Bai et al. (2023) , and corresponding images are collected from the KB; (b) textual, global image, and patch-level features are extracted, with patch features attending to textual features through cross-attention to yield multimodal features; and (c) entity token embeddings are incorporated into the textual features prior to cross-attention for richer contextualization; (d) the final document embedding jointly integrates textual, global, and multimodal features projected into the same embedding space.
InfoSeek
EVQA
RETINA
01pt. 01pt. 01pt.
Seen
Unseen
01pt. 01pt. Method
Document Modality
Recall@5
Acc
Recall@5
BEM
Recall@5
BEM
Recall@5
BEM
PreFLMR Lin et al. (2024b)
text
40.7
33.5
63.9
57.0
23.6
19.9
11.1
13.6
UniIR Wei et al. (2024)
text + single-image
34.5
26.4
22.0
25.7
13.8
14.6
4.7
11.2
ReT Caffagni et al. (2025)
text + single-image
50.1
40.4
46.3
43.7
15.1
15.2
6.8
11.4
MuKA Deng et al. (2025)
text + single-image
51.7
41.0
64.9
60.7
20.8
18.7
8.6
12.2
Table 2: Evaluating Robustness Against Visual Shortcuts . Retrieval and QA performance are reported. All models are jointly trained on EVQA and RETINA. The GT Document row represents an upper bound using gold knowledge. MIMIR consistently outperforms prior methods in both settings with and without visual shortcuts, showing the benefit of leveraging multi-image context over single-image or text-only approaches.
InfoSeek
EVQA
RETINA
Train
Method
Seen
Unseen
R@5
R@5
R@5
R@5
InfoSeek
PreFLMR
43.3
62.4
18.5
7.9
MuKA
54.0
63.8
17.4
7.7
MIMIR
53.9
64.3
35.2
32.7
EVQA
PreFLMR
41.2
64.4
19.9
9.0
Table 3: Cross-dataset Transfer. Rows indicate the training source, columns the evaluation target.
Figure 5: Image-only vs. Multimodal retrieval. On EVQA (left), image-only MuKA ( blue ) nearly matches the multimodal model ( green ), revealing the visual shortcut. On RETINA (right), MIMIR shows a substantial gap, indicating that multimodal processing is required.
Figure 6: Qualitative Comparison of Retrieval on RETINA Benchmark. Results for (a) MuKA Deng et al. (2025) and (b) MIMIR. GT documents are in green ; hard distractors containing the query image entity are in red ; query entity in GT documents are highlighted in green . While MuKA is biased toward visual shortcuts and often retrieves documents containing images similar to the query, MIMIR leverages related images (Fraxinus americana) and (Nunavut), which are augmented to the document embeddings.
InfoSeek
EVQA
RETINA
Method
Size-matched aug.
Seen
Unseen
PreFLMR
–
40.7
63.9
23.6
11.1
PreFLMR
+ repr.
41.0
64.2
24.2
12.4
PreFLMR
+ related passages
27.3
43.1
27.2
14.7
MuKA
–
51.7
64.9
20.8
8.6
MuKA
+ repr.
52.0
64.8
21.2
8.9
Table 4: Capacity-Controlled Comparison (R@5). Each control enlarges the per-document representation to MIMIR’s index size without adding related-entity visual content.
InfoSeek
EVQA
RETINA
Method
Seen
Unseen
Baseline (MuKA Deng et al. (2025) )
51.7
64.9
20.8
8.6
+ MI
51.5
65.1
38.1
33.3
+ MMF
51.6
65.3
41.1
34.5
+ ETE (Full MIMIR)
53.3
65.8
43.2
36.9
Table 5: MIMIR Component Ablation. Cumulative impact of components on retrieval performance: baseline, +MI (Multiple Images), +MMF (Multimodal Features), and +ETE (Entity Token Embedding).
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Example of a Valid Question Filtered Out due to Evaluation Constraints. While both the question and its answer are factually valid, multiple duck species may be as closely related to the mandarin duck (green) as the wood duck (red) is. This ambiguity complicates strict automated evaluation, which is why such samples are filtered from the test set.
Value
Number of Main Entities
19,408
Number of Extracted Relations
66,244
Avg. Train Question Length (tokens)
26.4
Avg. Train Answer Length (tokens)
5.8
Avg. Test Question Length (tokens)
28.0
Avg. Test Answer Length (tokens)
5.5
Appendix
Table 6: Summary of Dataset Statistics.
Method
Document Modality
InfoSeek
EVQA
RETINA (Seen)
RETINA (Unseen)
PreFLMR Lin et al. (2024b)
text
40.7
63.9
23.6
11.1
UniIR Wei et al. (2024)
text + single-image
34.5
22.0
13.8
4.7
ReT Caffagni et al. (2025)
text + single-image
50.1
46.3
15.1
6.8
MuKA Deng et al. (2025)
text + single-image
51.7
64.9
20.8
8.6
MIMIR (Ours)
text + multi-image
53.3 ( ±0.5 )
65.8 ( ±0.4 )
43.2 ( ±0.5 )
36.9 ( ±0.6 )
Appendix
Table 7: Standard deviation analysis . Average retrieval performance and standard deviations of MIMIR are reported.
Model
Retrieval
Seen
Unseen
VILA-13B Lin et al. (2024a)
✗
11.2
10.1
✓
32.7
31.6
InternVL3-8B Zhu et al. (2025)
✗
18.7
17.6
✓
41.3
39.2
Qwen3-VL-8B Yang et al. (2025)
✗
20.3
19.4
✓
45.7
43.9
Appendix
Table 8: MIMIR with Diverse VLM Backbones. QA performance (BEM Bulian et al. (2022) ) with and without retrieval.
(a) Image selection strategy (Recall@5)
Strategy
K=1
K=2
K=3
K=4
Random
22.5
27.8
35.9
39.2
Frequent
24.7
28.4
38.4
41.3
(b) Noise robustness (Recall@5)
Noise rate
0%
10%
20%
30%
MIMIR
43.2
42.8
40.9
36.8
Appendix
Table 9: Image Selection and Noise Robustness on RETINA. (a) Both selection strategies improve consistently with K , with frequency-based selection slightly ahead. (b) MIMIR degrades gracefully under noisy related-entity images, remaining stable up to 20%. Noise replaces the given fraction of related-entity images with images of unrelated entities while keeping the number of images per document, the architecture, and training fixed, so it varies only the correctness of the entity association.
InfoSeek
EVQA
RETINA
Method
Seen
Unseen
RAG-Anything Guo et al. (2025)
34.0
27.7
21.3
19.2
mKG-RAG Yuan et al. (2025)
45.7
35.8
25.8
22.8
mKG-RAG + MuKA
48.4
37.2
28.2
24.9
mKG-RAG + MIMIR
47.9
38.6
32.6
29.3
MIMIR
53.3
65.8
43.2
36.9
Appendix
Table 10: Comparison with Multimodal KG-RAG Methods (R@5) on the same KB under a shared top-5 retrieval protocol.
Figure 8: Robustness to Visual-similar Distractors. MIMIR ( green ) and MuKA ( red ) curves show recall for distractor (dotted) and GT (solid) documents. Distractors are non-GT documents whose main entity matches the query image’s entity.
Figure 9: Qualitative Examples of RETINA Training Set before Human Curation.
Figure 10: Entity Extraction Prompt.
Figure 11: Question Generation Prompt.
Figure 12: Paraphrasing Prompt.
Figure 13: VILA Training Prompt.
Figure 14: Retriever Instruction Prompts.
Figure 15: Human Curation User Interface An image, document title, a question, and an answer are presented for curation, along with guidelines for identifying common issues.
Knowledge-based visual question answering combines image understanding with external or learned knowledge to answer questions requiring facts beyond the image.