Knowledge-Based Visual Question Answering (KB-VQA) relies on retrieving external information to answer queries involving long-tail entities. However, existing retrieval pipelines predominantly employ CLIP-style dual encoders, which prioritize surface-level visual similarity over entity-level semantic alignment. This paradigm often fails when semantically identical concepts exhibit large visual variations or when distinct entities appear visually similar. To address this, we propose KBMR, the first MLLM-based embedding retriever tailored for KB-VQA. Leveraging the robust autoregressive capabilities of MLLMs, KBMR maps images into a semantic space that better preserves concept identity. To tackle the challenge of noisy supervision in Wikipedia-scale retrieval, we introduce an MLLM-based semantic discriminator that generates continuous entity-consistency weights. These weights guide a novel continuous semantic distillation objective, enabling effective hard negative sampling and soft supervision beyond rigid binary labels. Extensive experiments demonstrate that KBMR significantly outperforms CLIP baselines, yielding up to a 14.7% improvement in retrieval Recall@1 and a 9.4% gain in end-to-end VQA accuracy. Code is available at https://github.com/realHarryX/KBMR.
Figures & tables
Figure 1. Motivation and overview of KBMR for KB-VQA retrieval. Traditional CLIP retrieval prioritizes surface-level visual similarity, which is fragile to view inconsistency and often leads to entity confusion, recalling semantically mismatched evidence and producing incorrect answers. Our method leverages MLLM to retrieve entity-aware semantic knowledge entries, providing a higher-fidelity candidate pool and improving downstream answer grounding.
Figure 2. KBMR with hard negative sampling and continuous semantic distillation. Given an entity-centric KB-VQA query, Stage-1 uses a frozen encoder to retrieve a candidate pool. Stage-2 applies an MLLM-based Semantic Discriminator to score each query–candidate pair and produce entity consistency weights. Stage-3 samples hard negatives via weights to form training data. We train the retriever via continuous semantic distillation by aligning its similarity distribution with the discriminator-induced soft prior using a KL-based loss. Right: training data construction. Left: continuous semantic distillation training.
Method
w/ KBMR
E-VQA
InfoSeek
Single-Hop
All
Unseen-Q
Unseen-E
All
EchoSight † ( 61 )
✗
19.1
19.4
27.1
27.9
27.7
✓
26.7
27.6
35.1
36.3
36.0
Δ
+7.6
+8.2
+8.0
+8.4
+8.3
EchoSight ‡ ( 61 )
✗
42.0
41.8
30.7
31.5
31.3
✓
51.3
51.0
38.7
39.6
39.2
Table 1. VQA accuracy on E-VQA and InfoSeek. † indicates the EchoSight model without reranking, and ‡ indicates the EchoSight model with reranking. ⋄ indicates that MMKB-RAG is not open source, so we obtained the results by replicating the paper. Rows in blue correspond to replacing the retriever with KBMR. Red values denote absolute performance improvements.
Retriever
E-VQA
InfoSeek
R@1
R@5
R@10
R@20
R@1
R@5
R@10
R@20
CLIPs
CLIP I-T ( 42 )
3.3
7.7
12.1
16.5
32.0
54.0
61.6
68.2
CLIP ViT-L/14 ( 42 )
9.9
22.0
27.8
31.7
22.5
40.4
42.7
44.1
EVA-CLIP-8B ( 43 )
13.3
31.3
41.0
48.8
45.6
67.1
73.0
77.9
MLLM Retriever
Table 2. Retrieval performance on E-VQA and InfoSeek. ▲ denotes the universal retrieval methods.
Method
Pseudo Recall@5
VQA score
PreFLMR ( 28 )
70.9
61.9
OMGM ( 63 )
73.4
66.6
Wiki-PRF-7B ( 12 )
-
77.8
KBMR(Qwen2-VL-7B)
78.1
76.3 *
KBMR(LLaVA-OV-7B)
80.4
79.3 *
Table 3. Comparison of the retrieval and VQA results on OK-VQA. * Results obtained by replacing OMGM’s retriever.
Negative Sampling
Training Weights
E-VQA
InfoSeek
R@1
R@5
R@1
R@5
Random
Hard one-hot weights
9.1
18.7
20.3
36.8
CLIP similarity
Hard one-hot weights
14.7
34.9
47.1
70.3
SD
Hard one-hot weights
16.4
40.1
53.1
71.6
SD
30% inverted SD weights
12.1
30.4
43.3
62.1
SD
10% inverted SD weights
19.9
38.2
54.4
68.8
Table 4. Analysis of different hard-negative construction and supervision strategies for studying the effectiveness of SD signals. All settings use 8 hard negatives per query.
SD
E-VQA
InfoSeek
R@1
R@5
R@1
R@5
InternVL3-8B
20.7
44.3
54.4
67.9
InternVL3-14B
24.4
49.1
59.1
73.4
Qwen2.5-VL-7B
24.7
48.7
60.3
74.7
Table 5. Ablation study on different Semantic Discriminator.
#Negatives
E-VQA
InfoSeek
R@1
R@5
R@1
R@5
4
22.5
45.1
57.7
69.8
6
23.3
46.8
59.1
72.4
8
24.7
48.7
60.3
74.7
10
24.2
48.7
60.2
74.8
Table 6. Ablation study on the number of hard negatives.
Model
γ
E-VQA
InfoSeek
R@1
R@5
R@1
R@5
KBMR (LLaVA-OV-7B)
0.9
23.3
46.0
57.6
71.2
1.0
24.1
47.8
59.1
72.4
1.1
24.7
48.7
60.3
74.7
1.2
24.4
48.0
59.4
73.2
Table 7. Ablation on the semantic sharpness coefficient γ .
Figure 3. Global distribution of semantic consistency weights across the Top-50 highest-similarity negative candidates.
Figure 4. Qualitative analysis on KB-VQA retrieval . (a) shows the query with an image of a Corvus albicollis; (b) shows the GT document of Corvus albicollis; (c) and (d) show the top-5 retrieved images from EVA-CLIP and our proposed KBMR, respectively. The answer is highlighted in green.
Knowledge-based visual question answering combines image understanding with external or learned knowledge to answer questions requiring facts beyond the image.
State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA · School of Artificial Intelligence, University of Chinese Academy of Sciences · Beijing Key Laboratory of Super Intelligent Security of Multi-Modal Information +3