Organizations: Shanghai University (shu.edu.cn) · The Hong Kong University of Science and Technology (ust.hk) · Stanford University (stanford.edu) · University of Calgary (ucalgary.ca)
Knowledge-based visual question answering (KB-VQA) aims to combine visual information with external knowledge to answer user questions. Existing methods face two main bottlenecks: visual perception errors can propagate into hallucinations, and generalization across scenarios remains limited. To address these issues, we propose SAGE-VQA (Self-evolving Agent framework with Grounded Evidence), a training-free agentic framework that combines evidence verification with experience evolution. Our key insight is to explore an Evidence-Driven Adaptive Reasoning (EDAR) strategy. It accurately acquires and verifies question-relevant fine-grained visual information. It also extracts and refines reusable experience to support robust reasoning across tasks. Specifically, we design a Co-verified Visual Grounding module that verifies question-relevant visual information and provides grounded evidence for subsequent reasoning. We further introduce a Self-evolving Hierarchical Memory (SHM) mechanism. SHM organizes experiences into declarative and procedural memories based on their nature. These memories can be reused in subsequent reasoning and continually refined through feedback, without updating model parameters. Experimental results show that our method outperforms competing methods based on lightweight open-source vision-language models (VLMs) as well as those based on closed-source VLMs. Moreover, our method can serve as a plug-and-play reasoning framework for existing VLMs without modifying their architectures.
Figures & tables
Figure 1: Motivation for EDAR. Whole-image processing can allocate visual capacity to question-irrelevant regions and miss answer-relevant local details. Co-verified Visual Grounding locates and verifies candidate regions before answer generation.
Figure 2: Overview of SAGE-VQA. The framework comprises two core modules: Co-verified Visual Grounding for verifying question-relevant visual targets, and SHM for feedback-driven memory construction and refinement. EDAR integrates visual evidence, external knowledge, and relevant memories, then uses readiness assessment to guide further evidence gathering or answer generation.
Figure 3: Self-evolving Hierarchical Memory (SHM). SHM organizes episodic records and semantic rules as declarative memory, and executable methods as procedural memory. Based on feedback, SHM can refine existing memories or add new memory.
Method
TF
Subject
Context Modality
Grade
Avg.
NAT
SOC
LAN
TXT
IMG
NO
G1-6
G7-12
Human [ 24 ]
-
90.23
84.97
87.48
89.60
87.50
88.10
91.59
82.42
88.40
Multimodal-CoT [ 49 ]
✗
91.03
93.70
86.64
90.13
88.25
89.48
91.12
89.26
90.45
LLaVA [ 21 ]
✗
90.36
95.95
88.00
89.49
88.00
90.66
90.93
90.90
90.92
LLaVA+GPT-4 judge [ 21 ]
✗
91.56
96.74
91.09
90.62
88.99
93.52
92.73
92.16
92.53
CalibRL (Qwen2.5-VL-7B) [ 15 ]
✗
-
-
-
-
-
-
-
-
89.04
Table 1: ScienceQA test-set comparison. TF: training-free across all components. Subject: NAT/SOC/LAN denote natural/social/language science; context modality: TXT/IMG/NO denote text/image/no context; grade: G1–6/G7–12; Avg.: average accuracy.
Table 5
Pass
Correct
Acc. (%)
Δ (pp)
Corrected
Regressed
1
32
38.10
–
–
–
2
38
45.24
+7.14
15
9
3
40
47.62
+9.52
14
12
4
41
48.81
+10.71
11
10
5
40
47.62
+9.52
9
10
Table 6: Feedback-driven memory self-evolution on 84 recurring ScienceQA hard cases. Δ is measured from Pass 1; transitions are relative to the preceding pass.
Question (gold)
Baseline/prior state (wrong)
SAGE-VQA (correct)
Effective component
Which team is in white/orange? ( Knicks )
The bare VLM fixates on the opposing dark jersey (“JACKSON,” No. 1), never establishes the white/orange team’s identity, and predicts Raptors .
Although the VLM misreads a jersey as “UTAH,” MMKG retrieval links the image to the New York Knicks. The fact reasoner checks this identity against the choices and orange-white color scheme, yielding Knicks .
MMKG retrieval + evidence arbitration.
How many people have their arms around the woman? ( one )
It conflates the three riders with the relation-specific count and predicts three , even though its own trace observes that only the man behind the woman has an arm around her.
The VLM separates group membership from the queried relation: the rear man embraces the woman, whereas the front rider does not. SAM3 independently returns one relevant detection, supporting one .
EDAR enforces relation-aware counting.
Pair 1 S N S N 1.75 in Pair 2 S N S N 1.75 in Which pair has the greater magnetic-force magnitude? ( the same )
An earlier Qwen3.6-35B pass overweights a local cue and predicts Pair 1 , despite identical materials, orientations, and separations.
Procedural memory invokes compute_magnetic_force_magnitude . A successful execution compares the verified attributes and equal 1.75-in gaps, yielding the same .
Verify consistency between calculated values and options
Case-linked guidance to recompute requested quantities before matching an option. Its provenance remains episodic; recorded usage outcomes include one success and one failure.
Semantic Memory
Odds ratio calculation from 2×2 contingency tables
Generalized odds-ratio knowledge, OR=ad/(bc) , where (a,b) and (c,d) are case/control counts in exposed and unexposed groups. Recorded usage outcomes include seven successes and two failures.
Procedural Memory
Compass traverse angle computation
Executable operations for parsing degree-minute bearings and computing normalized interior angles. Its zero-success, five-failure history motivates reassessment of applicability and content.
Appendix
Table 11: Representative entries from the initial memory bank. Text is shortened for readability, while titles and statistics are retained.
Figure 5: A 512×512 three-phase segmented micro-CT slice (voxel = 1.6854 µm/px) from the MicroQuant-VQA benchmark. Voxels are labeled as void (black, pixel 0), biopolymer (gray, pixel 151), or aggregate (white, pixel 255). The question asks for the lineal-path probability L(r) of the aggregate phase at r=64.01 µm.
Dataset
Available questions
Evaluated split
Answer format
ScienceQA
21,208
official test
multiple choice
A-OKVQA
24,903
validation (1,145)
MC and direct answer
MMMU
11,500
validation (900)
multiple choice and open-ended
MicroQuant-VQA
4,970
validation (1,000)
multiple choice
Appendix
Table 12: Statistics and evaluation protocols for the four benchmarks.
Component
Hyperparameter
Setting
Fact-reasoner decoding
temperature
0.1
EDAR
maximum iterations T
3
EDAR
readiness gate θr
0.95
Memory retrieval
entries per query km
2
Memory retrieval
candidate budget km′
max(3km,5)
Memory retrieval
episodic/semantic/procedural type weights
1/1.15/1.20
Appendix
Table 13: Implementation settings used for all evaluations unless noted otherwise.
Configuration
ScienceQA (%)
MMMU (%)
Bare VLM (Qwen3.5-27B)
91.98
54.20
+ MMKG retrieval
93.00 ( +1.02 )
58.36 ( +4.16 )
+ EDAR (full strategy)
96.11 ( +3.11 )
68.12 ( +9.76 )
+ SHM (full SAGE-VQA)
98.56 ( +2.45 )
75.32 ( +7.20 )
Appendix
Table 14: Component contributions on Qwen3.5-27B. The + EDAR row adds visual verification and adaptive reasoning to MMKG retrieval, as in Table 5 . Parentheses report gains over the preceding configuration.
Memory configuration
ScienceQA (%)
MMMU (%)
No retrieved memory
92.76
61.29
+ Episodic + Semantic Memory
94.28 ( +1.52 )
63.21 ( +1.92 )
+ Episodic + Semantic + Procedural Memory
95.33 ( +1.05 )
66.30 ( +3.09 )
Appendix
Table 15: Contributions of declarative and procedural memory on Qwen3.5-9B. Episodic and Semantic Memory jointly form the declarative category; adding Procedural Memory tests its incremental value beyond declarative retrieval. Parentheses report gains over the preceding row. All configurations start from the same initial bank and update their memories separately during evaluation.
Figure 6: Retrieval-budget sensitivity on ScienceQA-test with Qwen3.5-9B. Gold markers indicate settings selected on the ScienceQA training split. The highest accuracy of 95.33% is achieved when km=2 and kg=15 .
Figure 7: Selective refinement beyond the initial evidence assessment. Under readiness-gated arbitration, most questions terminate after one outer iteration, which can already integrate perception, retrieval, and memory-informed reasoning. The small multi-iteration tail shows that further refinement is invoked selectively for unresolved cases. Results cover one Qwen3.5-9B run on the full ScienceQA test split.
Knowledge-based visual question answering combines image understanding with external or learned knowledge to answer questions requiring facts beyond the image.
Knowledge-based Visual Question Answering (KB-VQA) aims to answer queries that necessitate reasoning over external knowledge sources beyond the visual content. Typically, current methods fuse multimodal features to retrieve external information, subsequently leveraging Multimodal Large Language Models (MLLMs) to derive answers from the retrieved evidence. However, these methods often struggle to capture structural associations within complex contexts to effectively filter noise. Furthermore, they frequently fail to ensure that the reasoning process remains strictly faithful to the retrieved evidence. To address these challenges, we propose SAFE-G, a Structure-Aware Faithful Evidence-guided Generation framework, which enables precise evidence localization and trustworthy reasoning. Specifically, we first employ a coarse-grained hybrid search fusing visual and textual modalities to recall candidate documents, and subsequently implement a structure-aware fine-grained graph retrieval that captures structural dependencies to filter noise and pinpoint precise evidence. Moreover, we introduce a reinforcement learning (RL) strategy with an evidence-grounded reward that assigns credit to correct answers only when the selected evidence is correct. This strict alignment constraint compels the model to anchor its response in the retrieved context, effectively enhancing its capability to locate evidence via multimodal features and perform faithful reasoning. Extensive experiments on the Encyclopedic-VQA and InfoSeek benchmarks demonstrate that SAFE-G outperforms prior methods by a margin of 8.9% and 3.5%, substantially enhancing the overall reasoning accuracy. Our source code is publicly available at: https://github.com/MINE-USTC/SAFE-G.
Long Shu, Shuochen Liu, Wei Chen +4
State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China China · NIO China
Knowledge-based visual question answering (KB-VQA) requires vision-language models to understand images and use external knowledge, especially for rare entities and long-tail facts. Most existing retrieval-augmented generation (RAG) methods adopt a fixed pipeline that sequentially retrieves information, filters it, and then produces an answer. Such a design makes it difficult to adapt to diverse question types. Moreover, it separates retrieval from reasoning, making it hard for the model to decide when to search, how to refine queries, or when to stop. As a result, the retrieved evidence is often poorly aligned with the question. To address these limitations, we reformulate KB-VQA as a search-agent problem and model the solving process as a multi-step decision-making procedure. At each step, the agent selects one of four actions-Answer, Image Retrieval, Text Retrieval, and Caption-based on its current information state. We further design an automated pipeline to collect multi-step trajectories that record the agent's reasoning process, tool usage, and intermediate decisions. These trajectories are then used as supervision for fine-tuning. Experiments on InfoSeek and E-VQA demonstrate that our method achieves state-of-the-art performance, consistently outperforming prior baselines and confirming the effectiveness of our framework.
Zhuohong Chen, Zhenxian Wu, Yunyao Yu +6
Tsinghua University · Hefei University of Technology · University of Arizona
Knowledge-based Visual Question Answering (KB-VQA) requires Multimodal Large Language Models (MLLMs) to identify and combine fine-grained visual cues with retrieved textual evidence. However, retrieval often introduces noisy and partially relevant content, while images contain distracting visual regions, causing pretrained MLLMs to overlook the evidence that actually supports the answer. To address this, we introduce Look Twice (LoT), a training-free inference-time framework that turns the model's own internal attention into an explicit multimodal evidence-selection mechanism. LoT first leverages the model's internal attention patterns to identify query-relevant image regions and textual sentences, filters attention sinks and distracting content, and reformulates the input to explicitly highlight the selected evidence before answer generation. The method requires no parameter updates, auxiliary models, or architectural modifications. Across four KB-VQA benchmarks and ten off-the-shelf MLLMs ranging from 2B to 38B parameters, LoT improves every evaluated backbone, with average gains of up to +12.5 accuracy points. It also provides further gains when combined with established context-refinement strategies, yielding additional improvements over already refined inputs. These results establish LoT as a general and effective mechanism for enabling pretrained MLLMs to exploit available multimodal evidence more accurately. Source code is publicly available at https://aimagelab.github.io/LoT/.