Organizations: Shanghai University (shu.edu.cn) · The Hong Kong University of Science and Technology (ust.hk) · Stanford University (stanford.edu) · University of Calgary (ucalgary.ca)
Knowledge-based visual question answering (KB-VQA) aims to combine visual information with external knowledge to answer user questions. Existing methods face two main bottlenecks: visual perception errors can propagate into hallucinations, and generalization across scenarios remains limited. To address these issues, we propose SAGE-VQA (Self-evolving Agent framework with Grounded Evidence), a training-free agentic framework that combines evidence verification with experience evolution. Our key insight is to explore an Evidence-Driven Adaptive Reasoning (EDAR) strategy. It accurately acquires and verifies question-relevant fine-grained visual information. It also extracts and refines reusable experience to support robust reasoning across tasks. Specifically, we design a Co-verified Visual Grounding module that verifies question-relevant visual information and provides grounded evidence for subsequent reasoning. We further introduce a Self-evolving Hierarchical Memory (SHM) mechanism. SHM organizes experiences into declarative and procedural memories based on their nature. These memories can be reused in subsequent reasoning and continually refined through feedback, without updating model parameters. Experimental results show that our method outperforms competing methods based on lightweight open-source vision-language models (VLMs) as well as those based on closed-source VLMs. Moreover, our method can serve as a plug-and-play reasoning framework for existing VLMs without modifying their architectures.
Figures & tables
Figure 1: Motivation for EDAR. Whole-image processing can allocate visual capacity to question-irrelevant regions and miss answer-relevant local details. Co-verified Visual Grounding locates and verifies candidate regions before answer generation.
Figure 2: Overview of SAGE-VQA. The framework comprises two core modules: Co-verified Visual Grounding for verifying question-relevant visual targets, and SHM for feedback-driven memory construction and refinement. EDAR integrates visual evidence, external knowledge, and relevant memories, then uses readiness assessment to guide further evidence gathering or answer generation.
Figure 3: Self-evolving Hierarchical Memory (SHM). SHM organizes episodic records and semantic rules as declarative memory, and executable methods as procedural memory. Based on feedback, SHM can refine existing memories or add new memory.
Method
TF
Subject
Context Modality
Grade
Avg.
NAT
SOC
LAN
TXT
IMG
NO
G1-6
G7-12
Human [ 24 ]
-
90.23
84.97
87.48
89.60
87.50
88.10
91.59
82.42
88.40
Multimodal-CoT [ 49 ]
✗
91.03
93.70
86.64
90.13
88.25
89.48
91.12
89.26
90.45
LLaVA [ 21 ]
✗
90.36
95.95
88.00
89.49
88.00
90.66
90.93
90.90
90.92
LLaVA+GPT-4 judge [ 21 ]
✗
91.56
96.74
91.09
90.62
88.99
93.52
92.73
92.16
92.53
CalibRL (Qwen2.5-VL-7B) [ 15 ]
✗
-
-
-
-
-
-
-
-
89.04
Table 1: ScienceQA test-set comparison. TF: training-free across all components. Subject: NAT/SOC/LAN denote natural/social/language science; context modality: TXT/IMG/NO denote text/image/no context; grade: G1–6/G7–12; Avg.: average accuracy.
Table 5
Pass
Correct
Acc. (%)
Δ (pp)
Corrected
Regressed
1
32
38.10
–
–
–
2
38
45.24
+7.14
15
9
3
40
47.62
+9.52
14
12
4
41
48.81
+10.71
11
10
5
40
47.62
+9.52
9
10
Table 6: Feedback-driven memory self-evolution on 84 recurring ScienceQA hard cases. Δ is measured from Pass 1; transitions are relative to the preceding pass.
Question (gold)
Baseline/prior state (wrong)
SAGE-VQA (correct)
Effective component
Which team is in white/orange? ( Knicks )
The bare VLM fixates on the opposing dark jersey (“JACKSON,” No. 1), never establishes the white/orange team’s identity, and predicts Raptors .
Although the VLM misreads a jersey as “UTAH,” MMKG retrieval links the image to the New York Knicks. The fact reasoner checks this identity against the choices and orange-white color scheme, yielding Knicks .
MMKG retrieval + evidence arbitration.
How many people have their arms around the woman? ( one )
It conflates the three riders with the relation-specific count and predicts three , even though its own trace observes that only the man behind the woman has an arm around her.
The VLM separates group membership from the queried relation: the rear man embraces the woman, whereas the front rider does not. SAM3 independently returns one relevant detection, supporting one .
EDAR enforces relation-aware counting.
Pair 1 S N S N 1.75 in Pair 2 S N S N 1.75 in Which pair has the greater magnetic-force magnitude? ( the same )
An earlier Qwen3.6-35B pass overweights a local cue and predicts Pair 1 , despite identical materials, orientations, and separations.
Procedural memory invokes compute_magnetic_force_magnitude . A successful execution compares the verified attributes and equal 1.75-in gaps, yielding the same .
Verify consistency between calculated values and options
Case-linked guidance to recompute requested quantities before matching an option. Its provenance remains episodic; recorded usage outcomes include one success and one failure.
Semantic Memory
Odds ratio calculation from 2×2 contingency tables
Generalized odds-ratio knowledge, OR=ad/(bc) , where (a,b) and (c,d) are case/control counts in exposed and unexposed groups. Recorded usage outcomes include seven successes and two failures.
Procedural Memory
Compass traverse angle computation
Executable operations for parsing degree-minute bearings and computing normalized interior angles. Its zero-success, five-failure history motivates reassessment of applicability and content.
Appendix
Table 11: Representative entries from the initial memory bank. Text is shortened for readability, while titles and statistics are retained.
Figure 5: A 512×512 three-phase segmented micro-CT slice (voxel = 1.6854 µm/px) from the MicroQuant-VQA benchmark. Voxels are labeled as void (black, pixel 0), biopolymer (gray, pixel 151), or aggregate (white, pixel 255). The question asks for the lineal-path probability L(r) of the aggregate phase at r=64.01 µm.
Dataset
Available questions
Evaluated split
Answer format
ScienceQA
21,208
official test
multiple choice
A-OKVQA
24,903
validation (1,145)
MC and direct answer
MMMU
11,500
validation (900)
multiple choice and open-ended
MicroQuant-VQA
4,970
validation (1,000)
multiple choice
Appendix
Table 12: Statistics and evaluation protocols for the four benchmarks.
Component
Hyperparameter
Setting
Fact-reasoner decoding
temperature
0.1
EDAR
maximum iterations T
3
EDAR
readiness gate θr
0.95
Memory retrieval
entries per query km
2
Memory retrieval
candidate budget km′
max(3km,5)
Memory retrieval
episodic/semantic/procedural type weights
1/1.15/1.20
Appendix
Table 13: Implementation settings used for all evaluations unless noted otherwise.
Configuration
ScienceQA (%)
MMMU (%)
Bare VLM (Qwen3.5-27B)
91.98
54.20
+ MMKG retrieval
93.00 ( +1.02 )
58.36 ( +4.16 )
+ EDAR (full strategy)
96.11 ( +3.11 )
68.12 ( +9.76 )
+ SHM (full SAGE-VQA)
98.56 ( +2.45 )
75.32 ( +7.20 )
Appendix
Table 14: Component contributions on Qwen3.5-27B. The + EDAR row adds visual verification and adaptive reasoning to MMKG retrieval, as in Table 5 . Parentheses report gains over the preceding configuration.
Memory configuration
ScienceQA (%)
MMMU (%)
No retrieved memory
92.76
61.29
+ Episodic + Semantic Memory
94.28 ( +1.52 )
63.21 ( +1.92 )
+ Episodic + Semantic + Procedural Memory
95.33 ( +1.05 )
66.30 ( +3.09 )
Appendix
Table 15: Contributions of declarative and procedural memory on Qwen3.5-9B. Episodic and Semantic Memory jointly form the declarative category; adding Procedural Memory tests its incremental value beyond declarative retrieval. Parentheses report gains over the preceding row. All configurations start from the same initial bank and update their memories separately during evaluation.
Figure 6: Retrieval-budget sensitivity on ScienceQA-test with Qwen3.5-9B. Gold markers indicate settings selected on the ScienceQA training split. The highest accuracy of 95.33% is achieved when km=2 and kg=15 .
Figure 7: Selective refinement beyond the initial evidence assessment. Under readiness-gated arbitration, most questions terminate after one outer iteration, which can already integrate perception, retrieval, and memory-informed reasoning. The small multi-iteration tail shows that further refinement is invoked selectively for unresolved cases. Results cover one Qwen3.5-9B run on the full ScienceQA test split.
Knowledge-based visual question answering combines image understanding with external or learned knowledge to answer questions requiring facts beyond the image.