Look Twice: Training-Free Evidence Highlighting for Knowledge-based Visual Question Answering

Marco Morini; Sara Sarto; Marcella Cornia; Lorenzo Baraldi; Rita Cucchiara

cs.CVApr 1, 2026

Look Twice: Training-Free Evidence Highlighting for Knowledge-based Visual Question Answering

Authors: Marco Morini, Sara Sarto, Marcella Cornia, Lorenzo Baraldi, Rita Cucchiara

Organizations: University of Modena and Reggio Emilia, Italy

Abstract

Knowledge-based Visual Question Answering (KB-VQA) requires Multimodal Large Language Models (MLLMs) to identify and combine fine-grained visual cues with retrieved textual evidence. However, retrieval often introduces noisy and partially relevant content, while images contain distracting visual regions, causing pretrained MLLMs to overlook the evidence that actually supports the answer. To address this, we introduce Look Twice (LoT), a training-free inference-time framework that turns the model's own internal attention into an explicit multimodal evidence-selection mechanism. LoT first leverages the model's internal attention patterns to identify query-relevant image regions and textual sentences, filters attention sinks and distracting content, and reformulates the input to explicitly highlight the selected evidence before answer generation. The method requires no parameter updates, auxiliary models, or architectural modifications. Across four KB-VQA benchmarks and ten off-the-shelf MLLMs ranging from 2B to 38B parameters, LoT improves every evaluated backbone, with average gains of up to +12.5 accuracy points. It also provides further gains when combined with established context-refinement strategies, yielding additional improvements over already refined inputs. These results establish LoT as a general and effective mechanism for enabling pretrained MLLMs to exploit available multimodal evidence more accurately. Source code is publicly available at https://aimagelab.github.io/LoT/.

Figures & tables

Figure 1: Given an image, a question, and retrieved documents, an MLLM may overlook relevant multimodal evidence and answer incorrectly. LoT uses the model’s internal attention to identify relevant visual and textual content, then highlights the multimodal evidence to improve answer generation.

Figure 2: Overview of the proposed Look Twice (LoT). LoT inspects the MLLM internal attention to identify relevant visual regions and textual evidence, discards irrelevant content, and highlights the selected cues for final answer generation.

Figure 3: Qualitative examples of our visual evidence selection pipeline (right). In the attention map (left), irrelevant visual tokens (orange) exhibit disproportionately high activations in specific hidden-state dimensions, while relevant tokens (blue) remain moderate. The BOS token, known to act as an attention sink in LLMs, shows a similar pattern. In LoT, tokens with sink score exceeding a threshold $\tau$ are filtered.

Small-scale MLLMs (2-4B)
	E-VQA		InfoSeek			OVEN	ViQuAE
	Single	All	U-Q	U-E	All	All	All	Avg
$\blacktriangledown$ Qwen2-VL-2B	17.5	16.0	5.3	5.6	5.4	1.2	15.0	9.4
+ LoT (Ours)	20.5	18.5	21.8	21.4	21.6	14.5	29.1	20.9	$\Delta$ +11.5
$\blacktriangledown$ Qwen2.5-VL-3B	30.3	28.0	22.6	22.2	22.4	11.6	22.9	21.2
+ LoT (Ours)	32.6	30.2	30.3	29.8	30.1	29.5	34.7	31.1	$\Delta$ +9.9

Table 1: Performance comparison on KB-VQA benchmarks. For each backbone, we compare the original zero-shot retrieval-augmented model with the same model augmented with LoT. Results are reported on the E-VQA test set, InfoSeek validation set, OVEN validation set, and ViQuAE test set. Avg denotes the average score across the four “All” columns; improvements over baseline are in green.

Figure 4: Qualitative examples of LoT highlighting query-relevant visual regions and textual evidence, enabling the model to generate the correct answer.

	E-VQA		InfoSeek
	Base	+ LoT	Base	+ LoT
$\blacktriangledown$ Qwen2.5-VL-3B	28.0	30.2	22.4	30.1	$\Delta$ +4.95
+ CoT ( Wei et al. 2022 )	28.1	30.1	29.5	30.1	$\Delta$ +1.30
+ Critic ( Compagnoni et al. 2025 )	30.1	31.3	30.6	31.1	$\Delta$ +0.85
+ Critic ( Compagnoni et al. 2025 )	33.3	34.4	33.5	34.4	$\Delta$ +1.00
+ EchoSight Re-ranker ( Yan and Xie 2024 )	29.8	31.1	30.7	32.0	$\Delta$ +1.30

Table 2: Performance comparison of inference-time strategies combined with LoT. Gray rows denote zero-shot strategies that require no additional trained components.

	Highlight		E-VQA		InfoSeek			OVEN	ViQuAE
	Visual	Textual	Single	All	U-Q	U-E	All	All	All
$\blacktriangledown$ Qwen2.5-VL-3B	-	-	30.3	28.0	22.6	22.2	22.4	11.6	22.9
	✓	-	31.6	29.4	24.1	23.7	23.9	16.6	23.6
	-	✓	31.7	29.1	29.8	29.2	29.5	28.0	34.6
+ LoT (Ours)	✓	✓	32.6	30.2	30.3	29.8	30.1	29.5	34.7
$\blacktriangledown$ Qwen2.5-VL-7B	-	-	32.1	30.2	23.9	25.1	24.5	20.2	36.4

Table 3: Ablation evaluating visual and textual highlighting independently and jointly across KB-VQA benchmarks.

Figure 5: Performance on E-VQA (All) under different evidence-availability (left) and retrieval-depth settings (right).

	E-VQA		InfoSeek
	Single-Hop	All	Unseen-Q	Unseen-E	All
$\blacktriangledown$ Qwen2.5-VL-3B	30.3	28.0	22.6	22.2	22.4
Random	22.2	21.2	22.2	22.1	22.1
Leading	24.1	23.0	28.2	28.6	28.4
BM25	30.6	28.2	28.5	28.0	28.3
Retrieval Round-Robin	22.6	21.7	24.2	25.2	24.7

Table 4: Comparison of LoT with budget-matched non-attention-based selection strategies on E-VQA and InfoSeek. Textual methods retain

30

% of the context, while visual methods retain

46

% of the image area.

	E-VQA		InfoSeek
	Single	All	U-Q	U-E	All
$\blacktriangledown$ Qwen2.5-VL-3B (only input image)	30.3	28.0	22.6	22.2	22.4
w/ bbox on input image	29.6	27.4	22.8	22.2	22.5
w/ only selected evidence	30.7	28.3	22.7	22.3	22.5
w/ input image and evidence (no highlight)	30.0	27.8	21.4	20.8	21.2
w/ highlight on both input image and evidence	31.6	29.3	23.0	22.4	22.7

Table 5: Ablation study on visual highlighting strategies.

	E-VQA		InfoSeek
	Single	All	U-Q	U-E	All
$\blacktriangledown$ Qwen2.5-VL-3B	30.3	28.0	22.6	22.2	22.4
w/ highlight on all retrieved content	30.6	28.2	24.1	23.5	23.8
w/ highlight on entire passage	31.2	28.8	25.0	23.9	24.4
w/ highlight on evidence only (no discard)	31.7	29.1	24.5	24.0	24.2
w/ keep only evidence	30.5	28.5	27.0	26.3	26.7

Table 6: Ablation study on textual highlighting strategies.

	RealWorldQA	V-Star	TextVQA	ChartQA	Avg
$\blacktriangledown$ Qwen2.5-VL-3B	59.1	66.5	62.5	79.1	66.8
+ LoT (Ours)	61.7	71.2	66.4	79.5	69.7	$\Delta$ +2.9
$\blacktriangledown$ Qwen3-VL-4B	66.7	70.2	74.7	80.7	73.1
+ LoT (Ours)	71.6	78.5	76.5	82.3	77.2	$\Delta$ +4.2
$\blacktriangledown$ Qwen2.5-VL-7B	65.0	71.2	75.7	77.0	72.2
+ LoT (Ours)	67.5	73.3	77.9	79.4	74.5	$\Delta$ +2.3

Table 7: Performance comparison on standard MLLM benchmarks. For each backbone, we compare the original model with the same model augmented with LoT using only visual highlighting. Results are reported on RealWorldQA, V-Star, TextVQA, and ChartQA. Avg denotes the average score across the four benchmarks; improvements over the baseline are shown in green.

Dataset	Questions	Entity Identified	Fallback
E-VQA	5,750	5,746 (99.93%)	4 (0.07%)
InfoSeek	71,335	71,335 (100.00%)	0 (0.00%)
OVEN	3,291	2,995 (91.01%)	296 (8.99%)
ViQuAE	1,257	1,232 (98.01%)	25 (1.99%)
Overall	81,633	81,308 (99.60%)	325 (0.40%)

Table 8: Coverage of the dataset-specific entity-selection procedure. A fallback occurs when no valid entity is identified and the last input token is used to extract the attention map.

Model	Sink Dimensions
Qwen2-VL-2B-Instruct	1073, 534, 940
Qwen2-VL-7B-Instruct	2570, 458
Qwen2.5-VL-3B-Instruct	318, 1874, 1819
Qwen2.5-VL-7B-Instruct	458, 2570
Qwen2.5-VL-32B-Instruct	4675, 3094
Qwen3-VL-4B-Instruct	0

Table 9: Attention-sink dimensions filtered for each MLLM.

$\alpha$	Accuracy	Precision	Recall	Avg. Sentences
0.25	32.14	57.72	93.89	5.86
0.50	32.61	62.07	77.62	2.03
0.75	31.36	68.97	60.44	1.04

Table 10: Selection of

\alpha

on the E-VQA validation split using Qwen2.5-VL-3B.

$\alpha_{\mathrm{drop}}$	Accuracy	Context reduction
$\alpha/2.0$	31.59	74.47%
$\mathbf{\alpha/3.0}$	32.61	70.45%
$\alpha/4.0$	31.92	66.71%

Table 11: Selection of

\alpha_{\mathrm{drop}}

on the E-VQA validation split using Qwen2.5-VL-3B.

Panel of Figure 6 — Figure 6: Qualitative examples of attention maps filtering.

Method	IoU $\uparrow$	Coverage $\uparrow$	Precision $\uparrow$	Center Distance $\downarrow$
Min-Max Coordinates	0.352	0.891	0.500	0.090
Morphological Filtering	0.399	0.582	0.716	0.084
Weighted Centroid	0.487	0.814	0.626	0.071

Table 12: Comparison of bounding box extraction methods for target object localization.

$\tau$	$\beta$	IoU $\uparrow$	Acc@IoU $\geq$ 0.5 $\uparrow$
10	1	0.267	0.023
10	2	0.461	0.360
10	3	0.405	0.170
25	1	0.317	0.045
25	2	0.489	0.441
25	3	0.388	0.090

Table 13: Analysis of bounding-box extraction hyperparameters applied to the filtered attention maps. We report mean IoU and localization accuracy at

\mathrm{IoU}\geq 0.5

Configuration	IoU	[email protected]	Coverage	Precision
w/o sink filtering	0.388	0.083	0.999	0.388
w/ sink filtering	0.489	0.441	0.869	0.554

Table 14: Effect of attention-sink filtering on visual evidence localization.

	E-VQA
Layer Range	Single-Hop	All
Text Highlighting
First half	21.6	21.3
Middle half	30.1	28.4
Last half	35.3	32.6
Visual Highlighting

Table 15: Layer-range ablation on the E-VQA validation split using Qwen2.5-VL-3B. When varying the textual range, visual highlighting is fixed to the middle half; when varying the visual range, textual highlighting is fixed to the last half.

Figure 7: Qualitative examples of LoT highlighting query-relevant visual regions and textual evidence, enabling the model to generate the correct answer on E-VQA.

Knowledge-Based VQA

Knowledge-based visual question answering combines image understanding with external or learned knowledge to answer questions requiring facts beyond the image.

See all 42 papers

SAFE-G: Structure-aware Faithful Evidence-guided Generation for Knowledge-based Visual Question Answering
Aug 22, 2026Long Shu, Shuochen Liu, Wei Chen +4Knowledge-Based VQA Multimodal RAG
State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China China · NIO China
Ground Then Rank: Revisiting Knowledge-Based VQA with Training-Free Entity Identification
Jun 22, 2026Qian Ma, Qiong Wu, Zhengyi Zhou +1Multimodal RAG Visual Question Answering
Rensselaer Polytechnic Institute · AT&T Chief Data Office
WikiSeeker: Rethinking the Role of Vision-Language Models in Knowledge-Based Visual Question Answering
Apr 7, 2026Yingjian Zhu, Xinming Wang, Kun Ding +3Knowledge-Based VQA Multimodal RAG
School of Artificial Intelligence, University of Chinese Academy of Sciences · State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Science and Technology Beijing

Aug 22, 2026J/K move · Enter open · S save

SAFE-G: Structure-aware Faithful Evidence-guided Generation for Knowledge-based Visual Question Answering

Long Shu, Shuochen Liu, Wei Chen, Junda Lin+3

State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China China · NIO China

Knowledge-Based VQA Multimodal RAG Faithfulness in VLMs Graph Retrieval-Augmented Generation

Open paper AskPDF arXiv

Knowledge-based Visual Question Answering (KB-VQA) aims to answer queries that necessitate reasoning over external knowledge sources beyond the visual content. Typically, current methods fuse multimodal features to retrieve external information, subsequently leveraging Multimodal Large Language Models (MLLMs) to derive answers from the retrieved evidence. However, these methods often struggle to capture structural associations within complex contexts to effectively filter noise. Furthermore, they frequently fail to ensure that the reasoning process remains strictly faithful to the retrieved evidence. To address these challenges, we propose SAFE-G, a Structure-Aware Faithful Evidence-guided Generation framework, which enables precise evidence localization and trustworthy reasoning. Specifically, we first employ a coarse-grained hybrid search fusing visual and textual modalities to recall candidate documents, and subsequently implement a structure-aware fine-grained graph retrieval that captures structural dependencies to filter noise and pinpoint precise evidence. Moreover, we introduce a reinforcement learning (RL) strategy with an evidence-grounded reward that assigns credit to correct answers only when the selected evidence is correct. This strict alignment constraint compels the model to anchor its response in the retrieved context, effectively enhancing its capability to locate evidence via multimodal features and perform faithful reasoning. Extensive experiments on the Encyclopedic-VQA and InfoSeek benchmarks demonstrate that SAFE-G outperforms prior methods by a margin of 8.9% and 3.5%, substantially enhancing the overall reasoning accuracy. Our source code is publicly available at: https://github.com/MINE-USTC/SAFE-G.

Abstract

Knowledge-Based VQA

Explore similar work

SAFE-G: Structure-aware Faithful Evidence-guided Generation for Knowledge-based Visual Question Answering

Ground Then Rank: Revisiting Knowledge-Based VQA with Training-Free Entity Identification

WikiSeeker: Rethinking the Role of Vision-Language Models in Knowledge-Based Visual Question Answering