SAVER: Selective Auditing of Verbal Evidence for Error Recovery in VLM Change Reasoning

Youdi Li

cs.CLAug 24, 2026

SAVER: Selective Auditing of Verbal Evidence for Error Recovery in VLM Change Reasoning

Authors: Youdi Li

Organizations: Panasonic Connect Co., Ltd. Tokyo, Japan

Abstract

Vision-language models (VLMs) frequently fail at visual change reasoning, even when their vision encoders contain sufficient information. We observe that correct VLM outputs tend to contain explicit verbal evidence (object names, colors, spatial locations) that supports the claimed change, while incorrect outputs often lack such evidence. We propose SAVER (Selective Auditing of Verbal Evidence for Error Recovery), a lightweight, rule-based method that parses VLM responses for this evidence and triggers structured reprompting only when evidence is missing or inconsistent. Across three change detection benchmarks and four VLMs, SAVER significantly improves accuracy on tasks where errors stem from the model failing to articulate what it saw (expression failures), with gains up to +25.8% on CLEVR-Change. The evidence patterns can also be generated by an LLM in a single call, matching the hand-tuned gate on CLEVR-Change. Ablation experiments confirm that the evidence gate, not reprompting alone, drives the improvement.

Figures & tables

Figure 1: Expression failure in VLM change reasoning (one sample from CLEVR-Change). The ground truth is removal of the green cylinder, but GPT-4o misidentifies both the object and the change type. The model detects a change in the green region but cannot express it correctly.

Figure 2: The SAVER pipeline. Given an image pair, SAVER sends a baseline prompt and parses the response for verbal evidence using rule-based pattern matching. Sample A (color change): the response states “changed from red to yellow,” so the gate finds color_change evidence matching the predicted type and accepts the baseline (1 call). Sample B (drop): the baseline claims “the green cube…has changed” and mentions material terms (“matte,” “reflective”), but no evidence vector registers a match; the response never makes an explicit change claim such as “changed from matte to shiny.” The predicted type is unclear and no evidence is matched, so the gate triggers. The structured reprompt forces object enumeration, and the model correctly identifies the removed cylinder (2 calls).

$\hat{t}$	$\mathrm{req}(\hat{t})$	$\mathrm{conf}(\hat{t})$
drop	absence	presence
add	presence	absence
color	color_change	material_change
texture	material_change	color_change

Table 1: CLEVR-Change gate mappings. Each predicted type requires specific evidence and conflicts with a different category.

Dataset	Domain	$N$
CLEVR-Change	Synthetic 3D	632
MagicBrush	Edited photos	340
Spot-the-Diff	Natural photos	197

Table 2: Dataset summary.

Model	B0	B1	B2	SAVER	Trig.
GPT-4o	72.8	80.7	77.5	86.2	39%
Gemini	81.5	95.6	87.3	93.0	36%
Qwen	56.5	87.0	73.4	82.3	49%
LLaMA	82.3	86.2	90.5	88.9	37%

Table 3: CLEVR-Change accuracy (%,

N=632

). Bold = best; underline = SAVER. Trig. = gate trigger rate. All SAVER vs. B0:

p<0.001

Model	B0	B1	SAVER	Trig.
GPT-4o	57.6	70.3	62.9	22%
Gemini	67.9	79.7	68.8	11%
Qwen	38.5	68.8	56.2	38%
LLaMA	58.5	74.4	63.5	30%

Table 4: MagicBrush accuracy (%,

N=340

). Bold = best; underline = SAVER.

Model	B0	SAVER	$\Delta$	Trig.
GPT-4o	77.2	78.2	+1.0	14%
Gemini	67.5	70.1	+2.5	17%
Qwen	61.4	62.9	+1.5	12%
LLaMA	73.1	78.2	+5.1	57%

Table 5: Spot-the-Diff results (% recall,

N=197

). All

p>0.05

(Bonferroni). Fix/Broke counts: GPT-4o 2/0, Gemini 5/0, Qwen 3/0, LLaMA 14/4.

	Model	Fix	Broke	B1 Broke
CLEVR	GPT-4o	93	8	76
	Gemini	74	1	9
	Qwen	173	10	30
	LLaMA	59	17	67
MB	GPT-4o	21	3	21
	Gemini	5	2	14

Table 6: Fix/Broke counts for SAVER and always-structured (B1) across CLEVR-Change and MagicBrush (MB). Fix and Broke are for SAVER; B1 Broke shows how many correct baselines B1 overwrites.

Condition	Drop	Add	Color	Texture
B0 (baseline)	87.3	76.9	75.9	51.0
B1 (structured)	65.6 †	64.4 †	93.0	100.0
SC-5	93.0	82.5	80.4	53.5 †
SAVER	89.2	76.9	84.8	94.3

Table 7: CLEVR-Change per-type accuracy (%, GPT-4o). Bold = best; underline = SAVER; † = below baseline. SC-5 = self-consistency (

k

=5).

Condition	CLEVR	MB	Calls
B0 (baseline)	72.8	57.6	1.0
B1 (always-structured)	80.7	70.3	1.0
SAVER	86.2	62.9	1.4
SC-5	77.4	—	5.0
C1: Random	74.7	60.3	1.4
C2: Length	69.0	61.2	1.4

Table 8: Ablation results (% accuracy, GPT-4o). Calls = API calls per sample. MB = MagicBrush. Bold = best; † = below baseline. SC-5 = self-consistency with

k

=5. C4 uses SAVER’s gate but replaces the structured prompt with a generic retry.

	Model	B0	SAVER	SAVER-L	Trig.
CLEVR	GPT-4o	72.8	86.2	87.0	36%
	Gemini	81.5	93.0	92.4	23%
	Qwen	56.5	82.3	84.2	65%
	LLaMA	82.3	88.9	88.8	35%
MB	GPT-4o	57.6	62.9	58.5	9%
	Gemini	67.9	68.8	67.4	7%

Table 9: SAVER (LLM-gen) results (% accuracy). SAVER-L: SAVER with LLM-generated patterns. SAVER = manual patterns. Trig. = SAVER-L trigger rate. Bold = best. On CLEVR-Change, all SAVER-L vs. B0 gains are significant (

p<0.001

, Bonferroni) and no SAVER-L vs. SAVER difference is significant. On MagicBrush, SAVER-L is significantly below SAVER for Qwen (

p<0.001

, Bonferroni). ‡ = lower bound: Qwen2.5-VL-7B was retired before these experiments, so 23 of 31 gate-triggered samples could not be reprompted and keep their baseline response. MB = MagicBrush; STD = Spot-the-Diff.

Model	B0	SAVER	SAVER-L	+Critique
GPT-4o	57.6	62.9	58.5	60.0
Gemini	67.9	68.8	67.4	68.8
Qwen	38.5	56.2	40.6	39.1
LLaMA	58.5	63.5	61.2	63.5

Table 10: Self-critique on MagicBrush (% accuracy). SAVER-L: SAVER with LLM-generated patterns. +Critique = the SAVER-L patterns after one self-critique pass. Bold = best. The critique restores manual-gate accuracy for Gemini and LLaMA, roughly halves the gap for GPT-4o, and fails for Qwen.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Vector	Keywords / Patterns
absence	missing, absent, gone, disappeared, removed, no longer, not present
presence	new, added, appeared, introduced, additional, placed
color_change	{changed, switched, turned} $\sim$ {color}; also detects color-pair mentions (e.g., “red to blue”)
material_change	{changed, switched} $\sim$ {material, texture, matte, shiny}

Appendix

Table 11: CLEVR-Change evidence keywords.

Vector	Keywords / Patterns
absence	missing, absent, gone, disappeared, removed, no longer, not present, deleted, taken away, erased
presence	new, added, appeared, introduced, additional, placed, now has/contains/includes/shows
color_change	{changed, different, switched, turned, became} $\sim$ {color, colour, hue}; also from-to patterns
replacement	replaced, substituted, swapped, changed … to/into/for/with
attribute_change	modified, altered, transformed, different … shape/size/position

Appendix

Table 12: MagicBrush evidence keywords.

	Model	$\Delta$ %	95% CI
CLEVR	GPT-4o	+13.4 ∗∗∗	[10.8, 16.5]
	Gemini	+11.6 ∗∗∗	[9.3, 13.9]
	Qwen	+25.8 ∗∗∗	[22.2, 29.4]
	LLaMA	+6.6 ∗∗∗	[3.8, 9.3]
MB	GPT-4o	+5.3 ∗∗	[2.6, 8.2]
	Gemini	+0.9	[ $-$ 0.6, 2.4]

Appendix

Table 13: SAVER vs. baseline (

\Delta

% with bootstrap 95% CIs). ∗∗∗

p<0.001

; ∗∗

p<0.05

; no mark = n.s. MB = MagicBrush. Cohen’s

h

: 0.19–0.57 (CLEVR), 0.02–0.36 (MB).

Figure 3: SAVER improvement over B0 (baseline) in percentage points, with 95% bootstrap CIs across all 12 model–dataset combinations. Solid/filled = significant ( $p<0.05$ ); dashed/hollow = non-significant.

	Model	Trig.	Prec.	Rec.	$P_{\text{rand}}$
CLEVR	GPT-4o	0.39	0.508	0.733	0.272
	Gemini	0.36	0.371	0.726	0.185
	Qwen	0.49	0.691	0.782	0.435
	LLaMA	0.37	0.255	0.536	0.177
MB	GPT-4o	0.22	0.500	0.257	0.424
	Gemini	0.11	0.243	0.083	0.321

Appendix

Table 14: Manual-gate precision and recall against the baseline-correctness oracle, all 12 cells. Trig. = trigger rate.

P_{\text{rand}}

= precision of a random gate at the same trigger rate (the baseline error rate). Bold = precision above

P_{\text{rand}}

. MB = MagicBrush; STD = Spot-the-Diff.

	Model	LLM-gen	+Critique
CLEVR	GPT-4o	0.578	0.578
	Gemini	0.549	0.549
	Qwen	0.571	0.571
	LLaMA	0.287	0.278
MB	GPT-4o	0.355	0.487
	Gemini	0.217	0.353

Appendix

Table 15: Gate precision for the LLM-generated gates, single-shot (LLM-gen) and self-critiqued (+Critique), all 12 cells.

P_{\text{rand}}

as in Table 14 . The Spot-the-Diff Gemini single-shot gate triggers on only 6 of 197 samples. MB = MagicBrush; STD = Spot-the-Diff.

	Level	Trig.	Prec.	Rec.	Acc.
Conflict checks intact	L0 manual	0.39	0.508	0.733	86.2
	L1 windows	0.37	0.532	0.721	85.9
	L2 verbs	0.60	0.382	0.843	87.3
	L3 ubiq.	1.00	0.272	1.000	80.2
Conflict checks disabled	L0 manual	0.34	0.583	0.733	87.3
	L1 windows	0.29	0.663	0.709	87.0

Appendix

Table 16: Breadth degradation of the hand-tuned CLEVR-Change gate (GPT-4o). Levels are cumulative.

P_{\text{rand}}=0.272

throughout. Reference points: B0

=

72.8, B1

=

80.7.

Figure 4: Breadth degradation of the hand-tuned CLEVR-Change gate (GPT-4o), both conflict-check regimes. Dotted lines mark the baseline (B0) and always-structured (B1) accuracy. With conflict checks intact, broadening over-triggers and degrades gracefully toward B1. Without them, the gate under-triggers silently and falls back to B0.

Dropped	Trig.	Prec.	Acc.
0% (manual)	0.39	0.508	86.2
25%	0.66 $\pm$ 0.09	0.36 $\pm$ 0.04	82.8 $\pm$ 1.7
50%	0.53 $\pm$ 0.06	0.44 $\pm$ 0.05	86.6 $\pm$ 3.0
75%	0.92 $\pm$ 0.10	0.29 $\pm$ 0.02	81.6 $\pm$ 1.7

Appendix

Table 17: Coverage degradation of the hand-tuned CLEVR-Change gate (GPT-4o). Mean

\pm

sd over three draws.

P_{\text{rand}}=0.272

	Model	B1	SC-5	SAVER	Calls
CLEVR	GPT-4o	80.7	77.4	86.2	1.39
	Gemini	95.6	—	93.0	1.36
	Qwen	87.0	—	82.3	1.49
	LLaMA	86.2	—	88.9	1.37
MB	GPT-4o	70.3	—	62.9	1.22
	Gemini	79.7	—	68.8	1.11

Appendix

Table 18: Accuracy (%) vs. API cost. B1 uses 1.0 calls; SAVER averages 1.11–1.49 (Calls column); SC-5 uses 5.0 calls (GPT-4o CLEVR only). Bold = highest accuracy.

		CLEVR-Change				MagicBrush
Condition	Trig.	GPT-4o	Gemini	Qwen	LLaMA	GPT-4o	Gemini	Qwen	LLaMA
SAVER (ref)	11–49%	86.2	93.0	82.3	88.9	62.9	68.8	56.2	63.5
C5 Non-critical	matched	78.5	86.4	76.3	84.5	61.5	69.1	52.4	64.1
C6 Strict gate	55–100%	81.8	95.6	87.2	87.0	66.8	77.1	64.7	67.4
C7 Loose gate	5–33%	81.5	88.0	66.0	85.0	61.5	69.1	55.0	59.4

Appendix

Table 19: Gate design sensitivity analysis (% accuracy). C5 = non-critical keyword trigger; C6 = strict gate (any missing evidence); C7 = loose gate (conflict only). Bold = best per column. Spot-the-Diff is excluded (no B1 structured data for these conditions).

Figure 5: CLEVR-Change per-type accuracy for GPT-4o. Structured prompting (B1) hurts drop and add types but helps color and texture. SAVER selectively applies structured reasoning where it helps.

Condition	Accuracy	Fix	Broke	Trigger
Baseline (B0)	69.0%	—	—	—
Structured (B1)	76.5%	27	12	—
SAVER	77.5%	17	0	20%

Appendix

Table 20: Claude 3.5 Sonnet on CLEVR-Change (200 samples). Bold = best. SAVER outperforms B1 with zero breaks.

Model	Accepted	Wrong	Perc. Fail. %
GPT-4o	169	45	26.6
Gemini	163	61	37.4
Qwen	173	79	45.7
LLaMA	84	28	33.3

Appendix

Table 21: Perception failure rates on Spot-the-Diff. “Accepted” = gate found evidence and did not trigger. “Wrong” = accepted but incorrect. These are cases where the model describes changes confidently but inaccurately.

Model	API / Model ID
GPT-4o	OpenAI: gpt-4o
Gemini 2.0 Flash	OpenRouter: gemini-2.0-flash-001 †
Qwen2.5-VL-7B	OpenRouter: qwen-2.5-vl-7b-instruct †
LLaMA-4-Scout	OpenRouter: llama-4-scout
Claude 3.5 Sonnet	OpenRouter: claude-3.5-sonnet
GPT-4o (revision)	OpenRouter: openai/gpt-4o

Appendix

Table 22: Model versions and API endpoints. Top block: main experiments, January–February 2026. Bottom block: revision experiments, July 2026 (28 new GPT-4o Spot-the-Diff reprompts via OpenRouter; Claude Sonnet 4.5 as pattern generator and critic). † = retired by the provider before the revision experiments.

Explore similar work

Are VLMs Seeing or Just Saying? Uncovering the Illusion of Visual Re-examination
May 15, 2026Chufan Shi, Cheng Yang, Yaokang Wu +4VLM Evaluation VLM Robustness
University of Southern California · University of California San Diego · Carnegie Mellon Univer +1
Same Answer, Different Representations: Hidden instability in VLMs
Feb 6, 2026Farooq Ahmad Wani, Alessandro Suglia, Rohit Saxena +6VLM Evaluation VLM Robustness
Sapienza University of Rome · University of Edinburgh · University of Oxford +5
Reading Right, Answering Wrong: How Visual Configuration Changes Affect Evidence Use in VLMs
Sep 22, 2026Dingyang Lin, Yingfeng Luo, Chenglong Wang +4VLM Robustness Visual Question Answering
School of Computer Science and Engineering, Northeastern University, Shenyang, China · NiuTrans Research, Shenyang, China

May 15, 2026J/K move · Enter open · S save

Are VLMs Seeing or Just Saying? Uncovering the Illusion of Visual Re-examination

Chufan Shi, Cheng Yang, Yaokang Wu, Linghao Jin+3

University of Southern California · University of California San Diego · Carnegie Mellon Univer+1

VLM Evaluation VLM Robustness VLM Interpretability VLM Reasoning Visually Grounded Reasoning

Open paper AskPDF arXiv

Vision-Language Models (VLMs) often produce self-reflective statements like "let me check the figure again" during reasoning. Do such statements trigger genuine visual re-examination, or are they merely learned textual patterns? We investigate this via VisualSwap, an image-swap probing framework: after a model reasons over an image, we replace it with a visually similar but semantically different one and test whether the model notices. We introduce VS-Bench, 800 image pairs curated from MathVista, MathVerse, MathVision, and MMMU-Pro. Experiments on Qwen3-VL, Kimi-VL, and ERNIE-VL reveal a striking failure: models overwhelmingly miss the swap, with accuracy dropping by up to 60%. Counterintuitively, thinking models are nearly 3x more vulnerable than their instructed counterparts, and scaling offers no mitigation. Multi-turn user instructions restore visual grounding, but self-generated reflective statements during continuous generation do not. Attention analysis explains why: user instructions substantially elevate attention to visual tokens, whereas self-reflection does not. Current VLMs tend to say rather than actually see when claiming to perform visual re-examination. Our code and dataset are available at the project page: https://visualswap.github.io

Abstract

Explore similar work

Are VLMs Seeing or Just Saying? Uncovering the Illusion of Visual Re-examination

Same Answer, Different Representations: Hidden instability in VLMs

Reading Right, Answering Wrong: How Visual Configuration Changes Affect Evidence Use in VLMs