A spatial answer depends on the reference frame specified in language and its supporting visual evidence. Item accuracy alone obscures whether correctness persists when either changes. We introduce Frame Evidence, a benchmark of 320 matched triplets and 960 Chinese multiple-choice questions from captured and rendered scenes. One family compares three grounded reference frames over fixed images. The other pairs image deletion annotated as insufficient with answer-preserving addition. Joint metrics require correct answers across all three frames and paired evidence variants. Across 20 configurations on 288 held-out questions, GPT three-shot reaches 50.00% item accuracy, but the best three-frame group accuracy is 5/48 (10.42%), reached by GPT two-shot. Few-shot prompting and task-specific adaptation improve item accuracy without consistently improving joint correctness. Input controls show where images support matched success, while repeated executions reveal answer instability beneath similar scores.
Figures & tables
Figure 1: Matched protocols. FQ1 changes the queried frame while fixing the images. FQ2 deletes an annotated required view or adds a hard non-MSS view. The crossed-out image is withheld, and I1–I4 mark original positions for display. After I1 is deleted, “the first image” refers to I2.
Benchmark
Src.
MI
Match
FoR S
Min Ev
Del. →⊥
Hard +
Joint
Scale
COMFORT ( Zhang et al., 2025 )
Syn.
–
Y
Y
–
–
–
P
58.3K English cases
MuirBench ( Wang et al., 2025 )
Mix.
Y
Y
–
–
–
–
P
2,600 instances
MMSI-Bench ( Yang et al., 2026 )
Real
Y
–
–
P
–
–
–
1,000 QA
All-Angles ( Yeh et al., 2026 )
Real
Y
Y
P
–
–
–
Y
2,132 QA
SCOPE ( Kim et al., 2026 )
Mix.
Y
Y
P
–
–
–
P
20,144 QA
MultiView-Bench ( Zhang et al., 2026 )
Syn.
Y
P
–
–
–
–
–
500 main (+2K var.)
Table 1: Comparison of evaluation protocols most relevant to FrameEvidence.
Table 3
Figure 3: FQ2 image counts across the release (a) and train-fitted metadata controls (b). Test accuracy uses 144 items and NTS/DIS 48 groups. Schema is the visible insufficient-information option’s letter.
Model
Removed
Shifted
Stable
None
Groups
36
6
2
4
Qwen Base
1
0
0
0
Qwen LoRA
5
0
1
0
InternVL Base
4
0
0
0
InternVL LoRA
4
0
0
0
GPT 2-shot
8
3
0
0
Table 3: Joint deletion successes by reference change.
Classification
Matched success
Model
Setting
P
R
F1
Acc.
GEM
NTS
DIS
Open-weight baselines
Qwen2.5-VL-7B
Base
31.06
29.43
25.99
28.82
0.00
2.08
43.75
InternVL3-8B
Base
29.72
29.84
28.65
29.51
2.08
8.33
20.83
Qwen3-VL-8B
Base
34.60
34.30
33.83
34.03
0.00
14.58
39.58
LLaVA-OneVision-7B
Base
30.56
29.33
29.04
29.17
0.00
0.00
37.50
Table 4: All-images model results (%) on 288 questions, with 48 groups per family for joint metrics.
Model
Δ Acc.
Δ NTS
Δ DIS
Qwen2.5-VL-7B
−3.1
+2.1
+43.8
Qwen2.5-VL-7B LoRA
+3.1
+12.5
+12.5
InternVL3-8B
−1.4
+8.3
+12.5
InternVL3-8B LoRA
+2.4
+8.3
+8.3
Qwen3-VL-8B
+4.5
+14.6
+33.3
LLaVA-OneVision-7B
+0.0
+0.0
−6.2
Table 5: All-images minus text-only in percentage points, with questions and gold labels fixed.
Setting
Acc.
GEM
NTS
DIS
InternVL3-8B
Base
30.21
2.08
8.33
20.83
LoRA
32.41±1.31
0.69±1.20
9.72±1.20
12.50±2.08
Qwen2.5-VL-7B
Base
29.17
0.00
2.08
45.83
LoRA
36.00±1.40
0.00±0.00
10.42±3.61
12.50±0.00
Table 6: Ordinary and Circular evaluation (%) on 288 questions. Open-weight controls are matched within panels. LoRA gives three-seed mean ± sample standard deviation. GPT is zero-shot.
Figure 4: Selected GPT three-shot errors with released gold labels. I1–I4 mark original positions for display. Inputs use unlabeled images in their current order. Deletion in (b) removes the original camera anchor.
Figure 5: Initial and conditional success across 20 primary configurations. Horizontal rates use all 48 FQ2 groups. Vertical rates use each configuration’s MSS-correct subset, with counts on highlighted points. Arrows link Base and LoRA, each with its own conditioning subset.
Metric
Repeat
R1
R2
R3
At least one correct
62.15
55.90
57.29
59.03
Both correct
28.47
31.25
30.56
28.12
Consistency
53.47
59.38
54.86
51.39
Consistently wrong
25.00
28.12
24.31
23.26
Table 7: GPT paired runs (%, n=288 ). Nearly half of consistent pairs share a wrong answer.
Model / setting
MSS
Del.
Add.
Qwen2.5-VL-7B
2
3
2
InternVL3-8B
13
16
12
Qwen3-VL-8B
7
23
7
LLaVA-OneVision-7B
0
1
0
MiniCPM-V 4.5
0
6
0
Mistral Small 3.2 24B
29
24
21
Table 8: Insufficiency counts out of 48.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Displayed name
Configured repository or gateway identifier
Qwen2.5-VL-7B
Qwen/Qwen2.5-VL-7B-Instruct
InternVL3-8B
OpenGVLab/InternVL3-8B-hf
Qwen3-VL-8B
Qwen/Qwen3-VL-8B-Instruct
LLaVA-OneVision-7B
llava-hf/llava-onevision-qwen2-7b-ov-hf
MiniCPM-V 4.5
openbmb/MiniCPM-V-4_5
Mistral Small 3.2 24B
mistralai/Mistral-Small-3.2-24B-Instruct-2506
Appendix
Table 9: Model repositories and API aliases used in evaluation. Repositories identify model families. GPT and Claude names are provider-reported gateway aliases.
Input features
Train Acc.
Dev Acc.
Test Acc.
Test NTS
Test DIS
Majority letter
26.39
20.83
23.61
0.00
22.92
Image count
29.51
12.50
21.53
0.00
4.17
Count + insufficiency letter
47.92
43.75
37.50
6.25
2.08
Appendix
Table 10: Train-fitted FQ2 metadata controls (%). Fixed categorical majority rules use 288 training questions. Dev and test contain 48 and 144 questions, respectively. Joint test metrics use all 48 FQ2 groups. No pixels, question text or condition labels are used.
Family
Run
P
R
F1
Acc.
GEM
NTS
DIS
InternVL3-8B
Base
30.34
30.54
29.36
30.21
2.08
8.33
20.83
20260722
34.32
33.20
31.50
32.99
0.00
10.42
14.58
20260723
32.09
31.19
29.11
30.90
0.00
8.33
10.42
20260724
33.85
33.64
31.85
33.33
2.08
10.42
12.50
Mean
33.42±1.17
32.68±1.31
30.82±1.49
32.41±1.31
0.69±1.20
9.72±1.20
12.50±2.08
Qwen2.5-VL-7B
Base
30.87
29.77
26.35
29.17
0.00
2.08
45.83
Appendix
Table 11: Ordinary all-images multi-seed results (%). GEM denotes FQ1-GEM. Classification uses 288 questions and joint criteria use 48 groups per family. Means and sample deviations use three LoRA seeds. These ordinary runs are separate from Circular rotation zero.
Family
Seed / control
Vanilla
Mean rot.
Circular
Consistency
Drop
InternVL3-8B
Base
30.21
28.04
7.64
23.61
22.57
20260722
33.33
31.42
7.29
19.10
26.04
20260723
33.68
31.08
5.56
17.71
28.12
20260724
31.25
30.30
5.21
15.97
26.04
Qwen2.5-VL-7B
Base
14.93
18.75
2.08
5.90
12.85
20260722
34.38
35.76
16.67
33.68
17.71
Appendix
Table 12: Per-seed open-weight and zero-shot GPT Circular scores (%) for 288 items and four cyclic rotations. Drop uses percentage points. Qwen Base is summary-reported. Other rows use 1,152 retained valid responses each.
Quantity
Estimate
96 triplets
13 union components
Repeat accuracy
45.14
[39.58, 50.69]
[39.09, 50.56]
Primary–repeat disagreement
44.44
[38.54, 50.69]
[37.92, 54.49]
R0 –repeat disagreement
46.53
[40.28, 52.78]
[43.18, 51.96]
Repeat minus primary accuracy
4.51
[ −1.74 , 10.76]
[ −1.99 , 14.29]
Repeat minus R0 accuracy
−0.35
[ −6.94 , 6.25]
[ −12.28 , 6.98]
Same-order pair both correct
28.47
[23.26, 33.68]
[23.50, 33.90]
Appendix
Table 13: Paired GPT same-order control with 95% bootstrap intervals. Rates are percentages and differences are percentage points. Rotation contrasts subtract the corresponding R0 – Rj pair from the R0 –repeat pair.
Figure 6: Release composition. Each of six conditions contains 160 questions. Object-label occurrences are counted once per matched group across 126 distinct labels. Other pools labels outside the top ten.
Model
All images
First image
Text only
Acc.
GEM
NTS
DIS
Acc.
GEM
NTS
DIS
Acc.
GEM
NTS
DIS
Qwen2.5-VL-7B Base
28.82
0.00
2.08
43.75
29.17
0.00
2.08
37.50
31.94
0.00
0.00
0.00
Qwen2.5-VL-7B LoRA
35.42
0.00
12.50
12.50
31.25
0.00
0.00
0.00
32.29
2.08
0.00
0.00
InternVL3-8B Base
29.51
2.08
8.33
20.83
27.78
0.00
6.25
29.17
30.90
2.08
0.00
8.33
InternVL3-8B LoRA
32.64
0.00
8.33
8.33
27.78
0.00
4.17
10.42
30.21
0.00
0.00
0.00
Qwen3-VL-8B Base
34.03
0.00
14.58
39.58
28.82
0.00
8.33
10.42
29.51
0.00
0.00
6.25
Appendix
Table 14: Visual-input ablations on 288 test questions (%). Questions, options and gold answers are held fixed across all-images, first-image and text-only modes.
Metric
>
=
<
Median Δ
FQ1-GEM
2
13
5
0.0
NTS
5
8
7
0.0
DIS
17
3
0
+20.8
Appendix
Table 15: Axial–quadrant gaps over 20 configurations (24 groups per stratum). Signs count positive, tied and negative gaps. Median Δ is in percentage points.
Model
FQ1-GEM
NTS
DIS
2 n=15
3 n=16
4 n=17
2 n=17
3 n=16
4 n=15
2 n=17
3 n=16
4 n=15
Claude Fable 5.1 (0-shot)
0.0
0.0
0.0
11.8
12.5
0.0
23.5
56.2
46.7
Claude Fable 5.1 (1-shot)
6.7
12.5
0.0
0.0
6.2
0.0
29.4
37.5
46.7
Claude Fable 5.1 (2-shot)
0.0
12.5
5.9
17.6
18.8
0.0
35.3
37.5
40.0
Claude Fable 5.1 (3-shot)
0.0
6.2
0.0
11.8
12.5
0.0
29.4
56.2
33.3
GPT-5.6 (0-shot)
6.7
0.0
5.9
17.6
25.0
0.0
35.3
25.0
46.7
Appendix
Table 16: Evidence-set size strata (%). Each column gives its group denominator n .
Although Multimodal Large Language Models (MLLMs) have made substantial progress, their spatial reasoning may still produce intermediate judgments inconsistent with the input image, allowing errors to propagate through the reasoning chain and affect the final answer. Existing methods mainly improve spatial reasoning through training or additional spatial information, without considering whether the reasoning process itself is faithful to the model input. Our study shows that unfaithful reasoning chains significantly reduce final-answer accuracy. To address this issue, we propose a modular and training-free framework for spatial reasoning verification and correction. The framework constructs a Spatial Evidence Graph (SEG), which associates atomic spatial evidence extracted from Chain-of-Thought reasoning with visual entities, spatial relations, source steps, and visual evidence. Spatial Evidence Reliability Assessment (SERA) evaluates the reliability of visual evidence based on object existence, localization, and geometric measurements. The framework then identifies the earliest spatial evidence unit contradicted by reliable visual evidence and guides the original MLLM to revise the subsequent reasoning and final answer. Across 15 model-dataset settings, our method achieves an average accuracy of 68.94%, outperforming the compared baselines by 8.55 percentage points on average. Our code will be open-sourced.
Yang Yang, Jiawei Chen, Tairan Chen +1
Shanghai Key Laboratory of Multidimensional Information Processing, East China Normal University · Zhongguancun Academy · Stevens Institute of Technology
Closed yes/no spatial benchmarks can reward a correct answer even when the image adds little support beyond no-image contexts. Under a fixed forced-choice interface, Visual Credit Audit (VCA) separates two estimands: whether the benchmark image gives the model's declared decision more support than text-only and blank controls, and whether the model responds to relation-specific visual evidence. The first audit is training- and label-free and does not require an answer flip. Applying labels yields dependence-credited correctness (D-CC); on correct items, it equals same-control gold-aligned positive gain, while prediction alignment extends the audit to errors. Across four open MLLMs and two spatial benchmarks, 12.73-26.25% of decisions are correct yet uncredited. Matched same-split image permutation reduces D-CC by 21.25-47.80 points, with every paired 95% interval above zero. Fixed-pixel relation contrasts and a 3x3 evidence-source factorial show why null controls cannot identify relation response. Among controlled correct-but-uncredited agreement decisions, response to relation reversal spans 81.57-100.00%, while 32.11% pooled change answer. Independently audited outcomes on 108 geometry-compatible edits provide a bounded natural-image correspondence check. VCA thereby decomposes benchmark success into correctness, additional image support, and relation-consistent response.
Feixiang Liu, Qiang Qiu, Lanbo Sun +3
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences
When a multimodal large language model answers a visual reasoning question correctly, is the prediction actually supported by the task-critical visual evidence? Correct answers can coexist with flawed reasoning, making accuracy alone an incomplete test of grounding. We introduce VisualFLIP, a paired benchmark with 1,374 images arranged as same-question perturbation pairs across cardinality, attribute, spatial, and logic tasks. Each pair keeps the question fixed but minimally changes the evidence so the gold answer deterministically flips. We evaluate 24 MLLMs with pair accuracy, which requires solving both sides of a pair, and Collapse Rate (CR), which measures how often a model that solves at least one side repeats the same non-empty answer for both images. Together, these metrics show that paired correctness and evidence dependence are related but distinct: capable models can still fail to update after task-critical visual changes, and collapse becomes more severe for some models when the edited image follows an earlier answer in a sequential setting. Further details are available on our project page: https://didizhu-judy.github.io/VisualFLIP/