Frame Evidence:A Benchmark for Reference Frames and Visual Evidence in Multi-Image Spatial Reasoning

Chunxi Luo; Qishuai Ding; Xingchen Peng; Hengwei Ye; Ke Zhang; Wenxin Lian; Tianyi Lu; Chunlong Yu

cs.CVOct 14, 2026

Frame Evidence:A Benchmark for Reference Frames and Visual Evidence in Multi-Image Spatial Reasoning

Authors: Chunxi Luo, Qishuai Ding, Xingchen Peng, Hengwei Ye, Ke Zhang, Wenxin Lian, Tianyi Lu, Chunlong Yu,

Organizations: Shanghai University · Byaidata · East China University of Science and Technology · Shanghai Ocean University

Abstract

A spatial answer depends on the reference frame specified in language and its supporting visual evidence. Item accuracy alone obscures whether correctness persists when either changes. We introduce Frame Evidence, a benchmark of 320 matched triplets and 960 Chinese multiple-choice questions from captured and rendered scenes. One family compares three grounded reference frames over fixed images. The other pairs image deletion annotated as insufficient with answer-preserving addition. Joint metrics require correct answers across all three frames and paired evidence variants. Across 20 configurations on 288 held-out questions, GPT three-shot reaches 50.00% item accuracy, but the best three-frame group accuracy is 5/48 (10.42%), reached by GPT two-shot. Few-shot prompting and task-specific adaptation improve item accuracy without consistently improving joint correctness. Input controls show where images support matched success, while repeated executions reveal answer instability beneath similar scores.

Figures & tables

Figure 1: Matched protocols. FQ1 changes the queried frame while fixing the images. FQ2 deletes an annotated required view or adds a hard non-MSS view. The crossed-out image is withheld, and I1–I4 mark original positions for display. After I1 is deleted, “the first image” refers to I2.

Benchmark	Src.	MI	Match	FoR S	Min Ev	Del. $\to\bot$	Hard $+$	Joint	Scale
COMFORT ( Zhang et al., 2025 )	Syn.	–	Y	Y	–	–	–	P	58.3K English cases
MuirBench ( Wang et al., 2025 )	Mix.	Y	Y	–	–	–	–	P	2,600 instances
MMSI-Bench ( Yang et al., 2026 )	Real	Y	–	–	P	–	–	–	1,000 QA
All-Angles ( Yeh et al., 2026 )	Real	Y	Y	P	–	–	–	Y	2,132 QA
SCOPE ( Kim et al., 2026 )	Mix.	Y	Y	P	–	–	–	P	20,144 QA
MultiView-Bench ( Zhang et al., 2026 )	Syn.	Y	P	–	–	–	–	–	500 main (+2K var.)

Table 1: Comparison of evaluation protocols most relevant to FrameEvidence.

Figure 3: FQ2 image counts across the release (a) and train-fitted metadata controls (b). Test accuracy uses 144 items and NTS/DIS 48 groups. Schema is the visible insufficient-information option’s letter.

Model	Removed	Shifted	Stable	None
Groups	36	6	2	4
Qwen Base	1	0	0	0
Qwen LoRA	5	0	1	0
InternVL Base	4	0	0	0
InternVL LoRA	4	0	0	0
GPT 2-shot	8	3	0	0

Table 3: Joint deletion successes by reference change.

		Classification				Matched success
Model	Setting	P	R	F1	Acc.	GEM	NTS	DIS
Open-weight baselines
Qwen2.5-VL-7B	Base	31.06	29.43	25.99	28.82	0.00	2.08	43.75
InternVL3-8B	Base	29.72	29.84	28.65	29.51	2.08	8.33	20.83
Qwen3-VL-8B	Base	34.60	34.30	33.83	34.03	0.00	14.58	39.58
LLaVA-OneVision-7B	Base	30.56	29.33	29.04	29.17	0.00	0.00	37.50

Table 4: All-images model results (%) on 288 questions, with 48 groups per family for joint metrics.

Model	$\Delta$ Acc.	$\Delta$ NTS	$\Delta$ DIS
Qwen2.5-VL-7B	$-3.1$	$+2.1$	$+43.8$
Qwen2.5-VL-7B LoRA	$+3.1$	$+12.5$	$+12.5$
InternVL3-8B	$-1.4$	$+8.3$	$+12.5$
InternVL3-8B LoRA	$+2.4$	$+8.3$	$+8.3$
Qwen3-VL-8B	$+4.5$	$+14.6$	$+33.3$
LLaVA-OneVision-7B	$+0.0$	$+0.0$	$-6.2$

Table 5: All-images minus text-only in percentage points, with questions and gold labels fixed.

Setting	Acc.	GEM	NTS	DIS
InternVL3-8B
Base	30.21	2.08	8.33	20.83
LoRA	$32.41{\pm}1.31$	$0.69{\pm}1.20$	$9.72{\pm}1.20$	$12.50{\pm}2.08$
Qwen2.5-VL-7B
Base	29.17	0.00	2.08	45.83
LoRA	$36.00{\pm}1.40$	$0.00{\pm}0.00$	$10.42{\pm}3.61$	$12.50{\pm}0.00$

Table 6: Ordinary and Circular evaluation (%) on 288 questions. Open-weight controls are matched within panels. LoRA gives three-seed mean

\pm

sample standard deviation. GPT is zero-shot.

Figure 4: Selected GPT three-shot errors with released gold labels. I1–I4 mark original positions for display. Inputs use unlabeled images in their current order. Deletion in (b) removes the original camera anchor.

Figure 5: Initial and conditional success across 20 primary configurations. Horizontal rates use all 48 FQ2 groups. Vertical rates use each configuration’s MSS-correct subset, with counts on highlighted points. Arrows link Base and LoRA, each with its own conditioning subset.

Metric	Repeat	$R_{1}$	$R_{2}$	$R_{3}$
At least one correct	62.15	55.90	57.29	59.03
Both correct	28.47	31.25	30.56	28.12
Consistency	53.47	59.38	54.86	51.39
Consistently wrong	25.00	28.12	24.31	23.26

Table 7: GPT paired runs (%,

n=288

). Nearly half of consistent pairs share a wrong answer.

Model / setting	MSS	Del.	Add.
Qwen2.5-VL-7B	2	3	2
InternVL3-8B	13	16	12
Qwen3-VL-8B	7	23	7
LLaVA-OneVision-7B	0	1	0
MiniCPM-V 4.5	0	6	0
Mistral Small 3.2 24B	29	24	21

Table 8: Insufficiency counts out of 48.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Displayed name	Configured repository or gateway identifier
Qwen2.5-VL-7B	Qwen/Qwen2.5-VL-7B-Instruct
InternVL3-8B	OpenGVLab/InternVL3-8B-hf
Qwen3-VL-8B	Qwen/Qwen3-VL-8B-Instruct
LLaVA-OneVision-7B	llava-hf/llava-onevision-qwen2-7b-ov-hf
MiniCPM-V 4.5	openbmb/MiniCPM-V-4_5
Mistral Small 3.2 24B	mistralai/Mistral-Small-3.2-24B-Instruct-2506

Appendix

Table 9: Model repositories and API aliases used in evaluation. Repositories identify model families. GPT and Claude names are provider-reported gateway aliases.

Input features	Train Acc.	Dev Acc.	Test Acc.	Test NTS	Test DIS
Majority letter	26.39	20.83	23.61	0.00	22.92
Image count	29.51	12.50	21.53	0.00	4.17
Count + insufficiency letter	47.92	43.75	37.50	6.25	2.08

Appendix

Table 10: Train-fitted FQ2 metadata controls (%). Fixed categorical majority rules use 288 training questions. Dev and test contain 48 and 144 questions, respectively. Joint test metrics use all 48 FQ2 groups. No pixels, question text or condition labels are used.

Family	Run	P	R	F1	Acc.	GEM	NTS	DIS
InternVL3-8B	Base	30.34	30.54	29.36	30.21	2.08	8.33	20.83
	20260722	34.32	33.20	31.50	32.99	0.00	10.42	14.58
	20260723	32.09	31.19	29.11	30.90	0.00	8.33	10.42
	20260724	33.85	33.64	31.85	33.33	2.08	10.42	12.50
	Mean	$33.42{\pm}1.17$	$32.68{\pm}1.31$	$30.82{\pm}1.49$	$32.41{\pm}1.31$	$0.69{\pm}1.20$	$9.72{\pm}1.20$	$12.50{\pm}2.08$
Qwen2.5-VL-7B	Base	30.87	29.77	26.35	29.17	0.00	2.08	45.83

Appendix

Table 11: Ordinary all-images multi-seed results (%). GEM denotes FQ1-GEM. Classification uses 288 questions and joint criteria use 48 groups per family. Means and sample deviations use three LoRA seeds. These ordinary runs are separate from Circular rotation zero.

Family	Seed / control	Vanilla	Mean rot.	Circular	Consistency	Drop
InternVL3-8B	Base	30.21	28.04	7.64	23.61	22.57
	20260722	33.33	31.42	7.29	19.10	26.04
	20260723	33.68	31.08	5.56	17.71	28.12
	20260724	31.25	30.30	5.21	15.97	26.04
Qwen2.5-VL-7B	Base	14.93	18.75	2.08	5.90	12.85
	20260722	34.38	35.76	16.67	33.68	17.71

Appendix

Table 12: Per-seed open-weight and zero-shot GPT Circular scores (%) for 288 items and four cyclic rotations. Drop uses percentage points. Qwen Base is summary-reported. Other rows use 1,152 retained valid responses each.

Quantity	Estimate	96 triplets	13 union components
Repeat accuracy	45.14	[39.58, 50.69]	[39.09, 50.56]
Primary–repeat disagreement	44.44	[38.54, 50.69]	[37.92, 54.49]
$R_{0}$ –repeat disagreement	46.53	[40.28, 52.78]	[43.18, 51.96]
Repeat minus primary accuracy	4.51	[ $-1.74$ , 10.76]	[ $-1.99$ , 14.29]
Repeat minus $R_{0}$ accuracy	$-0.35$	[ $-6.94$ , 6.25]	[ $-12.28$ , 6.98]
Same-order pair both correct	28.47	[23.26, 33.68]	[23.50, 33.90]

Appendix

Table 13: Paired GPT same-order control with 95% bootstrap intervals. Rates are percentages and differences are percentage points. Rotation contrasts subtract the corresponding

R_{0}

–

R_{j}

pair from the

R_{0}

–repeat pair.

Figure 6: Release composition. Each of six conditions contains 160 questions. Object-label occurrences are counted once per matched group across 126 distinct labels. Other pools labels outside the top ten.

Model	All images				First image				Text only
	Acc.	GEM	NTS	DIS	Acc.	GEM	NTS	DIS	Acc.	GEM	NTS	DIS
Qwen2.5-VL-7B Base	28.82	0.00	2.08	43.75	29.17	0.00	2.08	37.50	31.94	0.00	0.00	0.00
Qwen2.5-VL-7B LoRA	35.42	0.00	12.50	12.50	31.25	0.00	0.00	0.00	32.29	2.08	0.00	0.00
InternVL3-8B Base	29.51	2.08	8.33	20.83	27.78	0.00	6.25	29.17	30.90	2.08	0.00	8.33
InternVL3-8B LoRA	32.64	0.00	8.33	8.33	27.78	0.00	4.17	10.42	30.21	0.00	0.00	0.00
Qwen3-VL-8B Base	34.03	0.00	14.58	39.58	28.82	0.00	8.33	10.42	29.51	0.00	0.00	6.25

Appendix

Table 14: Visual-input ablations on 288 test questions (%). Questions, options and gold answers are held fixed across all-images, first-image and text-only modes.

Metric	$>$	$=$	$<$	Median $\Delta$
FQ1-GEM	2	13	5	0.0
NTS	5	8	7	0.0
DIS	17	3	0	+20.8

Appendix

Table 15: Axial–quadrant gaps over 20 configurations (24 groups per stratum). Signs count positive, tied and negative gaps. Median

\Delta

is in percentage points.

Model	FQ1-GEM			NTS			DIS
	2 $n=15$	3 $n=16$	4 $n=17$	2 $n=17$	3 $n=16$	4 $n=15$	2 $n=17$	3 $n=16$	4 $n=15$
Claude Fable 5.1 (0-shot)	0.0	0.0	0.0	11.8	12.5	0.0	23.5	56.2	46.7
Claude Fable 5.1 (1-shot)	6.7	12.5	0.0	0.0	6.2	0.0	29.4	37.5	46.7
Claude Fable 5.1 (2-shot)	0.0	12.5	5.9	17.6	18.8	0.0	35.3	37.5	40.0
Claude Fable 5.1 (3-shot)	0.0	6.2	0.0	11.8	12.5	0.0	29.4	56.2	33.3
GPT-5.6 (0-shot)	6.7	0.0	5.9	17.6	25.0	0.0	35.3	25.0	46.7

Appendix

Table 16: Evidence-set size strata (%). Each column gives its group denominator

n

Spatial Reasoning Benchmarks

Spatial reasoning benchmarks test whether AI systems can perceive, represent, and act on geometric, relational, and viewpoint-dependent information.

See all 162 papers

Trace, Verify, and Correct: A Training-Free Framework for Spatial Reasoning in Multimodal LLMs
Aug 5, 2026Yang Yang, Jiawei Chen, Tairan Chen +1Multimodal Grounding Visual Spatial Reasoning
Shanghai Key Laboratory of Multidimensional Information Processing, East China Normal University · Zhongguancun Academy · Stevens Institute of Technology
Visual Credit Audit for Multimodal Spatial Reasoning
Jul 29, 2026Feixiang Liu, Qiang Qiu, Lanbo Sun +3VLM Evaluation Spatial Reasoning Benchmarks
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences
VisualFLIP: Do Predictions Depend on Task-Critical Visual Evidence in Multimodal Reasoning?
Jun 5, 2026Didi Zhu, Changrui Chen, Stefanos Zafeiriou +1VLM Evaluation Multimodal Grounding
Imperial College London

Aug 5, 2026J/K move · Enter open · S save

Trace, Verify, and Correct: A Training-Free Framework for Spatial Reasoning in Multimodal LLMs

Yang Yang, Jiawei Chen, Tairan Chen, Zhaoxia Yin

Shanghai Key Laboratory of Multidimensional Information Processing, East China Normal University · Zhongguancun Academy · Stevens Institute of Technology

Multimodal Grounding Visual Spatial Reasoning Multimodal Large Language Models Multimodal CoT Reasoning Spatial Reasoning in Language Models Multimodal Spatial Reasoning

Open paper AskPDF arXiv

Although Multimodal Large Language Models (MLLMs) have made substantial progress, their spatial reasoning may still produce intermediate judgments inconsistent with the input image, allowing errors to propagate through the reasoning chain and affect the final answer. Existing methods mainly improve spatial reasoning through training or additional spatial information, without considering whether the reasoning process itself is faithful to the model input. Our study shows that unfaithful reasoning chains significantly reduce final-answer accuracy. To address this issue, we propose a modular and training-free framework for spatial reasoning verification and correction. The framework constructs a Spatial Evidence Graph (SEG), which associates atomic spatial evidence extracted from Chain-of-Thought reasoning with visual entities, spatial relations, source steps, and visual evidence. Spatial Evidence Reliability Assessment (SERA) evaluates the reliability of visual evidence based on object existence, localization, and geometric measurements. The framework then identifies the earliest spatial evidence unit contradicted by reliable visual evidence and guides the original MLLM to revise the subsequent reasoning and final answer. Across 15 model-dataset settings, our method achieves an average accuracy of 68.94%, outperforming the compared baselines by 8.55 percentage points on average. Our code will be open-sourced.

Abstract

Spatial Reasoning Benchmarks

Explore similar work

Trace, Verify, and Correct: A Training-Free Framework for Spatial Reasoning in Multimodal LLMs

Visual Credit Audit for Multimodal Spatial Reasoning

VisualFLIP: Do Predictions Depend on Task-Critical Visual Evidence in Multimodal Reasoning?