MentalBlackboard: Evaluating Spatial Visualization via Mathematical Transformations

Nilay Yilmaz; Maitreya Patel; Naga Sai Abhiram Kusumba; Yixuan He; Yezhou Yang

cs.CVFeb 22, 2026

MentalBlackboard: Evaluating Spatial Visualization via Mathematical Transformations

Authors: Nilay Yilmaz, Maitreya Patel, Naga Sai Abhiram Kusumba, Yixuan He, Yezhou Yang

Organizations: Arizona State University.

Abstract

Spatial visualization is the mental ability to imagine, transform, and manipulate the spatial characteristics of objects and actions. This intelligence is a part of human cognition where actions and perception are connected on a mental level. To explore whether state-of-the-art Vision-Language Models (VLMs) exhibit this ability, we develop MentalBlackboard, an open-ended spatial visualization benchmark for Paper Folding and Hole Punching tests within two core tasks: prediction and planning. Our prediction experiments reveal that models struggle with applying symmetrical transformations, even when they predict the sequence of unfolding steps correctly. Also, rotations introduce a significant challenge to the physical situational awareness for models. The planning task reveals limitations of models in analyzing symmetrical relationships and in implementing the multi-stage symmetry process, with Claude Opus 4.1 achieving the highest planning score at an accuracy of 10%. The top-performing model, o3, attains a peak performance of 71.6% on the generalization task, which does not require spatial visualization but transfers spatial data; however, it achieves only 25% accuracy on text-based prediction tasks.

Figures & tables

	Symmetry	Memory	Rotation	Size	Dimension	Evaluation	SV Tasks
SPACE ( Ramakrishnan et al., 2025 )	✗	✗	✓	172/50 (visual)	2D	MC	MRT + MPFB ( Patereon et al., 1930 )
STARE ( Li et al., 2025b )	✗	✗	✓	320	2D	Yes/No	Cube Folding
Mind the Gap ( Stogiannidis et al., 2025 )	✓+ ✗	✓+ ✗	✗+ ✓	200/400	2D	MC	PFT + MRT
BSA ( Xu et al., 2025 )	✗	✗+✓	✓	90	2D	MC	SBST + R-Cube-Vis ( Fehringer, 2020 )
Omni Spatial ( Jia et al., 2025 )	✓+ ✗	✓+ ✗	✗+ ✓	$<245$	2D	MC	PFT + MRT
SpatialViz-Bench ( Wang et al., 2025a )	✓+ ✗+ ✗	✓+ ✗+✓	✗+ ✓+ ✓	$1180$	2D	MC	PFT + VP-MRT + CUR-MA

Table 1: Comparison of spatial reasoning datasets and their SV tasks on cognitive processes, size, space, and evaluation approach. Meanings of the abbreviations: Symmetry = Does the task involve mental transformation based on symmetry?, Memory = Does the task in datasets require tracking actions?, Rotation = Does mental rotation exhibit in the task?, MC = multiple-choice, MPFB = Minnesota Paper Form Board, SBST = Santa Barbara Solid Test ( Cohen and Hegarty, 2012 ) , R-Cub-Vis = R-Cube Visualization Short Test, CUR-MA = Cube Unfolding and Reconstruction and Mental Animation, VP-MRT = Visual Penetration and Mental Rotation Task. If a dataset includes multiple SV tasks that obtain different mental operations, their results are concatenated with a plus symbol.

Figure 1: The MentalBlackboard creation pipeline, which consists of environment setup, dataset configuration, dataset generation, and 2D and text mapping.

Figure 2: Examples of a prediction task and a planning task.

Model	Exact Match	Overall Partial Accuracy	Hole Count Errors (%)		Unfolding Accuracy (%)		Field-wise Partial Accuracy (%)
	(%)	(%)	Extra	Missing	Exact	Steps	Shape	Size	Location	Direction
Video-Based Prediction
Qwen2.5-VL	0.00	14.29	0.00	100	3.78	16.99	22.44	23.39	17.95	21.93
Claude Opus 4.1	1.22	18.60	57.00	8.22	16.33	38.66	53.87	53.98	29.69	36.50
Claude 4	1.56	19.99	44.11	6.11	16.89	29.44	66.92	67.00	32.72	41.52
LLaVA-OneVision	0.00	19.03	0.00	100	2.78	12.69	25.49	25.95	20.32	25.41

Table 2: Model performances across video-based, 2D image-based, and text-based prediction tasks. Metrics are grouped into exact match accuracy, custom score, hole count errors, unfolding accuracy, and hole attribute accuracy. Dashes (–) indicate that the field was not evaluated for that model. Top scores are shown in purple, low scores in red.

Figure 3: Group-wise performance loss of o3 models across task modalities under exact hole and unfolding-step matching.

Figure 4: Partial accuracy on planning task per field

Figure 5: An example of a generalization task employs a visual analogical reasoning framework, which requires making an inference from the first scenario to identify the missing part in the second scenario

Appendix figures & tables46 assets

Supplementary material from the paper’s appendix.

Appendix

Table 8

Table 9

Panel of Figure 6 — Figure 6: Benchmark statistics of the prediction task. (a) The distribution of the folding and rotation types across four main tasks and eleven sub-tasks. (b) The visualization of the initial hole locations in the paper design. The darker cells represent paper regions with a higher number of punched locations. (c) The distribution of the initial hole shapes across nine categories. Square and circle shapes are excluded from rotation tasks, as their appearance is not affected by rotations.

Figure 7: Paper design with numbered locations.

Row	Count of Steps	Rotation	Number of Structure	Action Types
Group1	One	✗	16	F
Group2	Two	✗	160	FF
Group3	Three	✗	1408	FFF
Group4	Four	✗	10752	FFFF
Group5	Two	✓	48	FR
Group6	Three	✓	768	FRF FFR

Appendix

Table 7: Configuration of the MentalBlackboard benchmark. F means a type of paper folding, while R indicates a type of rotation.

Folding		Unfolding
horizontal top to bottom		vertical right to left
horizontal bottom to top		vertical left to right
vertical left to right		horizontal top to bottom
vertical right to left		horizontal bottom to top
diagonal top left to bottom right		diagonal top right to bottom left
diagonal top right to bottom left		diagonal bottom right to top left

Appendix

Table 8: Rotation 90°- counterclockwise

Folding		Unfolding
horizontal top to bottom		horizontal top to bottom
horizontal bottom to top		horizontal bottom to top
vertical left to right		vertical left to right
vertical right to left		vertical right to left
diagonal top left to bottom right		diagonal top left to bottom right
diagonal top right to bottom left		diagonal top right to bottom left

Appendix

Table 9: Rotation 180°- counterclockwise

Folding		Unfolding
horizontal top to bottom		vertical left to right
horizontal bottom to top		vertical right to left
vertical left to right		horizontal bottom to top
vertical right to left		horizontal top to bottom
diagonal top left to bottom right		diagonal bottom left to top right
diagonal top right to bottom left		diagonal top left to bottom right

Appendix

Table 10: Rotation 270°- counterclockwise

Model	Video Prediction	Image Prediction	Text Prediction	Planning
Human	75.00	75.00	75.00	75.00
o3	16.66	21.69	40.56	14.50
Claude Opus 4.1	2.66	5.66	33.01	18.00

Appendix

Table 11: Model performance across different prediction and planning tasks.

Figure 8: Exact match accuracy on planning task

Model	One Step	Two Steps	Three Steps	Four Steps	Extra Holes	Missing Holes
o3	9	20	6	3	5.25	76.00
GPT-5.1	8	0	0	1	5.25	81.00
GPT-4o	1	1	1	0	2.75	88.25
Claude 4	17	4	2	0	3.25	84.00
Claude Opus 4.1	23	13	4	0	5.00	73.25
LLaVa-OneVision	3	0	2	0	7.75	81.50

Appendix

Table 12: Percentage of records with group-wise performance (% Correct) and extra and missing hole predictions across models in the planning task.

	Extra Holes			Missing Holes			Equal Holes
Model	Extra Unfolds	Fold Depth	Exact Unfolds	Extra Unfolds	Fold Depth	Exact Unfolds	Fold Depth
o3	0.25	12.78	5.32	0.00	3.89	0.62	4.33
Claude Opus 4.1	19.11	16.89	5.11	0.67	2.56	0.67	5.33

Appendix

Table 13: Model performances grouped by hole conditions and unfolding strategies.

Panel of Figure 9 — Figure 9: Group-wise evaluation for two models in the video prediction task.

o3	Exact Match	Unfolding Exact	Extra Hole (%)	Missing Hole (%)	Shape (%)	Size (%)	Location (%)	Direction (%)
D	0.00	6.25	0.00	8.33	98.6	98.6	52.1	80.0
H	40.91	90.91	4.55	0.00	97.7	97.7	90.9	87.1
V	56.00	100.00	0.00	0.00	100.0	100.0	100.0	89.0
DR	6.25	41.67	0.00	4.17	99.3	99.3	64.2	73.6
HR	10.71	28.57	0.00	0.00	100.0	100.0	69.6	77.4
VR	16.67	37.50	0.00	0.00	100.0	100.0	66.7	83.7

Appendix

Table 14: Model performances across video-based prediction folds (o3). Metrics are grouped into exact match accuracy, unfolding accuracy, hole count errors, and hole attribute accuracy. D = Diagonal, H = Horizontal, V = Vertical and R = Rotation

Claude Opus 4.1	Exact Match	Unfolding Exact	Extra Hole (%)	Missing Hole (%)	Shape (%)	Size (%)	Location (%)	Direction (%)
D	0.00	44.90	0.00	14.29	96.6	96.6	52.0	70.1
H	0.00	0.00	100.00	0.00	31.4	31.9	31.2	29.9
V	20.00	28.00	64.00	0.00	57.0	57.0	48.1	52.5
DR	0.00	33.33	2.08	4.17	97.9	97.9	55.0	64.1
HR	0.00	0.00	96.43	0.00	32.6	32.6	23.6	24.6
VR	0.00	8.33	75.00	0.00	47.9	47.9	34.0	36.5

Appendix

Table 15: Model performances across video-based prediction folds (Claude Opus 4.1). Metrics are grouped into exact match accuracy, unfolding accuracy, hole count errors, and hole attribute accuracy. D = Diagonal, H = Horizontal, V = Vertical and R = Rotation

Panel of Figure 10 — Figure 10: Performance of the models on the generalization task: (a) exact match accuracy of baseline models; (b) the accuracy of the models for each category.

Model	Exact Match	Partial Accuracy	Hole Count Errors (%)		Unfolding Accuracy (%)
	(%)	(%)	Extra	Missing	Exact	Steps
o3	11.11	26.79	41.11	13.33	20.56	34.19
GPT-5.1	1.11	20.49	28.89	22.78	12.78	31.63.
Claude Opus 4.1	0.56	6.51	70.56	7.22	6.67	26.28
InternVL3	0.00	7.13	0.56	94.97	4.47	7.73
LLaVA-OneVision	0.00	5.98	41.34	58.66	0.00	18.27

Appendix

Table 16: Performance results of the backward prediction task.

Models	2D Image		Text		Video
Models	Exact Match (%)	Overall Score (%)	Exact Match (%)	Overall Score (%)	Exact Match (%)	Overall Score (%)
o3	19.05	39.39	25.07	47.13	15.87	40.60
Claude Opus 4.1	4.44	28.53	17.46	33.75	2.86	26.00
Claude 4	2.86	27.44	12.38	30.90	2.22	27.18
GPT-4o	0.32	20.40	0.32	20.42	0.32	17.58
GPT-5.1	1.27	25.62	3.17	25.51	1.27	26.37

Appendix

Table 17: Model Performance across Text, Video, and 2D Image Tasks

Figure 11: An example of a correct model case for the prediction task

Figure 12: An example of an incorrect case where unfolding actions are accurate but the result is incorrect

Figure 13: An example of an incorrect case where the model creates missing holes

Figure 14: An example of an incorrect case where the model creates extra holes despite accurate unfolding steps

Figure 15: An example of an incorrect case where the model creates extra holes by predicting more unfolding steps

Figure 16: An example of an incorrect case where the model creates extra holes by applying symmetry without considering the physical state of the paper

Figure 17: An example of a correct case where the paper is rotated

Figure 18: An example of an incorrect case where the paper is rotated

Figure 19: An example of a correct case for the planning task

Figure 20: An example of an incorrect case where the model creates extra holes

Figure 21: An example of an incorrect case where the model folds paper fewer times than required.

Figure 22: An example of an incorrect case where the physical dynamics of layer depth are not considered.

Figure 23: An example of an incorrect case where the model’s fold combinations physically distort the paper structure.

Figure 24: An example of an incorrect case where the model generates missing holes by calculating the initial location of the hole (trapezoid) wrong.

Figure 25: An example of a correct backward prediction task

Figure 26: An example of an incorrect backward prediction task

Figure 27: An example of a correct generalization task - direction

Figure 28: An example of a correct generalization task - location

Figure 29: An example of an incorrect generalization task - direction

Figure 30: An example of an incorrect generalization task - location

Figure 31: Text-based prediction prompt.

Figure 32: 2D image-based prediction prompt.

Figure 33: Video frame-based prediction prompt.

Figure 34: Video-based prediction prompt.

Figure 35: Video-based backward prediction prompt.

Figure 36: Video frame-based backward prediction prompt.

Figure 37: Planning prompt.

Figure 38: Generalization prompt.

Visual Spatial Reasoning

Visual spatial reasoning studies how AI systems infer geometry, object relations, viewpoints, and physical change from images and video.

See all 251 papers

SpaceCast-Bench: Evaluating Predictive Spatial Reasoning in Vision-Language Models
Oct 8, 2026Hongxing Li, Jinyue Su, Dingming Li +5Spatial Reasoning Spatial Reasoning Benchmarks
Zhejiang University
From Sight to Foresight: Predictive Spatial Reasoning in Vision-Language Models
Oct 2, 2026Feiran Wang, Xiaoqi Wang, Ziwei Li +3Visual Spatial Reasoning VLM Reasoning
University of Illinois at Chicago · Bosch Research North America · Bosch Center for Artificial Intelligence (BCAI)
3ViewSense: Spatial and Mental Perspective Reasoning from Orthographic Views in Vision-Language Models
Mar 8, 2026Shaoxiong Zhan, Yanlin Lai, Zheng Liu +6Visual Spatial Reasoning Spatial Reasoning in Language Models
Shenzhen International Graduate School, Tsinghua University, Shenzhen, China · Pengcheng Laboratory, Shenzhen, China · School of Software Engineering, Chongqing University, Chongqing, China

Oct 8, 2026J/K move · Enter open · S save

SpaceCast-Bench: Evaluating Predictive Spatial Reasoning in Vision-Language Models

Hongxing Li, Jinyue Su, Dingming Li, Wenqi Zhang+4

Zhejiang University

Spatial Reasoning Spatial Reasoning Benchmarks Visual Spatial Reasoning Spatial Reasoning in Language Models VLM Reasoning 3D Spatial Reasoning

Open paper AskPDF arXiv

Existing spatial reasoning benchmarks mainly test spatial perception: reading off relations already visible in the input. Yet real-world spatial intelligence demands predictive spatial reasoning: constructing a scene from observations, anticipating how an intervention changes it, and reasoning about the unseen outcome. We introduce SpaceCast-Bench, the first benchmark to directly and diagnostically evaluate this capability. Built around an observe-transform-infer framework, its 3,862 questions from 182 real-world scenes span 16 task types at three levels: static perception, local prediction, and global prediction, progressively requiring scene understanding, spatial state updating, and relational inference over unobserved outcomes. Evaluating 21 models exposes a stark gap: the strongest model reaches only 58.0% against 87.2% human performance, while spatially specialized models remain near random chance. Controlled analyses further reveal that bridge views are critical for integrating distributed observations, and that explicit 3D evidence benefits models more reliably than generated outcome images or videos. Fine-tuning on our programmatically generated data lifts Qwen3-VL-4B from 34.0% to 65.7% with macro-average gains across six out-of-domain benchmarks.

Abstract

Visual Spatial Reasoning

Explore similar work

SpaceCast-Bench: Evaluating Predictive Spatial Reasoning in Vision-Language Models

From Sight to Foresight: Predictive Spatial Reasoning in Vision-Language Models

3ViewSense: Spatial and Mental Perspective Reasoning from Orthographic Views in Vision-Language Models