Spatial Competence Benchmark

Jash Vira; Ashley Harris

Axiomatic
Task	$N$	Claude Sonnet 4.5	Gemini 3 Pro Preview	GPT-5.2
Topology Enumeration	10	40.0 +20.0	100.0 -50.0	30.0 $\pm$ 0.0
Enumerate Edges	8	12.5 +50.0	75.0 $\pm$ 0.0	100.0 -12.5
Classify Behaviour	13	38.5 $\pm$ 0.0	46.2 $\pm$ 0.0	53.8 -7.7
Half-Subdivision Neighbours	17	58.8 -23.5	88.2 $\pm$ 0.0	82.4 -5.9
Two Segments	27	63.0 +11.1	88.9 $\pm$ 0.0	88.9 $\pm$ 0.0

Appendix

Table 2: Per-task accuracy (%) on SCBench (no tools). Each subtask yields a score in

[0,1]

: eleven tasks use binary pass/fail; the remaining eleven award partial credit (or mixed binary+partial) via task-specific metrics (see individual task appendices). Per-task accuracy is the mean subtask score; bucket and overall accuracy weight every subtask equally. Coloured deltas denote tools minus no-tools (percentage points).

N

= number of subtasks.

Figure 4: Local-Only failure example on Delaunay Triangulation (curated 28-point subpass, seed 1009; Claude Sonnet 4.5 no-tools). Left: ground-truth triangulation. Right: model output. Judge label: Local-Only (Global Constraint Integration Failure) with confidence 0.90.

	OpenAI	Anthropic	Google
Model alias	gpt-5.2	claude-sonnet-4-5	gemini-3-pro-preview
Temperature	Provider default	Provider default	Provider default
Reasoning	effort=xhigh	thinking enabled (max)	thinking_budget=HIGH
Tools	code_interpreter ,	code_execution ,	code_execution ,
	web_search	web_search	google_search

Appendix

Table 3: Per-provider evaluation settings for the six runs (single-turn, zero-shot, one sample per instance, no self-correction).

Figure 5: Lego Hemispherical Shell (1, 2). (Left) A valid Lego hemisphere, 1524 bricks. (Right) Constructive solid geometry (CSG) intersection of Lego structure and hemispherical goal.

Step	Check
1	Each brick not on the ground must connect to another brick (up or down)
2	Bricks must align in X/Y to the 8mm stud grid of any blocks they intersect the studs of
3	No two bricks may occupy the same volume
4	Connected bricks must not form a part with a centre of mass outside of the part’s ground footprint
5	No bricks can be below ground, entirely within the inner hemisphere, or entirely outside of the outer hemisphere

Appendix

Table 8

Figure 6: Lego Hemispherical Shell (3, 4). (Left) An invalid Lego construction will fall over. (Right) Stacked bricks that do not align with the stud grid.

Figure 7: CSG Union (1, 2). (Left) Two cylinders intersecting at right angles at the origin, as described by the prompt. (Right) Two cubes intersection.

Consistency check
Empty lists, not 3D, invalid point Ids, NaN positions
Degenerate faces and duplicate vertices
Edge manifold check
Watertightness
Self intersecting face check
Non-planar face check

Appendix

Table 11

Figure 8: CSG Union (3). Three rectangular prism union.

Figure 9: The shadow cast from dozens of non-intersecting tetrahedra can form complex 2D shapes, like this washer-shape formed from a spiral.

Step	Check
1	Orientation quaternions are checked for normalisation, within 1% ( $-$ 75% penalty)
2	Intersection check ( $-$ 50% penalty)
3	Projection and 2D CSG is performed in OpenSCAD
4	Score calculated from 2D CSG areas of intersection, overlap, and symmetric difference

Appendix

Table 14

Common mistake
The tetrahedron centre of rotation is not its centre of mass
Normalisation of quaternions; this is clearly asked for in the prompt
If they try to solve the problem in 2D without considering 3D, the tetrahedra will intersect

Appendix

Table 15

Figure 10: Voxel Grid Projection. Valid solution to 20x20x20, 500 voxels, and no voxel having “7” in str(x + y + z). “Noise” voxels are used to plug gaps and break symmetry.

Step	Check
1	Exact number of voxels required
2	No repeated voxel coordinates
3	All coordinates within [0, N-1]
4	Every (x,y) pair must have at least one voxel at some z
5	Every (x,z) pair must have at least one voxel at some y
6	Every (y,z) pair must have at least one voxel at some x

Appendix

Table 17

Figure 11: Voxel Grid Projection (subtasks 3, 4). (Left) The model lost focus while building this structure. Its poor coverage of the XZ and YZ plane scored it a 0. (Right) Poor coverage of a 16x16x16 grid. Voxel coordinates span [4,12] instead of [0,15].

Figure 12: 3D Maze (1). A valid 3D maze with 7 jumps (shown in yellow).

Rule
Grid must be filled in fully and of size Width $\times$ Height
No single elevation value may occupy > 30% of cells
There must only be one solution
The path must visit > 20% of cells
You can walk to a neighbouring cell if the elevation is +/ $-$ 1
If a neighbouring cell has a drop $\geq$ 2, and then there’s a cell at the same elevation, you can jump

Appendix

Table 20

Figure 13: 3D Maze (2, 3). (Left) A 5x5 maze with 2 jumps, a valid solution to subpass 0. (Right) Two parallel jumps (shown in red) mean multiple solutions or loops.

Step	Check
1	Trivial things are checked: the grid is square, correctly dimensioned, heights are distributed
2	Breadth-first search (BFS) is used to find the shortest path, including jumps
3	Depth-first search (DFS) is used to find articulation points and discover any ‘bridges’ implying multiple paths

Appendix

Table 22

Figure 14: 3D Maze. A model’s attempt at subpass 1, failed due to duplicate paths.

Figure 15: Polynomial Curve Fitting (2). Expected (left) vs. actual (right): a model’s attempt at a 128x128 space partition.

Step	Check
1	First the grader checks for forbidden characters. After removing ‘return’, ‘def’ and ‘f’, there must only be x, y, digits, operators, and whitespace
2	Assuming that safety check passes, the code is compiled and executed
3	f(x,y) is evaluated at every grid cell
4	The score is recorded as a fraction of correctly classified cells

Appendix

Table 25

Figure 16: Hamiltonian Loop (1). A 16x16 grid with 228 visitable cells and a solved Hamiltonian loop.

Step	Action
1	Check a cache of known, solved, subproblems against the grid, seeing if the particular patterns of obstacles have been seen before. Check from largest to smallest. If so, excellent. Mark that chunk as solved
2	Split remaining grid into 2x2 chunks, mark as either solvable (all 4 empty) or unsolvable (has obstacles)
3	Unsolvable chunks grow by absorbing neighbors until solvable
4	Once solvable, attempt to solve using DFS solver. If solved, cache it, otherwise grow
5	Chunks merge when they collide during growth. If chunk size gets too big for DFS, subdivide or restart
6	Once obstacle chunks are solved, grow open space chunks into similar-sized chunks

Appendix

Table 27

Step	Check
1	Path length must equal the number of valid (non-blocked) cells
2	All coordinates within bounds of the grid
3	Path may not visit blocked cells
4	Each consecutive pair must be 4-connected / side adjacent
5	Each cell visited exactly once
6	The last cell must be adjacent to the first

Appendix

Table 28

Figure 17: Hyper-Snake (1). (Left) 2D snake is trivial. The snake (start = blue, head = yellow, body as gradient) collected the food (green) and filled the entire space. (Right) The snake might struggle to access the food due to being blocked by 3 boxes (in X, Y, and Z). However this is a 4D snake, and the W axis is not blocked, allowing the food to be accessed as the snake’s final act.

Step	Check
1	Path must begin at specified start
2	Each step must have correct number of coordinates
3	Each step changes exactly one coordinate by $\pm$ 1
4	All positions within grid extents
5	No position may coincide with a wall
6	Snake body grows; cannot revisit own path

Appendix

Table 30

Figure 18: Hyper-Snake (2). Mean score by subpass across six model runs, with dimensionality progression from 2D to 10D.

Figure 19: Pipe Loop Fitting (1). 14 lengths of pipe laid out end-to-end in a 3x3 square.

Stage	Check
1. Basic validation	Minimum point count
	Loop closure (last point 1m away from first)
	No duplicate vertices
	Within bounds
2. Segment analysis	Compute all line segments
	Detect and count intersections

Appendix

Table 33

Figure 20: Pipe Loop Fitting (2, 3). (Left) 390 pipes in 35 x 35, solved with spring solver (subpass 37). (Right) Mean score across all 40 subpasses under six model runs.

Pass	People	Building width
0	4	2m
1	20	4m
2	40	6m
3	80	8m
4	150	10m
5	200	7m

Appendix

Table 35

Figure 21: Hide and Seek (1). View from above the crowd (left) and sniper’s view (right). This is a clear failure and scores 0.

Step	Check
1	Correct number of people
2	All positions are numeric and 2D
3	Each person within maximum distance from building center
4	Person bounding boxes may not intersect building
5	No overlapping bounding boxes

Appendix

Table 37

Figure 22: Pack Rectangular Prisms (1). 108 prisms (8 size classes) packed 96% effectively.

Step	Check
1	Individual box sizes are counted, and we check to make sure that the correct count of each size is present
2	We check for overlaps: no two boxes may intersect or overlap
3	Compute bounding box of all placed boxes. Efficiency = total_prism_volume / bounding_box_volume
4	If perfect packing is theoretically possible: score = efficiency. Otherwise: score scaled relative to theoretical maximum

Appendix

Table 39

Figure 23: Pack Rectangular Prisms (2, 3). (Left) Mean score by subtask across six model runs. (Right) A model’s perfect packing for subtask 1.

Figure 24: Fluid Simulation (1). A successful earthworks project that diverts rainfall from the centre of the map to 3 unconnected lakes.

#	Requirement
1	A lake of at least 36 voxels in surface area
2	3 unconnected bodies of water, each at least $2{\times}2$
3	A lake of at least 2 voxels at $z>5$
4	An underground lake of at least 10 voxels in volume but no water voxels are visible from above
5	3 lakes on 3 different z levels
6	A lake at least 6 voxels deep

Appendix

Table 42

Step	Check
1	Apply earthworks: modify terrain grid per model instructions
2	Rock stability simulation: floating rock (disconnected) falls until supported
3	Water simulation: drop individual water voxels from above (water falls until hitting rock or other water); flood-fill for pooling behavior; water reaching grid edge drains away; iterate until water stabilizes
4	Evaluation against target: count water surface area; count disconnected water bodies; check water at specific Z levels; check if water is visible from above; measure exact volumes

Appendix

Table 43

Figure 25: Terrain Leveling (1, 2). (Left) Typical starting world, with the largest flat city (5 cells) shown in green, 16x16 with z range 1 to 10. (Right) Typical blasting plan.

Figure 26: Terrain Leveling (3, 4). (Left) Blue cells represent the new city after leveling. (Right) Terrain delta for the blast pattern. Red represents elevation decrease, blue represents elevation increase.

Step	Action
1	When detonation occurs at (x, y, depth), the heightmap is reduced by depth
2	(Depth $\times$ 4) spheres in a vertical stack are added to a PyBullet sim at x,y
3	The simulation runs until stable
4	Where each sphere ended up, the heightmap there is raised pro-rata
5	Rocks that fall off the world are lost

Appendix

Table 46

Figure 27: Interlocking Parts (1, 2). (Left) A single threaded rod cannot hold these panels together, and the corner bar is twice as wide as the other bars. (Right) These panels interlock when oriented for assembly and can be held by a single threaded rod.

Subpass	Check
0: Basic validation	Exactly 10 parts; each part is or generates valid STL; non-zero volume; watertight meshes; consistent face winding; no non-manifold geometry; single connected body per part
1: Printability	Parts fit in build volume ( $400{\times}400{\times}50$ ); rest on $z{=}0$ plane; sufficient bed contact area ( $\geq 500$ mm 2 ); no large unsupported overhangs; will not fall over during printing
2: Bar gap verification	100 randomly placed cubes, per part, are used to check that: 40mm cubes can pass through gaps (gap $\geq$ 50mm) due to at least one intersection test resulting in a zero volume per part; 101mm cubes cannot pass through (gap $\leq$ 100mm), as all intersections must have a positive volume
3: Assembly size	Assembled cage fits 350 $\times$ 350 $\times$ 700 internal space; not larger than 380 $\times$ 380 $\times$ 740 external
4: Rod clearance	6mm rod passes through corners without intersection; 16mm rod blocked (adequate material around holes)
5: Part interference	No two parts intersect when assembled. All 90 combinations are checked

Appendix

Table 48

Figure 28: Interlocking Parts (3). Rod engagement tests performed using OpenSCAD (subtask 6). Zero rods result in a panel falling out; one rod results in a hinge.

Figure 29: Interlocking Parts (4). A valid (human-created) solution, showing all 10 parts and the completed assembly.

Digit 8 constraints
Cannot be printed on its long or short side (bridge overhangs)
When printed flat, must not occur after any number except 8 (or the start)

Appendix

Table 51

Regex branch
^[012345679,]+
^8[012345679,]+
^88[012345679,]+
^888,[012345679,]+
^8,888[012345679,]+
^88,888[012345679,]+

Appendix

Table 52

Figure 30: Largest 3D-Printable Prime (1, 2). (Left) A model’s suggestion of 3,317 is not 3D-printable: the brown digit 1 sags without support, scoring 0. (Right) The largest 3D-printable prime without repeating triplets.

Step	Check
1	Triplet constraint. No sequence of 3 consecutive digits repeated (constrains the search space to prevent infinite solutions)
2	Printability. Check for overhangs in each digit’s orientation. 8 cannot be printed on its long or short side; 7 can be printed on its head and side but not its base; 3 can only go on its long side
3	Stackability. Flat-on-flat: determined by which segments are in use. 7 can go on 3, but not vice versa. 6 cannot go on 9 flat, but can when rotated 180° or flipped. Only 7s, 3s, and 1s can go non-flat. Only vertical 1 “spires” can go on top of non-flat 3s or 7s
4	Height calculation. Flat: $+1.2$ mm; rotate90X: $+21.2$ mm; rotate90Y: $+11.2$ mm (or $1.2$ mm for “1”). A 0.2 mm gap in $Z$ produces parts that hold together during printing but snap apart afterwards
5	Primality test (Baillie–PSW). Miller–Rabin base 2; strong Lucas test; results cached
6	Scoring. Score $=$ len(answer) / len(known_best); known best: 42-digit number; 10% penalty if height ${<}\,70$ mm

Appendix

Table 54

Step	Action
1	Skip if smaller than the current best
2	With comma positions now known, check for repeated triplets
3	Run primality test on the $\scriptstyle\sim$ 40-digit candidate
4	Record as new best; continue searching

Appendix

Table 55

Figure 31: Topology Enumeration (1, 2). (Left) Labelling the 4 corners with alternating 1s and 0s ensures that a border between classes must touch all edges, and thus a boundary between the two classes must exist inside the square. (Right) 3 corners labeled 0 and 1 labeled 1 infers a class boundary intersecting 2 edges, and thus there are two regions inside the square.

Figure 32: Enumerate Edges (1). Given the vertex labels, boundaries between regions are unavoidable. List all the connections that are guaranteed to exist (green lines).

Figure 33: Classify Behaviour (1). Classifying the intersections.

Figure 59

Split cycle
Z, Y, X, Z, Y, X …
X, Y, Z, X, Y, Z …
X, X, Y, Z, X, X, Y, Z …
Z, Y, X, X, Z, Z, Y, X, Z …
Y, Y, X, Y, Z, Y, Y, X, Y, Z

Appendix

Table 60

Figure 34: Half Subdivision Neighbours (2). Subdivision neighbour test performed in 3D.

Figure 35: Delaunay Triangulation (1). This triangulation required 5 triangles; 6 were provided.

Check
All triangles use valid point indices
Triangulation is complete (covers convex hull)
Delaunay criterion: no point inside circumcircle of any triangle
This can’t use the “ground truth” reference without first confirming there are no co-circular points

Appendix

Table 63

Figure 36: Shikaku Rectangles. A solved

7{\times}7

instance: each coloured rectangle contains exactly one clue equal to its area.

Check
Rectangles tile the grid exactly (no gaps, no overlaps)
Each rectangle contains exactly one clue number
Each rectangle’s area equals its contained clue

Appendix

Table 65

Figure 37: Two Segments. The model must place two segments on the square boundary to partition the interior into a target set of regions (here: 1 pentagon, 2 quadrilaterals, and 1 triangle).

Bucket	$N$	Claude Sonnet 4.5	Gemini 3 Pro Preview	GPT-5.2
Axiomatic	75	49.3 +6.7	81.3 -6.7	74.7 -4.0
Constructive	171	30.2 +4.3	51.4 +12.3	51.9 +15.0
Planning	39	27.5 -7.8	39.0 +23.6	50.0 -8.3
Overall	285	34.9 +3.2	57.6 +8.8	57.6 +6.8

Spatial Competence Benchmark

Abstract

Figures & tables

Appendix

Spatial Reasoning Benchmarks

Spatial-IQ: Deconstructing Spatial Intelligence via Hierarchical Capability Tests

SpatialWorld: Benchmarking Interactive Spatial Reasoning of Multimodal Agents in Real-World Tasks

SpaceCast-Bench: Evaluating Predictive Spatial Reasoning in Vision-Language Models

Abstract

Spatial Reasoning Benchmarks

Explore similar work

Spatial-IQ: Deconstructing Spatial Intelligence via Hierarchical Capability Tests

SpatialWorld: Benchmarking Interactive Spatial Reasoning of Multimodal Agents in Real-World Tasks

SpaceCast-Bench: Evaluating Predictive Spatial Reasoning in Vision-Language Models