PolyComp: A Polycube-based Benchmark for Compositional 3D Spatial Reasoning in Multimodal Models

Siddharth Patel

cs.CVAug 13, 2026

PolyComp: A Polycube-based Benchmark for Compositional 3D Spatial Reasoning in Multimodal Models

Authors: Siddharth Patel

Abstract

We introduce PolyComp, a procedurally generated and verified benchmark that stresses visual recognition and compositional spatial reasoning. In each problem, a model must identify which of four options shows a pair of polycube components that can be combined to form a target solid. The benchmark contains 120 problems across four geometry families, and each problem has three different presentation formats using either a single image or multiple images. The random guessing baseline is 25%. Across the three presentations (360 presented problems per model), GPT-5.6 Sol with max effort attains 50.0% accuracy (95% problem-cluster CI 43.3-56.7%) at a mean cost of $0.951 per presented problem, Claude Fable 5 with max effort attains 39.4% (33.1-46.1%) at $0.701, and Gemini 3.1 Pro Preview with thinking level high attains 27.5% (22.8-32.5%), near the 25% random guessing baseline, at $0.350. The observed accuracy spread across geometry families is larger than across presentation formats. We present a problem development and evaluation protocol, cost and token accounting, and release the 120 problems.

Figures & tables

Figure 1: Example PolyComp problem. The model is given the two views of the target solid and then asked to identify which of options A through D shows two components that together can form the target solid. This is one of the simpler problems missed by the three evaluated models, all of which picked D instead of the correct option A.

Figure 2: Problem generation and validation. The flow chart shows target solid construction, partitioning, incorrect option generation, view selection, geometry verification, rasterization, and asset QA.

Field	OpenAI	Anthropic	Gemini
Endpoint and mode	Batch wrapper around POST /v1/responses	/v1/messages/batches	:batchGenerateContent
Model; dates	gpt-5.6-sol ; July 20–25	claude-fable-5 ; July 20	gemini-3.1-pro-preview ; July 20
Reasoning and output cap	reasoning.effort=max ; 128,000 tokens	output_config.effort=max ; managed thinking; 128,000 tokens	thinkingLevel=high ; 65,536 tokens
Image accounting	base64 PNG data URL; detail=original	base64 image source; provider-managed resolution	base64 PNG inline_data ; MEDIA_RESOLUTION_HIGH
Regular list rates	input $5; cached$ 0.50; cache write $6.25; output$ 30	input $10; cache read$ 1; 5m/1h write $12.50/$ 20; output $50	input $2; cached$ 0.20; output (including thinking) $12

Table 1: Protocol. Dates identify the evaluation window. Prices are regular API list rates per million tokens, captured July 16 and rechecked July 25, 2026. Sampling parameters were omitted.

Model	Accuracy	95% CI	Mean tok.	Median tok.	Mean cost	Median cost
GPT-5.6 Sol	50.0%	[43.3, 56.7]	42,682	40,549	$0.951	$0.881
Claude Fable 5	39.4%	[33.1, 46.1]	19,016	17,961	$0.701	$0.634
Gemini 3.1 Pro Preview	27.5%	[22.8, 32.5]	32,566	30,430	$0.350	$0.329

Table 2: Primary results. Confidence intervals are 95% problem-cluster percentile-bootstrap intervals over 120 problems, with all presentations for a sampled problem resampled together. Token summaries are per final response. Costs are normalized to regular, non-batch API list prices for every provider.

Figure 3: Accuracy versus cost. Each point summarizes one model over 360 problem-presentation responses. The x-axis is mean list-price cost per presented problem; vertical whiskers are 95% problem-cluster bootstrap intervals. The dashed line is the 25% uniform random-guessing baseline.

Figure 4: Accuracy by model and presentation formulation. Whiskers are pointwise 95% problem-bootstrap intervals over 120 problems. The dashed line marks the 25% random guessing baseline. There is weak evidence that the multi-image presentations improved performance for GPT-5.6 Sol and Claude Fable 5.

Spatial Reasoning Benchmarks

Spatial reasoning benchmarks test whether AI systems can perceive, represent, and act on geometric, relational, and viewpoint-dependent information.

See all 162 papers

Spatial-IQ: Deconstructing Spatial Intelligence via Hierarchical Capability Tests
Jul 24, 2026Patrick Rim, Tom Long, Ekta Prashnani +63D Spatial Reasoning Visual Spatial Reasoning
1NVIDIA Research · 2Yale University
StateSight: Benchmarking Latent Spatial-State Reconstruction in Vision-Language Models
Aug 15, 2026Michelle LinVLM Evaluation Visual Spatial Reasoning
Thomas Jefferson High School for Science and Technology Alexandria, Virginia, USA
Spatial Competence Benchmark
Mar 5, 2026Jash Vira, Ashley HarrisSpatial Reasoning Benchmarks Spatial Reasoning
Independent Researcher · Maptek

Jul 24, 2026J/K move · Enter open · S save

Spatial-IQ: Deconstructing Spatial Intelligence via Hierarchical Capability Tests

Patrick Rim, Tom Long, Ekta Prashnani, Ruth Rosenholtz+5

1NVIDIA Research · 2Yale University

3D Spatial Reasoning Visual Spatial Reasoning Object Counting Multimodal Model Evaluation Spatial Reasoning Benchmarks Spatial Reasoning in Language Models+1

Open paper AskPDF arXiv

Multimodal large language models (MLLMs) excel at visual interpretation but fail on spatial reasoning tasks that humans solve reliably. Existing benchmarks evaluate these models as black boxes, limiting their ability to identify the underlying causes of lower performance: when a model fails a spatial reasoning task, it remains difficult to ascertain whether the hurdle is perceptual, such as recognizing object boundaries, or cognitive, such as reasoning about occlusion to infer hidden geometry. We introduce Spatial-IQ, a hierarchical diagnostic framework that decomposes object counting in stacked 3D structures into 9 perceptual and cognitive sub-tasks organized by the developmental stages of human spatial cognition, with mental rotation as an additional target probe. Using NVIDIA Isaac Sim, we procedurally generated a diverse dataset of roughly 80,000 stacked 3D structures with per-task ground truth. We evaluate models across three output formats (free-response text, multiple-choice images, and image editing) alongside a human baseline. The Spatial-IQ framework shows that top-performing models often succeed at the target task (object counting) without succeeding on the lower-level sub-tasks intended to support it, and that models differ in how much of these hierarchical chains they preserve, often revealing shortcut behavior that raw target-task accuracy alone would obscure. Finally, we demonstrate that training models with chain-of-thought (CoT) supervision over our hierarchical sub-tasks, combined with reinforcement learning with verifiable rewards, significantly improves both spatial consistency across sub-tasks and target-task accuracy, supporting the value of the proposed decomposition as both a diagnostic tool and a training signal.

Abstract

Spatial Reasoning Benchmarks

Explore similar work

Spatial-IQ: Deconstructing Spatial Intelligence via Hierarchical Capability Tests

StateSight: Benchmarking Latent Spatial-State Reconstruction in Vision-Language Models

Spatial Competence Benchmark