How Far Are Vision-Language Models from Constructing the Real World? A Benchmark for Physical Generative Reasoning

Luyu Yang; Yutong Dai; An Yan; Viraj Prabhu; Ran Xu; Zeyuan Chen

Figure 2: DreamHouse Dataset Overview. (Top) Dataset statistics showing member count distributions (box plots, left axis) and style proportions (dashed line, right axis) across all 13 architectural styles, ordered by decreasing structural complexity. Member counts range from 133 to 1,548 (mean 673). (Bottom) Representative Cycles-rendered timber frame structures for each style: Courtyard, Cruciform, Colonial, Split-Level, Barn, Ranch, Townhouse, Z-Plan, Carriage, Farmhouse, Saltbox, Shotgun, A-Frame , spanning complex multi-wing configurations to compact single-story forms.

Figure 3: Structural Validation Suite. 10 tests across four pillars. IRC Compliance (left): load path, span limits, completeness, stability score. Structure Physics (center-left): L/360 deflection, cantilever ratio, point load bearing. Geometric Integrity (center-right): dual-end restraint, gap detection, roof coverage. LoD 350 (right): fabrication-grade member geometry.

Figure 4: Task formalization example Planner-Managed. Top (agent loop): Evaluation is formalized as a recurrent agentic process. At each turn

t

, the agent

\mathcal{A}

(VLM) receives observation

o_{t}=(I_{0},\,f_{t-1})

, the original multi-view task image

I_{0}

and structured validation feedback

f_{t-1}

from the previous turn, and generates a Blender Python action

a_{t}

. The executor

\mathcal{E}

applies

a_{t}

to transition the scene graph

s_{t-1}\to s_{t}

, and the validator

\mathcal{V}

produces feedback

f_{t}

reporting per-test pass/fail and violation counts. This feedback becomes the input observation for the next turn, closing the loop. On failure (e.g.,

f_{2}

, red), the agent retries from the same scene state

s_{2}

without resetting context.

\mathcal{A}

,

\mathcal{E}

, and

\mathcal{V}

are omitted from the diagram for visual clarity; see Section 3.2 for full formalization. Bottom (task instantiation): The task input

I_{0}

consists of five rendered views of the target structure paired with building context and rules. The agent reasons over these to produce and iteratively revise construction code; the environment executes the code in Blender and the validator returns structured diagnostic feedback driving the next revision.

Figure 5: Planner-Managed qualitative example AF-01-0060 (A-frame style). All three models begin from scratch and receive stepwise visual feedback toward the same target structure. Despite reaching a valid result at step 6, Gemini and Claude employ markedly different construction strategies. Gemini pursues a top-down, shape-first approach: it approximates the overall silhouette early and refines toward it. Claude reasons bottom-up: it first establishes a structurally sound interior frame, then lays the roof rafters over it in the final step – a sequence closer to how a builder would physically construct an A-frame. GPT-5 fails to recognize the defining constraint of A-frame geometry, that the roof planes double as load-bearing walls, and from step 6 onward enters a false loop of adding conventional wall studs along the perimeter. Unable to escape this structural misconception, it exhausts all attempts without producing a valid result. This example highlights that identical visual feedback can elicit fundamentally different reasoning strategies, and that success depends not just on visual matching ability but on implicit architectural knowledge.

Model	Protocol	Structural Pass Rate		Visual Fidelity		Topological Fidelity		Joint Pass Rate
Model	Protocol	Frame	Facade	Frame	Facade	Frame	Facade	Frame	Facade
GPT-5	Planner-Atomic	0.792	0.637	0.312	0.287	0.143	0.151	0.035	0.019
	Planner-Reactive	0.302	0.247	0.293	0.266	0.141	0.143	0.003	0.003
	Planner-Managed	0.333	0.157	0.179	0.175	0.178	0.157	0.008	0.003
Claude	Planner-Atomic	0.716	0.779	0.406	0.377	0.164	0.168	0.071	0.064
	Planner-Reactive	0.428	0.494	0.239	0.245	0.133	0.141	0.003	0.013

Table 1: Consolidated Performance Metrics for GPT-5, Claude Opus 4.5, and Gemini 3 Flash. Frame = bare timber framing; Facade = finished exterior. Bold = best per column; shaded rows = best model per protocol by average across all conditions. Structural Pass Rate in

[0,1]

: passes only if all 16 validation tests clear. Visual Fidelity : average visual similarity score across 5 rendered views. Topological Fidelity : structural similarity across member counts, spatial alignment, and volumetric overlap. Joint Pass Rate in

[0,1]

: fraction satisfying both structural and visual criteria simultaneously.

Model	Baseline	Final	Best	$\Delta$	% Improved Tasks	Structures Retained
	Mean Visual Score
Claude	0.408	0.441	0.461	+0.033	70.3%	68.5%
GPT-5	0.392	0.405	0.425	+0.013	61.4%	78.1%
Gemini	0.433	0.443	0.471	+0.010	56.1%	73.8%

Table 2: Iterative refinement with visual feedback. Each model starts from its Planner-Atomic structural passes and iterates up to 10 visual-feedback revisions. Baseline = oneshot output before revision; Final = last attempt; Best = highest-scoring attempt across all iterations.

\Delta

is Final

-

Baseline mean score. Improved Tasks = fraction of tasks that improved their visual fidelity score. Structures Retained = fraction retaining structural validity at the final attempt. Bold = best per column.

Figure 6: Iterative Refinement with Visual Feedback (Section 4.2 , qualitative example BN-01-0273 (barn style). Starting from a structurally valid Planner-Atomic output, each model receives up to 10 rounds of side-by-side visual feedback (current render vs. target) and iteratively revises its structure while maintaining structural validity; steps 9–10 are omitted as changes from step 8 are negligible. Claude demonstrates meaningful visual self-correction: it produces a plausible roof (called: gambrel style) by step 2 but with an incorrect ridge orientation, then progressively corrects across subsequent steps, converging to the almost right form by step 8. GPT-5 commits to an incorrect orientation from step 1 and fails to recover, instead collapsing into a hybrid that blends both orientations without resolving either. Gemini makes minimal modifications across all 10 attempts, suggesting it does not meaningfully incorporate the visual feedback signal. This example illustrates the key behavioral divergence in this experiment: models differ not only in final score, but in whether they engage with feedback at all.

Figure 7: Structural failure analysis across pipelines and models. (a) Per-structure failure rate for each of the 10 structural validation tests, shown for the Planner-Atomic and Planner-Managedprotocols across all three models (

N

= 1,200 per model for Planner-Atomic;

N

\approx

1,160–1,197 for Planner-Managed, excluding Phase-0 failures). Each cell reports the percentage of structures for which that test failed at least once across all retry attempts. Values are independent marginal failure rates per test and do not sum to 100% per row, as a single structure can fail multiple tests simultaneously. Bold column headers indicate the five dominant failure tests. (b) Co-failure pattern distribution: each structure is assigned to the exact set of tests it failed, ranked by frequency. Orange bars show Planner-Atomic(

N

= 3,551); blue bars show Planner-Managed(

N

= 3,484), each normalized to their respective totals. The top four Planner-Atomicpatterns share a Span + Deflection core, reflecting near-universal geometry violations. Planner-Managedfailures are dominated by Load + Stability errors, indicating structural connectivity issues rather than geometric ones.

Figure 8: Typical failure modes of Claude 4.5 opus (Planner-Reactive). Each pair shows a ground-truth target structure (left) alongside Claude’s generated output (right) for configurations RN_01_0035 (left pair) and AF_05_0713 (right pair). Claude’s outputs exhibit two characteristic failure patterns: RN_01_0035 produces an overly sparse frame with missing members, while AF_05_0713 produces an overly dense packing of repeated elements. Both reflect insufficient adherence to IRC structural rules, the model optimizes for visual plausibility rather than code-compliant member placement, resulting in structures that fail span limit and deflection validation despite appearing superficially similar to the target geometry.

Figure 9: Representative failure modes of Gemini-3-flash on A-frame structures under the Planner-Reactive protocol . Each pair shows the target structure (left) and the model output (right). Left pair AF_03_0392 : Gemini correctly identifies the A-frame geometry and rafter count but misplaces the side wing, attaching it to the wrong elevation face and inverting the attachment axis, producing a structure that passes visual inspection from one view but fails load-path and dual-end connection tests. Right pair AF_04_0460 : The ridge direction is rotated

90^{\circ}

relative to the target, causing all rafters to span the wrong axis; the resulting assembly passes geometric tests (

T_{3}

,

T_{4}

) but fails roof coverage (

T_{6}

) and gap detection (

T_{7}

) entirely. These errors illustrate a systematic failure mode: Gemini recovers what structural elements are present but not where they are oriented, consistent with

O3

: the model attends to member topology but not to the 3-D spatial relationships encoded in the multi-view input.

Figure 10: Representative failure modes of GPT-5 on A-frame structures under the Planner-Reactive protocol. Each pair shows the target structure (left) and the model output (right). Left pair AF_02_0182 : GPT-5 reproduces the correct rafter count and overall A-frame silhouette but generates collar ties and floor joists that extend beyond the structure’s bounding envelope, with members protruding outside the intended footprint. The assembly fails dual-end connection (

T_{10}

) and gap detection (

T_{7}

) as a result. Right pair AF_03_0318 : A similar pattern: the rafter and joist layout is topologically correct but spatially unconstrained, members overshoot the ridge and sill attachment zones, leaving floating ends unconnected to the load path. Both failures share a common root cause: GPT-5 reasons about which members to place but not about the spatial envelope that bounds them, treating member generation as an independent local decision rather than a globally constrained packing problem. This geometric incapability, the inability to enforce that every member end terminates within a valid connection zone, is the dominant failure signature of GPT-5 across A-frame styles and is consistent with

O2

: structural reasoning is not monolithic, and spatial constraint satisfaction is a capability that does not co-occur with topological correctness.

Test	Name	Pass Condition	Category
$T_{1}$	Load Path	$\sigma(v_{i})=1\;\forall i$	IRC / Structural
$T_{2}$	Span Limits	$L_{i}^{\text{eff}}\leq(1+\tau)L^{*}$	IRC / Structural
$T_{3}$	O.C. Spacing	$s_{\ell}\in\mathcal{S}\pm 0.05\,\text{m}$	IRC / Geometric
$T_{4}$	Std. Dimensions	$(w_{i},d_{i})\in\Lambda$	Geometric
$T_{5}$	Deflection L/360	$\delta_{i}\leq(1+\tau_{\delta})\,L_{i}/360$	Physics
$T_{6}$	Roof Coverage	$\rho\geq 0.70$	Geometric

Table 12

LoD 350 Requirement	Tests	What Failure Means
(i) Accurate geometry	$T_{4}$ , $T_{2}$ , $T_{5}$	Non-standard cross-sections; spans exceeding IRC limits; serviceability failure under design load
(ii) No hard clashes	$T_{7}$ , $T_{3}$	Members overlap or leave gaps wider than one stud bay; framing cannot be physically assembled
(iii) Interface definition	$T_{10}$ , $T_{6}$	Rafters/studs free at one end (hinge failure under load); roof plane incompletely framed
(iv) Load-path integrity	$T_{1}$ , $T_{9}$ , $T_{8}$	Floating members; unsupported elevated sections; Topological Stability Index $<1.0$

Table 3: Mapping of LoD 350 constructibility requirements to the DreamHouse validation tests.

View	Azimuth	Elevation	$c_{v}$
Front	$0^{\circ}$	$12^{\circ}$	1.05
Back	$180^{\circ}$	$12^{\circ}$	1.05
Left	$270^{\circ}$	$12^{\circ}$	1.05
Right	$90^{\circ}$	$12^{\circ}$	1.05
Front-right	$45^{\circ}$	$18^{\circ}$	1.10

Table 4: Camera configuration for the five views.

c_{v}

is the distance multiplier.

Figure 11: Representative renders from the bridge generation pilot: glulam Howe truss (left), steel Pratt truss (center-left), steel arch (center-right), and steel Warren truss (right). Each bridge is fully parameterized by a low-dimensional BridgeParams dataclass whose scalar fields: span_length , truss_height , num_panels , deck_width project directly to measurable visual quantities in the render. Frontier VLMs recovered valid parameters from a single reference image in one shot with no iterative refinement, exposing the domain as a visual measurement task rather than a test of structural reasoning. This failure mode motivated the shift to timber-frame construction, where the structurally critical quantities are interior to the assembly and not directly legible from exterior renders.

Figure 12: Iterative Refinement with Visual Feedback trajectories for Claude 4.5 opus on four similar barn-style structures. Each pair of rows shows the target (leftmost, lighter color) followed by successive iterations (left to right, darker). Dashed horizontal lines separate groups of structurally distinct targets. Within each group, targets share the same architectural style and similar proportions, yet Claude consistently pursues different refinement strategies across instances: some trajectories make large structural revisions between iterations (roof skeleton replaced); others converge quickly and make only minor adjustments; others oscillate without monotonic improvement. This variability within a homogeneous style group reveals that Claude’s behavior is not driven by a stable visual error signal, it does not reliably identify the same class of discrepancy across similar targets and apply a consistent corrective action. Rather, each iteration represents an independent re-interpretation of the feedback, which can move the output toward or away from the target depending on which aspect of the render the model attends to. This is consistent with

O1

and

O3

: visual similarity and structural validity are orthogonal, and the ability to see a structural error in a rendered image does not imply the ability to correct it in the next generation.

Figure 13: The 13 architectural style archetypes visualized in Blender, shown as wireframe meshes in the 3D viewport. Each archetype defines the canonical massing, roof type, section layout, and story count for one style category in the DreamHouse dataset. The procedural generation pipeline instantiates each archetype across a range of lot sizes, proportions, and complexity levels to produce the 26,000+ structure dataset. Structures are rendered headless via the Blender Python API ( bpy ); the console output visible at the bottom shows a representative member placement call during procedural construction.

Model	Protocol	Struct. Pass	Visual Fidelity	Visual Pass ( $\geq$ 0.6)	Joint Pass
GPT-5	Planner-Managed	0.333	0.179	0.022	0.008
GPT-5	+ visual feedback	0.317	0.173	0.030	0.009
Claude	Planner-Managed	0.713	0.278	0.043	0.031
Claude	+ visual feedback	0.780	0.288	0.122	0.094
Gemini	Planner-Managed	0.785	0.313	0.054	0.043
Gemini	+ visual feedback	0.740	0.311	0.133	0.089

Table 5: Stepwise (Planner-Managed) without per-step visual feedback vs. with in Frame (timber framing as input). Joint pass rate is the primary metric: structures that are both structurally valid and visually similar (

S\geq 0.6

).

Figure 14: Representative outputs of Qwen3.5-397B-A17B under Planner-Managed, showing five target–output pairs across two views (front elevation, top; diagonal, bottom). Failures are more fundamental than those observed in frontier models: columns 1, 2, 4 show incorrect rafter orientation — rafters span the wrong axis entirely, producing a roof that is geometrically inverted relative to the target; column 3 exhibits severe global disproportion, with the generated structure roughly half the height and twice the footprint of the target, indicating a failure to infer absolute scale from the multi-view input; column 5 shows a collapsed depth dimension, the 3D volume is flattened into a near-planar assembly that passes a front-view silhouette check but has no structural depth. Unlike the frontier model failures in Figures 8 and 9 , which involve spatially misplaced but topologically coherent members, Qwen3.5 errors reflect a lack of basic 3D spatial grounding, the model does not reliably recover the orientation, scale, or depth of the target structure from the provided views.

Protocol	Model	Temp.	Max Tokens	Max Retries	$\tau_{\mathrm{retry}}$	Context History	Render Engine
Planner-Atomic	GPT-5	N/A	16,384	5	0.8	full	-
Planner-Atomic	Gemini	0.0	16,384	5	0.8	full	-
Planner-Atomic	Claude	0.0	16,384	5	0.8	full	-
Planner-Reactive	GPT-5	N/A	64,000	10	0.6	full	-
Planner-Reactive	Gemini	0.0	64,000	10	0.6	full	-
Planner-Reactive	Claude	0.0	64,000	10	0.6	full	-

Table 6: API hyperparameters per model and protocol.

\tau_{\mathrm{retry}}

: visual similarity threshold below which the pipeline retries the current step or full generation. N/A : parameter not supported by the API. — : not applicable for this protocol.

How Far Are Vision-Language Models from Constructing the Real World? A Benchmark for Physical Generative Reasoning

Abstract

Figures & tables

VLM Evaluation

PhysVista: Benchmarking Physical Intelligence in VLMs via a Perception-Reasoning-Assessment Loop

From Perception to Action: An Interactive Benchmark for Vision Reasoning

LMBuild: Evaluating LLM Agents for Generating Buildable and Functional Structures

Abstract

VLM Evaluation

Explore similar work

PhysVista: Benchmarking Physical Intelligence in VLMs via a Perception-Reasoning-Assessment Loop

From Perception to Action: An Interactive Benchmark for Vision Reasoning

LMBuild: Evaluating LLM Agents for Generating Buildable and Functional Structures