The physical world is not merely visual; it is governed by rigorous structural and procedural constraints. Yet, the evaluation of vision-language models (VLMs) remains heavily skewed toward perceptual realism, prioritizing the generation of visually plausible 3D layouts, shapes, and appearances. Current benchmarks rarely test whether models grasp the step-by-step processes and physical dependencies required to actually build these artifacts, a capability essential for automating design-to-construction pipelines. To address this, we introduce DreamHouse, a novel benchmark for physical generative reasoning: the capacity to synthesize artifacts that concurrently satisfy geometric, structural, constructability, and code-compliance constraints. We ground this benchmark in residential timber-frame construction, a domain with fully codified engineering standards and objectively verifiable correctness. We curate over 26,000 structures spanning 13 architectural styles, ach verified to construction-document standards (LOD 350) and develop a deterministic 10-test structural validation framework. Unlike static benchmarks that assess only final outputs, DreamHouse supports iterative agentic interaction. Models observe intermediate build states, generate construction actions, and receive structured environmental feedback, enabling a fine-grained evaluation of planning, structural reasoning, and self-correction. Extensive experiments with state-of-the-art VLMs reveal substantial capability gaps that are largely invisible on existing leaderboards. These findings establish physical validity as a critical evaluation axis orthogonal to visual realism, highlighting physical generative reasoning as a distinct and underdeveloped frontier in multimodal intelligence. Available at https://luluyuyuyang.github.io/dreamhouse
Figures & tables
Figure 2: DreamHouse Dataset Overview. (Top) Dataset statistics showing member count distributions (box plots, left axis) and style proportions (dashed line, right axis) across all 13 architectural styles, ordered by decreasing structural complexity. Member counts range from 133 to 1,548 (mean 673). (Bottom) Representative Cycles-rendered timber frame structures for each style: Courtyard, Cruciform, Colonial, Split-Level, Barn, Ranch, Townhouse, Z-Plan, Carriage, Farmhouse, Saltbox, Shotgun, A-Frame , spanning complex multi-wing configurations to compact single-story forms.
Figure 3: Structural Validation Suite. 10 tests across four pillars. IRC Compliance (left): load path, span limits, completeness, stability score. Structure Physics (center-left): L/360 deflection, cantilever ratio, point load bearing. Geometric Integrity (center-right): dual-end restraint, gap detection, roof coverage. LoD 350 (right): fabrication-grade member geometry.
Figure 4: Task formalization example Planner-Managed. Top (agent loop): Evaluation is formalized as a recurrent agentic process. At each turn t , the agent A (VLM) receives observation ot=(I0,ft−1) , the original multi-view task image I0 and structured validation feedback ft−1 from the previous turn, and generates a Blender Python action at . The executor E applies at to transition the scene graph st−1→st , and the validator V produces feedback ft reporting per-test pass/fail and violation counts. This feedback becomes the input observation for the next turn, closing the loop. On failure (e.g., f2 , red), the agent retries from the same scene state s2 without resetting context. A , E , and V are omitted from the diagram for visual clarity; see Section 3.2 for full formalization. Bottom (task instantiation): The task input I0 consists of five rendered views of the target structure paired with building context and rules. The agent reasons over these to produce and iteratively revise construction code; the environment executes the code in Blender and the validator returns structured diagnostic feedback driving the next revision.
Figure 5: Planner-Managed qualitative example AF-01-0060 (A-frame style). All three models begin from scratch and receive stepwise visual feedback toward the same target structure. Despite reaching a valid result at step 6, Gemini and Claude employ markedly different construction strategies. Gemini pursues a top-down, shape-first approach: it approximates the overall silhouette early and refines toward it. Claude reasons bottom-up: it first establishes a structurally sound interior frame, then lays the roof rafters over it in the final step – a sequence closer to how a builder would physically construct an A-frame. GPT-5 fails to recognize the defining constraint of A-frame geometry, that the roof planes double as load-bearing walls, and from step 6 onward enters a false loop of adding conventional wall studs along the perimeter. Unable to escape this structural misconception, it exhausts all attempts without producing a valid result. This example highlights that identical visual feedback can elicit fundamentally different reasoning strategies, and that success depends not just on visual matching ability but on implicit architectural knowledge.
Model
Protocol
Structural Pass Rate
Visual Fidelity
Topological Fidelity
Joint Pass Rate
Frame
Facade
Frame
Facade
Frame
Facade
Frame
Facade
GPT-5
Planner-Atomic
0.792
0.637
0.312
0.287
0.143
0.151
0.035
0.019
Planner-Reactive
0.302
0.247
0.293
0.266
0.141
0.143
0.003
0.003
Planner-Managed
0.333
0.157
0.179
0.175
0.178
0.157
0.008
0.003
Claude
Planner-Atomic
0.716
0.779
0.406
0.377
0.164
0.168
0.071
0.064
Planner-Reactive
0.428
0.494
0.239
0.245
0.133
0.141
0.003
0.013
Table 1: Consolidated Performance Metrics for GPT-5, Claude Opus 4.5, and Gemini 3 Flash. Frame = bare timber framing; Facade = finished exterior. Bold = best per column; shaded rows = best model per protocol by average across all conditions. Structural Pass Rate in [0,1] : passes only if all 16 validation tests clear. Visual Fidelity : average visual similarity score across 5 rendered views. Topological Fidelity : structural similarity across member counts, spatial alignment, and volumetric overlap. Joint Pass Rate in [0,1] : fraction satisfying both structural and visual criteria simultaneously.
Mean Visual Score
Model
Baseline
Final
Best
Δ
% Improved Tasks
Structures Retained
Claude
0.408
0.441
0.461
+0.033
70.3%
68.5%
GPT-5
0.392
0.405
0.425
+0.013
61.4%
78.1%
Gemini
0.433
0.443
0.471
+0.010
56.1%
73.8%
Table 2: Iterative refinement with visual feedback. Each model starts from its Planner-Atomic structural passes and iterates up to 10 visual-feedback revisions. Baseline = oneshot output before revision; Final = last attempt; Best = highest-scoring attempt across all iterations. Δ is Final − Baseline mean score. Improved Tasks = fraction of tasks that improved their visual fidelity score. Structures Retained = fraction retaining structural validity at the final attempt. Bold = best per column.
Figure 6: Iterative Refinement with Visual Feedback (Section 4.2 , qualitative example BN-01-0273 (barn style). Starting from a structurally valid Planner-Atomic output, each model receives up to 10 rounds of side-by-side visual feedback (current render vs. target) and iteratively revises its structure while maintaining structural validity; steps 9–10 are omitted as changes from step 8 are negligible. Claude demonstrates meaningful visual self-correction: it produces a plausible roof (called: gambrel style) by step 2 but with an incorrect ridge orientation, then progressively corrects across subsequent steps, converging to the almost right form by step 8. GPT-5 commits to an incorrect orientation from step 1 and fails to recover, instead collapsing into a hybrid that blends both orientations without resolving either. Gemini makes minimal modifications across all 10 attempts, suggesting it does not meaningfully incorporate the visual feedback signal. This example illustrates the key behavioral divergence in this experiment: models differ not only in final score, but in whether they engage with feedback at all.
Figure 7: Structural failure analysis across pipelines and models. (a) Per-structure failure rate for each of the 10 structural validation tests, shown for the Planner-Atomic and Planner-Managedprotocols across all three models ( N = 1,200 per model for Planner-Atomic; N≈ 1,160–1,197 for Planner-Managed, excluding Phase-0 failures). Each cell reports the percentage of structures for which that test failed at least once across all retry attempts. Values are independent marginal failure rates per test and do not sum to 100% per row, as a single structure can fail multiple tests simultaneously. Bold column headers indicate the five dominant failure tests. (b) Co-failure pattern distribution: each structure is assigned to the exact set of tests it failed, ranked by frequency. Orange bars show Planner-Atomic( N = 3,551); blue bars show Planner-Managed( N = 3,484), each normalized to their respective totals. The top four Planner-Atomicpatterns share a Span + Deflection core, reflecting near-universal geometry violations. Planner-Managedfailures are dominated by Load + Stability errors, indicating structural connectivity issues rather than geometric ones.
Figure 8: Typical failure modes of Claude 4.5 opus (Planner-Reactive). Each pair shows a ground-truth target structure (left) alongside Claude’s generated output (right) for configurations RN_01_0035 (left pair) and AF_05_0713 (right pair). Claude’s outputs exhibit two characteristic failure patterns: RN_01_0035 produces an overly sparse frame with missing members, while AF_05_0713 produces an overly dense packing of repeated elements. Both reflect insufficient adherence to IRC structural rules, the model optimizes for visual plausibility rather than code-compliant member placement, resulting in structures that fail span limit and deflection validation despite appearing superficially similar to the target geometry.
Figure 9: Representative failure modes of Gemini-3-flash on A-frame structures under the Planner-Reactive protocol . Each pair shows the target structure (left) and the model output (right). Left pair AF_03_0392 : Gemini correctly identifies the A-frame geometry and rafter count but misplaces the side wing, attaching it to the wrong elevation face and inverting the attachment axis, producing a structure that passes visual inspection from one view but fails load-path and dual-end connection tests. Right pair AF_04_0460 : The ridge direction is rotated 90∘ relative to the target, causing all rafters to span the wrong axis; the resulting assembly passes geometric tests ( T3 , T4 ) but fails roof coverage ( T6 ) and gap detection ( T7 ) entirely. These errors illustrate a systematic failure mode: Gemini recovers what structural elements are present but not where they are oriented, consistent with O3 : the model attends to member topology but not to the 3-D spatial relationships encoded in the multi-view input.
Figure 10: Representative failure modes of GPT-5 on A-frame structures under the Planner-Reactive protocol. Each pair shows the target structure (left) and the model output (right). Left pair AF_02_0182 : GPT-5 reproduces the correct rafter count and overall A-frame silhouette but generates collar ties and floor joists that extend beyond the structure’s bounding envelope, with members protruding outside the intended footprint. The assembly fails dual-end connection ( T10 ) and gap detection ( T7 ) as a result. Right pair AF_03_0318 : A similar pattern: the rafter and joist layout is topologically correct but spatially unconstrained, members overshoot the ridge and sill attachment zones, leaving floating ends unconnected to the load path. Both failures share a common root cause: GPT-5 reasons about which members to place but not about the spatial envelope that bounds them, treating member generation as an independent local decision rather than a globally constrained packing problem. This geometric incapability, the inability to enforce that every member end terminates within a valid connection zone, is the dominant failure signature of GPT-5 across A-frame styles and is consistent with O2 : structural reasoning is not monolithic, and spatial constraint satisfaction is a capability that does not co-occur with topological correctness.
Members overlap or leave gaps wider than one stud bay; framing cannot be physically assembled
(iii) Interface definition
T10 , T6
Rafters/studs free at one end (hinge failure under load); roof plane incompletely framed
(iv) Load-path integrity
T1 , T9 , T8
Floating members; unsupported elevated sections; Topological Stability Index <1.0
Table 3: Mapping of LoD 350 constructibility requirements to the DreamHouse validation tests.
View
Azimuth
Elevation
cv
Front
0∘
12∘
1.05
Back
180∘
12∘
1.05
Left
270∘
12∘
1.05
Right
90∘
12∘
1.05
Front-right
45∘
18∘
1.10
Table 4: Camera configuration for the five views. cv is the distance multiplier.
Figure 11: Representative renders from the bridge generation pilot: glulam Howe truss (left), steel Pratt truss (center-left), steel arch (center-right), and steel Warren truss (right). Each bridge is fully parameterized by a low-dimensional BridgeParams dataclass whose scalar fields: span_length , truss_height , num_panels , deck_width project directly to measurable visual quantities in the render. Frontier VLMs recovered valid parameters from a single reference image in one shot with no iterative refinement, exposing the domain as a visual measurement task rather than a test of structural reasoning. This failure mode motivated the shift to timber-frame construction, where the structurally critical quantities are interior to the assembly and not directly legible from exterior renders.
Figure 12: Iterative Refinement with Visual Feedback trajectories for Claude 4.5 opus on four similar barn-style structures. Each pair of rows shows the target (leftmost, lighter color) followed by successive iterations (left to right, darker). Dashed horizontal lines separate groups of structurally distinct targets. Within each group, targets share the same architectural style and similar proportions, yet Claude consistently pursues different refinement strategies across instances: some trajectories make large structural revisions between iterations (roof skeleton replaced); others converge quickly and make only minor adjustments; others oscillate without monotonic improvement. This variability within a homogeneous style group reveals that Claude’s behavior is not driven by a stable visual error signal, it does not reliably identify the same class of discrepancy across similar targets and apply a consistent corrective action. Rather, each iteration represents an independent re-interpretation of the feedback, which can move the output toward or away from the target depending on which aspect of the render the model attends to. This is consistent with O1 and O3 : visual similarity and structural validity are orthogonal, and the ability to see a structural error in a rendered image does not imply the ability to correct it in the next generation.
Figure 13: The 13 architectural style archetypes visualized in Blender, shown as wireframe meshes in the 3D viewport. Each archetype defines the canonical massing, roof type, section layout, and story count for one style category in the DreamHouse dataset. The procedural generation pipeline instantiates each archetype across a range of lot sizes, proportions, and complexity levels to produce the 26,000+ structure dataset. Structures are rendered headless via the Blender Python API ( bpy ); the console output visible at the bottom shows a representative member placement call during procedural construction.
Model
Protocol
Struct. Pass
Visual Fidelity
Visual Pass ( ≥ 0.6)
Joint Pass
GPT-5
Planner-Managed
0.333
0.179
0.022
0.008
+ visual feedback
0.317
0.173
0.030
0.009
Claude
Planner-Managed
0.713
0.278
0.043
0.031
+ visual feedback
0.780
0.288
0.122
0.094
Gemini
Planner-Managed
0.785
0.313
0.054
0.043
+ visual feedback
0.740
0.311
0.133
0.089
Table 5: Stepwise (Planner-Managed) without per-step visual feedback vs. with in Frame (timber framing as input). Joint pass rate is the primary metric: structures that are both structurally valid and visually similar ( S≥0.6 ).
Figure 14: Representative outputs of Qwen3.5-397B-A17B under Planner-Managed, showing five target–output pairs across two views (front elevation, top; diagonal, bottom). Failures are more fundamental than those observed in frontier models: columns 1, 2, 4 show incorrect rafter orientation — rafters span the wrong axis entirely, producing a roof that is geometrically inverted relative to the target; column 3 exhibits severe global disproportion, with the generated structure roughly half the height and twice the footprint of the target, indicating a failure to infer absolute scale from the multi-view input; column 5 shows a collapsed depth dimension, the 3D volume is flattened into a near-planar assembly that passes a front-view silhouette check but has no structural depth. Unlike the frontier model failures in Figures 8 and 9 , which involve spatially misplaced but topologically coherent members, Qwen3.5 errors reflect a lack of basic 3D spatial grounding, the model does not reliably recover the orientation, scale, or depth of the target structure from the provided views.
Protocol
Model
Temp.
Max Tokens
Max Retries
τretry
Context History
Render Engine
Planner-Atomic
GPT-5
N/A
16,384
5
0.8
full
-
Planner-Atomic
Gemini
0.0
16,384
5
0.8
full
-
Planner-Atomic
Claude
0.0
16,384
5
0.8
full
-
Planner-Reactive
GPT-5
N/A
64,000
10
0.6
full
-
Planner-Reactive
Gemini
0.0
64,000
10
0.6
full
-
Planner-Reactive
Claude
0.0
64,000
10
0.6
full
-
Table 6: API hyperparameters per model and protocol. τretry : visual similarity threshold below which the pipeline retries the current step or full generation. N/A : parameter not supported by the API. — : not applicable for this protocol.
Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.
Vision-Language Models (VLMs) have shown strong multimodal reasoning capabilities, yet whether they truly capture the physical consistency underlying real-world dynamics remains unclear. Existing benchmark paradigms often suffer from fragmented evaluation, focusing on isolated cognitive stages while overlooking the inherent synergy between perception, reasoning, and physical judgment. The lack of a holistic perspective limits the ability to diagnose whether VLMs can reliably serve as evaluators of physical plausibility in generative content. To address these issues, we introduce PhysVista, a benchmark designed to evaluate physical intelligence in VLMs through a closed cognitive loop framework inspired by the human seeing-reasoning-assessment process. PhysVista restores this loop by jointly evaluating physical state perception, physical dynamics reasoning, and physical plausibility assessment. It further distinguishes event-level reasoning and scale-level reasoning to enable fine-grained analysis of physical understanding. In addition, PhysVista incorporates both real-world and AI-generated videos, allowing evaluation across diverse domains and emerging generative scenarios. Extensive experiments across a diverse set of VLMs reveal substantial limitations in physical reasoning and plausibility assessment, highlighting a persistent gap between visual recognition and genuine physical understanding, and pointing toward more principled designs for physically grounded multimodal intelligence.
Xinge Peng, Yiting Lu, Tianwu Zhi +4
University of Science and Technology of China · ByteDance · City University of Hong Kong
Understanding the physical structure is essential for real-world applications such as embodied agents, interactive design, and long-horizon manipulation. Yet, prevailing Vision-Language Model (VLM) evaluations still center on structure-agnostic, single-turn setups (e.g., VQA), which fail to assess agents' ability to reason about how geometry, contact, and support relations jointly constrain what actions are possible in a dynamic environment. To address this gap, we introduce the Causal Hierarchy of Actions and Interactions (CHAIN) benchmark, an interactive 3D, physics-driven testbed designed to evaluate whether models can understand, plan, and execute structured action sequences grounded in physical constraints. CHAIN shifts evaluation from passive perception to active problem solving, spanning tasks such as interlocking mechanical puzzles and 3D stacking and packing. We conduct a comprehensive study of state-of-the-art VLMs and diffusion-based models under unified interactive settings. Our results show that top-performing models still struggle to internalize physical structure and causal constraints, often failing to produce reliable long-horizon plans and cannot robustly translate perceived structure into effective actions. The project is available at https://social-ai-studio.github.io/CHAIN/.
Yuhao Wu, Maojia Song, Yihuai Lan +8
Singapore University of Technology and Design (SUTD), Singapore · Singapore Management University (SMU), Singapore · University of Science and Technology of China (USTC), China +1
LLM-based agents are increasingly capable of generating complex 3D structures, with the potential to reshape how objects are designed and realized in the physical world. Yet, producing elegant geometry is fundamentally different from producing objects that can be built and perform their intended functions. Existing evaluations largely focus on geometric quality while overlooking physical realizability. We introduce LMBuild, a benchmark for evaluating LLM agents on generating buildable and functional structures. LMBuild represents generated objects as assembled structures comprising part decompositions, joints, materials, and sequences. To support reproducible evaluation, we provide a unified framework consisting of: (1) an interactive environment in which agents can use tools to retrieve, create, and place components to construct objects; (2) a curated benchmark that repurposes established CAD datasets and augments them with knowledge from Wikipedia; and (3) a evaluation framework covering structural soundness, functional affordance, design quality, and physical realization. Evaluations across 30 systems reveal several intriguing findings: (a) Soundness and alignment are no longer the primary bottlenecks for frontier closed-source models, while functional affordance and physical operability remain substantially more challenging; (b) stronger models more effectively create new components, whereas weaker models tend to rely on retrieval; and (c) providing functional specifications substantially improves part completeness, kinematics, and physical operability. These results show that generating real-world structures requires deeper reasoning about functional affordances, mechanics, and designing and creating novel components. We expect LMBuild to provide a foundation for measuring progress and incentivizing research toward agents that generate buildable and functional structures.