Fysiverse-3D-Vision Technical Report: Generating Executable 3D Worlds from Images through Unified Spatial Reasoning

Dingkang Yang; Yizhou Liu; Wendong Cheng; Zizhi Chen; Shunli Wang; Yang Liu; Hongsheng Li; Lihua Zhang

cs.CVSep 22, 2026

Fysiverse-3D-Vision Technical Report: Generating Executable 3D Worlds from Images through Unified Spatial Reasoning

Authors: Dingkang Yang, Yizhou Liu, Wendong Cheng, Zizhi Chen, Shunli Wang, Yang Liu, Hongsheng Li, Lihua Zhang

Organizations: Physical Superintelligence Lab, Fysics AI · Multimedia Laboratory (MMLab), The Chinese University of Hong Kong · College of Intelligent Robotics and Advanced Manufacturing, Fudan University

Abstract

Generative models have advanced image-conditioned 3D content creation, yet generating controllable and executable 3D scenes from a single image remains challenging. Existing 3D generative approaches can synthesize visually plausible objects and scenes, but their spatial layout estimation is coupled with specific asset generators. They struggle to jointly model object semantics, metric geometry, and scene-level spatial relationships, which are essential for interactive editing, physical simulation, and embodied applications. We propose Fysiverse-3D-Vision, a unified vision-language-geometry framework for generative 3D scene reconstruction and executable asset construction from a single image. We establish a shared representation where spatial reasoning and geometric reconstruction mutually enhance each other, allowing object layouts to be inferred beyond the constraints of individual asset generators. Our model integrates textual supervision, semantic visual cues, and geometric representations within a unified Transformer to capture scene context, metric geometry, and object-level interactions. An object-conditioned layout module performs cross-attention between target object representations and global geometric features to predict object translation, rotation, and scale. Training progressively learns geometry-language alignment, introduces layout reasoning while preserving reconstruction capability, and refines physical consistency through collision-aware optimization. By separating spatial layout reasoning from asset synthesis, Fysiverse-3D-Vision provides an adaptable interface for interactive scene editing, object-level manipulations, and executable 3D content generation. Experiments demonstrate that our framework achieves superior geometric consistency, layout estimation, rendering quality, and physical property understanding compared with existing approaches.

Figures & tables

Figure 1: From a single image, Fysiverse-3D-Vision reconstructs individual objects, independently predicts their translation, rotation, and scale, and attaches physical, material, affordance, and part-level attributes for editing and simulation.

Figure 2: Three-stage simulation-data curation pipeline: heterogeneous indoor datasets are merged, compact object-centric regions are cropped, and samples are categorized by structural completeness, visibility, and semantic quality.

Figure 3: Our method decouples executable asset generation from 3D layout prediction. Completed object crops are used to generate textured, physical, and part-level assets, while a unified vision-language-geometry Transformer predicts object translation, rotation, and scale from scene geometry and object-condition tokens. Training proceeds through geometry-language pretraining, layout injection, and real-data refinement.

Methods	Inst. Spec.	Geometry					Visual Metrics
Methods	Inst. Spec.	CD-S $\downarrow$	CD-O $\downarrow$	F-Score-S $\uparrow$	F-Score-O $\uparrow$	IoU-B $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	CLIP-S $\uparrow$
PartCrafter [ 23 ]	$\times$	0.2027	—	35.86	—	—	—	—	—	—
Gen3DSR [ 10 ]	$\checkmark$	0.1260	0.1511	43.81	38.95	0.3162	10.15	0.7874	0.4225	0.5972
MIDI ∗ [ 19 ]	$\checkmark$	0.0832	0.1073	52.38	60.43	0.2357	10.97	0.8139	0.3867	0.5894
SceneGen [ 29 ]	$\checkmark$	0.0678	0.0616	58.33	71.89	0.5785	10.36	0.7708	0.3719	0.6050
SAM3D [ 5 ]	$\checkmark$	0.0776	0.0593	71.54	60.58	0.5813	11.59	0.8183	0.3391	0.6096

Table 1: Quantitative evaluation of image-based 3D scene reconstruction on the 3D-FUTURE test set. The comparison covers geometric accuracy, spatial layout consistency, and visual reconstruction quality. ∗ indicates adopting MV-Adapter [ 20 ] for texture rendering.

Figure 4: Qualitative comparison of scene reconstruction results under diverse environments. The examples include in-domain and out-of-domain scenes with different object arrangements and visual appearances. Compared with existing approaches, F3V better preserves object positions, scales, and spatial relationships, demonstrating stronger geometric reasoning and layout consistency.

Figure 5: Additional qualitative examples. The results demonstrate the robustness of Fysiverse-3D-Vision under diverse scene configurations and object arrangements.

Components			Metrics
Step 1	Step 2	Step 3	CD-S $\downarrow$	CD-O $\downarrow$	F-Score-S $\uparrow$	F-Score-O $\uparrow$	IoU-B $\uparrow$
$\checkmark$			0.1321	0.1117	42.40	52.07	0.3551
	$\checkmark$		0.2495	0.1824	36.52	25.18	0.3186
		$\checkmark$	0.0865	0.0773	58.75	59.30	0.3855
$\checkmark$	$\checkmark$		0.0796	0.0814	64.67	57.02	0.4530
$\checkmark$	$\checkmark$	$\checkmark$	0.0258	0.0579	75.16	69.56	0.5904

Table 2: Ablation analysis of the proposed multi-stage training strategy. Each stage is progressively introduced to examine its contribution to geometric reconstruction and spatial layout prediction.

Methods	Physical Attributes
Methods	Scale $\downarrow$	Mat. $\uparrow$	Aff. $\uparrow$	Kin. $\uparrow$	Desc. $\uparrow$
MIDI [ 19 ]	—	—	—	—	—
MIDI + PhysPre	12.54	7.17	10.03	0.27	7.05
F3V (ours)	8.09	14.51	11.90	0.32	10.94

Table 3: Evaluation of physical attribute understanding on executable 3D assets. Scale denotes absolute-scale error, while the remaining metrics measure the prediction accuracy of material, affordance, kinematic properties, and textual descriptions.

Figure 6: Extension to part-level executable scene reconstruction. By integrating part-aware asset generation with the predicted spatial layout, F3V enables fine-grained reconstruction beyond object-level placement.

3D Scene Generation

3D scene generation creates structured, explorable environments from text, images, maps, or other inputs.

See all 190 papers

Jul 15, 2026

ThinkBLOX: 3D Indoor Scene Generation with Progressive Reasoning

While traditional graphics methods often synthesize 3D indoor scenes autoregressively or hierarchically, recent vision-language model (VLM)-based generators predominantly adopt a one-shot paradigm where the full layout is planned at once. This one-shot approach often requires global re-optimization or complete reconstruction during interactive editing (e.g., inserting or moving objects) and can lead to physically or semantically poorly organized arrangements. To address these challenges, we propose ThinkBLOX, a VLM-based progressive reasoning framework that iteratively designs and refines 3D scenes. ThinkBLOX treats layout generation as a state-conditioned, step-by-step reasoningand-action process. To power this, we construct the ThinkBLOX-Data-200K dataset, containing 224,757 procedural placement pairs annotated with multi-view scene context, explicit Chain-of-Thought (CoT) rationales, and structured JSON layouts. Through supervised fine-tuning (SFT) on this dataset, the VLM learns to bridge the reasoning-action gap under incremental updates. Furthermore, recognizing that scene synthesis is inherently a multisolution task where SFT suffers from reward conflict, we introduce Tier-Decoupled GDPO. This reinforcement learning scheme organizes heterogeneous rewards into distinct tiers, stabilizing policy optimization across physical validity, semantic plausibility, and reasoning-action consistency. Extensive experiments show that ThinkBLOX significantly outperforms recent one-shot and iterative baselines in physical plausibility, semantic alignment, and interactive editability. Additionally, we show that it supports diverse applications, including both global and local generation and rearrangement of 3D scenes.

Yuan Xiao, Can Wang, Xiangyu Kong +1

Department of Computer Science, City University of Hong Kong, Hong Kong · School of Computing and Data Science, University of Hong Kong · School of Computer Science, Beijing Information Science and Technology University

3D Scene Editing 3D Scene Generation 3D Scene Layout Generation Indoor Scene Generation

Ask PDF

Nov 18, 2025

Let Language Constrain Geometry: Vision-Language Models as Semantic and Spatial Critics for 3D Generation

Text-to-3D generation has advanced rapidly, yet state-of-the-art models, encompassing both optimization-based and feed-forward architectures, still face two fundamental limitations. First, they struggle with coarse semantic alignment, often failing to capture fine-grained prompt details. Second, they lack robust 3D spatial understanding, leading to geometric inconsistencies and catastrophic failures in part assembly and spatial relationships. To address these challenges, we propose VLM3D, a general framework that repurposes large vision-language models (VLMs) as powerful, differentiable semantic and spatial critics. Our core contribution is a dual-query critic signal derived from the VLM's Yes or No log-odds, which assesses both semantic fidelity and geometric coherence. We demonstrate the generality of this guidance signal across two distinct paradigms: (1) As a reward objective for optimization-based pipelines, VLM3D significantly outperforms existing methods on standard benchmarks. (2) As a test-time guidance module for feed-forward pipelines, it actively steers the iterative sampling process of SOTA native 3D models to correct severe spatial errors. VLM3D establishes a principled and generalizable path to inject the VLM's rich, language-grounded understanding of both semantics and space into diverse 3D generative pipelines.

Weimin Bai, Yubo Li, Weijian Luo +4

Peking University, Beijing · hi-lab, Xiaohongshu Inc. · MMLab, CUHK +1

3D Spatial Reasoning Text-to-3D Generation Inference-Time Guidance

Ask PDF

Oct 23, 2026

Easy-3Dfy: Object-Level Geometric Composition for Single-Image 3D Scene Construction

Single-image 3D scene construction is a fundamental problem in 3D vision. Recent methods such as SAM-3Dfy enable end-to-end reconstruction of 3D assets and their scene layouts, but still struggle with fine-grained scene geometry and asset quality. Meanwhile, specialized 3D object generators provide detailed object geometry, while 3D foundation models offer strong and generalizable spatial understanding. We propose Easy-3Dfy, a compositional workflow that decouples object generation from scene layout estimation, allowing specialized models to fully exploit their respective strengths. An RPF-based aligner then bridges these capabilities by establishing geometric correspondence between object-local meshes and partial scene pointmaps, recovering object layout for scene composition. Without requiring complete scene-level supervision, the proposed workflow consistently outperforms SAM-3Dfy across multiple geometric metrics on Scene-Maker and 3D-FUTURE200. Our results demonstrate that explicit geometric composition can effectively combine high-quality object generation with strong spatial understanding, while enabling the workflow to benefit independently from advances in both 3D object generation and foundation models.

Ziang Zhang, Wenyou Yi, Zehan Wang +4

Zhejiang University · Tencent

Image-to-3D Scene Generation 3D Scene Layout Generation Single-View 3D Reconstruction

Ask PDF

Abstract

3D Scene Generation

Explore similar work

ThinkBLOX: 3D Indoor Scene Generation with Progressive Reasoning

Let Language Constrain Geometry: Vision-Language Models as Semantic and Spatial Critics for 3D Generation

Easy-3Dfy: Object-Level Geometric Composition for Single-Image 3D Scene Construction