GRASE: Geometry-native Real-to-sim Agentic Scene Modeling from Video

Zhaorui Lin; Haohan Liu; Zhen Chen; Xiangru Huang

cs.CVOct 15, 2026

GRASE: Geometry-native Real-to-sim Agentic Scene Modeling from Video

Authors: Zhaorui Lin, Haohan Liu, Zhen Chen, Xiangru Huang

Organizations: Xi’an Jiaotong University (xjtu.edu.cn) · Fuzhou University (fzu.edu.cn) · Adobe Systems (adobe.com) · Westlake University (westlake.edu)

Abstract

Real-to-sim conversion aims to turn a captured video of a real environment into an editable 3D scene while preserving the geometry and layout of what was observed. Existing methods often follow a one-pass, loosely coupled paradigm: each stage receives only limited feedback from previous stages, allowing errors to propagate without reliable correction and leaving the final scene inconsistent. We present GRASE, an agentic framework that closes this loop by treating the captured multi-view video observations as a standing reference throughout scene modeling. Instead of committing to intermediate estimates, GRASE revises its current hypothesis whenever rendered evidence disagrees with the capture, overcoming the fixed intermediate assumptions of conventional pipelines. Specifically, GRASE first fits the architectural shell and its openings from depth and multi-view cues. An agent then places generated assets in a shared coordinate frame while checking their spatial relationships. Finally, each asset's geometry and placement are refined iteratively to improve agreement across the observed views. By coupling these steps through feedback, GRASE localizes structural discrepancies and revises only the responsible components, while preserving cross-view consistency. The resulting 3D scenes are faithful to the input, consistent across views, and suitable for simulations. In our experiments, GRASE substantially outperforms other state-of-the-art methods, including a direct real-to-sim conversion using GPT6-Astra.

Figures & tables

Figure 1: GRASE transforms an input video into an editable 3D scene, recovering room structure, object layout, and fine-grained details. The example shows the progression from video observations to reconstruction, together with scene overviews, a close-up, and multiview renderings.

Figure 2: Overview of GRASE. (a) Scene preprocessing extracts a point cloud, a scene DAG, and object masks. (b) Room surface topology and geometry refinement reconstructs the shell and openings. (c) Object-centric layout registration and part-wise geometric refinement align object meshes and correct local shapes. Agentic verification guides both reconstruction stages.

Figure 3: Room-surface reconstruction and refinement. Input video frames and the segmented point cloud guide patch initialization. The patches are iteratively refined using geometric evidence and comparisons between source frames and renderings. Observed opening boundaries are back-projected onto the fitted surfaces to construct explicit openings.

Figure 4: Object generation and placement. The large objects in “Scene with Large Obj.” are built from primitives through Blender MCP. The VLM checks them against source views; only models with incorrect shapes or structures are replaced using TRELLIS.2. Small objects are generated directly with TRELLIS.2. All objects are then registered and refined to form the final scene.

Figure 5: Part-wise geometric refinement of a large object. After object registration, a local mismatch remains in the sofa backrest. The VLM identifies the affected part and guides a constrained stretch that improves its alignment with the observed geometry while preserving the rest of the object.

Imaginarium
	Object-level				Scene-level			Input-view rendering
Method	mIoU $\uparrow$	NC $\uparrow$	CD (cm) $\downarrow$	F@5cm $\uparrow$	CD (cm) $\downarrow$	NC $\uparrow$	F@5cm $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
SAM3D	0.0482	0.6182	7.1482	0.0648	72.1585	0.5892	0.1006	–	–	–
GPT-6 vanilla	0.1066	0.6490	18.4091	0.1144	18.1588	0.6844	0.3449	11.8223	0.5342	0.6481
GPT-6 + TRELLIS.2 + PC	0.2066	0.6850	11.8184	0.2062	10.9977	0.7946	0.5615	12.5762	0.5994	0.5606
GRASE (Ours)	0.7425	0.7929	2.1936	0.8536	3.1412	0.9044	0.8866	15.5692	0.6667	0.3381

Table 1: Object- and scene-level reconstruction and input-view rendering on Imaginarium and Replica. PC denotes point cloud; – indicates unavailable rendering metrics.

(a) Imaginarium
Method	CD (cm) $\downarrow$	NC $\uparrow$	F@5cm $\uparrow$
GPT-6 vanilla	29.6426	0.7916	0.3474
GPT-6 + TRELLIS.2 + PC	6.5474	0.8785	0.6729
GRASE (Ours)	2.9734	0.9548	0.8263

Table 2: Architectural shell reconstruction: (a) Imaginarium comparison; (b) seven-scene critic ablation. See Appendix A.7 for the ablation protocol and units.

Figure 6: Qualitative comparison on four scenes. Each row shows the input, matched-camera renderings from SAM3D, GPT-6 vanilla, GPT-6 + TRELLIS.2 + point cloud, and GRASE, followed by a novel-view overview of the GRASE scene. Figure 7 provides additional input-camera viewpoints for the same scenes.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Scene preprocessing
Quantity	Symbol	Value
Frames sampled per video (real / simulated)	$T$	100 / 60
Mask erosion kernel	$r_{\mathrm{erode}}$	$3\times 3$
Confidence floor / quantile	$\tau_{\mathrm{conf}}$ , $\kappa$	$1.2/0.35$
Points per view	$N_{\mathrm{view}}$	$650$
Object generation

Appendix

Table 3: Constants used by GRASE. Symbols follow Sec. 3 and Sec. A .

Figure 7: Full qualitative multiview comparison for the four scenes of Figure 6 . Each group of three rows shows one scene from three input-camera viewpoints. The first five columns share camera poses and intrinsics within each row; Ours NVS shows Blender scene overviews.

Figure 8: Our scene collection. Each row shows an input frame followed by five renderings of the reconstructed scene.

Figure 9: Replica scenes. Columns follow Figure 8 .

Figure 10: Imaginarium scenes. Columns follow Figure 8 .

Real-to-Sim Transfer

See all 14 papers

Lucida: Parse, Generate, and Place for Composable Real-to-Sim Scene Modeling
Aug 31, 2026Minghan Qin, Yuang Wang, Xiuyu Yang +63D Asset Generation 3D Reconstruction
1ByteDance Seed · 2Peking University · 3Zhejiang University
SimRecon: SimReady Compositional Scene Reconstruction from Real Videos
Mar 2, 2026Chong Xia, Kai Zhu, Zizhuo Wang +33D Reconstruction
Tsinghua University · Galbot
GeoWorld: Providing Full-frame Geometry Features to Facilitate 3D Scene Generation
Nov 28, 2025Yuhao Wan, Lijuan Liu, Jingzhi Zhou +6Video Diffusion Models 3D Scene Generation
1VCIP & AAIS, Nankai University · 2ByteDance Inc. · 3Renmin University of China +2

Aug 31, 2026J/K move · Enter open · S save

Lucida: Parse, Generate, and Place for Composable Real-to-Sim Scene Modeling

Minghan Qin, Yuang Wang, Xiuyu Yang, Yushi Long+5

1ByteDance Seed · 2Peking University · 3Zhejiang University

3D Asset Generation 3D Reconstruction Object Pose Estimation Indoor Scene Reconstruction

Open paper AskPDF arXiv

Composable scene modeling aims to recover a real indoor scene as complete, editable object assets arranged as observed, giving robot simulation and embodied AI a simulation-ready replica of the real environment whose objects can be manipulated individually. Existing pipelines decompose the task into three steps---parse the observations into instances, generate an asset for each, and place each asset back---but every step presumes an input that a cluttered capture rarely provides: accurate instance geometry, unoccluded views, and assets that accurately match the observations. We propose Lucida, which keeps this order but redistributes the requirements, so each step consumes only what a real capture reliably provides and precision is reached at the end of the pipeline rather than demanded at its start. Lucida parses the video into a scene graph whose nodes carry per-instance multi-view evidence, generates a complete asset for each instance from its evidence, and places assets with GizmoAct, a VLM policy that casts placement as multi-turn GUI interaction, manipulating the object's gizmo in a closed loop and deciding itself when alignment is reached. Across scene-level 3D object detection, object pose estimation, and scene reconstruction, Lucida improves mAP over Boxer by 69% on R2S-Scene, raises [email protected] from 57.8% to 83.4% on CA-1M, and increases scene F-Score from 0.794 for SAM3D to 0.924.

Abstract

Real-to-Sim Transfer

Explore similar work

Lucida: Parse, Generate, and Place for Composable Real-to-Sim Scene Modeling

SimRecon: SimReady Compositional Scene Reconstruction from Real Videos

GeoWorld: Providing Full-frame Geometry Features to Facilitate 3D Scene Generation