Organizations: Zhejiang University · Shanghai Innovation Institute · Westlake University · LIGHTSPEED · City University of Hong Kong · University of Hong Kong · Fudan University · Adobe
We present Unravel, a training-free framework that turns a single image into an interactive 3D scene, where every object is an independent and movable asset. Existing generators either produce one entangled geometry in which no object can be moved, or predict all objects in one feed-forward pass, where a wrong guess cannot be detected or repaired. Unravel instead takes a different approach: it first takes the scene apart, and then puts it back together. Starting from the input image, an agentic harness that iteratively peels the image layer by layer, and adopts point cloud as major modality for spatial relationship recovery. Specifically, a vision-language model plans which objects are fully visible, an image editor removes them and restores the occluded regions. Every edit is verified before being incorporated into the reconstruction. This process gradually turns occluded parts of the scene into direct observation, allowing objects to be reconstructed when they are best exposed rather than hallucinated from the original image alone. The scene is then rebuilt in reverse removal order, so that the supporting objects are naturally placed before the objects they support. Since a single-view image only reveals part of an object, we treat the observed points as a lower bound on its size: an asset may grow to cover its points, but can not shrink away from them. Finally, we physically settle the reconstructed scene to remove floating objects and interpenetrations, followed by an agentic audit that corrects remaining placement errors. Extensive experiments on diverse indoor scenes show that Unravel recovers more objects, places them more faithfully, and produces scenes that are more physically-plausible than existing methods.
Figures & tables
Figure 1: Unravel turns a single image into an interactive 3D scene. In the top row, objects are removed from the input image layer by layer until the room is empty, and the scene is then rebuilt with every object as an independent asset (exploded view, right). The rows below show more input images and their recovered interactive scenes.
Figure 2: Single-image interactive scene reconstruction. One-shot generators such as MIDI ( Huang et al., 2025 ) and SAM 3D ( Chen et al., 2026 ) recover independent assets, but under occlusion they misplace objects and distort the geometry of hidden parts (red highlights). Unravel recovers occluded objects with faithful geometry and placement (green highlights).
Figure 3: Overview of Unravel. (a) Agentic layer-wise deconstruction. An agentic loop plans, edits, and verifies removals until the room is empty. Independent 3D assets are extracted before removal. Per-round reconstructions are aligned and fused into an instance-labeled point cloud. (b) Support-aware scene recomposition. Assets return to the environment shell in reverse removal order. One-sided extent constraints guide registration to their point clusters. Physical settling and a final audit refine placement.
Figure 4: Layer-wise deconstruction and point cloud fusion. The upper block shows planner outputs, removal targets, and edited images across six rounds. Red boxes mark targets, and green boxes mark protected objects. The lower block shows aligned per-round reconstructions and cumulative fused point clouds, starting with the original-image reconstruction. Fusion retains removed-object geometry and adds surfaces reconstructed from edited images.
2D masks
3D assets ↑
Scene geometry
Method
Area ratio ↑
UCR (%) ↓
VLM
Human
Points (M) ↑
UCR (%) ↓
Direct reconstruction
0.772
11.3
4.7
4.9
1.53
10.1
Unravel
1.000
0.0
8.2
8.6
2.81
0.0
Table 1: Ablation on five scenes. Area ratio and UCR summarize coverage for objects removed after the first round, using Unravel’s masks and point clusters as references. UCR (under-coverage rate) denotes the percentage with coverage below 70%. VLM and Human report mean fidelity scores (1 to 10) over all 160 objects. Point counts are scene averages.
Figure 5: Qualitative comparison of single-image scene reconstruction. Each input is followed by two views of each reconstructed scene. The two Astra variants separate direct scene generation from placement with shared assets. GPT-6 Astra (Ultra) + TRELLIS2 receives the same generated meshes and corresponding 2D instance masks used by Unravel. Red boxes highlight placement discrepancies in the agentic results.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Generated assets with and without deconstruction. Each column shows one object. The top row is generated from the mask in the input view, and the bottom row from the mask captured at the best-observed moment. Incomplete masks lead to broken geometry, spurious parts, and hallucinated contents, while Unravel keeps the assets faithful.
Figure 7: Comparison with Seeing Through Clutter. The input image is on the left, followed by two views of Seeing Through Clutter in the middle and two views of Unravel on the right. Seeing Through Clutter is shown as colored geometry and Unravel as a textured scene.
Figure 8: Complete deconstruction traces on four scenes. For each scene we show the input image, three views rendered from the reconstructed scene, the edited image of every round, and the assets extracted in each round. Red boxes mark the removal targets of each round, and duplicate assets are omitted for space.
Figure 9: Deconstruction traces on four more scenes. For each scene we show the input image, three views rendered from the reconstructed scene, and the edited image of every round. Red boxes mark the removal targets of each round.
Figure 10: Additional reconstructions on eight further scenes. For each scene we show the input image and three views rendered from the reconstructed scene.
Figure 11: Reconstructions from real photographs. Each row shows an input image on the left and three views of the reconstructed scene on the right.
Reconstructing interactive, simulation-ready 3D scenes from a single image is a critical bottleneck for robotic manipulation. While recent single-image lifters recover plausible per-object shapes, composing them yields scenes that collapse under physical simulation due to interpenetrating, hovering, or sinking objects. Existing physics-aware methods address this strictly as a post-hoc layout correction, leaving the underlying geometric errors unresolved. To address this, we introduce SimuScene, a compositional 3D reconstruction pipeline that puts physics in the loop of shape and layout estimation. Rather than using physics merely for layout cleanup, we utilize the physics engine as a diagnostic measurement tool during the generative process itself. By diagnostically simulating reconstructed objects under gravity, we convert penetration and support failures into quantitative correction signals that drive gravity-axis stretching and amodal shape resampling. This physics-informed feedback loop mitigates accumulated reconstruction errors and produces a stable, simulation-ready compositional 3D scene. Extensive experiments demonstrate state-of-the-art performance on physical stability and geometric alignment benchmarks. We further highlight SimuScene's utility by deploying reconstructed environments in humanoid control and robot-arm manipulation tasks.
Reconstructing physically stable 3D scenes from a single RGB image enables casual images to be converted into simulation-ready digital assets for applications such as immersive interaction and content creation. However, existing single-image reconstruction methods fall short in capturing the physical structure of a scene. As a result, they often produce geometrically plausible but physically inconsistent results, including object floating and penetration, which lead to unstable behavior in physics simulations. Image-conditioned scene generation methods improve physical plausibility but often rely on strong scene priors, yielding plausible yet inaccurate object arrangements that fail to match the input image. We propose REST3D, a single-image reconstruction framework that can reconstruct physically stable 3D scenes by integrating physical scene understanding with physics-constrained refinement. We first introduce an agentic physical scene understanding technique that constructs a scene-tree representation capturing object physical states and inter-object relationships from a gravity-support perspective, providing a structural prior for reconstruction. Leveraging this structure, we initialize the scene using image-to-3D models, followed by scene-tree-guided alignment and physics-constrained optimization to resolve physical violations while preserving visual consistency with the input image. Experiments show that our method significantly reduces physical errors and improves simulation stability on both synthetic and real-world datasets while maintaining strong reconstruction quality. We further demonstrate the reconstructed scenes in VR-based human-object interaction, showing their potential for immersive applications.
Generating explorable 3D scenes from a single image is a highly challenging problem in 3D vision. Existing methods struggle to support free exploration, often producing severe geometric distortions and noisy artifacts when the viewpoint moves far from the original perspective. We introduce One2Scene, an effective framework that decomposes this ill-posed problem into three tractable sub-tasks to enable immersive explorable scene generation. We first use a panorama generator to produce anchor views from a single input image as initialization. Then, we lift these 2D anchors into an explicit 3D geometric scaffold via a generalizable, feed-forward Gaussian Splatting network. Instead of treating the panorama as a single image for reconstruction, we project it into multiple sparse anchor views and reformulate the reconstruction task as multi-view stereo matching, which allows us to leverage robust geometric priors learned from large-scale multi-view datasets. A bidirectional feature fusion module is used to enforce cross-view consistency, yielding an efficient and geometrically reliable scaffold. Finally, the scaffold serves as a strong prior for a novel view generator to produce photorealistic and geometrically accurate views at arbitrary cameras. By explicitly conditioning on a 3D-consistent scaffold to perform reconstruction, One2Scene works stably under large camera motions, supporting immersive scene exploration. Extensive experiments show that One2Scene substantially outperforms state-of-the-art methods in panorama depth estimation, feed-forward 360° reconstruction, and explorable 3D scene generation. Project page: https://one2scene5406.github.io/