cs.CVSep 22, 2026

HARMONY: Hierarchical Agentic Reasoning for MONocular Image-to-Scene Synthesis

Authors: Shufan Sun, Chen Wang, Enxin Song, Jiatao Gu, Lingjie Liu

Organizations: University of Pennsylvania

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Compositional 3D scene reconstruction has recently been explored from two directions: agentic reasoning that provides semantic understanding of spatial relationships but lacks precise alignment with input images; and visual geometry foundation models that predict dense point maps from input images but the reconstruction quality is limited. Therefore, recovering a complete 3D scene from a single monocular image with accurate inter-object relationships and high-fidelity reconstruction quality remains challenging. In this paper, we present HARMONY, a hierarchical chain-of-thought framework that leverages both agentic reasoning and visual geometry foundation. Given an image of an indoor scene, starting from an empty 3D floorplan, HARMONY first calibrates the camera against the reference image to establish a semantically-grounded spatial frame, then uses agentic VLM reasoning to recover the 3D room layout and an initial placement order. It then places the objects in a hierarchical order, from wall-mounted elements, free-standing furniture, to dependent decorations on top of furniture. We also use depth-first traversal for furniture so each placement conditions on previously resolved structure and a reflective feedback loop to avoid error accumulation. After each object placement by VLM, we use the point cloud estimations to perform geometry-based refinement so that the rendered image aligns better with the input. HARMONY can produce 3D scenes that are semantically consistent and perceptually aligned with the reference image, extending single-image compositional reconstruction to complex indoor scene images. Experiments on synthetic and real-world images demonstrate that HARMONY outperforms the evaluated reconstruction baselines, while qualitative comparisons with GPT-6 Astra suggest more faithful object arrangements and better preservation of scene details.

Figures & tables

Figure 1: Given a single input image, we reconstruct individual objects and use a VLM to reason object placements and orientations, yielding a high-quality compositional 3D scene.
Figure 2: Overview of our pipeline. HARMONY takes a monocular image as input. It first creates an empty 3D room layout and anchors the camera to the 3D scene based on the estimated point cloud and Manhattan frame. It then segments and inpaints the objects in the scene and reconstructs their 3D meshes. Next, a VLM hierarchically plans and places the objects in three stages, with point clouds used for geometric correction. Finally, HARMONY relights the reconstructed scene using VLM-estimated material and emission properties.
Figure 3: Qualitative comparison between HARMONY and baselines.
VariantN-CLIP ↓\downarrowPL ↓\downarrowVariantN-CLIP ↓\downarrowPL ↓\downarrow
HARMONY (full)0.10340.0462w/o VGGT refinement0.10690.0509
w/o depth-first traversal0.10450.0523w/o VLM placement reasoning0.10790.0522
w/o placement refinement0.10510.0516w/o feedback loop0.11840.0509
w/o placement order0.10650.0477w/o camera calibration0.13810.0596
Table 3: Ablation of our key design choices on 10 scenes.
Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

ModelRole in HARMONYAccess
gpt-5.5 ( OpenAI, 2026 )scene / layout / lighting VLMAPI
Gemini-Flash-2.5 ( Google, 2026 )amodal completion, appearanceAPI
Hunyuan3D-2 ( Tencent Hunyuan3D Team, 2025b )image-to-3D mesh + texturelocal server
VGGT ( Wang et al., 2025 )Manhattan / metric alignmentlocal
LocateAnything ( Wang et al., 2026 )open-vocab. box proposalslocal
SAM2 ( Ravi et al., 2024 )instance segmentationlocal
Appendix
Table 4: Foundation models used by HARMONY. The two prompted models (VLM and image editor) can be served locally or through hosted APIs with no change to the pipeline.
Figure 4: Failure cases of HARMONY. Case 1 (gym) shows rare items that fail to detect or inpaint; In Case 2, the red box marks an object present in the input image and reconstructed scene in top-down layout but occluded in the rendered reconstruction, since it sits against a wall occluded from render camera.
Figure 5: Visualization of VLM-selected front face for each object for PCA-aligned meshes.
Figure 6: Visualization of the calibration process during layout generation stage.
Figure 7: Visualization of the layout initialization stage, left to right and top to bottom. Given a reference image, the Manhattan corner estimated from the converted VGGT point cloud backprojects onto the render plane and shows clear orthogonal structure, with its floor and ceiling anchors and the edges extending from the floor corner. The camera initialized on the canonical wall-box mesh by the VLM is then aligned to this same floor corner, its extending edges, and the ceiling point during camera calibration, anchoring itself onto the backprojected reference until convergence – shown as the corner reprojected onto both the reference photo and the calibrated 3D render. The empty-room image is inpainted by Gemini-Flash-2.5 from the reference image to remove furniture while preserving room structure, then backprojected onto the aligned floorplan to give it partial texture. Gemini then completes the floor’s and each wall’s texture into a full, coherent tileable material.
Figure 8: Wall labeling example. Left: top-down floor plan with the camera position and viewing direction. Right: the corresponding reference-textured empty-room render, with the same left/back wall labels and their shared corner marked.
Figure 9: Wall-mounted detection-to-placement pipeline. Top: the reference photo and the final render with wall-mounted objects placed. Bottom: all detected segmentation candidates before filtering, and the subset actually used for placement after containment/fragment removal and verification.
Figure 10: Decoration Placement Correction. Top left: Reference image with VLM-selected furniture to reason and place decorations; Top right: Base render before placing decorations onto the desk for the VLM to compare and reason the decorations that should be placed; Bottom left: Initial placement of decorations onto corresponding supporting furniture; Bottom right: pyrender rendering of the scene after per-object orientation correction of the lamp and matched-group reordering of the pillows.
Figure 11: HARMONY300 example: reference photo, instance segmentation colormap, and HARMONY’s physically-lit render.
Figure 12: HARMONY300 example: reference photo, instance segmentation colormap, and HARMONY’s physically-lit render.
Figure 13: An example study trial: the input photo shown to participants (top) and the six candidate reconstructions they rank (bottom, 2 ×\times 3) including HARMONY and five baselines. In the live interface each panel appears unlabeled as an anonymized letter A–F, reshuffled per (participant, scene); method identities are shown here only for illustration.
Figure 14: Scenes from HARMONY300’s easy split. For each pair: input photo (left) and HARMONY’s render (right)
Figure 15: Scenes from HARMONY300’s medium and complicated splits, spanning diverse room types, layouts and lighting conditions. For each pair: input photo (left) and HARMONY’s render (right).
Figure 16: Reconstructed HARMONY300 scenes rendered from a canonical elevated viewpoint rather than the input camera, with the ceiling and near wall removed so the full room layout is visible.
Figure 17: Multi-room layout generalization. HARMONY adapts the reconstructed scene to diverse room configurations by refining camera poses with respect to a canonical layout.
Figure 18: Six evenly-spaced frames from rigid-body animations simulating earthquakes.
Figure 19: Humanoid interaction in a reconstructed room. A Unitree H1 (19 DoF) operating in a scene reconstructed from a single photograph by HARMONY. Ordered left to right, top to bottom: the robot crouches to reach the chair backrest, draws the chair away from the desk, releases and stands, steps around the chair, crosses to its front, turns, seats itself, and places both hands on the desk. All contacts are resolved against the reconstructed geometry: hands meet the backrest at, thighs rest on the seat pan, and both feet remain on the floor across all 230 frames.
Figure 20: Manipulator grasp of a reconstructed object. A Franka Panda descends onto the office chair’s backrest, aligns its fingers to straddle the panel, closes the jaw, and draws the chair backward. The grasp height is set from the measured panel cross-section: at 0.868 m the backrest is 68 mm thick, within the 80 mm jaw, whereas 35 mm lower it thickens to 87 mm and cannot be grasped. End-effector poses are solved with Lula IK (no failures across 140 frames), and the fingertips hold 0.868 m throughout the 0.45 m pull.

Explore similar work

CardsList
Jun 19, 2026cs.CV

φφ-Scene: Physically Grounded Image-to-3D Scene Reconstruction

Reconstructing compositional 3D scenes from a single image is a fundamental challenge in 3D world modeling. Recent methods can recover high-fidelity, complete 3D objects and predict plausible scene arrangements, but most still treat scene reconstruction primarily as a visual and geometric prediction problem. Their outputs may therefore contain floating objects, interpenetrations, or unstable-contact artifacts, limiting their physical validity and downstream usability in simulation, robotics, and interactive environments. We present φφ-Scene, a physically grounded approach to open-vocabulary and compositional image-to-3D scene reconstruction. The key premise is that a reconstructed scene should not be treated merely as a set of objects with predicted poses, but as a stable physical system. Accordingly, φφ-Scene formulates reconstruction as topology-driven physical assembly: it infers how objects support one another, orders them accordingly, and progressively settles each object against its already stabilized support context. For each object in topological order, SDF-based optimization first resolves penetrations against the pre-settled support context, and rigid-body simulation then settles the object into a stable contact configuration under real-world physical constraints. Experiments on 3D-Front show that φφ-Scene achieves the strongest overall performance among out-of-domain methods and remains highly competitive with in-domain baselines. Human and VLM evaluations further show strong preference for φφ-Scene in visual quality, reference alignment, and physical plausibility. Finally, dedicated physical plausibility metrics covering static contact quality and dynamic stability demonstrate that φφ-Scene substantially reduces penetration artifacts while producing much lower post-simulation drift, indicating more stable and physically grounded 3D scenes.

Haodong Li, Lulu Shao, Haolin Lu +4

University of California San Diego, USA

Scene Reconstruction3D Reconstruction3D WorldScene Understanding
AskPDF
Sep 29, 2026cs.CV

Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering

Reasoning about the 3D world from multi-view images remains a fundamental challenge for Multimodal Large Language Models (MLLMs). While modern MLLMs handle single-image inputs effectively, they struggle to integrate evidence across viewpoints into a coherent 3D understanding. A growing body of work attempts to close this gap by injecting 3D awareness into MLLMs, either by boosting fine-grained pixel-level cross-view correspondence or by fusing features from 3D geometry foundation models, yet a substantial gap to human reasoning persists. In this work, we revisit human spatial reasoning, which suggests that rather than relying on fine-grained geometry cues, humans roughly identify common objects across views, infer the relative geometry between viewpoints, and assemble a coarse 3D layout of the scene. Inspired by this process, we introduce Imagine3D-LLM, an MLLM that learns to assemble a similar compact 3D representation of the scene and conditions its answer on this representation. Concretely, we append a small set of learnable summary tokens after the image tokens, decode them into a compact 3D Gaussian Splatting representation supervised by a photometric reconstruction loss, and train jointly with the standard next-token prediction objective. Notably, although only the summary tokens receive direct reconstruction supervision, this objective also induces stronger cross-frame correspondence within the LLM's underlying image features, suggesting that learning to reconstruct propagates 3D-aware signals throughout the model. As a result, Imagine3D-LLM consistently outperforms prior approaches across multiple spatial reasoning and 3D understanding benchmarks, suggesting that imagining the scene can be more effective than being told its pixel-wise geometry.

Jaewoo Jung, Hyeonseo Yu, Honggyu An +10

KAIST AI · ETH Zürich · Google +2

Spatial ReasoningMultimodal Large Language ModelsGeometric CuesAnswer
AskPDF
Jun 2, 2026cs.CV

SimuScene: Simulation-Ready Compositional 3D Scene Reconstruction from a Single Image

Reconstructing interactive, simulation-ready 3D scenes from a single image is a critical bottleneck for robotic manipulation. While recent single-image lifters recover plausible per-object shapes, composing them yields scenes that collapse under physical simulation due to interpenetrating, hovering, or sinking objects. Existing physics-aware methods address this strictly as a post-hoc layout correction, leaving the underlying geometric errors unresolved. To address this, we introduce SimuScene, a compositional 3D reconstruction pipeline that puts physics in the loop of shape and layout estimation. Rather than using physics merely for layout cleanup, we utilize the physics engine as a diagnostic measurement tool during the generative process itself. By diagnostically simulating reconstructed objects under gravity, we convert penetration and support failures into quantitative correction signals that drive gravity-axis stretching and amodal shape resampling. This physics-informed feedback loop mitigates accumulated reconstruction errors and produces a stable, simulation-ready compositional 3D scene. Extensive experiments demonstrate state-of-the-art performance on physical stability and geometric alignment benchmarks. We further highlight SimuScene's utility by deploying reconstructed environments in humanoid control and robot-arm manipulation tasks.

Inhee Lee, Sangwon Baik, Sungjoo Kim +3

Seoul National University

3D ReconstructionSingle ImagesRobotic ManipulationPhysics-Informed Learning
AskPDF