Embodied AI training and evaluation require object-centric digital twin environments with accurate metric geometry and semantic grounding. Recent transformer-based feedforward reconstruction methods can efficiently predict global point clouds from sparse monocular videos, yet these geometries suffer from inherent scale ambiguity and inconsistent coordinate conventions. This mismatch prevents the reliable fusion of these dimensionless point cloud predictions with locally reconstructed object meshes. We propose a novel scale-aware 3D fusion framework that registers visually grounded object meshes with transformer-predicted global point clouds to construct metrically consistent digital twins. Our method introduces a Vision-Language Model (VLM)-guided geometric anchor mechanism that resolves this fundamental coordinate mismatch by recovering an accurate real-world metric scale. To fuse these networks, we propose a geometry-aware registration pipeline that explicitly enforces physical plausibility through gravity-aligned vertical estimation, Manhattan-world structural constraints, and collision-free local refinement. Experiments on real indoor kitchen environments demonstrate improved cross-network object alignment and geometric consistency for downstream tasks, including multi-primitive fitting and metric measurement. We additionally introduce an open-source indoor digital twin dataset with metrically scaled scenes and semantically grounded and registered object-centric mesh annotations.
Figures & tables
Fig. 1: Comparison of scenarios. Left image: Output result of digital twin. Middle: Traditional fusion reconstruction method, in which objects (such as a bottle) share the geometric structure with the environment, thus unable to achieve physical independence. Right image: Our digital twin model based on objects, in which the objects are structurally independent and are physically operable meshes.
Fig. 2: Our proposed pipeline for semantic and geometrically grounded object-centric scene reconstruction. Stream A (Top) reconstructs the global metric geometry via VLM-aided scale recovery. Stream B (Bottom) generates high-fidelity object meshes from optimal 2D views. Stage C (Right) fuses these streams through a geometrically grounded hierarchical registration process, enforcing physical plausibility and geometric constraints to produce a refined digital twin.
Fig. 3: Visual progression of our 3D fusion framework compared to the baseline. From left to right: (1) The original Input Frame captured from the sequence. (2) The dense, unscaled Point Cloud reconstructed by Pi-Long. (3) Our Mesh , demonstrating that after scale recovery and geometries-aware registration, the lifted object meshes perfectly align within the physical 2D bounding box (red) when projected back to the camera view. (4) The naive SAM3D Mesh baseline, which fails to establish a coherent metric space, resulting in severe misalignment or absence within the ground truth bounding box when re-rendered.
Method
Bottle 1
Fridge
Coffee M.
Cab. 1
DishW.
Coffee
Cab. 2
Stool
Cab. 3
Oven
Bottle 2
Cab. 4
mIoU ↑
SAM3D
0.0000
0.0000
0.0000
0.0000
0.2943
0.0000
0.0001
0.0000
0.0000
0.0000
0.0014
0.0000
0.0246
Ours
0.6321
0.6205
0.5751
0.1502
0.3979
0.7519
0.3079
0.6743
0.7798
0.5842
0.7283
0.6562
0.5715
TABLE I: Quantitative comparison of 3D reconstruction semantic consistency. We report the Novel View Semantic Synthesis IoU (NVSS-IoU) for 12 individual objects and the overall mean IoU (mIoU). Our geometrically grounded alignment method significantly outperforms the baseline by establishing a coherent metric space.
Object
RMSE (m) ↓
Scale ( s )
Vertices
Faces
Bottle 1
0.0165
0.24
3,970
7,028
Bottle 2
0.0461
0.25
5,156
9,058
Coffee Maker
0.0482
0.43
2,658
4,490
Coffee
0.0910
0.43
6,016
10,336
Cabinet 3
0.1637
0.69
4,165
7,054
Step Stool
0.1699
0.45
27,366
35,710
TABLE II: Geometric registration metrics and mesh complexity. We report the TrICP Root Mean Square Error (RMSE), SAM3D -to-World Scale Factor ( s ), and topological complexity (Vertices and Faces) of the generated meshes for all 12 tracked objects in the scene.
Fig. 4: Qualitative comparison of the assembled 3D digital twin. Left: The baseline fails to establish a metric space, resulting in floating, unscaled, and overlapping artifacts. Right: With geometric grounding, our method produces a physically plausible, Manhattan-aligned, and tightly registered object-centric scene without subsurface penetration.
Object
Ours (Full)
w/o metric Anchor
w/o TrICP
w/o Geom. Grounding
Bottle 1
0.6321
0.0015
0
0.4141
Fridge
0.6205
0.2686
0
0.6048
Coffee Maker
0.5751
0
0.0643
0.1393
Cabinet 1
0.1502
0
0
0
Dishwasher
0.3979
0
0
0
Coffee
0.7519
0
0
0
TABLE III: Quantitative ablation of semantic and geometric consistency shows the Novel View Semantic Synthesis IoU (NVSS-IoU) for the full pipeline compared to three ablated variants.
Building high-fidelity digital twins of articulated objects from visual data remains a central challenge. Existing approaches depend on multi-view captures of the object in discrete, static states, which severely constrains their real-world scalability. In this paper, we introduce Articulat3D, a novel framework that constructs such digital twins from casually captured monocular videos by jointly enforcing explicit 3D geometric and motion constraints. We first propose Motion Prior-Driven Initialization, which leverages 3D point tracks to exploit the low-dimensional structure of articulated motion. By modeling scene dynamics with a compact set of motion bases, we facilitate soft decomposition of the scene into multiple rigidly moving groups. Building on this initialization, we introduce Geometric and Motion Constraints Refinement, which enforces physically plausible articulation through learnable kinematic primitives parameterized by a joint axis, a pivot point, and per-frame motion scalars, yielding reconstructions that are both geometrically accurate and temporally coherent. Extensive experiments demonstrate that Articulat3D achieves state-of-the-art performance on synthetic benchmarks and real-world casually captured monocular videos, significantly advancing the feasibility of digital twin creation under uncontrolled real-world conditions. Our project page is available at https://maxwell-zhao.github.io/Articulat3D/.
Lijun Guo, Haoyu Zhao, Xingyue Zhao +5
Wuhan University · Hong Kong Embodied AI Lab · The Chinese University of Hong Kong +3
3D foundation models (3DFMs) excel at predicting camera poses and dense depth from multiple views of a scene, showcasing strong zero-shot generalization. However, as metric scale is not observable from monocular images, their absolute scale predictions are typically inaccurate. Inertial measurement units (IMUs), present in most devices, naturally complement monocular cameras by observing scaled motion. We introduce VI3, a model-agnostic framework that metrically anchors a pretrained 3DFM using only IMU readings. VI3 initializes and preintegrates the IMU to obtain a metric motion reference, which is then used to recover the scale of the 3DFM outputs. Our method includes adaptable anchoring strategies tailored to diverse 3DFM architectures. Experiments on synthetic and real aerial datasets demonstrate that VI3 recovers metric scale without ground-truth supervision while preserving geometric consistency, acting as a fine refinement under well-conditioned motion and as a strong prior when motion is less informative.
This paper addresses metric 3D reconstruction of indoor scenes by exploiting their inherent geometric regularities with compact representations. Using planar 3D primitives - a well-suited representation for man-made environments - we introduce PLANA3R, a pose-free framework for metric Planar 3D Reconstruction from unposed two-view images. Our approach employs Vision Transformers to extract a set of sparse planar primitives, estimate relative camera poses, and supervise geometry learning via planar splatting, where gradients are propagated through high-resolution rendered depth and normal maps of primitives. Unlike prior feedforward methods that require 3D plane annotations during training, PLANA3R learns planar 3D structures without explicit plane supervision, enabling scalable training on large-scale stereo datasets using only depth and normal annotations. We validate PLANA3R on multiple indoor-scene datasets with metric supervision and demonstrate strong generalization to out-of-domain indoor environments across diverse tasks under metric evaluation protocols, including 3D surface reconstruction, depth estimation, and relative pose estimation. Furthermore, by formulating with planar 3D representation, our method emerges with the ability for accurate plane segmentation. The project page is available at https://lck666666.github.io/plana3r
Changkun Liu, Bin Tan, Zeran Ke +6
The Hong Kong University of Science and Technology · Ant Group · Wuhan University +2