Physics simulators and motion planners require convex collision geometry, yet image-to-3D generative models output dense, frequently non-manifold visual meshes. Bridging the two today takes a slow, brittle reconstruct-then-decompose pipeline of repair, decimation, and approximate convex decomposition. We present I2CD, which predicts a convex decomposition directly from a single RGB image. Rather than train a new image-to-3D model, I2CD freezes the pretrained Hunyuan3D-2 image-conditioned diffusion transformer and shape decoder and trains only a lightweight cross-attention head (38M parameters, under ten GPU-hours) whose learned "convex-slot" tokens emit the halfplane parameters of K convex polytopes. The output is compact, convex by construction, and loads into physics engines without any post-processing, in ∼0.5s per image. On 227 held-out OmniObject3D and Google Scanned Objects instances, I2CD attains the highest volumetric IoU among eight reconstruct-then-decompose pipelines while running 6-37× faster end-to-end. In a cross-simulator study in MuJoCo, PyBullet, Genesis, and Isaac Sim, every engine uses I2CD geometry as delivered, whereas raw generated meshes "load" everywhere but are silently replaced by a different collision shape in most cases or need seconds to minutes of per-object preprocessing. On a physical xArm7, I2CD produces planner-ready geometry for a 20-object cluttered scene in 11s versus 328s for the strongest baseline, at comparable pick-and-place execution success (85 vs. 90 of 100 trials).
Figures & tables
Fig. 1: From a single RGB image, I2CD predicts a union of convex polytopes in ∼0.5 s that physics simulators consume directly, with no mesh extraction, repair, or post-hoc decomposition.
Fig. 2: Pipeline. The frozen Hunyuan3D-2 DiT samples a shape latent from the image and the frozen ShapeVAE decoder D expands it into per-token shape features. Our trainable convex head uses K learned convex-slot tokens that cross-attend to these features and emit halfplane parameters {nk,h,dk,h,tk,βk} for K convex polytopes Ck ; their union is the predicted occupancy, from which per-convex meshes are extracted for simulator use. During training (top), the latent comes from the frozen encoder E instead.
Method
IoU (%) ↑
Omni
GSO
F (%) ↑
CD (mm) ↓
ttotal (s) ↓
HY3D + CoACD
62.8±19.0
62.9
62.7∗
16.4±12.9
6.67±7.07
7.82
HY3D + V-HACD
61.0±18.6
61.1
60.8∗
16.1±15.3
6.76±6.77
7.50
HY3D + CvxNet
58.7±15.9
56.1
62.0∗
13.2±10.9
8.17±11.63
3.05
HY3D + BSP-Net
59.7±16.4
59.7
59.7∗
14.4±11.3
8.73±9.46
3.67
SAM3D + CoACD
61.5±20.2
61.3
62.0
19.2±14.7
5.87±6.82
19.04
SAM3D + V-HACD
62.2±18.3
61.0
63.8∗
17.1±14.8
6.12±6.78
13.76
TABLE I: Single-image convex decomposition on OmniObject3D + GSO ( n=227 ), mean ± std over objects, with IoU also split by dataset (OmniObject3D uses a hull-proxy reference, GSO the exact mesh). ttotal for baselines is mesh generation ( 1.56 s HY3D, 13.30 s SAM3D) plus decomposition. ∗ GSO means for baselines are recovered from the pooled and OmniObject3D means (exact up to rounding).
Fig. 3: Qualitative comparison. Each color is one convex part; the time under each column is the rounded mean end-to-end runtime. I2CD produces compact, volumetrically faithful decompositions in one pass, whereas reconstruct-then-decompose pipelines inherit and amplify reconstruction artifacts into fragmented or misaligned parts.
Engine
Geometry
Preserved
Load
Penetr.
At rest
(%)
(s)
(mm)
(%)
MuJoCo
ours
99.2
0.05
0.22
99.2
raw, default ( → hull)
40.2
2.77
0.26
83.5
PyBullet
ours
99.2
0.03
0.01
100.0
raw, default ( → hull)
40.2
2.48
0.01
78.7
raw, concave mesh
100 †
0.01
0.01
100.0
TABLE II: Simulation-readiness beyond loading ( 127 OmniObject3D objects, four engines). “Raw” is the HY3D mesh; “default” is the engine’s untouched mesh handling. “Preserved”: round-trip IoU ≥0.9 between engine-effective and input geometry (not fidelity to the object). All rows load 100% of objects except ‡ , where PhysX cooking crashed on 1 (hull) and 6 (SDF) objects, excluded from that row. Genesis’s SDF collider is omitted: in a 20 -object pilot it exceeded 180 GB RAM or 15 min on 13 objects and took 1 – 6 min on the rest. † exact shape, but dynamic concave meshes register zero object–object contacts in PyBullet.
Fig. 4: Real cluttered scene. From one RGB-D observation (left), SAM3D reconstructs dense per-object meshes; auto-hull collapses each to one convex hull; SAM3D+CoACD decomposes post hoc; I2CD predicts compact convex geometry directly. Times are wall-clock to a planner-ready 20 -object scene.
Fig. 5: Physical execution. The xArm7 executing a planned pick in the 20 -object scene of Fig. 4 . Every trial in Table III is run on this robot.
Method
tgeom (s) ↓
tload (s) ↓
Success ↑
SAM3D + auto-hull
198
0.9
5/100
SAM3D + CoACD
328
1.7
90/100
I2CD (ours)
11
0.9
85/100
TABLE III: Physical pick-and-place in a 20 -object scene. tgeom : time to planner-ready geometry for all objects; tload : one-time MuJoCo load. Success over 20 targets ×5 seeds, executed on the xArm7.
Loss
IoU (%) ↑
F (%) ↑
CD (mm) ↓
BCE + NCE (ours)
63.8
12.5
5.98
Class-balanced BCE only
62.2
11.4
6.49
Unbalanced BCE
54.7
9.7
8.89
MSE
failed: 127/127 empty
TABLE IV: Training objective on OmniObject3D ( n=127 ); each row is that objective’s validation-best checkpoint (ours is the Table I model; its 63.8 vs. 63.1 there is DiT sampling noise).
(a) Parts K vs. halfplanes H
(b) Head depth
Cell: IoU / F / CD
(cross-attn, self-attn)
K\H
6
12
24
Head (layers)
IoU
6
61.4 / 11.9 / 6.66
65.5 / 13.1 / 6.52
65.3 / 12.8 / 6.76
(1, 0) minimal
61.4
12
61.3 / 12.4 / 6.79
63.1 / 13.1 / 6.23
64.4 / 12.8 / 6.00
24
58.6 / 12.4 / 6.49
60.2 / 12.0 / 6.59
61.6 / 12.6 / 6.11
(2, 1) default
62.0
TABLE V: Capacity on OmniObject3D at a matched 100 -epoch budget.
Fig. 6: Qualitative ablation of the objective. I2CD yields compact, coherent parts; class-balanced BCE alone tiles the interior with thin slabs; unbalanced BCE (uBCE) under-fills the shape.
This work proposes a new formulation to the long-standing problem of convex decomposition through learning feature fields, enabling the first feed-forward model for open-world convex decomposition. Our method produces high-quality decompositions of 3D shapes into a union of convex bodies, which are essential to accelerate collision detection in physical simulation, amongst many other applications. The key insight is to adopt a feature learning approach and learn a continuous feature field that can later be clustered to yield a good convex decomposition via our self-supervised, purely-geometric objective derived from the classical definition of convexity. Our formulation can be used for single shape optimization, but more importantly, feature prediction unlocks scalable, self-supervised learning on large datasets resulting in the first learned open-world model for convex decomposition. Experiments show that our decompositions are higher-quality than alternatives and generalize across open-world objects as well as across representations to meshes, CAD models, and even Gaussian splats. https://research.nvidia.com/labs/sil/projects/learning-convex-decomp/
Reconstructing interactive, simulation-ready 3D scenes from a single image is a critical bottleneck for robotic manipulation. While recent single-image lifters recover plausible per-object shapes, composing them yields scenes that collapse under physical simulation due to interpenetrating, hovering, or sinking objects. Existing physics-aware methods address this strictly as a post-hoc layout correction, leaving the underlying geometric errors unresolved. To address this, we introduce SimuScene, a compositional 3D reconstruction pipeline that puts physics in the loop of shape and layout estimation. Rather than using physics merely for layout cleanup, we utilize the physics engine as a diagnostic measurement tool during the generative process itself. By diagnostically simulating reconstructed objects under gravity, we convert penetration and support failures into quantitative correction signals that drive gravity-axis stretching and amodal shape resampling. This physics-informed feedback loop mitigates accumulated reconstruction errors and produces a stable, simulation-ready compositional 3D scene. Extensive experiments demonstrate state-of-the-art performance on physical stability and geometric alignment benchmarks. We further highlight SimuScene's utility by deploying reconstructed environments in humanoid control and robot-arm manipulation tasks.
We propose Tetris3D, a generative framework for single-image 3D scene reconstruction that recovers objects which are physically and geometrically coherent as a scene. Existing methods often generate objects independently or couple them implicitly, providing limited guidance for ensuring fine-grained spatial compatibility between neighboring objects that interact with one another. To address this, we explicitly condition the generation of each object on the geometry of surrounding objects and their physical relationships, guiding its shape and pose to remain geometrically and physically plausible within the scene. Moreover, we introduce ComOb, a physics simulation-based dataset of 1.2M scenes featuring physical interactions across diverse object categories, with per-object meshes and pairwise physical relation annotations. Comprehensive experiments on synthetic and realworld scenes show that Tetris3D recovers coherent object shapes and poses even when interacting regions are occluded, and achieves state-of-the-art performance in both generation quality and physical stability.