cs.CVNov 2, 2026

CoLa3D: Composable Latent 3D Decomposition

Authors: Xudong Cai, Yongcai Wang, Deying Li, Chuanxia Zheng

Organizations: School of Information, Renmin University of China · PVG, Nanyang Technological University · Hebei Key Laboratory of Real-virtual Integrated Autonomous Systems (RIAS)

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

We introduce CoLa3D, a simple and effective model for composable 3D scene decomposition that, given a single whole-scene latent from either a single image or an existing scene mesh as input, directly outputs a set of object meshes with consistent spatial layout that can be assembled back into a coherent scene. CoLa3D introduces a novel paradigm for 3D scene decomposition within a shared whole-scene latent, unlike alternatives that perform per-object generation in canonical spaces and then glue the objects back together with fragile pose and scale estimation. The network is a lightweight, flexible, and scalable query-based transformer that accepts various types of queries, including 2D masks, 3D points, and learnable queries, and learns to decompose the scene into objects in a compact latent space. On both synthetic and real-world datasets, CoLa3D achieves state-of-the-art performance on composable 3D scene decomposition, substantially improving global scene-level consistency and object-level quality while running 2 times faster and using less than 1% of SAM 3D's training data. Visible depth on par with MoGe2 further confirms that latent decomposition preserves accurate visible scene geometry while producing physically complete, decomposable object meshes.

Figures & tables

Figure 1: Composable 3D scene generated by CoLa3D. Given a single input image ( left ), CoLa3D decomposes its whole -scene latent into individual object meshes ( bottom ) within a shared coordinate frame, faithfully reproducing the input layout ( middle left ) while keeping each object physically accessible for downstream tasks such as rearrangement ( middle right ) and physical simulation ( right ).
Figure 2: Overview of CoLa3D. Starting from an RGB image or a holistic scene mesh, CoLa3D first obtains a structured scene latent zgeos\bm{z}^{s}_{\text{geo}} with TRELLIS.2 [ 76 ] and augments it with 3D semantic features zsems\bm{z}^{s}_{\text{sem}} from Utonia [ 87 ] . A lightweight decomposition transformer then uses mask, learnable, or point queries to decompose zs=zgeos+zsems\bm{z}^{s}=\bm{z}^{s}_{\text{geo}}+\bm{z}^{s}_{\text{sem}} through LL blocks of self-attention, cross-attention to frozen DINOv3 [ 52 ] features fimg\bm{f}^{\text{img}} , and alternating Latent-to-Query/Query-to-Latent cross-attention. Finally, a multi-hot assignment extracts per-instance latents {zko}\{\bm{z}^{o}_{k}\} , which are decoded in parallel by the shared decoder D\mathcal{D} into aligned object meshes and can be recomposed in the global 3D space.
Figure 3: Qualitative comparisons on synthetic data. Compared with baselines, CoLa3D produces spatially coherent object meshes with correct relative positions and scales.
Table 1: Comparisons on synthetic datasets. Prior column: denotes a 2D instance mask, a 3D point prompt, and no prior. We highlight the Best , Second-Best , and Third-Best results.
Figure 4: Qualitative results on real-world scenes. CoLa3D produces coherent decompositions on Replica and ScanNet++ images, demonstrating robust generalization from synthetic training data.
Table 2: Zero-shot generalization on real-world scenes. Prior column: denotes a 2D instance mask, a 3D point prompt, and no prior. We highlight the Best , Second-Best , and Third-Best results. Here, CD is one-side Chamfer Distance (GT →\to Pred).
Table 3: Comparisons on MetaScenes-hard. Prior column: denotes a 2D instance mask, a 3D point prompt, and no prior. We highlight the Best , Second-Best , and Third-Best results.
Table 4: Additional results and ablations on 3D-FRONT. Left: visible-region depth accuracy. MoGe2 is shown as a reference and excluded from ranking. Right: ablation study on the InstPIFU split. We highlight the Best , Second-Best , and Third-Best results among 3D methods.
Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Architecture details of the attention block used in the Query-Based 3D Latent Decomposition Transformer.
Table 5: Detailed per-dataset inference time. We report the average per-scene inference time in seconds.
Table 6: Geometric interpenetration on synthetic datasets. We report collision rate (CR) and normalized interpenetration volume (NIV), both in percent; lower is better. Prior column: denotes a 2D instance mask, a 3D point prompt, and no prior. We highlight the Best , Second-Best , and Third-Best results.
Figure 6: Decomposition from holistic scene meshes. CoLa3D produces coherent object-level meshes across generated and existing holistic scene meshes.
Figure 7: Additional qualitative comparisons on synthetic scenes. We show more results on the 3D-FRONT InstPIFU split and MetaScenes following the same layout as fig. 3 .
Figure 8: Additional qualitative comparisons on real-world scenes. We show more zero-shot results on Replica and ScanNet++ following the same layout as fig. 4 .
Figure 9: Qualitative depth comparison on 3D-FRONT. We visualize the input image, ground-truth depth, rendered depth maps from Gen3DSR, MIDI, SceneGen, SAM 3D, and CoLa3D with mask, learnable, and point queries, and the MoGe2 prediction as a monocular-depth reference. CoLa3D yields visible-region depth on par with MoGe2 while additionally producing decomposable per-object geometry.
More in this topic

3D Scene Understanding

3D scene understanding builds geometric and semantic representations of environments for spatial reasoning, perception, and interaction.

See all 304 papers

Explore similar work

CardsList

  1. DistScene: Object-to-Scene Distillation for 3D Scene Generation

    Oct 3, 2026Kunming Luo, Hongyu Yan, Ken Deng +63D Scene GenerationImage-to-3D Scene Generation

    The Hong Kong University of Science and Technology · TeleAI, China Telecom

  2. Mira-Scene: Pixel-Aligned Layouts for Generative 3D Scene Reconstruction

    Sep 20, 2026Yang-Tian Sun, Tianjia Liu, Zehuan Huang +73D Reconstruction3D Scene Layout Generation

    The University of Hong Kong · VAST

  3. OmniLatent: Unified Latent for 3D Part Modeling

    Nov 6, 2026Yunhan Yang, Yunfei Zhao, Zeqiang Lai +63D Part SegmentationPart-Level 3D Generation

    University of Hong Kong, Hong Kong · Tencent Technology, China · The Chinese University of Hong Kong, Hong Kong +1