Recent progress in 3D generation has motivated growing interest in compact generative representations of complex 3D scenes. For native 3D Gaussian scenes, however, learning such representations remains challenging due to widely varying primitive counts, nonuniform spatial density, and differences in geometry and appearance even among spatially neighboring primitives. In this work, we introduce TriFold, a native 3D Gaussian VAE that encodes Gaussian scenes into a fixed-layout triplane latent and jointly reconstructs their geometry and appearance. The key insight is to impose structural regularity on the latent space while preserving the spatial adaptivity of the underlying Gaussians. During encoding, we use cell-wise learned queries to aggregate projected Gaussians into multiple slots per cell, reducing the information loss caused by single-feature pooling. During decoding, occupancy prediction identifies active regions, where query-based set prediction and candidate selection reconstruct local Gaussian sets with spatially varying primitive counts. Building on this representation, we train a latent flow-matching model for conditional scene-level 3DGS generation, enabling point-cloud-conditioned synthesis and single-image-conditioned amodal generation. Experiments demonstrate that TriFold achieves high-fidelity reconstruction and generation from a single image that provides only a partial observation of the scene, validating its effectiveness for native 3D Gaussian representation learning. https://fantasy-amap.github.io/TriFold
Figures & tables
Figure 1: Overview of TriFold. (a) TriFold encodes variable-cardinality Gaussian scenes into fixed-layout triplane latents that jointly represent geometry and appearance, and decodes content-adaptive Gaussian sets with variable local densities. (b) The learned representation supports detailed scene reconstruction. (c) Our single-image-conditioned amodal generation preserves observed content while inferring plausible geometry and appearance for unobserved regions.
Figure 2: Overview of TriFold. The triplane encoder maps a variable-cardinality Gaussian set to a fixed-layout variational latent, and the triplane decoder reconstructs spatially adaptive local Gaussian sets. The learned latent further supports image-conditioned scene generation with flow matching.
Figure 3: Qualitative VAE reconstruction comparison. Compared with GaussianGPT, our method better captures scene geometry and appearance.
Figure 4: Adaptive 3DGS allocation. Visualization of the target scene, VAE reconstruction, and retained Gaussian count per occupied voxel.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
CD ↓
Can3Tok
4.53
0.012
0.920
10.96
DiffGS
12.43
0.719
0.374
3.21
GaussianGPT
21.40
0.856
0.157
2.62
TriFold (Ours)
26.55
0.913
0.088
2.23
Table 1: Quantitative comparison with existing 3D Gaussian representation methods. Chamfer Distance (CD) values are multiplied by 103 . Best results are in bold , second-best are underlined .
Observed Appearance
Unobserved Appearance
Unobserved Geometry
Condition
FID ↓
KID ( ×103 ) ↓
FID ↓
KID ( ×103 ) ↓
COV-CD (%) ↑
MMD-CD ( ×103 ) ↓
Full point cloud
54.12
16.76
N/A (complete conditioning)
Single image
60.38
18.20
67.81
21.43
85.24
12.75
Table 2: Conditional generation evaluation. For single-image conditioning, appearance is evaluated separately in regions observed and unobserved from the conditioning view. Full-point-cloud conditioning covers the complete scene and therefore has no unobserved split. Detailed evaluation protocols are provided in Appendix A.6 .
Figure 5: Point-cloud-conditioned generation. Given a full point cloud, the latent flow model generates native 3DGS with coherent geometry and appearance across novel views.
Figure 6: Single-image amodal generation. Given one input image, the latent flow model generates plausible room structure and appearance across novel views.
Table 9
ID
Architecture
Training objective
Rendering quality
Gaussian reconstruction
Triplane Encoder
Triplane Decoder
Phantom
Lpos
Lrender
PSNR ↑
SSIM ↑
LPIPS ↓
Chamfer Distance (CD) ↓
baseline
✓
12.10
0.617
0.491
3.06
1
✓
✓
14.72
0.752
0.378
2.58
2
✓
✓
13.53
0.669
0.330
2.77
3
✓
✓
✓
17.81
0.803
0.231
2.48
4
✓
✓
✓
22.64
0.885
0.134
2.41
Table 5: Component analysis. Each row adds or replaces one model or training component while keeping the remaining evaluation protocol fixed. For the Lpos -only baseline, we use the same correspondence to directly supervise all Gaussian attributes.
Variant
PSNR ↑
SSIM ↑
LPIPS ↓
CD ↓
Encoder Head
Sparse 3D CNN
23.16
0.830
0.139
2.48
PointNet
24.37
0.878
0.117
2.41
PTv3
25.83
0.904
0.096
2.27
Aggregation
Mean pooling
24.60
0.857
0.102
2.39
Table 6: Architectural design choices. Comparison of alternative encoder, projection, and decoding designs under the same evaluation settings.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Variable cardinality and nonuniform spatial density in scene-level 3DGS. (a-c) Representative InteriorGS scene crops with different primitive counts, visualized using Gaussian centers voxelized on a shared 2563 grid. (d) Across 100 random scene crops, the number of primitives ranges from 351,306 to 1,457,726, with a median of 682,845. (e) The complementary cumulative distribution of occupied-voxel density shows a pronounced long tail: 97.8% of occupied voxels contain at most 24 Gaussians, while the maximum density can even reach 389 Gaussians per voxel. These statistics motivate a representation that accommodates both scene-dependent cardinality and non-uniform local Gaussian allocation.
Figure 8: Additional VAE Reconstructions. Under the identical evaluation setup, our VAE preserves more visible layout, appearance, and fine-grained local structures in these examples.
Figure 9: Representative reconstruction failures of the adapted Can3Tok and DiffGS baselines. Both baselines are trained using their officially released implementations on the same scene-level training split and with the same optimization budget used in our comparison. Under this setting, distinct input scenes can be reconstructed as similar low-detail structures. These examples illustrate the observed behavior under our scene-level adaptation and are not intended to characterize the methods outside this setting.
Figure 10: Encoder feature comparison. We visualize features extracted by PTv3, PointNet, and a sparse 3D CNN from the same Gaussian input. PTv3 retains clearer spatial variation around surfaces, objects, and scene boundaries in this example; Table 6 reports the corresponding quantitative comparison.
Setting
PSNR ↑
SSIM ↑
LPIPS ↓
CD ↓
K=8
25.06
0.887
0.112
2.31
K=16
25.61
0.895
0.106
2.33
K=24 (Ours)
25.83
0.904
0.096
2.27
K=32
25.96
0.907
0.092
2.27
Appendix
Table 11: Effect of local Gaussian decoder capacity. K is the maximum number of Gaussian candidates per selected voxel, while rendering supervision always uses the complete target scene. CD values are multiplied by 103 . Best results are in bold , second-best are underlined .
Figure 11: Single-slot removal. This intervention complements the slot-count ablation in Table 10 . We zero one of the Q encoder slots while keeping all remaining latent values fixed. The top row visualizes decoded feature energy on the xz plane, and the bottom row shows the corresponding Gaussian reconstruction. Different removals produce distinct changes, indicating that the slots are not identical copies of the same local summary.
Figure 12: Latent interpolation and perturbation analysis. (a) Linear interpolation produces smooth changes in decoded triplane features. (b) The latent representation remains stable under moderate perturbations, with pronounced degradation under Gaussian noise at σ=1.0 .
Figure 13: Additional point-cloud-conditioned generations. Each panel shows a Gaussian scene sampled from a different point-cloud condition. The examples illustrate the diversity of generated indoor layouts and scene content.
Figure 14: Single-image-conditioned generation. Each row shows the conditioning view followed by top, front, and side renderings of one sampled Gaussian scene. The novel views assess whether input-specific scene cues are preserved within a coherent 3D completion.
Most recent advances in 3D generative modeling rely on diffusion or flow-matching formulations. We instead explore a fully autoregressive alternative and introduce GaussianGPT, a transformer-based model that directly generates 3D Gaussians via next-token prediction, thus facilitating full 3D scene generation. We first compress Gaussian primitives into a discrete latent grid using a sparse 3D convolutional autoencoder with vector quantization. The resulting tokens are serialized and modeled using a causal transformer with 3D rotary positional embedding, enabling sequential generation of spatial structure and appearance. Unlike diffusion-based methods that refine scenes holistically, our formulation constructs scenes step-by-step, naturally supporting completion, outpainting, controllable sampling via temperature, and flexible generation horizons. This formulation leverages the compositional inductive biases and scalability of autoregressive modeling while operating on explicit representations compatible with modern neural rendering pipelines, positioning autoregressive transformers as a complementary paradigm for controllable and context-aware 3D generation.
Nicolas von Lützow, Barbara Rössle, Katharina Schmid +1
Many scalable latent 3D generators operate on structured tensors, whereas pre-optimized 3D Gaussian Splatting (3DGS) reconstructions are unordered, spatially irregular, and vary widely in primitive count. We present GS-Voxel, a fitting-free structured latent framework, and evaluate it for large-scale aerial 3D Gaussian scene generation. GS-Voxel deterministically converts a compatible pre-optimized 3DGS reconstruction into sparse active voxels without additional per-scene optimization, retaining the sub-voxel positions and rendering attributes of the selected primitives. A GS-specific factorized VAE then separately encodes voxel geometry and local Gaussian attributes into sparse 3D latents whose size grows with the number of occupied voxels rather than being limited by a fixed scene-wide primitive count. We train image-conditioned flow models in the GS-Voxel latent space to generate aerial 3DGS scenes. A key application enabled by GS-Voxel is large-area scene generation: overlap-aware tiled inference extends synthesis beyond a single training crop conditioned on satellite-view images. Our results show that GS-Voxel provides structured latents for pre-optimized aerial 3DGS reconstructions, with latent capacity that grows with the number of occupied voxels.
Ming Qian, Zijian Wang, Minchao Sun +5
Amap, Alibaba · Zhejiang University · Peking University
We present Density-Sampled Gaussians (DeG), a novel 3D representation designed to bridge the gap between adaptive rendering primitives and scalable generative modeling. Unlike existing approaches that constrain 3D Gaussians to fixed voxel grids or arrays, DeG models Gaussian centers as samples from a learnable probability density function defined over an octree. This formulation provides a rigorous mathematical framework for adaptive density control: by jointly optimizing the spatial density and Gaussian attributes under rendering supervision, our model naturally concentrates primitives in regions of high geometric complexity. We achieve this via a new render loss contribution gradient that serves as a fully differentiable analogue to the discrete densification and pruning heuristics used in standard Gaussian Splatting. The resulting representation is highly flexible, supporting variable-resolution decoding from a single latent code by simply adjusting the sampling budget. To enable generative synthesis, we train a latent diffusion model on DeG. We identify a critical challenge in applying diffusion to unordered set-structured latents, which can significantly slow convergence, and propose VecSeq, a canonical re-indexing mechanism that anchors latent tokens to a deterministic 3D Sobol sequence. This transforms the ambiguous set-generation problem into a robust sequence modeling task. Extensive experiments demonstrate that our pipeline achieves state-of-the-art quality in single-image-to-3D generation, combining the structural adaptivity of unstructured primitives with the training stability of grid-based methods.
Runjie Yan, Yan-Pei Cao, Peng Wang +2
Institute for Interdisciplinary Information Sciences, Tsinghua University, China · VAST