Fusion-Aware Direct 3D Gaussian Generation with Structured Patch Latent Flows

Yizhao Wang; Jingbo Wang; Guantao Zhang

cs.CVAug 18, 2026

Fusion-Aware Direct 3D Gaussian Generation with Structured Patch Latent Flows

Authors: Yizhao Wang, Jingbo Wang, Guantao Zhang

Organizations: School of Computer Science and Technology, Henan Institute of Science and Technology

Abstract

Class-guided 3D object generation is important for intelligent content creation, virtual environments, and digital asset design. Although 3D Gaussian Splatting (3DGS) offers an explicit and render-efficient representation, directly generating 3D Gaussian objects is difficult because Gaussian primitives are unordered, variable-sized, locally dense, and highly sensitive to rendering. Existing 3DGS generation methods usually depend on multi-view synthesis, reconstruction, or lifted 2D priors, fusing information mainly from observed views rather than modeling the intrinsic structural distribution of 3D Gaussian objects. This paper proposes a fusion-aware hierarchical Gaussian patch representation for direct class-guided 3DGS generation with rectified flow. Irregular Gaussian sets are decomposed into canonical local patches and encoded as structured tokens. The resulting hierarchical latent space fuses global class semantics, patch-level geometry and appearance, spatial correspondence, and rendering-sensitive cues. On this basis, we design a structure-aware rectified flow model with patch-position conditioning, global-local coupled velocity prediction, and density-aware velocity weighting, enabling direct latent generation of class-conditioned 3DGS objects within seconds. A render-feedback fusion strategy further aligns latent flow learning with decoded multi-view rendering quality. Experiments show that the proposed method generates 3D Gaussian objects with more coherent geometry, sharper local details, and better multi-view consistency than baseline latent generative models. Ablation studies confirm the contributions of hierarchical information fusion, global-local coupling, density-aware supervision, and render-feedback learning while preserving practical sampling efficiency overall.

Figures & tables

Figure 1: Overview of the hierarchical Gaussian patch VAE representation learning stage. This stage establishes a structured and continuous latent space for subsequent class-guided rectified flow generation.

Figure 2: Patch encoding visualization. The first row shows representative input 3DGS objects from different categories, and the second row shows their corresponding patch-encoded representations, where different colors indicate local Gaussian patches.

Figure 3: Hierarchical decoder architecture. The global branch derives object-level context and coarse patch frames from the global latent $\mathbf{z}^{g}$ , while the local branch reconstructs each canonical Gaussian patch from the local latent $\mathbf{z}^{p}_{m}$ under cross-attention conditioning from the global context. The decoded local Gaussian attributes are denormalized with the predicted patch frame and then merged into the final 3D Gaussian object.

Figure 4: Illustration of the class-guided structure-aware rectified flow model. The local flow branch extracts patch-level structural tokens, while the global flow branch fuses the global latent, local structural context, time embedding, and class embedding to predict the global velocity and modulate local velocity prediction.

Figure 5: Visualization of the class-guided rectified-flow generation trajectory. Each row corresponds to a class condition, and each column shows the decoded 3D Gaussian object at a different integration step. The generated object progressively evolves from noisy Gaussian primitives to a coherent class-specific structure.

Component	Setting
Patch anchors	FPS on normalized Gaussian centers
Patch ordering	Morton order after local coordinate normalization
Latent dimensions	$d_{g}=256$ , $d_{p}=128$
Condition embeddings	$\dim(\mathbf{e}_{c})=128$ ; $\mathrm{PE}(\bar{\boldsymbol{\mu}}_{m})$ projected to 128 dimensions
Density-aware weights	$\alpha=1.0$ , $\beta=0.5$ , $\gamma=0.5$
Flow loss weights	$\lambda_{g}=1.0$ , $\lambda_{\ell}=2.0$

Table 1: Implementation details and hyperparameter settings of the proposed method.

Method	Geometry		Rendering		Structure
Method	CD $\downarrow$	F@1 $\uparrow$	LPIPS $\downarrow$	SSIM $\uparrow$	FE $\downarrow$	PE $\downarrow$
Global VAE	3.94	57.8	0.205	0.733	0.081	0.126
Flat patch VAE	3.31	62.4	0.183	0.756	0.064	0.103
w/o canonicalization	3.12	64.8	0.174	0.768	0.058	0.097
w/o frame supervision	2.95	66.1	0.166	0.779	0.071	0.089
Ours	2.41	72.5	0.128	0.826	0.039	0.061

Table 2: Comparison of different 3DGS autoencoding representations. Best results are shown in bold.

Figure 6: Qualitative comparison of 3DGS reconstruction results using different autoencoding representations. The examples cover structurally different categories, including medical devices, karts, bookshelves, cars, and consoles.

Method	Semantic		Geometry		Rendering		Consistency		Efficiency
Method	CAcc $\uparrow$	CLIP $\uparrow$	CD $\downarrow$	F@1 $\uparrow$	LPIPS $\downarrow$	SSIM $\uparrow$	MVC $\uparrow$	DCS $\uparrow$	Time $\downarrow$	Mem. $\downarrow$
DiT-3DGS	83.2	0.312	3.91	57.6	0.207	0.724	0.746	0.711	35.4	10.8
U-ViT-3DGS	84.5	0.319	3.72	59.1	0.198	0.737	0.759	0.728	31.7	10.1
Vanilla RF-3DGS	85.8	0.327	3.56	61.0	0.189	0.751	0.774	0.742	16.9	8.9
GaussianCube	86.1	0.329	3.42	62.7	0.183	0.758	0.791	0.754	18.6	8.4
DiffGS	87.3	0.336	3.18	64.1	0.171	0.771	0.806	0.768	24.8	9.1

Table 3: Quantitative comparison with state-of-the-art methods. Best results are shown in bold.

Figure 7: Qualitative comparison with representative 3DGS generation baselines under the same class conditions. The examples include climbing frame, window, display cabinet, armchair, dining table, and shower partition categories.

Variant	CD $\downarrow$	LPIPS $\downarrow$	MVC $\uparrow$	Time $\downarrow$	CAcc $\uparrow$
Full model	2.47	0.132	0.872	3.8	91.6
w/o hierarchical representation	3.18	0.171	0.806	4.6	87.4
w/o global-local coupling	2.94	0.158	0.829	4.1	88.6
w/o patch-position conditioning	2.83	0.153	0.817	3.9	88.1
w/o density-aware weighting	2.72	0.147	0.842	3.8	89.3
w/o render-feedback	2.68	0.151	0.833	3.7	89.0

Table 4: Main ablation study on the proposed components. Best results are shown in bold.

Setting	CD $\downarrow$	LPIPS $\downarrow$	MVC $\uparrow$	Stability $\uparrow$	CAcc $\uparrow$
Two-stage training	2.47	0.132	0.872	0.94	91.6
Joint training	2.91	0.158	0.824	0.78	88.2
Frozen decoder in stage 2	2.69	0.146	0.846	0.90	89.4
$M=128$ patches	2.78	0.149	0.839	0.88	89.1
$M=512$ patches	2.55	0.137	0.861	0.86	90.4
$S=16$ ODE steps	2.63	0.142	0.853	0.93	90.1

Table 5: Additional ablation study on training strategy and implementation choices. Best results are shown in bold.

Figure 8: Qualitative visualization of component ablation results. The comparison includes the full model and variants without hierarchical representation, global-local coupling, patch-position conditioning, density-aware weighting, and render-feedback supervision.

Figure 9: Multi-view rendering results of generated 3D Gaussian objects. Each row shows the same generated object rendered from five viewpoints, including $0^{\circ}$ , $45^{\circ}$ , $90^{\circ}$ , $135^{\circ}$ , and $180^{\circ}$ .

Figure 10: Local detail visualization of generated 3D Gaussian objects. Each row shows a full generated object and four zoomed-in regions that contain texture-sensitive surfaces, object boundaries, repeated small parts, or thin structural elements.

Figure 11: Training convergence curves of the proposed method and representative variants. CD, LPIPS, and MVC are evaluated during training to compare convergence speed, final generation quality, and multi-view consistency.

Figure 12: Sensitivity analysis of key hyperparameters, including patch number

M

, patch size

K

, ODE step number

S

, and render-feedback frequency. The red star denotes the default setting used in the main experiments.

Explore similar work

Aug 17, 2026

Visibility-Guided Structured Measure Flow for Class-Conditioned 3D Gaussian Generation

3D Gaussian Splatting (3DGS) has made real-time, high-fidelity 3D rendering practical, yet turning this explicit representation into a native generative space remains an open challenge. Directly generating 3DGS objects is difficult because Gaussian primitives are unordered, variable-sized, locally dense, and highly sensitive to rendering behavior. We present VISTA-GS, a visibility-guided structured measure flow framework for class-conditioned 3D Gaussian generation. Instead of treating a 3DGS object as a flat primitive sequence or a generic latent token grid, we formulate it as a structured Gaussian measure weighted by opacity, anisotropic covariance, and multi-view visibility. Based on this formulation, we introduce a visibility-aware measure VAE that learns permutation-invariant, variable-size-compatible, and rendering-aware latent representations of 3DGS objects. We further develop a renderer-consistent measure flow that transports class-conditioned priors toward the learned 3DGS measure distribution while aligning the decoded objects with their multi-view rendering distributions. To preserve object layout and local details, VISTA-GS incorporates structure-preserving patch transport that couples global class semantics, local Gaussian measure patches, and spatial anchors during flow prediction. On VISTA-Obj30, VISTA-GS improves over the strongest baseline by roughly 60--72% across geometry, appearance, view-consistency error, and generation speed. This design enables efficient generation of coherent, detailed, and view-consistent 3D Gaussian objects without relying on per-instance optimization, multi-view image synthesis, or reconstruction-based lifting pipelines. Project code and model checkpoints will be released.

Yizhao Wang

School of Computer Science, Henan Institute of Science and Technology

3D Asset Generation 3D Gaussian Splatting Flow Matching Variational Autoencoders

Ask PDF

May 8, 2026

Generative 3D Gaussians with Learned Density Control

We present Density-Sampled Gaussians (DeG), a novel 3D representation designed to bridge the gap between adaptive rendering primitives and scalable generative modeling. Unlike existing approaches that constrain 3D Gaussians to fixed voxel grids or arrays, DeG models Gaussian centers as samples from a learnable probability density function defined over an octree. This formulation provides a rigorous mathematical framework for adaptive density control: by jointly optimizing the spatial density and Gaussian attributes under rendering supervision, our model naturally concentrates primitives in regions of high geometric complexity. We achieve this via a new render loss contribution gradient that serves as a fully differentiable analogue to the discrete densification and pruning heuristics used in standard Gaussian Splatting. The resulting representation is highly flexible, supporting variable-resolution decoding from a single latent code by simply adjusting the sampling budget. To enable generative synthesis, we train a latent diffusion model on DeG. We identify a critical challenge in applying diffusion to unordered set-structured latents, which can significantly slow convergence, and propose VecSeq, a canonical re-indexing mechanism that anchors latent tokens to a deterministic 3D Sobol sequence. This transforms the ambiguous set-generation problem into a robust sequence modeling task. Extensive experiments demonstrate that our pipeline achieves state-of-the-art quality in single-image-to-3D generation, combining the structural adaptivity of unstructured primitives with the training stability of grid-based methods.

Runjie Yan, Yan-Pei Cao, Peng Wang +2

Institute for Interdisciplinary Information Sciences, Tsinghua University, China · VAST

3D Gaussian Splatting 3D Asset Generation Latent Diffusion Models Adaptive Gaussian Densification Single-View 3D Reconstruction

Ask PDF

Jun 23, 2026

FLUX3D: High-Fidelity 3D Gaussian Generation with Diffusion-Aligned Sparse Representation

Sparse voxel representation has emerged as a scalable foundation for image-to-3D Gaussian Splatting (3DGS) generation, yet current methods struggle to preserve high-frequency visual details of input images due to two structural bottlenecks. First, they adopt discriminative 2D features optimized for semantic abstraction to construct sparse voxel latents, which suppress reconstructive cues and induce a representation bottleneck. Second, in the generation stage, standard diffusion transformers lack effective mechanisms to align dense 2D image tokens with sparse 3D voxel latents, resulting in a cross-modal correspondence bottleneck. To address these issues, we propose FLUX3D, a scalable image-to-3DGS framework that boosts both representation learning and cross-modal alignment during generation. We first revisit 2D feature selection for sparse-voxel-based 3D representation learning, propose Diffusion-Aligned Structured Latents (DA-SLAT) and couple it with a decoder-only architecture to improve 3DGS reconstruction fidelity. We also design a sparse-structure-aware diffusion framework, which integrates the Sparse-structure Multimodal Diffusion Transformer (SMDiT) and Modal-Aware Rotary Positional Embedding (MARoPE) to achieve geometry-agnostic 2D-3D alignment. Extensive benchmark experiments demonstrate that FLUX3D yields substantial improvements in appearance fidelity and significantly outperforms all state-of-the-art (SOTA) methods in generating high-quality 3DGS assets.

Haorui Ji, Weizhe Liu, Hongdong Li +1

The Australian National University · ByteDance

Diffusion Model Alignment 3D Gaussian Splatting 3D Asset Generation 3D Scene Representation

Ask PDF