Fusion-Aware Direct 3D Gaussian Generation with Structured Patch Latent Flows

Yizhao Wang; Jingbo Wang; Guantao Zhang

cs.CVAug 18, 2026

Fusion-Aware Direct 3D Gaussian Generation with Structured Patch Latent Flows

Authors: Yizhao Wang, Jingbo Wang, Guantao Zhang

Organizations: School of Computer Science and Technology, Henan Institute of Science and Technology

Abstract

Class-guided 3D object generation is important for intelligent content creation, virtual environments, and digital asset design. Although 3D Gaussian Splatting (3DGS) offers an explicit and render-efficient representation, directly generating 3D Gaussian objects is difficult because Gaussian primitives are unordered, variable-sized, locally dense, and highly sensitive to rendering. Existing 3DGS generation methods usually depend on multi-view synthesis, reconstruction, or lifted 2D priors, fusing information mainly from observed views rather than modeling the intrinsic structural distribution of 3D Gaussian objects. This paper proposes a fusion-aware hierarchical Gaussian patch representation for direct class-guided 3DGS generation with rectified flow. Irregular Gaussian sets are decomposed into canonical local patches and encoded as structured tokens. The resulting hierarchical latent space fuses global class semantics, patch-level geometry and appearance, spatial correspondence, and rendering-sensitive cues. On this basis, we design a structure-aware rectified flow model with patch-position conditioning, global-local coupled velocity prediction, and density-aware velocity weighting, enabling direct latent generation of class-conditioned 3DGS objects within seconds. A render-feedback fusion strategy further aligns latent flow learning with decoded multi-view rendering quality. Experiments show that the proposed method generates 3D Gaussian objects with more coherent geometry, sharper local details, and better multi-view consistency than baseline latent generative models. Ablation studies confirm the contributions of hierarchical information fusion, global-local coupling, density-aware supervision, and render-feedback learning while preserving practical sampling efficiency overall.

Figures & tables

Figure 1: Overview of the hierarchical Gaussian patch VAE representation learning stage. This stage establishes a structured and continuous latent space for subsequent class-guided rectified flow generation.

Figure 2: Patch encoding visualization. The first row shows representative input 3DGS objects from different categories, and the second row shows their corresponding patch-encoded representations, where different colors indicate local Gaussian patches.

Figure 3: Hierarchical decoder architecture. The global branch derives object-level context and coarse patch frames from the global latent $\mathbf{z}^{g}$ , while the local branch reconstructs each canonical Gaussian patch from the local latent $\mathbf{z}^{p}_{m}$ under cross-attention conditioning from the global context. The decoded local Gaussian attributes are denormalized with the predicted patch frame and then merged into the final 3D Gaussian object.

Figure 4: Illustration of the class-guided structure-aware rectified flow model. The local flow branch extracts patch-level structural tokens, while the global flow branch fuses the global latent, local structural context, time embedding, and class embedding to predict the global velocity and modulate local velocity prediction.

Figure 5: Visualization of the class-guided rectified-flow generation trajectory. Each row corresponds to a class condition, and each column shows the decoded 3D Gaussian object at a different integration step. The generated object progressively evolves from noisy Gaussian primitives to a coherent class-specific structure.

Component	Setting
Patch anchors	FPS on normalized Gaussian centers
Patch ordering	Morton order after local coordinate normalization
Latent dimensions	$d_{g}=256$ , $d_{p}=128$
Condition embeddings	$\dim(\mathbf{e}_{c})=128$ ; $\mathrm{PE}(\bar{\boldsymbol{\mu}}_{m})$ projected to 128 dimensions
Density-aware weights	$\alpha=1.0$ , $\beta=0.5$ , $\gamma=0.5$
Flow loss weights	$\lambda_{g}=1.0$ , $\lambda_{\ell}=2.0$

Table 1: Implementation details and hyperparameter settings of the proposed method.

Method	Geometry		Rendering		Structure
Method	CD $\downarrow$	F@1 $\uparrow$	LPIPS $\downarrow$	SSIM $\uparrow$	FE $\downarrow$	PE $\downarrow$
Global VAE	3.94	57.8	0.205	0.733	0.081	0.126
Flat patch VAE	3.31	62.4	0.183	0.756	0.064	0.103
w/o canonicalization	3.12	64.8	0.174	0.768	0.058	0.097
w/o frame supervision	2.95	66.1	0.166	0.779	0.071	0.089
Ours	2.41	72.5	0.128	0.826	0.039	0.061

Table 2: Comparison of different 3DGS autoencoding representations. Best results are shown in bold.

Figure 6: Qualitative comparison of 3DGS reconstruction results using different autoencoding representations. The examples cover structurally different categories, including medical devices, karts, bookshelves, cars, and consoles.

Method	Semantic		Geometry		Rendering		Consistency		Efficiency
Method	CAcc $\uparrow$	CLIP $\uparrow$	CD $\downarrow$	F@1 $\uparrow$	LPIPS $\downarrow$	SSIM $\uparrow$	MVC $\uparrow$	DCS $\uparrow$	Time $\downarrow$	Mem. $\downarrow$
DiT-3DGS	83.2	0.312	3.91	57.6	0.207	0.724	0.746	0.711	35.4	10.8
U-ViT-3DGS	84.5	0.319	3.72	59.1	0.198	0.737	0.759	0.728	31.7	10.1
Vanilla RF-3DGS	85.8	0.327	3.56	61.0	0.189	0.751	0.774	0.742	16.9	8.9
GaussianCube	86.1	0.329	3.42	62.7	0.183	0.758	0.791	0.754	18.6	8.4
DiffGS	87.3	0.336	3.18	64.1	0.171	0.771	0.806	0.768	24.8	9.1

Table 3: Quantitative comparison with state-of-the-art methods. Best results are shown in bold.

Figure 7: Qualitative comparison with representative 3DGS generation baselines under the same class conditions. The examples include climbing frame, window, display cabinet, armchair, dining table, and shower partition categories.

Variant	CD $\downarrow$	LPIPS $\downarrow$	MVC $\uparrow$	Time $\downarrow$	CAcc $\uparrow$
Full model	2.47	0.132	0.872	3.8	91.6
w/o hierarchical representation	3.18	0.171	0.806	4.6	87.4
w/o global-local coupling	2.94	0.158	0.829	4.1	88.6
w/o patch-position conditioning	2.83	0.153	0.817	3.9	88.1
w/o density-aware weighting	2.72	0.147	0.842	3.8	89.3
w/o render-feedback	2.68	0.151	0.833	3.7	89.0

Table 4: Main ablation study on the proposed components. Best results are shown in bold.

Setting	CD $\downarrow$	LPIPS $\downarrow$	MVC $\uparrow$	Stability $\uparrow$	CAcc $\uparrow$
Two-stage training	2.47	0.132	0.872	0.94	91.6
Joint training	2.91	0.158	0.824	0.78	88.2
Frozen decoder in stage 2	2.69	0.146	0.846	0.90	89.4
$M=128$ patches	2.78	0.149	0.839	0.88	89.1
$M=512$ patches	2.55	0.137	0.861	0.86	90.4
$S=16$ ODE steps	2.63	0.142	0.853	0.93	90.1

Table 5: Additional ablation study on training strategy and implementation choices. Best results are shown in bold.

Figure 8: Qualitative visualization of component ablation results. The comparison includes the full model and variants without hierarchical representation, global-local coupling, patch-position conditioning, density-aware weighting, and render-feedback supervision.

Figure 9: Multi-view rendering results of generated 3D Gaussian objects. Each row shows the same generated object rendered from five viewpoints, including $0^{\circ}$ , $45^{\circ}$ , $90^{\circ}$ , $135^{\circ}$ , and $180^{\circ}$ .

Figure 10: Local detail visualization of generated 3D Gaussian objects. Each row shows a full generated object and four zoomed-in regions that contain texture-sensitive surfaces, object boundaries, repeated small parts, or thin structural elements.

Figure 11: Training convergence curves of the proposed method and representative variants. CD, LPIPS, and MVC are evaluated during training to compare convergence speed, final generation quality, and multi-view consistency.

Figure 12: Sensitivity analysis of key hyperparameters, including patch number

M

, patch size

K

, ODE step number

S

, and render-feedback frequency. The red star denotes the default setting used in the main experiments.

Explore similar work

Visibility-Guided Structured Measure Flow for Class-Conditioned 3D Gaussian Generation
Aug 17, 2026Yizhao Wang3D Asset Generation 3D Gaussian Splatting
School of Computer Science, Henan Institute of Science and Technology
Generative 3D Gaussians with Learned Density Control
May 8, 2026Runjie Yan, Yan-Pei Cao, Peng Wang +23D Gaussian Splatting 3D Asset Generation
Institute for Interdisciplinary Information Sciences, Tsinghua University, China · VAST
FLUX3D: High-Fidelity 3D Gaussian Generation with Diffusion-Aligned Sparse Representation
Jun 23, 2026Haorui Ji, Weizhe Liu, Hongdong Li +1Diffusion Model Alignment 3D Gaussian Splatting
The Australian National University · ByteDance

Aug 17, 2026J/K move · Enter open · S save

Visibility-Guided Structured Measure Flow for Class-Conditioned 3D Gaussian Generation

Yizhao Wang

School of Computer Science, Henan Institute of Science and Technology

3D Asset Generation 3D Gaussian Splatting Flow Matching Variational Autoencoders

Open paper AskPDF arXiv

3D Gaussian Splatting (3DGS) has made real-time, high-fidelity 3D rendering practical, yet turning this explicit representation into a native generative space remains an open challenge. Directly generating 3DGS objects is difficult because Gaussian primitives are unordered, variable-sized, locally dense, and highly sensitive to rendering behavior. We present VISTA-GS, a visibility-guided structured measure flow framework for class-conditioned 3D Gaussian generation. Instead of treating a 3DGS object as a flat primitive sequence or a generic latent token grid, we formulate it as a structured Gaussian measure weighted by opacity, anisotropic covariance, and multi-view visibility. Based on this formulation, we introduce a visibility-aware measure VAE that learns permutation-invariant, variable-size-compatible, and rendering-aware latent representations of 3DGS objects. We further develop a renderer-consistent measure flow that transports class-conditioned priors toward the learned 3DGS measure distribution while aligning the decoded objects with their multi-view rendering distributions. To preserve object layout and local details, VISTA-GS incorporates structure-preserving patch transport that couples global class semantics, local Gaussian measure patches, and spatial anchors during flow prediction. On VISTA-Obj30, VISTA-GS improves over the strongest baseline by roughly 60--72% across geometry, appearance, view-consistency error, and generation speed. This design enables efficient generation of coherent, detailed, and view-consistent 3D Gaussian objects without relying on per-instance optimization, multi-view image synthesis, or reconstruction-based lifting pipelines. Project code and model checkpoints will be released.

Abstract

Explore similar work

Visibility-Guided Structured Measure Flow for Class-Conditioned 3D Gaussian Generation

Generative 3D Gaussians with Learned Density Control

FLUX3D: High-Fidelity 3D Gaussian Generation with Diffusion-Aligned Sparse Representation