Gaussian-JEPA: Joint-Embedding Predictive Learning for 3D Gaussian Splats

Bin Ren; Qi Ma; Yue Li; Zongyan Han; Yidi Li; Yuqian Fu; Rao Muhammad Anwer; Theo Gevers

cs.CVAug 16, 2026

Gaussian-JEPA: Joint-Embedding Predictive Learning for 3D Gaussian Splats

Authors: Bin Ren, Qi Ma, Yue Li, Zongyan Han, Yidi Li, Yuqian Fu, Rao Muhammad Anwer, Theo Gevers, +2 more

Organizations: Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), AE · ETH Zürich, CH · University of Amsterdam, NL · Taiyuan University of Technology, CN · King Abdullah University of Science and Technology (KAUST), SA

Abstract

3D Gaussian Splatting (3DGS) represents 3D content with anisotropic primitives that jointly encode geometry and appearance. Fixed-budget encoders consume sampled observations of Gaussian assets, so the same object may be observed through different primitive realizations. Existing self-supervised methods mainly reconstruct masked Gaussian attributes, tying supervision to one sampled realization and requiring an input-space decoder. Latent prediction offers an alternative, but its application to Gaussian tokens requires targets that accommodate coupled attributes and heterogeneous spatial support. We introduce Gaussian-JEPA, which predicts representations of held-out Gaussian token blocks from visible context. An online encoder processes the context, while a shared exponential-moving-average encoder supplies stop-gradient features for multi-scale targets. Complementary target projections and feature-space grounding provide latent supervision without reconstructing Gaussian attributes. We evaluate the features under Gaussian resampling, partial observations, and renderable shape completion, together with transfer to part segmentation and object classification. Compared with matched reconstruction pretraining, Gaussian-JEPA is more consistent across resampled inputs, retains more instance information under partial observations, and provides stronger frozen features for Gaussian completion. These results support latent prediction as an effective objective for reusable 3D Gaussian representations. Code is on the project page (https://amazingren.github.io/Gaussian-JEPA/).

Figures & tables

Figure 1: Motivation and learning paradigms. (a) Independent fixed-budget sampling yields different 1K GS observations of the same 34.3K GS asset. (b) Gaussian-MAE reconstructs masked attributes with an input-space decoder. (c) Gaussian-JEPA predicts block-indexed latent targets from visible context; the shared EMA encoder processes the full 1K GS input for each block. The lower panels use the same input and partition, isolating the learning objective.

Figure 2: Overview of Gaussian-JEPA. (a) A 1K-Gaussian observation is grouped into 64 local tokens and partitioned into a 32-token context and four non-overlapping target blocks. (b) The online encoder processes the shared context once. For each $\mathcal{B}_{m}$ , the shared EMA encoder processes all tokens into $H_{m}^{\xi}$ , whose indexed features form $Z_{m}^{\xi}$ . A shared predictor combines context features with target-conditioned queries. Linear prediction heads and MLP target projectors define two complementary latent spaces. $\mathcal{L}_{\mathrm{pred}}$ aligns predicted and stop-gradient target features, while $\mathcal{L}_{\mathrm{ground}}$ regularizes target projections before detachment.

Figure 3: Gaussian resampling on ModelNet40-GS. Rows show car_0209 and guitar_0156 . (a) Full GS (15.7K/18.1K); (b)–(d) independent 1K resamplings.

Figure 4: Relative embedding drift under Gaussian resampling. Each same-object distance is normalized by the mean distance to non-matching gallery objects in the same embedding space; dashed lines denote method means.

Method	Rel. drift $\downarrow$	R@1 (%) $\uparrow$	R@5 (%) $\uparrow$
Gaussian-MAE	0.3400	92.95	98.89
Gaussian-JEPA ( Ours )	0.2594	93.00	99.36

Table 1: Frozen representation consistency under Gaussian resampling. Relative drift normalizes each same-object distance by its mean distance to all non-matching gallery objects in the corresponding embedding space. Retrieval uses one reference and four independentl resampled queries per ModelNet40-GS object.

Method	R@1 (%) $\uparrow$	R@5 (%) $\uparrow$	R@10 (%) $\uparrow$
Gaussian-MAE	19.80	39.02	49.14
Gaussian-JEPA ( Ours )	39.82	65.28	75.95

Table 2: Retrieval from partial Gaussian observations. At 55% missing groups, five queries per object retrieve from the complete 2,467-object ModelNet40-GS gallery.

Figure 5: Robustness to partial Gaussian observations. Lines show five-seed mean R@1.

Figure 6: Qualitative Gaussian shape completion. Given a partial input of 512 Gaussians, decoders with identical architectures predict complete 1K-Gaussian representations from frozen Gaussian-MAE and Gaussian-JEPA features. The original full asset is shown only as a visual reference. Gaussian-JEPA recovers more coherent missing geometry.

Method	CD $\downarrow$	F 1% $\uparrow$	PSNR $\uparrow$	PSNR fg $\uparrow$	SSIM $\uparrow$
Gaussian-MAE	0.0732	6.62	16.03	11.31	0.7148
Gaussian-JEPA ( Ours )	0.0678	7.42	17.24	12.38	0.7469

Table 3: Shape completion on ShapeNet55-GS. CD and F 1% (%) are averaged over three independently trained decoders; full statistics are provided in the Supp. Mat. Render metrics use the seed-0 decoder at 50% visibility; PSNR is in dB.

Method	mIoU C (%) $\uparrow$	mIoU I (%) $\uparrow$
Supervised representation learning
PointNet ( Qi et al. 2017a )	80.4	83.7
PointNet++ ( Qi et al. 2017b )	81.9	85.1
Transformer ( Vaswani 2017 )	83.4	85.1
PTv1 ( Zhao et al. 2021 )	83.7	86.6
Self-supervised pretraining

Table 4: Part segmentation on ShapeNet-Part. † denotes published point-cloud results under their original protocols. Both Gaussian methods use the same downstream setting; parenthesized values show gains over Gaussian-MAE.

Figure 7: Clustering of encoded local features. We apply $k$ -means ( $k=6$ ) to features from Point-MAE, Point-JEPA, Gaussian-MAE, and Gaussian-JEPA. Each point or Gaussian is colored by its cluster assignment, providing a qualitative view of spatial organization in the learned local features.

Method	Input	MN10	MN40
Supervised Learning Only
PointNet † ( Qi et al. 2017a )	1K P	–	89.2
PointNet++ † ( Qi et al. 2017b )	1K P	–	91.9
PTv1 † ( Zhao et al. 2021 )	1K P	–	90.6
PTv2 † ( Wu et al. 2022 )	1K P	–	91.6
Full fine-tuning ( Full )

Table 5: Classification on ModelNet (accuracy, %). P / G : points/Gaussians. † denotes published point-cloud results. Gaussian methods use 1K-Gaussian pretraining and transfer; parenthesized values show gains over Gaussian-MAE.

Variant	MN10 LP $\uparrow$	Resamp. R@1 $\uparrow$
Target spaces and projector supervision
Single + feature-space grounding	93.06	90.85
Dual + attribute reconstruction	93.38	88.18
Dual + feature-space grounding	93.50	93.23
Target count; 32 masked groups
$M=2$ : $[16,16]$	93.17	92.17

Table 6: Ablations on latent targets, target sampling, and Gaussian attributes (in %). MN10 LP measures semantic transfer; resampling R@1 measures retrieval across Gaussian resamples. Target-count and scale-spread comparisons retain 32 masked groups; selected settings are shaded.

Component	Setting	Value
Input and target construction
Input	Gaussian budget / feature dimension	1,024 / 14 (all attributes)
Grouping	token groups / neighbors / soft KNN	64 / 32 / No
Target sampler	blocks / sizes / effective mask	4 / $[11,9,7,5]$ / 50%
Network architecture
Online / EMA encoders	depth / width / heads / drop path	12 / 384 / 6 / 0.1

Table A: Pretraining configuration. The integer target schedule fixes the effective mask ratio.

Method	GPU	Encoder (M)	Trainable / total (M)	Step (s) $\downarrow$	Peak memory (GiB) $\downarrow$
Gaussian-MAE ( Ma et al. 2025 )	H200 NVL	21.83	29.14 / 29.14	0.860	8.05
Gaussian-JEPA	H200 NVL	21.83	24.44 / 46.27	1.048	11.96

Table B: Parameters and pretraining resources. Measurements use FP32, batch size 256, 1K Gaussians, and one NVIDIA H200 NVL. Step time and peak allocated memory cover forward, backward, AdamW, and the EMA update, excluding data loading; timing averages 30 steps after 8 warm-up steps. Encoder parameters are those retained for downstream evaluation.

Metric	Gaussian-MAE	Gaussian-JEPA	Paired improvement [95% CI]
Resampling relative drift $\downarrow$	0.3400	0.2594	$0.0806\ [0.0789,\ 0.0821]$ reduction
Partial R@1 at 55% missing $\uparrow$	19.80	39.82	$20.02\ [18.89,\ 21.16]$ points
Completion CD $\downarrow$	$0.0732{\pm}0.0026$	$\mathbf{0.0678{\pm}0.0023}$	$0.00542\ [0.00516,\ 0.00567]$ reduction
Completion F 1% $\uparrow$	$6.62{\pm}0.63$	$\mathbf{7.42{\pm}0.44}$	$0.80\ [0.68,\ 0.91]$ points

Table C: Paired object-level uncertainty for Gaussian-specific evaluations. Positive improvements favor Gaussian-JEPA. Retrieval and F-score values are percentages. Completion entries report mean

\pm

standard deviation over three decoder seeds; confidence intervals use 10,000 paired object bootstrap replicates.

Method	MN10	MN40
4K-Gaussian transfer
Gaussian-MAE (paper) ( Ma et al. 2025 )	95.37	93.35
Gaussian-MAE (our rerun)	95.31	92.46
Gaussian-JEPA (Ours)	95.15	93.21
1K-Gaussian transfer
Gaussian-MAE	94.16	92.54

Table D: ModelNet full fine-tuning across Gaussian input budgets (accuracy, %). All encoders are pretrained with 1K Gaussians. The published row is taken from Gaussian-MAE; our rerun uses its official code under the same 1K-to-4K protocol. The main comparison transfers both methods with 1K Gaussians.

Encoder initialization	Transfer	MN10	MN40
Random initialization	Full	93.61	88.20
Gaussian-JEPA pretraining	Full	94.94	92.63

Table E: Effect of Gaussian pretraining (accuracy, %). The backbone and full fine-tuning protocol are identical; only encoder initialization differs.

Method	Transfer	MN10	MN40
Point-JEPA-style $E(C)$	Full	90.86	84.35
Gaussian-JEPA $E(C)$	Full	93.91	92.03
Point-JEPA-style $E(C)$	Linear	89.87	82.29
Gaussian-JEPA $E(C)$	Linear	92.87	88.78
Point-JEPA-style $E(C)$	MLP-3	90.20	81.80
Gaussian-JEPA $E(C)$	MLP-3	92.96	90.14

Table F: Centroid-only Gaussian transfer (accuracy, %). Both models are pretrained for 300 epochs and evaluated with the same 1K-Gaussian downstream protocols.

Method	MN40	OBJ_BG	OBJ_ONLY	PB_T50_RS
Gaussian-MAE $E(C)$	90.88	84.24	88.64	82.15
Gaussian-JEPA $E(C)$	92.17	87.43	87.26	83.14

Table G: Centroid-only transfer (accuracy, %). Gaussian-pretrained encoders are fully fine-tuned on ModelNet40 and three ScanObjectNN variants.

Figure A: PCA visualization of frozen encoder features. Columns compare Point-MAE, Point-JEPA, Gaussian-MAE, and Gaussian-JEPA; rows show ShapeNet and Objaverse objects. The first three principal components of propagated token features are mapped to RGB. Colors are interpreted within each panel and are not aligned across methods.

Figure B: t-SNE visualization of frozen ModelNet10-GS representations. Both panels contain the same 908 test objects, colored by category. PCA and t-SNE are fitted independently for each method with identical hyperparameters; coordinates are therefore not compared across panels.

Figure C: Class-conditional relative cosine affinity in the frozen ModelNet10-GS feature spaces. The same 908 objects and three-resampling aggregation as Fig. B are used. Rows and columns denote categories; outlined diagonal cells are within-class affinities. Both panels use the same scale.

Figure D: Additional part-segmentation results on ShapeNet-Part. Columns compare Point-MAE, Point-JEPA, Gaussian-MAE, Gaussian-JEPA, and ground truth. All methods are visualized on the same 2,048 annotated points with a shared camera and part-color mapping.

Target setting	Effective mask	MN10 linear $\uparrow$
$[6,6,6,6]$	$24/64=37.5\%$	93.61
$[8,8,8,8]$	$32/64=50.0\%$	93.50
$[10,10,10,10]$	$40/64=62.5\%$	93.39

Table H: Target-budget sensitivity. All settings use four equal-size targets and feature-space grounding.

Figure E: Additional Gaussian shape-completion results on ShapeNet55-GS. From left to right: the partial input (512 GS), the Gaussian-MAE prediction (1K GS), the Gaussian-JEPA prediction (1K GS), and the full source asset shown as a geometric reference. The same observation, camera viewpoint, and crop are used within each row. Orange highlights the completed regions.

Explore similar work

Apr 16, 2026

TokenGS: Decoupling 3D Gaussian Prediction from Pixels with Learnable Tokens

In this work, we revisit several key design choices of modern Transformer-based approaches for feed-forward 3D Gaussian Splatting (3DGS) prediction. We argue that the common practice of regressing Gaussian means as depths along camera rays is suboptimal, and instead propose to directly regress 3D mean coordinates using only a self-supervised rendering loss. This formulation allows us to move from the standard encoder-only design to an encoder-decoder architecture with learnable Gaussian tokens, thereby unbinding the number of predicted primitives from input image resolution and number of views. Our resulting method, TokenGS, demonstrates improved robustness to pose noise and multiview inconsistencies, while naturally supporting efficient test-time optimization in token space without degrading learned priors. TokenGS achieves state-of-the-art feed-forward reconstruction performance on both static and dynamic scenes, producing more regularized geometry and more balanced 3DGS distribution, while seamlessly recovering emergent scene attributes such as static-dynamic decomposition and scene flow.

Jiawei Ren, Michal Jan Tyszkiewicz, Jiahui Huang +1

NVIDIA

3D Gaussian Splatting 3D ViTs Dynamic Scene Reconstruction Feed-Forward 3D Reconstruction

Ask PDF

Jul 7, 2026

GaussFusion: Towards Multimodal 3D Gaussian Pretraining

3D Gaussian Splatting provides an explicit representation that jointly models geometry and appearance, serving as a scalable foundation for 3D representation learning. Existing pre-training methods for Gaussian representations, such as masked Gaussian reconstruction, primarily capture local structures but offer limited semantic supervision. In this paper, we propose GaussFusion, a multimodal pre-training framework for 3D Gaussian representations. GaussFusion integrates image and text supervision into masked Gaussian modeling through cross-modal semantic alignment, enabling the Gaussian encoder to learn both visual and language-level semantic information during pre-training. To better adapt masked modeling to the non-uniform distribution of Gaussian primitives, we further propose Gaussian Salience-guided Multi-scale Hole Masking (GSHM). GSHM constructs spatially continuous masked regions based on Gaussian salience. By applying hole masks at multiple scales, GSHM encourages the encoder to capture both fine-grained local patterns and broader structural dependencies. Extensive experiments on downstream tasks demonstrate that GaussFusion improves the transferability of Gaussian representations. Notably, GaussFusion outperforms Gaussian-MAE on ModelNet40 and ScanObjectNN (PB-T50-RS) by 0.61% and 3.85%, respectively.

Zhixuan You, Jihua Zhu, Yiding Sun +5

School of Software Engineering, Xi’an Jiaotong University, Xi’an, 710100, Shaanxi, China · Wuhu HIT Robot Technology Research Institute Co., Ltd., Wuhu, Anhui, China

3D Gaussian Splatting Cross-Modal Representation Learning Multimodal Pretraining 3D Representation Learning

Ask PDF

Sep 8, 2026

GSComplete: Gaussian Splat Completion with 2D Diffusion Priors

Gaussian splats provide a fast, high-fidelity representation for 3D objects but are often constructed from incomplete input data in practice, leaving missing regions. Existing completion methods either do not preserve the original splats or require scarcely available 3D training data. We propose GSComplete, which combines 3D generation based on Score Distillation Sampling with a novel preservation loss that encourages the original splats to be preserved where they should be visible. This effectively completes the Gaussian splat object using only 2D diffusion priors while fully preserving existing splats and generating new splats only in missing regions, without occluding the input. To evaluate our approach, we introduce a new dataset of partial Gaussian splat objects and show that GSComplete achieves significantly more accurate preservation of the input than existing methods with comparable plausibility of the completed result. Our code and dataset will be made available upon acceptance.

Elias Brugger, Philipp Erler, Stefan Ohrhallinger +1

1TU Wien, Austria · 2Adobe Research, United Kingdom

3D Gaussian Splatting 3D Shape Completion

Ask PDF