Can a neural network learn to render 3D Gaussian splats on its own, with no rasterizer in the renderer? We present GaussianRenderFormer, a transformer that takes a set of Gaussians and a camera and outputs the image directly. Each Gaussian becomes one token in a view-independent encoder, and a view-dependent decoder turns small patches of camera rays into pixels by attending to those tokens. Trained on 26,820 Objaverse objects pruned to 20,000 Gaussians each, GaussianRenderFormer renders 300 unseen objects at 38.08 dB PSNR against the full, unpruned splats, while rasterizing the same pruned input reaches 44.02 dB. The gap is not one of generalization, and our diagnostics are consistent with a bottleneck in the view decoder: telling it the projected footprint of each Gaussian, which it could in principle infer, adds 1.4 dB. Under-training is not ruled out. Code, trained models and videos: https://svlwoof.github.io/gaussianrenderformer
Figures & tables
Figure 1: GaussianRenderFormer renders 3D Gaussian splats with a neural network, without a rasterizer. Top: the reference, a rasterization of each object’s full splat. Bottom (ours): GaussianRenderFormer from a 20,000-Gaussian input, on six objects never seen in training, chosen by hand for variety. Right: a 3× zoom of the most detailed region of the last object, where the model softens the stone’s texture.
Figure 2: Overview of GaussianRenderFormer. Each Gaussian’s 14 parameters are embedded as one token, and a view-independent scene encoder (top) lets these tokens exchange information, using a 3D rotary embedding of their centers. For each view, the image is split into 4×4 -pixel patches, and each patch becomes a token built from its camera rays. The view decoder (bottom) lets these tokens attend to the scene tokens, using a 2D rotary embedding of where each Gaussian lands in the image and an embedding of its projected footprint, and a DPT head turns them into pixels.
Stage
Trains
Steps
Resolution
Loss
1
input layer only
33.5k
256
log-L1
2
whole model
402k
256
log-L1
3
whole model, new decoder changes
3k
256
log-L1, background weight
4
whole model
385k
512
log-L1 + LPIPS, background weight
Table 1: Training stages. Stage 3 switches on the changes of Section 4.2 .
Figure 3: Three fresh held-out objects, chosen at the 10th, 50th and 90th percentile of GaussianRenderFormer’s per-object PSNR. From left to right: the reference (the full, unpruned splat rasterized with gsplat), gsplat on the pruned 20,000-Gaussian input, GaussianRenderFormer on the same input, and the absolute error of each against the reference, scaled by 5. GaussianRenderFormer scores 34.5, 39.5 and 41.7 dB on these views, and gsplat on the input 39.7, 45.5 and 45.7 dB. GaussianRenderFormer gets shapes and colors right, and its error concentrates on edges and fine texture.
PSNR (dB)
Masked
Renderer and objects
4 views
28 views
PSNR (dB)
SSIM
LPIPS
gsplat on the input, fresh held-out
44.97
44.02
39.19
0.9954
0.0046
gsplat on the input, training objects
45.01
44.12
–
0.9954
0.0046
Ours, fresh held-out
38.36
38.08
32.68
0.9814
0.0113
Ours, development held-out
38.36
38.02
32.66
0.9802
0.0116
Ours, training objects
38.55
38.25
–
0.9821
0.0108
Table 2: Main result, against the unpruned render. PSNR over the 4 views at distance 1.7 used in the rest of the paper and over all 28 views at the three training distances, masked PSNR over the object’s pixels on the whole image, and SSIM and LPIPS over the 28 views. Fresh held-out: 300 objects never used during development. Development held-out: the 300 held-out objects of the other sections. No full splat is kept for the training objects, so they have no masked PSNR. Without the footprint: the same recipe and schedule without the projected footprint (Section 4.2 ). Higher PSNR and SSIM and lower LPIPS are better.
Views per call
Ours (ms)
per view (ms)
peak memory (GB)
gsplat (ms)
1
275
275
2.8
0.7
4
685
171
7.7
0.9
14
2,012
144
13.0
1.2
Table 3: Rendering time and memory on one NVIDIA L40S GPU, 20,000 Gaussians, 512×512 pixels. Median over 10 held-out objects. Of the 275 ms of one view, 151 ms are the scene encoder and 124 ms the view decoder. The model has 195M parameters, 113M in the scene encoder and 82M in the view decoder.
Held-out objects
Training objects
Model
PSNR
Difference
LPIPS
PSNR
All changes
34.42
0.0224
35.69
No footprint
32.71
−1.71±0.07
0.0316
34.43
No 2D rotary embedding
29.22
−5.20±0.18
0.0467
30.50
8×8 patches
31.37
−3.05±0.10
0.0379
32.49
Windowed cross-attention
34.62
+0.20±0.02
0.0218
35.85
Table 4: Ablations on the 1,000-object harness. PSNR in dB against the unpruned render, on the 300 held-out objects and on 300 of the training objects (their training views). Difference: held-out PSNR minus that of the arm with all changes, with a 95% bootstrap interval over objects for this single run. The last rows vary how the footprint is given and repeat the arm with all changes with another seed, which differs from the first run by 0.10 dB per object on average.
Figure 4: PSNR against the unpruned render when the end of stage 2, which has seen all 26,820 objects, is fine-tuned on fewer objects, with the same number of steps for every point. With one object the model fits its training images and forgets the others. The two curves meet at the full set, far below rasterizing the same input. The triangles show the final model, trained about 10 times longer with the footprint.
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
Ours
gsplat on the input
PSNR per frame (dB)
38.52
44.12
Worst frame per object (dB)
35.70
Spread over frames, standard deviation (dB)
1.38
Frame-to-frame consistency, PSNR (dB)
42.98
47.74
Appendix
Table 5: Rendering along a 96-frame camera path, mean over 30 held-out objects.
Figure 5: Every 12th frame of a 96-frame camera path around a held-out object. Top: the reference. Bottom: GaussianRenderFormer, which renders each frame independently.
Scene
Splat
Photos
Region
Input
Ours
gsplat on the input
Bonsai
2.00M
31.04
346,027
20k / 80k / 160k
34.12 / 33.84 / 31.09
35.70 / 43.67 / 48.93
Counter
1.81M
28.11
732,862
20k / 80k / 160k
31.78 / 29.92 / 28.40
34.37 / 37.96 / 40.86
Garden
7.81M
26.84
994,061
20k / 80k / 160k
30.56 / 29.65 / 27.33
32.58 / 37.53 / 40.23
Kitchen
2.38M
30.63
1,284,974
20k / 80k / 160k
31.99 / 28.61 / 25.81
36.74 / 41.55 / 44.86
Appendix
Table 6: Real captures from Mip-NeRF 360. Splat: our gsplat reconstruction and its PSNR on held-out photos. Region: the cube cut out of it. PSNR in dB over the whole frame, mean over 7 views.
Figure 6: A region of the bonsai scene from the Mip-NeRF 360 dataset, a real capture, seen from two orbit cameras. Left: the full region (346,027 Gaussians) rasterized with gsplat, used as the reference. Middle: the region pruned to 20,000 Gaussians, GaussianRenderFormer’s training input size, rasterized with gsplat. Right: GaussianRenderFormer on the same pruned input. Over the full frame, GaussianRenderFormer scores 34.1 dB and gsplat 35.7 dB against the reference.
Figure 7: The four real captures with 20,000 Gaussians, each at its best and its worst view by GaussianRenderFormer’s PSNR on the evaluation crop, with errors scaled by 5 as in Figure 3 . Bonsai: views 4 and 0, GaussianRenderFormer 31.6 and 28.3 dB, gsplat on the input 32.8 and 29.8 dB. Counter: views 6 and 4, 30.7 and 29.1 dB against 33.2 and 31.9 dB. Garden: views 4 and 0, 29.4 and 28.4 dB against 31.4 and 30.3 dB. Kitchen: views 6 and 3, 31.9 and 29.8 dB against 36.9 and 33.7 dB.
Objects and views
Model
PSNR
Difference
LPIPS
Development held-out, 4 views
Before the footprint, 94k steps
34.73
0.022
without the 2D embedding
30.40
−4.3 (4.2 to 4.5)
0.039
with 8×8 patches
32.31
−2.4 (2.3 to 2.5)
0.033
Fresh held-out, 28 views
Final model
38.08
0.0113
without the footprint
36.67
−1.40 (1.33 to 1.48)
0.0140
Training objects, 28 views
Final model
38.25
0.0108
Appendix
Table 7: Ablations at full scale. PSNR in dB against the unpruned render, the difference from the model with all changes with a 95% bootstrap interval over objects, and LPIPS.
Change
Ours
gsplat on the input
Change
Ours
gsplat on the input
Training setup
38.74
45.24
5,000 Gaussians
31.66
36.20
256 pixels
33.39
41.03
10,000 Gaussians
35.59
40.10
1,024 pixels
32.72
41.18
40,000 Gaussians
35.68
49.91
Distance 0.9
37.17
40.28
Field of view 35∘
37.60
45.18
Distance 1.15
38.88
42.63
Field of view 40∘
38.25
45.20
Appendix
Table 8: One change at a time from the training setup (20,000 Gaussians, distance 1.7, 512×512 pixels, 45∘ field of view), on 30 held-out objects. PSNR in dB against a rasterization of the full splat. Training distances range from 1.15 to 2.45. For the field of view, the camera distance is scaled so the object fills the frame as at 45∘ . The inputs of other sizes are pruned again from the full splat with our pipeline, and pruning again to 20,000 Gaussians gives 38.76 and 45.28 dB, the reference for their differences in the text.
Gaussians
Ours (ms)
Ours memory (GB)
gsplat (ms)
gsplat memory (GB)
20,000
275
2.9
0.7
0.007
40,000
613
3.0
0.8
0.009
80,000
1,793
3.9
1.2
0.014
160,000
5,294
6.7
1.5
0.030
320,000
19,367
12.4
1.9
0.044
640,000
74,853
23.7
2.5
0.082
Appendix
Table 9: Time and peak memory for one 512×512 view as the input grows, on one NVIDIA L40S GPU. Inputs are a region of the Mip-NeRF 360 garden scene pruned to each size. The model’s memory includes its weights, gsplat’s excludes the input.
Initialization and schedule
PSNR (dB)
Difference
LPIPS
Validation loss
RenderFormer, input layer first (ours)
29.98
0.0783
0.00100
RenderFormer, whole model from the start
26.85
−3.13±0.10
0.1257
0.00148
RenderFormer, whole model, input layer at 10−3
26.30
−3.68±0.12
0.1383
0.00162
Random weights
13.12
0.5733
0.01357
Appendix
Table 10: The first training stages, 50,000 steps each at 256×256 pixels on all 26,820 objects. Held-out PSNR and LPIPS at 256 pixels against the unpruned render, the difference from our recipe with a 95% bootstrap interval over objects, and validation loss at the end of training.
Figure 8: Validation loss of the final run at 512×512 pixels (stage 4), on the 100 validation objects, every 2,000 steps, and of the same recipe trained without the footprint (dashed). The log-L1 term of the first 16,000 steps was not logged.
Steps before the decay
94,000
188,000
282,000
375,000
PSNR (dB)
36.20
37.12
37.65
38.08
Appendix
Table 11: PSNR of the final run on the fresh held-out objects over their 28 views, at four points, each after a 10,000-step decay.
Figure 9: The arms of Table 4 and the full-scale models on the first and third objects of Figure 3 : the first object (top), and 4× close-ups of the region of each of the two objects with the most detail (middle and bottom). The final model keeps the painted pattern of the first object, and the same recipe trained without the footprint renders it slightly softer. The models of the 1,000-object harness, trained for about a tenth as long, are softer, and lose much of the pattern without the footprint, without the 2D embedding and with 8×8 patches. The last two columns are the recipe before the footprint, trained on all the data for 94,000 steps, with and without the 2D embedding.
Figure 10: PSNR of each of the 300 fresh held-out objects (mean over its 28 views), in bins of 2 dB.
Share of error
In-phase gain
Unmatched power
Error, Ours
Band
Ours
gsplat
Ours
gsplat
Ours
gsplat
over gsplat
0 to 1/64
0.046
0.015
0.997
1.001
0.000
0.000
16.4
1/64 to 1/32
0.039
0.022
0.999
1.000
0.002
0.000
9.1
1/32 to 1/16
0.069
0.049
0.997
0.999
0.006
0.001
6.7
1/16 to 1/8
0.131
0.113
0.989
0.997
0.015
0.003
5.4
1/8 to 1/4
0.270
0.255
0.960
0.985
0.048
0.010
4.9
Appendix
Table 12: Error by frequency band, GaussianRenderFormer against gsplat on the same pruned input, both compared with the unpruned render. Frequencies in cycles per pixel, the last band includes the corners of the spectrum.
Gaussians per pixel
Share of pixels
Ours
gsplat
Difference
1
7.5%
53.30
57.36
4.06
2 to 3
5.6%
46.95
49.42
2.48
4 to 7
4.3%
35.18
43.25
8.07
8 to 15
3.1%
32.01
37.38
5.37
16 to 31
2.8%
30.25
33.84
3.59
32 to 63
5.0%
28.99
34.94
5.95
Appendix
Table 13: PSNR in dB against the unpruned render, by the number of input Gaussians reaching a pixel, over all object pixels of the 1,200 held-out views.
Figure 11: Failure cases. The four fresh held-out objects with the lowest mean PSNR of GaussianRenderFormer, each at its worst view. Columns as in Figure 3 , with errors scaled by 5. GaussianRenderFormer scores 25.4, 27.2, 29.1 and 28.5 dB on these views, and gsplat on the input 37.9, 40.0, 38.6 and 40.4 dB. The shapes are right on all four. The error comes from dense texture that GaussianRenderFormer renders less precisely than the reference (the clouds on the globe, the facade of the building, the houses and trees of the town) and from soft, low-contrast structure (the streaks on the ground under the sign).
Change
Setup
Effect
More depth: 18 + 9 layers (287M parameters)
100 objects, earlier model
Fit 13.43 vs 13.40
Wider feed-forward network ( 8× , 322M)
100 objects, earlier model
Fit 13.35 vs 13.40
Small MLP as input layer, log-scale input
100 objects, earlier model
Fit 13.43 vs 13.40
Training from scratch
Several widths, with and without a 256-pixel warm-up
Never left a poor plateau
Changing the pretrained position embedding (doubled frequency)
10 objects
Destroyed the pretrained behavior
Geometry-biased cross-attention
100 objects
Fit and held-out unchanged, the model learned to ignore it
Appendix
Table 14: Changes that did not help, with the setup they were tested in. “Fit” is the margin on the training objects and “held-out” the margin on the 300 held-out objects, in dB.
Figure 12: A model trained on 14 views of one object until it memorized them. On a training view (top) its render cannot be told apart from the reference. On new views between the training views (middle and bottom), it is much sharper than rasterizing its input, and restores much of the detail that pruning removed. This experiment used an earlier input, pruned without the fine-tune of Section 5 , which is why the rasterized input is blurry. PSNR over the whole frame, on this view (52.5 to 53.0 dB on average over the 14 training views). Tomatoes scan by simonbethke, CC BY 4.0.
Object
Source
Training
Difference (dB)
Views better
Slipper
real scan, 1.17M Gaussians
120,000 steps
+1.82
35 / 40
Molecule
Objaverse_Splats
150,000 steps
+1.47
29 / 40
Tomatoes
real scan, 219,000 Gaussians
several cycles
+1.30
34 / 40
Tent
Objaverse_Splats
150,000 steps
+0.71
28 / 40
Octopus
real scan, 1.84M Gaussians
90,000 steps
−1.21
11 / 40
8 other objects
Objaverse_Splats
60,000 steps
−2.1 to −9.1
0 to 7 / 40
Appendix
Table 15: Per-object training: PSNR of the model minus gsplat on the same 20,000-Gaussian input, mean over 40 views not used in training (higher is better), and the number of views where the model is better.
Figure 13: A model trained on the slipper scan alone, on three views not used in training: close-up (top), at a random distance (middle) and distant (bottom). The reference is the full scan of 1.17M Gaussians, and both gsplat and the model get it pruned to 20,000. Close up, the pruned input shows long streaks where detail was removed, which the model fills in with the fleece texture. Slipper scan by Andre, CC BY 4.0.
Model
Token width, attention heads, feed-forward width
768, 6, 3,072
Scene encoder
12 layers, 16 register tokens, 3D RoPE on Gaussian centers
View decoder
6 layers, 4×4 -pixel patches, DPT head with 128 features
Ray embedding
pretrained 8×8 embedding, rays upsampled from 4×4
2D RoPE of projected positions
16 channel pairs (32 channels) per head, patch coordinates scaled by 0.25
Projected footprint
4 numbers per Gaussian on the cross-attention keys, a zero-initialized linear layer per decoder layer (23,040 parameters)
Feed-forward Gaussian reconstruction builds 3D scenes directly from input images for novel view rendering, but learning Gaussian attributes from predicted geometry for high-quality rendering under large viewpoint changes remains challenging. We analyze limitations in Gaussian attribute learning and introduce InfiniSplat v2, which improves rendering quality through architectural and training changes. Our method decouples geometry and attribute prediction with a frozen geometry branch, aligns predicted Gaussians using ground-truth camera poses during training, and adopts large-angle target view supervision. Across six datasets, the feed-forward outputs of the same trained model achieve state-of-the-art mean LPIPS and DISTS among the evaluated methods in both pose-conditioned and pose-free settings. In the pose-conditioned setting, mean LPIPS and DISTS decrease by 25.8% and 32.9%, respectively, compared with the strongest evaluated baselines. We further find that opacity learned with only image reconstruction supervision suppresses geometrically inconsistent points. Project page: https://zju3dv.github.io/InfiniSplat2/
Yongzhen Hu, Hao Yu, Jiawei Wang +7
Zhejiang University · Hello TransTech · Shenzhen University +1
In this work, we revisit several key design choices of modern Transformer-based approaches for feed-forward 3D Gaussian Splatting (3DGS) prediction. We argue that the common practice of regressing Gaussian means as depths along camera rays is suboptimal, and instead propose to directly regress 3D mean coordinates using only a self-supervised rendering loss. This formulation allows us to move from the standard encoder-only design to an encoder-decoder architecture with learnable Gaussian tokens, thereby unbinding the number of predicted primitives from input image resolution and number of views. Our resulting method, TokenGS, demonstrates improved robustness to pose noise and multiview inconsistencies, while naturally supporting efficient test-time optimization in token space without degrading learned priors. TokenGS achieves state-of-the-art feed-forward reconstruction performance on both static and dynamic scenes, producing more regularized geometry and more balanced 3DGS distribution, while seamlessly recovering emergent scene attributes such as static-dynamic decomposition and scene flow.
Jiawei Ren, Michal Jan Tyszkiewicz, Jiahui Huang +1
Feed-forward 3D Gaussian Splatting reconstructs scenes from images in seconds. Pixel-aligned methods keep every Gaussian grounded in an observation, but produce redundant, potentially misaligned Gaussians across views. Token-based methods instead aggregate all views into a fixed set of learned tokens but lack explicit observation support and use a fixed reconstruction budget. We introduce ZipSplat, a method that predicts a compact, observation-grounded scene representation whose size is set at inference. It derives each scene token from an image patch rather than a learned query, and decodes it into freely placed 3D Gaussians. Tokens across views are merged into a compact set whose size is adjustable at inference. However, without geometric supervision, decoded Gaussians can drift from their source patch, and merging can mix tokens from different surfaces. We therefore introduce two geometric losses to ground and shape this representation. A cone loss ties each token's Gaussians to its source patch, and GeoCE aligns token similarity with 3D proximity so that tokens observing the same surface merge together. Building on these observation--geometry associations, we further introduce TokenBA, a token-level bundle adjustment that refines scene tokens and camera poses in a few seconds. ZipSplat requires neither camera poses nor intrinsics, yet outperforms pixel-aligned and token-based baselines on DL3DV and RealEstate10K, even those given ground-truth cameras. It further generalizes zero-shot to Mip-NeRF360 and ScanNet++. Our project page is at https://veichta.com/zipsplat.