GaussianRenderFormer: Transformer-based Rendering of 3D Gaussian Splats

Shahaf Valencia Levy; Sagie Benaim

cs.CVNov 3, 2026

GaussianRenderFormer: Transformer-based Rendering of 3D Gaussian Splats

Authors: Shahaf Valencia Levy, Sagie Benaim

Organizations: The Hebrew University of Jerusalem

Abstract

Can a neural network learn to render 3D Gaussian splats on its own, with no rasterizer in the renderer? We present GaussianRenderFormer, a transformer that takes a set of Gaussians and a camera and outputs the image directly. Each Gaussian becomes one token in a view-independent encoder, and a view-dependent decoder turns small patches of camera rays into pixels by attending to those tokens. Trained on 26,820 Objaverse objects pruned to 20,000 Gaussians each, GaussianRenderFormer renders 300 unseen objects at 38.08 dB PSNR against the full, unpruned splats, while rasterizing the same pruned input reaches 44.02 dB. The gap is not one of generalization, and our diagnostics are consistent with a bottleneck in the view decoder: telling it the projected footprint of each Gaussian, which it could in principle infer, adds 1.4 dB. Under-training is not ruled out. Code, trained models and videos: https://svlwoof.github.io/gaussianrenderformer

Figures & tables

Panel of Figure 1 — Figure 1: GaussianRenderFormer renders 3D Gaussian splats with a neural network, without a rasterizer. Top: the reference, a rasterization of each object’s full splat. Bottom (ours): GaussianRenderFormer from a 20,000-Gaussian input, on six objects never seen in training, chosen by hand for variety. Right: a $3\times$ zoom of the most detailed region of the last object, where the model softens the stone’s texture.

Panel of Figure 2 — Figure 2: Overview of GaussianRenderFormer. Each Gaussian’s 14 parameters are embedded as one token, and a view-independent scene encoder (top) lets these tokens exchange information, using a 3D rotary embedding of their centers. For each view, the image is split into $4\times 4$ -pixel patches, and each patch becomes a token built from its camera rays. The view decoder (bottom) lets these tokens attend to the scene tokens, using a 2D rotary embedding of where each Gaussian lands in the image and an embedding of its projected footprint, and a DPT head turns them into pixels.

Stage	Trains	Steps	Resolution	Loss
1	input layer only	33.5k	256	log-L1
2	whole model	402k	256	log-L1
3	whole model, new decoder changes	3k	256	log-L1, background weight
4	whole model	385k	512	log-L1 + LPIPS, background weight

Table 1: Training stages. Stage 3 switches on the changes of Section 4.2 .

Panel of Figure 3 — Figure 3: Three fresh held-out objects, chosen at the 10th, 50th and 90th percentile of GaussianRenderFormer’s per-object PSNR. From left to right: the reference (the full, unpruned splat rasterized with gsplat), gsplat on the pruned 20,000-Gaussian input, GaussianRenderFormer on the same input, and the absolute error of each against the reference, scaled by 5. GaussianRenderFormer scores 34.5, 39.5 and 41.7 dB on these views, and gsplat on the input 39.7, 45.5 and 45.7 dB. GaussianRenderFormer gets shapes and colors right, and its error concentrates on edges and fine texture.

	PSNR (dB)		Masked
Renderer and objects	4 views	28 views	PSNR (dB)	SSIM	LPIPS
gsplat on the input, fresh held-out	44.97	44.02	39.19	0.9954	0.0046
gsplat on the input, training objects	45.01	44.12	–	0.9954	0.0046
Ours, fresh held-out	38.36	38.08	32.68	0.9814	0.0113
Ours, development held-out	38.36	38.02	32.66	0.9802	0.0116
Ours, training objects	38.55	38.25	–	0.9821	0.0108

Table 2: Main result, against the unpruned render. PSNR over the 4 views at distance 1.7 used in the rest of the paper and over all 28 views at the three training distances, masked PSNR over the object’s pixels on the whole image, and SSIM and LPIPS over the 28 views. Fresh held-out: 300 objects never used during development. Development held-out: the 300 held-out objects of the other sections. No full splat is kept for the training objects, so they have no masked PSNR. Without the footprint: the same recipe and schedule without the projected footprint (Section 4.2 ). Higher PSNR and SSIM and lower LPIPS are better.

Views per call	Ours (ms)	per view (ms)	peak memory (GB)	gsplat (ms)
1	275	275	2.8	0.7
4	685	171	7.7	0.9
14	2,012	144	13.0	1.2

Table 3: Rendering time and memory on one NVIDIA L40S GPU, 20,000 Gaussians,

512\times 512

pixels. Median over 10 held-out objects. Of the 275 ms of one view, 151 ms are the scene encoder and 124 ms the view decoder. The model has 195M parameters, 113M in the scene encoder and 82M in the view decoder.

	Held-out objects			Training objects
Model	PSNR	Difference	LPIPS	PSNR
All changes	34.42		0.0224	35.69
No footprint	32.71	$-1.71\pm 0.07$	0.0316	34.43
No 2D rotary embedding	29.22	$-5.20\pm 0.18$	0.0467	30.50
$8\times 8$ patches	31.37	$-3.05\pm 0.10$	0.0379	32.49
Windowed cross-attention	34.62	$+0.20\pm 0.02$	0.0218	35.85

Table 4: Ablations on the 1,000-object harness. PSNR in dB against the unpruned render, on the 300 held-out objects and on 300 of the training objects (their training views). Difference: held-out PSNR minus that of the arm with all changes, with a 95% bootstrap interval over objects for this single run. The last rows vary how the footprint is given and repeat the arm with all changes with another seed, which differs from the first run by 0.10 dB per object on average.

Panel of Figure 4 — Figure 4: PSNR against the unpruned render when the end of stage 2, which has seen all 26,820 objects, is fine-tuned on fewer objects, with the same number of steps for every point. With one object the model fits its training images and forgets the others. The two curves meet at the full set, far below rasterizing the same input. The triangles show the final model, trained about 10 times longer with the footprint.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

	Ours	gsplat on the input
PSNR per frame (dB)	38.52	44.12
Worst frame per object (dB)	35.70
Spread over frames, standard deviation (dB)	1.38
Frame-to-frame consistency, PSNR (dB)	42.98	47.74

Appendix

Table 5: Rendering along a 96-frame camera path, mean over 30 held-out objects.

Panel of Figure 5 — Figure 5: Every 12th frame of a 96-frame camera path around a held-out object. Top: the reference. Bottom: GaussianRenderFormer, which renders each frame independently.

Scene	Splat	Photos	Region	Input	Ours	gsplat on the input
Bonsai	2.00M	31.04	346,027	20k / 80k / 160k	34.12 / 33.84 / 31.09	35.70 / 43.67 / 48.93
Counter	1.81M	28.11	732,862	20k / 80k / 160k	31.78 / 29.92 / 28.40	34.37 / 37.96 / 40.86
Garden	7.81M	26.84	994,061	20k / 80k / 160k	30.56 / 29.65 / 27.33	32.58 / 37.53 / 40.23
Kitchen	2.38M	30.63	1,284,974	20k / 80k / 160k	31.99 / 28.61 / 25.81	36.74 / 41.55 / 44.86

Appendix

Table 6: Real captures from Mip-NeRF 360. Splat: our gsplat reconstruction and its PSNR on held-out photos. Region: the cube cut out of it. PSNR in dB over the whole frame, mean over 7 views.

Panel of Figure 6 — Figure 6: A region of the bonsai scene from the Mip-NeRF 360 dataset, a real capture, seen from two orbit cameras. Left: the full region (346,027 Gaussians) rasterized with gsplat, used as the reference. Middle: the region pruned to 20,000 Gaussians, GaussianRenderFormer’s training input size, rasterized with gsplat. Right: GaussianRenderFormer on the same pruned input. Over the full frame, GaussianRenderFormer scores 34.1 dB and gsplat 35.7 dB against the reference.

Panel of Figure 7 — Figure 7: The four real captures with 20,000 Gaussians, each at its best and its worst view by GaussianRenderFormer’s PSNR on the evaluation crop, with errors scaled by 5 as in Figure 3 . Bonsai: views 4 and 0, GaussianRenderFormer 31.6 and 28.3 dB, gsplat on the input 32.8 and 29.8 dB. Counter: views 6 and 4, 30.7 and 29.1 dB against 33.2 and 31.9 dB. Garden: views 4 and 0, 29.4 and 28.4 dB against 31.4 and 30.3 dB. Kitchen: views 6 and 3, 31.9 and 29.8 dB against 36.9 and 33.7 dB.

Objects and views	Model	PSNR	Difference	LPIPS
Development held-out, 4 views	Before the footprint, 94k steps	34.73		0.022
	without the 2D embedding	30.40	$-4.3$ (4.2 to 4.5)	0.039
	with $8\times 8$ patches	32.31	$-2.4$ (2.3 to 2.5)	0.033
Fresh held-out, 28 views	Final model	38.08		0.0113
	without the footprint	36.67	$-1.40$ (1.33 to 1.48)	0.0140
Training objects, 28 views	Final model	38.25		0.0108

Appendix

Table 7: Ablations at full scale. PSNR in dB against the unpruned render, the difference from the model with all changes with a 95% bootstrap interval over objects, and LPIPS.

Change	Ours	gsplat on the input	Change	Ours	gsplat on the input
Training setup	38.74	45.24
5,000 Gaussians	31.66	36.20	256 pixels	33.39	41.03
10,000 Gaussians	35.59	40.10	1,024 pixels	32.72	41.18
40,000 Gaussians	35.68	49.91
Distance 0.9	37.17	40.28	Field of view $35^{\circ}$	37.60	45.18
Distance 1.15	38.88	42.63	Field of view $40^{\circ}$	38.25	45.20

Appendix

Table 8: One change at a time from the training setup (20,000 Gaussians, distance 1.7,

512\times 512

pixels,

45^{\circ}

field of view), on 30 held-out objects. PSNR in dB against a rasterization of the full splat. Training distances range from 1.15 to 2.45. For the field of view, the camera distance is scaled so the object fills the frame as at

45^{\circ}

. The inputs of other sizes are pruned again from the full splat with our pipeline, and pruning again to 20,000 Gaussians gives 38.76 and 45.28 dB, the reference for their differences in the text.

Gaussians	Ours (ms)	Ours memory (GB)	gsplat (ms)	gsplat memory (GB)
20,000	275	2.9	0.7	0.007
40,000	613	3.0	0.8	0.009
80,000	1,793	3.9	1.2	0.014
160,000	5,294	6.7	1.5	0.030
320,000	19,367	12.4	1.9	0.044
640,000	74,853	23.7	2.5	0.082

Appendix

Table 9: Time and peak memory for one

512\times 512

view as the input grows, on one NVIDIA L40S GPU. Inputs are a region of the Mip-NeRF 360 garden scene pruned to each size. The model’s memory includes its weights, gsplat’s excludes the input.

Initialization and schedule	PSNR (dB)	Difference	LPIPS	Validation loss
RenderFormer, input layer first (ours)	29.98		0.0783	0.00100
RenderFormer, whole model from the start	26.85	$-3.13\pm 0.10$	0.1257	0.00148
RenderFormer, whole model, input layer at $10^{-3}$	26.30	$-3.68\pm 0.12$	0.1383	0.00162
Random weights	13.12		0.5733	0.01357

Appendix

Table 10: The first training stages, 50,000 steps each at

256\times 256

pixels on all 26,820 objects. Held-out PSNR and LPIPS at 256 pixels against the unpruned render, the difference from our recipe with a 95% bootstrap interval over objects, and validation loss at the end of training.

Panel of Figure 8 — Figure 8: Validation loss of the final run at $512\times 512$ pixels (stage 4), on the 100 validation objects, every 2,000 steps, and of the same recipe trained without the footprint (dashed). The log-L1 term of the first 16,000 steps was not logged.

Steps before the decay	94,000	188,000	282,000	375,000
PSNR (dB)	36.20	37.12	37.65	38.08

Appendix

Table 11: PSNR of the final run on the fresh held-out objects over their 28 views, at four points, each after a 10,000-step decay.

Panel of Figure 9 — Figure 9: The arms of Table 4 and the full-scale models on the first and third objects of Figure 3 : the first object (top), and $4\times$ close-ups of the region of each of the two objects with the most detail (middle and bottom). The final model keeps the painted pattern of the first object, and the same recipe trained without the footprint renders it slightly softer. The models of the 1,000-object harness, trained for about a tenth as long, are softer, and lose much of the pattern without the footprint, without the 2D embedding and with $8\times 8$ patches. The last two columns are the recipe before the footprint, trained on all the data for 94,000 steps, with and without the 2D embedding.

Panel of Figure 10 — Figure 10: PSNR of each of the 300 fresh held-out objects (mean over its 28 views), in bins of 2 dB.

	Share of error		In-phase gain		Unmatched power		Error, Ours
Band	Ours	gsplat	Ours	gsplat	Ours	gsplat	over gsplat
0 to 1/64	0.046	0.015	0.997	1.001	0.000	0.000	16.4
1/64 to 1/32	0.039	0.022	0.999	1.000	0.002	0.000	9.1
1/32 to 1/16	0.069	0.049	0.997	0.999	0.006	0.001	6.7
1/16 to 1/8	0.131	0.113	0.989	0.997	0.015	0.003	5.4
1/8 to 1/4	0.270	0.255	0.960	0.985	0.048	0.010	4.9

Appendix

Table 12: Error by frequency band, GaussianRenderFormer against gsplat on the same pruned input, both compared with the unpruned render. Frequencies in cycles per pixel, the last band includes the corners of the spectrum.

Gaussians per pixel	Share of pixels	Ours	gsplat	Difference
1	7.5%	53.30	57.36	4.06
2 to 3	5.6%	46.95	49.42	2.48
4 to 7	4.3%	35.18	43.25	8.07
8 to 15	3.1%	32.01	37.38	5.37
16 to 31	2.8%	30.25	33.84	3.59
32 to 63	5.0%	28.99	34.94	5.95

Appendix

Table 13: PSNR in dB against the unpruned render, by the number of input Gaussians reaching a pixel, over all object pixels of the 1,200 held-out views.

Panel of Figure 11 — Figure 11: Failure cases. The four fresh held-out objects with the lowest mean PSNR of GaussianRenderFormer, each at its worst view. Columns as in Figure 3 , with errors scaled by 5. GaussianRenderFormer scores 25.4, 27.2, 29.1 and 28.5 dB on these views, and gsplat on the input 37.9, 40.0, 38.6 and 40.4 dB. The shapes are right on all four. The error comes from dense texture that GaussianRenderFormer renders less precisely than the reference (the clouds on the globe, the facade of the building, the houses and trees of the town) and from soft, low-contrast structure (the streaks on the ground under the sign).

Change	Setup	Effect
More depth: 18 + 9 layers (287M parameters)	100 objects, earlier model	Fit 13.43 vs 13.40
Wider feed-forward network ( $8\times$ , 322M)	100 objects, earlier model	Fit 13.35 vs 13.40
Small MLP as input layer, log-scale input	100 objects, earlier model	Fit 13.43 vs 13.40
Training from scratch	Several widths, with and without a 256-pixel warm-up	Never left a poor plateau
Changing the pretrained position embedding (doubled frequency)	10 objects	Destroyed the pretrained behavior
Geometry-biased cross-attention	100 objects	Fit and held-out unchanged, the model learned to ignore it

Appendix

Table 14: Changes that did not help, with the setup they were tested in. “Fit” is the margin on the training objects and “held-out” the margin on the 300 held-out objects, in dB.

Panel of Figure 12 — Figure 12: A model trained on 14 views of one object until it memorized them. On a training view (top) its render cannot be told apart from the reference. On new views between the training views (middle and bottom), it is much sharper than rasterizing its input, and restores much of the detail that pruning removed. This experiment used an earlier input, pruned without the fine-tune of Section 5 , which is why the rasterized input is blurry. PSNR over the whole frame, on this view (52.5 to 53.0 dB on average over the 14 training views). Tomatoes scan by simonbethke, CC BY 4.0.

Object	Source	Training	Difference (dB)	Views better
Slipper	real scan, 1.17M Gaussians	120,000 steps	$+1.82$	35 / 40
Molecule	Objaverse_Splats	150,000 steps	$+1.47$	29 / 40
Tomatoes	real scan, 219,000 Gaussians	several cycles	$+1.30$	34 / 40
Tent	Objaverse_Splats	150,000 steps	$+0.71$	28 / 40
Octopus	real scan, 1.84M Gaussians	90,000 steps	$-1.21$	11 / 40
8 other objects	Objaverse_Splats	60,000 steps	$-2.1$ to $-9.1$	0 to 7 / 40

Appendix

Table 15: Per-object training: PSNR of the model minus gsplat on the same 20,000-Gaussian input, mean over 40 views not used in training (higher is better), and the number of views where the model is better.

Panel of Figure 13 — Figure 13: A model trained on the slipper scan alone, on three views not used in training: close-up (top), at a random distance (middle) and distant (bottom). The reference is the full scan of 1.17M Gaussians, and both gsplat and the model get it pruned to 20,000. Close up, the pruned input shows long streaks where detail was removed, which the model fills in with the fleece texture. Slipper scan by Andre, CC BY 4.0.

Model
Token width, attention heads, feed-forward width	768, 6, 3,072
Scene encoder	12 layers, 16 register tokens, 3D RoPE on Gaussian centers
View decoder	6 layers, $4\times 4$ -pixel patches, DPT head with 128 features
Ray embedding	pretrained $8\times 8$ embedding, rays upsampled from $4\times 4$
2D RoPE of projected positions	16 channel pairs (32 channels) per head, patch coordinates scaled by 0.25
Projected footprint	4 numbers per Gaussian on the cross-attention keys, a zero-initialized linear layer per decoder layer (23,040 parameters)

Appendix

Table 16: Settings of the final model.

Neural Rendering

Neural rendering uses learned 3D scene and light representations to synthesize, edit, and relight images from new viewpoints.

See all 123 papers

InfiniSplat v2: Analyzing and Improving Feed-Forward 3DGS from Any Views
Oct 14, 2026Yongzhen Hu, Hao Yu, Jiawei Wang +73D Gaussian Splatting Feed-Forward 3D Reconstruction
Zhejiang University · Hello TransTech · Shenzhen University +1
TokenGS: Decoupling 3D Gaussian Prediction from Pixels with Learnable Tokens
Apr 16, 2026Jiawei Ren, Michal Jan Tyszkiewicz, Jiahui Huang +13D Gaussian Splatting 3D ViTs
NVIDIA
ZipSplat: Fewer Gaussians, Better Splats
Jun 3, 2026Alexander Veicht, Sunghwan Hong, Dániel Baráth +13D Gaussian Splatting Multi-View 3D Reconstruction
ETH Zürich · Microsoft

Oct 14, 2026J/K move · Enter open · S save

InfiniSplat v2: Analyzing and Improving Feed-Forward 3DGS from Any Views

Yongzhen Hu, Hao Yu, Jiawei Wang, Xianming Zeng+6

Zhejiang University · Hello TransTech · Shenzhen University+1

3D Gaussian Splatting Feed-Forward 3D Reconstruction Novel View Synthesis 3DGS Training

Open paper AskPDF arXiv

Feed-forward Gaussian reconstruction builds 3D scenes directly from input images for novel view rendering, but learning Gaussian attributes from predicted geometry for high-quality rendering under large viewpoint changes remains challenging. We analyze limitations in Gaussian attribute learning and introduce InfiniSplat v2, which improves rendering quality through architectural and training changes. Our method decouples geometry and attribute prediction with a frozen geometry branch, aligns predicted Gaussians using ground-truth camera poses during training, and adopts large-angle target view supervision. Across six datasets, the feed-forward outputs of the same trained model achieve state-of-the-art mean LPIPS and DISTS among the evaluated methods in both pose-conditioned and pose-free settings. In the pose-conditioned setting, mean LPIPS and DISTS decrease by 25.8% and 32.9%, respectively, compared with the strongest evaluated baselines. We further find that opacity learned with only image reconstruction supervision suppresses geometrically inconsistent points. Project page: https://zju3dv.github.io/InfiniSplat2/

Abstract

Neural Rendering

Explore similar work

InfiniSplat v2: Analyzing and Improving Feed-Forward 3DGS from Any Views

TokenGS: Decoupling 3D Gaussian Prediction from Pixels with Learnable Tokens

ZipSplat: Fewer Gaussians, Better Splats