Scaling View Synthesis Transformers

Evan Kim; Hyunwoo Ryu; Thomas W. Mitchel; Vincent Sitzmann

Figure 1: Scaling Laws for View Synthesis Transformers. Evaluated on RealEstate10K [ 34 ] , our SVSM exhibits a

3\times

more compute-optimal Pareto frontier than LVSM while retaining the same scaling behavior (similar slope and curvature everywhere).

Figure 2: Architectures of the current SOTA, the decoder-only LVSM [ 10 ] (a) and SVSM (ours, b). Our cross-attention based decoder enables parallel rendering of multiple target views after a single scene encoding. Each target view is decoded independently given the shared scene representation, but the cross-attention allows these independent decodings to be executed in parallel.

Figure 3: Effective Batch Size . Training loss (smoothed with a rolling-average) and test PSNR measured throughout training across various paired

B

and

V_{T}

runs provide evidence for our effective batch size hypothesis: Models trained with the same product of number of scenes in the batch

B

and number of reconstruction target views

V_{T}

, i.e. runs with the same effective batch size

B_{\text{eff}}

, perform the same and are colored identically. On

V_{C}=8

(top) , we sweep across

B_{\text{eff}}=

128 , 256 on DL3DV, and on

V_{C}=2

(bottom) , we sweep across

B_{\text{eff}}=

128 , 1024 on RealEstate10K.

	Scale Parameters			Reconstruction Quality			Rendering FPS (↑)
Model	Model Size	Train Iters	Train FLOPs (↓)	PSNR (↑)	SSIM (↑)	LPIPS (↓)	$V_{C}$ =2	$V_{C}$ =4	$V_{C}$ =8
LVSM Encoder-Decoder [ 10 ]	173M	100k	2.53 zflops	28.58	0.893	0.114	53.7	52.9	52.7
LVSM Decoder-Only [ 10 ]	171M	100k	1.60 zflops	29.67	0.906	0.098	37.9	19.5	8.6
SVSM Enc-Dec ( ours , Iter-matched)	740M	100k	0.74 zflops	29.80	0.907	0.098	48.6	42.7	35.0
SVSM Enc-Dec ( ours , Pareto-optimal)	416M	170k	0.77 zflops	30.01	0.910	0.096	71.0	61.8	49.7

Table 1: Stereo (

V_{C}=2

) NVS Results of the Largest Models. All models use a patch size of

8

with input images at

256\times 256

resolution. Our models achieve the highest reconstruction metrics while using less than half of the training compute. The rendering FPS of both SVSM models is also much faster than that of the LVSM decoder-only model, though both are slower than the LVSM encoder-decoder when

V_{C}

is large.

Figure 4: Data and Model Scaling Plots. While our model ( blue ) is optimal when sufficient data is available, decoder-only LVSM ( red ) performs better with less data. The Pareto frontier analysis shows that our model is more data-hungry. Our model is also less parameter-efficient, although the gap closes as we increase the training compute. However, with sufficient data and compute, our model ( blue ) is overall superior in terms of training compute-optimality and rendering speed.

Figure 5: Qualitative Scaling Behavior ,

V_{C}=2

. From left to right, both models steadily increase in rendering quality until reaching near photo-realistic results. Compared vertically, for a given compute-budget, SVSM renderings consistently contain less artifacts.

	Recon Quality
Model	PSNR (↑)	SSIM (↑)	LPIPS (↓)
pixelNeRF [ 29 ]	20.43	0.589	0.550
pixelSplat [ 2 ]	26.09	0.863	0.136
MVSplat [ 3 ]	26.39	0.869	0.128
GS-LRM [ 31 ]	28.10	0.892	0.114
SVSM Enc-Dec (ours)	30.01	0.910	0.096

Table 2: Comparison to Geometry-Aware Methods. Our method achieves a new state-of-the-art on RealEstate10K [ 34 ] with the set from Charatan et al. [2] , outperforming not only LVSM, but also prior work with explicit 3D structure.

Model	Parameter Coeff. $a$	Data Coeff. $b$
LVSM	0.65	0.33
SVSM	0.52	0.47

Table 3: Parameter and Data Scaling Coefficients. As regressed from the plots in Fig. 4 , we find power law coefficients for scaling models and data with respect to compute.

Figure 6: Multiview PRoPE. We find that multiview projective RoPE embeddings [ 18 , 13 , 14 ] are critical for our model to scale with compute and data in the multiview setting (

V_{C}>2

). For each layer of the multiview transformer encoder, PRoPE embeddings use context camera poses to transform context view tokens into a common coordinate frames before the attention layer, and apply the inverse transformation before each MLP. To render, both context features and query tokens of the target view are transformed by PRoPE before cross-attention.

Figure 7: Multiview Scaling Behavior. Conducted on DL3DV [ 15 ] . (a) For

V_{C}{>}2

, without PRoPE, SVSM saturates and stops scaling much more quickly than LVSM. (b) When PRoPE is added, SVSM continues scaling with a better Pareto-frontier.

	Scale Parameters			Reconstruction Quality			Rendering FPS (↑)
Model	Model Size	Train Iters	Train FLOPs (↓)	PSNR (↑)	SSIM (↑)	LPIPS (↓)	$V_{C}$ =4	$V_{C}$ =8	$V_{C}$ =16
LVSM Decoder-only + PRoPE [ 10 , 14 ]	171M	100k	43 eflops	26.19	0.830	0.145	104.7	52.6	23.8
SVSM Enc-Dec ( ours , $\approx$ Iter-matched)	711M	100k	32 eflops	26.29	0.835	0.141	280.4	261.2	230.4
SVSM Enc-Dec ( ours , Pareto-optimal)	400M	233k	44 eflops	26.87	0.853	0.129	411.1	381.1	333

Table 4: Multiview (

V_{C}{>}2

) NVS Results of the Largest Models. Our compute-matched model achieves significantly better rendering quality (

+0.68\text{ PSNR}

,

-0.016\text{ LPIPS}

), while maintaining nearly four times the rendering speed at inference-time.

Figure 8: Qualitative Scaling Behavior ,

V_{C}=4

. The performance of both LVSM and our method steadily increases with compute (left to right). Compared vertically, for a given compute budget SVSM renderings are consistently less blurry.

Figure 9: Fixed-size Latent Scaling Experiments. Conducted on Objaverse [ 4 ] . (a) For

V_{C}{=}8

, SVSM and LVSM decoder-only scale equally while SVSM’s frontier is shifted by

5\times

on the compute axis. (b) When a fixed latent bottleneck is used, SVSM-fixed and LVSM encoder-decoder scale equally, but significantly worse than the unbottlenecked designs. SVSM again maintains a superior pareto frontier.

	Encoder			Decoder
Params	dim	dim_head	n_layers	dim	dim_head	n_layers
15M	384	64	3	384	64	3
27M	384	64	6	384	64	6
35M	384	64	8	384	64	8
62M	512	64	8	384	64	8
79M	512	64	8	512	64	12

Table 5: RealEstate10K

\boldsymbol{V_{C}}\mathbf{{=}2}

, SVSM Encoder-Decoder. SVSM Encoder-Decoder model settings used to sweep scaling laws for the stereo (

V_{C}{=}2

) novel view synthesis setting. Bolded is the row for the compute-matched, Pareto-optimal SVSM model compared against the 24-layer LVSM Decoder-only model.

Params	dim	dim_head	n_layers	dim	dim_head	n_layers
	Encoder			Decoder
8M	-	-	-	384	64	3
13M	-	-	-	384	64	6
22M	-	-	-	512	64	6
28M	-	-	-	512	64	8
53M	-	-	-	640	64	10

Table 6: RealEstate10K

\boldsymbol{V_{C}}\mathbf{{=}2}

, LVSM Decoder-only. LVSM Decoder-only model settings used to sweep scaling laws for the stereo (

V_{C}{=}2

) novel view synthesis setting.

	Encoder			Decoder
Params	dim	dim_head	n_layers	dim	dim_head	n_layers
15M	384	64	3	384	64	3
32M	384	64	6	384	64	8
85M	512	64	10	512	64	12
168M	640	64	12	640	64	16
280M	768	64	12	768	64	20

Table 7: DL3DV

\boldsymbol{V_{C}}\mathbf{{=}4}

, SVSM Encoder-Decoder. SVSM Encoder-Decoder model settings used to sweep scaling laws for the multi-view (

V_{C}>2

) novel view synthesis setting. Bolded is the row for the compute-matched, Pareto-optimal SVSM model compared against the 24-layer LVSM Decoder-only model.

Params	dim	dim_head	n_layers	dim	dim_head	n_layers
	Encoder			Decoder
8M	-	-	-	384	64	3
22M	-	-	-	512	64	6
43M	-	-	-	640	64	10
90M	-	-	-	768	64	12
175M	-	-	-	768	64	24

Table 8: DL3DV

\boldsymbol{V_{C}}\mathbf{{=}4}

, LVSM Decoder-only. LVSM Decoder-only model settings used to sweep scaling laws for the multi-view (

V_{C}>2

) novel view synthesis setting.

Figure 10: Linear Power Scaling Laws. We fit scaling laws onto sections of the Pareto-frontiers of the model families. We see that both models have approximately the same slope in each of their corresponding sections, indicating equal scaling.

Model	$c$ for $P>0.14$	$c$ for $P\leq 0.14$
LVSM	-0.23	-0.12
SVSM	-0.22	-0.12

Table 9: LPIPS vs. compute scaling coefficients. As regressed from Fig. 10 , we find power law coefficients for LPIPS

P(\chi{})\propto\chi{}^{c}

, and we report

c

in this table for the first half, which is determined by test LPIPS loss greater than

0.14

and the second half which is test LPIPS loss less than

0.14

.

Model	PSNR (↑)	SSIM (↑)	LPIPS (↓)
Vanilla SVSM	23.50	0.727	0.254
Vanilla w/ concat. pose	23.47	0.726	0.254
PRoPE on Encoder	23.61	0.733	0.249
PRoPE on Decoder	24.31	0.771	0.220
PRoPE on both	24.62	0.782	0.210
GTA on both	24.63	0.782	0.207

Table 10: PRoPE ablations. We vary where we apply PRoPE, try a different relative attention method (GTA), and also test pose information flow. GTA varies negligibly from PRoPE, indicating that epipolar geometry is not crucial, and the skip pose connection also has negligible impact, indicating that pose information flow is not responsible.

Figure 11: All Scaling Laws. We collect scaling laws across the three datasets we tested on, and in all cases, SVSM is compute-optimal.

Figure 12: Qualitative results on RE10K (

V_{C}{=}2

) across scale.

Figure 13: Qualitative results on DL3DV (

V_{C}{=}4

) across scale.

Figure 14: Multiview consistency results on Objaverse (

V_{C}{=}8

).

Scaling View Synthesis Transformers

Abstract

Figures & tables

Novel View Synthesis

DVSM: Decoder-only View Synthesis Model Done Right

AIMS: Anchor-Integrated Multi-View Synthesis for Scalable Novel View Rendering

Resolving Representation Ambiguity in Feedforward Novel View Synthesis Transformer via Semantic-Spatial Decoupling

Abstract

Novel View Synthesis

Explore similar work

DVSM: Decoder-only View Synthesis Model Done Right

AIMS: Anchor-Integrated Multi-View Synthesis for Scalable Novel View Rendering

Resolving Representation Ambiguity in Feedforward Novel View Synthesis Transformer via Semantic-Spatial Decoupling