Geometry-free view synthesis transformers have recently achieved state-of-the-art performance in Novel View Synthesis (NVS), outperforming traditional approaches that rely on explicit geometry modeling. Yet the factors governing their scaling with compute remain unclear. We present a systematic study of scaling laws for view synthesis transformers and derive design principles for training compute-optimal NVS models. Contrary to prior findings, we show that encoder-decoder architectures can be compute-optimal; we trace earlier negative results to suboptimal architectural choices and comparisons across unequal training compute budgets. Across several compute levels, we demonstrate that our encoder-decoder architecture, which we call the Scalable View Synthesis Model (SVSM), scales as effectively as decoder-only models, achieves a superior performance-compute Pareto frontier, and surpasses the previous state-of-the-art on real-world NVS benchmarks with substantially reduced training compute.
Figures & tables
Figure 1: Scaling Laws for View Synthesis Transformers. Evaluated on RealEstate10K [ 34 ] , our SVSM exhibits a 3× more compute-optimal Pareto frontier than LVSM while retaining the same scaling behavior (similar slope and curvature everywhere).
Figure 2: Architectures of the current SOTA, the decoder-only LVSM [ 10 ] (a) and SVSM (ours, b). Our cross-attention based decoder enables parallel rendering of multiple target views after a single scene encoding. Each target view is decoded independently given the shared scene representation, but the cross-attention allows these independent decodings to be executed in parallel.
Figure 3: Effective Batch Size . Training loss (smoothed with a rolling-average) and test PSNR measured throughout training across various paired B and VT runs provide evidence for our effective batch size hypothesis: Models trained with the same product of number of scenes in the batch B and number of reconstruction target views VT , i.e. runs with the same effective batch size Beff , perform the same and are colored identically. On VC=8 (top) , we sweep across Beff= 128 , 256 on DL3DV, and on VC=2 (bottom) , we sweep across Beff= 128 , 1024 on RealEstate10K.
Scale Parameters
Reconstruction Quality
Rendering FPS (↑)
Model
Model Size
Train Iters
Train FLOPs (↓)
PSNR (↑)
SSIM (↑)
LPIPS (↓)
VC =2
VC =4
VC =8
LVSM Encoder-Decoder [ 10 ]
173M
100k
2.53 zflops
28.58
0.893
0.114
53.7
52.9
52.7
LVSM Decoder-Only [ 10 ]
171M
100k
1.60 zflops
29.67
0.906
0.098
37.9
19.5
8.6
SVSM Enc-Dec ( ours , Iter-matched)
740M
100k
0.74 zflops
29.80
0.907
0.098
48.6
42.7
35.0
SVSM Enc-Dec ( ours , Pareto-optimal)
416M
170k
0.77 zflops
30.01
0.910
0.096
71.0
61.8
49.7
Table 1: Stereo ( VC=2 ) NVS Results of the Largest Models. All models use a patch size of 8 with input images at 256×256 resolution. Our models achieve the highest reconstruction metrics while using less than half of the training compute. The rendering FPS of both SVSM models is also much faster than that of the LVSM decoder-only model, though both are slower than the LVSM encoder-decoder when VC is large.
Figure 4: Data and Model Scaling Plots. While our model ( blue ) is optimal when sufficient data is available, decoder-only LVSM ( red ) performs better with less data. The Pareto frontier analysis shows that our model is more data-hungry. Our model is also less parameter-efficient, although the gap closes as we increase the training compute. However, with sufficient data and compute, our model ( blue ) is overall superior in terms of training compute-optimality and rendering speed.
Figure 5: Qualitative Scaling Behavior , VC=2 . From left to right, both models steadily increase in rendering quality until reaching near photo-realistic results. Compared vertically, for a given compute-budget, SVSM renderings consistently contain less artifacts.
Recon Quality
Model
PSNR (↑)
SSIM (↑)
LPIPS (↓)
pixelNeRF [ 29 ]
20.43
0.589
0.550
pixelSplat [ 2 ]
26.09
0.863
0.136
MVSplat [ 3 ]
26.39
0.869
0.128
GS-LRM [ 31 ]
28.10
0.892
0.114
SVSM Enc-Dec (ours)
30.01
0.910
0.096
Table 2: Comparison to Geometry-Aware Methods. Our method achieves a new state-of-the-art on RealEstate10K [ 34 ] with the set from Charatan et al. [2] , outperforming not only LVSM, but also prior work with explicit 3D structure.
Model
Parameter Coeff. a
Data Coeff. b
LVSM
0.65
0.33
SVSM
0.52
0.47
Table 3: Parameter and Data Scaling Coefficients. As regressed from the plots in Fig. 4 , we find power law coefficients for scaling models and data with respect to compute.
Figure 6: Multiview PRoPE. We find that multiview projective RoPE embeddings [ 18 , 13 , 14 ] are critical for our model to scale with compute and data in the multiview setting ( VC>2 ). For each layer of the multiview transformer encoder, PRoPE embeddings use context camera poses to transform context view tokens into a common coordinate frames before the attention layer, and apply the inverse transformation before each MLP. To render, both context features and query tokens of the target view are transformed by PRoPE before cross-attention.
Figure 7: Multiview Scaling Behavior. Conducted on DL3DV [ 15 ] . (a) For VC>2 , without PRoPE, SVSM saturates and stops scaling much more quickly than LVSM. (b) When PRoPE is added, SVSM continues scaling with a better Pareto-frontier.
Scale Parameters
Reconstruction Quality
Rendering FPS (↑)
Model
Model Size
Train Iters
Train FLOPs (↓)
PSNR (↑)
SSIM (↑)
LPIPS (↓)
VC =4
VC =8
VC =16
LVSM Decoder-only + PRoPE [ 10 , 14 ]
171M
100k
43 eflops
26.19
0.830
0.145
104.7
52.6
23.8
SVSM Enc-Dec ( ours , ≈ Iter-matched)
711M
100k
32 eflops
26.29
0.835
0.141
280.4
261.2
230.4
SVSM Enc-Dec ( ours , Pareto-optimal)
400M
233k
44 eflops
26.87
0.853
0.129
411.1
381.1
333
Table 4: Multiview ( VC>2 ) NVS Results of the Largest Models. Our compute-matched model achieves significantly better rendering quality ( +0.68 PSNR , −0.016 LPIPS ), while maintaining nearly four times the rendering speed at inference-time.
Figure 8: Qualitative Scaling Behavior , VC=4 . The performance of both LVSM and our method steadily increases with compute (left to right). Compared vertically, for a given compute budget SVSM renderings are consistently less blurry.
Figure 9: Fixed-size Latent Scaling Experiments. Conducted on Objaverse [ 4 ] . (a) For VC=8 , SVSM and LVSM decoder-only scale equally while SVSM’s frontier is shifted by 5× on the compute axis. (b) When a fixed latent bottleneck is used, SVSM-fixed and LVSM encoder-decoder scale equally, but significantly worse than the unbottlenecked designs. SVSM again maintains a superior pareto frontier.
Encoder
Decoder
Params
dim
dim_head
n_layers
dim
dim_head
n_layers
15M
384
64
3
384
64
3
27M
384
64
6
384
64
6
35M
384
64
8
384
64
8
62M
512
64
8
384
64
8
79M
512
64
8
512
64
12
Table 5: RealEstate10K VC=2 , SVSM Encoder-Decoder. SVSM Encoder-Decoder model settings used to sweep scaling laws for the stereo ( VC=2 ) novel view synthesis setting. Bolded is the row for the compute-matched, Pareto-optimal SVSM model compared against the 24-layer LVSM Decoder-only model.
Encoder
Decoder
Params
dim
dim_head
n_layers
dim
dim_head
n_layers
8M
-
-
-
384
64
3
13M
-
-
-
384
64
6
22M
-
-
-
512
64
6
28M
-
-
-
512
64
8
53M
-
-
-
640
64
10
Table 6: RealEstate10K VC=2 , LVSM Decoder-only. LVSM Decoder-only model settings used to sweep scaling laws for the stereo ( VC=2 ) novel view synthesis setting.
Encoder
Decoder
Params
dim
dim_head
n_layers
dim
dim_head
n_layers
15M
384
64
3
384
64
3
32M
384
64
6
384
64
8
85M
512
64
10
512
64
12
168M
640
64
12
640
64
16
280M
768
64
12
768
64
20
Table 7: DL3DV VC=4 , SVSM Encoder-Decoder. SVSM Encoder-Decoder model settings used to sweep scaling laws for the multi-view ( VC>2 ) novel view synthesis setting. Bolded is the row for the compute-matched, Pareto-optimal SVSM model compared against the 24-layer LVSM Decoder-only model.
Encoder
Decoder
Params
dim
dim_head
n_layers
dim
dim_head
n_layers
8M
-
-
-
384
64
3
22M
-
-
-
512
64
6
43M
-
-
-
640
64
10
90M
-
-
-
768
64
12
175M
-
-
-
768
64
24
Table 8: DL3DV VC=4 , LVSM Decoder-only. LVSM Decoder-only model settings used to sweep scaling laws for the multi-view ( VC>2 ) novel view synthesis setting.
Figure 10: Linear Power Scaling Laws. We fit scaling laws onto sections of the Pareto-frontiers of the model families. We see that both models have approximately the same slope in each of their corresponding sections, indicating equal scaling.
Model
c for P>0.14
c for P≤0.14
LVSM
-0.23
-0.12
SVSM
-0.22
-0.12
Table 9: LPIPS vs. compute scaling coefficients. As regressed from Fig. 10 , we find power law coefficients for LPIPS P(χ)∝χc , and we report c in this table for the first half, which is determined by test LPIPS loss greater than 0.14 and the second half which is test LPIPS loss less than 0.14 .
Model
PSNR (↑)
SSIM (↑)
LPIPS (↓)
Vanilla SVSM
23.50
0.727
0.254
Vanilla w/ concat. pose
23.47
0.726
0.254
PRoPE on Encoder
23.61
0.733
0.249
PRoPE on Decoder
24.31
0.771
0.220
PRoPE on both
24.62
0.782
0.210
GTA on both
24.63
0.782
0.207
Table 10: PRoPE ablations. We vary where we apply PRoPE, try a different relative attention method (GTA), and also test pose information flow. GTA varies negligibly from PRoPE, indicating that epipolar geometry is not crucial, and the skip pose connection also has negligible impact, indicating that pose information flow is not responsible.
Figure 11: All Scaling Laws. We collect scaling laws across the three datasets we tested on, and in all cases, SVSM is compute-optimal.
Figure 12: Qualitative results on RE10K ( VC=2 ) across scale.
Figure 13: Qualitative results on DL3DV ( VC=4 ) across scale.
Figure 14: Multiview consistency results on Objaverse ( VC=8 ).
Recent Large View Synthesis Models (LVSMs) advocate an encoder-decoder architecture that separates reconstruction and rendering into distinct networks. We re-examine this design. Through controlled experiments, we show that a decoder-only architecture, which represents scenes implicitly as a KV-cache, outperforms encoder-decoder variants while using fewer parameters at identical rendering complexity. Further analysis shows that sharing weights between the color-input reconstruction network and the camera-only rendering network better aligns their features at the same viewpoint, facilitating image synthesis. Building on this finding, our model, dubbed DVSM, further incorporates foundation model priors and stage-wise patch sizing for an improved efficiency-quality tradeoff. Our results establish a new state of the art for novel-view synthesis across multiple benchmarks, in some cases even outperforming per-scene-optimized 3DGS under dense input views.
Feed-forward novel view synthesis methods achieve strong generalization from posed multi-view inputs, but scaling them to large input view sets remains challenging. Transformer-based approaches that jointly process all input-view tokens incur rapidly increasing computation and memory as the number of views grows, while simple view subsampling discards potentially useful observations. We introduce Anchor-Integrated Multi-View Synthesis (AIMS), a scalable framework that decouples the number of available observations from the number of views processed by the global synthesis model. AIMS selects a fixed set of spatially distributed anchor views using farthest point sampling, groups nearby observations around each anchor, and uses a lightweight learnable integrator to fuse their information into enriched anchor representations. This allows additional observations to contribute to synthesis while keeping the downstream global view budget fixed. Evaluations on RealEstate10K and ScanNet demonstrate a favorable quality--efficiency trade-off against transformer-based and Gaussian-based baselines. AIMS achieves 29.41 dB and 17.73 dB PSNR on the two datasets, respectively, with rendering averaging 7.24 ms per view.
Transformer-based models have advanced feedforward novel view synthesis (NVS). Current architectures such as GS-LRM and LVSM mix semantic information (e.g., RGB) and spatial information (e.g., Plücker rays) into a shared feature space. Since Plücker rays naturally carry lattice-like spatial structure, these designs can make the spatial bias interfere with appearance representation and degrade rendering fidelity. To this end, we propose to decouple the representation of feedforward NVS transformers into separate semantic and spatial tokens. The decoupled design keeps semantic and spatial information explicit in their branches while preserving cross-branch interaction through shared attention routing. Built on this design, we introduce optional categorized supervision and bidirectional modulation: the former provides branch-specific training signals, while the latter improves interaction between the two branches. Notably, the base decoupled design introduces virtually zero additional inference latency due to its architectural design. The proposed designs achieve consistent improvements, demonstrating effectiveness across decoder-only and encoder-decoder feedforward NVS models.
Yihang Wu, Yihang Sun, Shaofeng Zhang +4
Institute of Trustworthy Embodied Artificial Intelligence (TEAI), Fudan University · 3Sch. of Artificial Intelligence & Sch. of Computer Science, Shanghai Jiao Tong University · University of Science and Technology of China +1