Recent work has shown that neural networks can perform 3D tasks such as Novel View Synthesis (NVS) without explicit 3D reconstruction. Even so, we argue that strong 3D inductive biases are still helpful in the design of such networks. We show this point by introducing LagerNVS, an encoder-decoder neural network for NVS that builds on `3D-aware' latent features. The encoder is initialized from a 3D reconstruction network pre-trained using explicit 3D supervision. This is paired with a lightweight decoder, and trained end-to-end with photometric losses. LagerNVS achieves state-of-the-art deterministic feed-forward Novel View Synthesis (including 31.4 PSNR on Re10k), with and without known cameras, renders in real time, generalizes to in-the-wild data, and can be paired with a diffusion decoder for generative extrapolation.
Figures & tables
Figure 2: Method. The model takes any number of images and, optionally, their camera parameters as input. A large network initialized from a reconstruction model [ 70 ] outputs an intermediate feature representation with implicit 3D information. A lightweight network is queried on a target camera pose and renders a 512×512 image at more than 30 FPS on a single H100 GPU when provided with up to 9 input images.
Figure 3: Generalization. Our model generalizes to a wide range of input types and applications, including (a) internet photos with unknown camera poses, (b) embodied applications (logos removed as postprocessing), (c) 360 ∘ scenes and (d) non-square aspect ratios.
Figure 4: Architectures. Our highway encoder-decoder (left) maximizes information flow from source images to the latent representation (R), making it more expressive than the bottleneck encoder-decoder (middle). Unlike decoder-only models (right), our design allows scaling the encoder without slowing decoding.
Figure 5: Comparison to LVSM. Our model estimates geometry better than LVSM in regions where 2D matching is challenging (top). It also uses monocular cues better for depth (bottom).
Method
Batch
PSNR ↑
SSIM ↑
LPIPS ↓
(a)
LVSM [ 34 ] Enc-dec. bottleneck
64
28.32
0.888
0.117
(b)
LVSM [ 34 ] Decoder-only
64
28.89
0.894
0.108
(c)
Ours Enc-dec. highway— x-attn.
64
30.11
0.912
0.089
(d)
Ours Enc-dec. highway— full
64
30.48
0.918
0.086
(e)
LVSM [ 34 ] Enc-dec. bottleneck
512
28.58
0.893
0.114
(f)
LVSM [ 34 ] Decoder-only
512
29.67
0.906
0.098
Table 1: Comparison to LVSM. Our model outperforms LVSM (SoTA) both at small (batch size 64) and large (batch size 512) training scales.
Figure 6: Single-view NVS is supported for small camera motion with LagerNVS.
Encoder
E2E
X-attn
Pre-Tr.
PSNR ↑
SSIM ↑
LPIPS ↓
(a)
Highway
✓
✓
3D
21.02
0.652
0.257
(b)
Highway
✓
✗
3D
21.30
0.667
0.248
(c)
Decoder only
✓
✗
—
18.39
0.533
0.407
(d)
Bottleneck
✓
✗
—
17.53
0.480
0.461
(e)
Highway
✓
✓
—
18.03
0.551
0.405
(f)
Highway
✓
✓
2D
18.17
0.515
0.388
Table 2: Ablations. Even when handicapped by using faster cross-attention, our highway encoder-decoder model outperforms the decoder-only and bottleneck encoder-decoder variants, which are analogous to the models introduced by LVSM [ 34 ] . Using 3D-aware pre-training (Pre-tr.) is crucial, but only effective if the features are fine-tuned end-to-end (“E2E”).
Figure 7: Ablations. Our architecture (1), pre-training (2) and end-to-end training (3) result in the best NVS quality.
Method
Test data
PSNR ↑
SSIM ↑
LPIPS ↓
w cameras single dataset
DepthSplat [ 80 ]
DL3DV 4-view
22.30
0.765
0.189
Ours
27.56
0.869
0.095
DepthSplat [ 80 ]
DL3DV 6-view
23.47
0.812
0.154
Ours
29.45
0.904
0.068
w/o cameras generalizable
AnySplat [ 33 ]
Re10k 2-view
17.05
0.626
0.349
Flare [ 90 ]
23.77
0.801
0.191
Table 3: Feed-forward 3DGS comparison. Our model outperforms state-of-the-art feed-forward 3DGS models across the board, both with and without input camera poses available, illustrating the strength of latent 3D representations. NoPoSplat only trained on RealEstate10k (a potentially unfair comparison), but we include it here for completeness. See Sec. C.3 for details of ‘Ours (v2)’.
Figure 8: Comparison to feed-forward 3DGS. Our model better handles thin parts, reflections, and occlusions, and supports inference with and without source camera poses.
Figure 9: Diffusion. Our decoder can be fine-tuned with a diffusion objective, enabling hallucination of unobserved regions.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Views
Posed
Split
PSNR
SSIM
LPIPS
Re10k
2
✓
[ 9 ]
28.99 (29.05)
0.900 (0.901)
0.149 (0.147)
Re10k
2
✗
[ 9 ]
27.88 (28.28)
0.875 (0.885)
0.161 (0.155)
Re10k
2
✓
[ 90 ]
26.36 (26.40)
0.866 (0.867)
0.190 (0.188)
Re10k
2
✗
[ 90 ]
25.11 (25.64)
0.833 (0.848)
0.210 (0.201)
DL3DV
2
✓
[ 80 ]
21.66 (21.77)
0.688 (0.692)
0.290 (0.287)
DL3DV
2
✗
[ 80 ]
21.27 (21.33)
0.666 (0.670)
0.303 (0.301)
Appendix
Table A1 : Generalizable model performance at 512×512 . To facilitate future comparisons, we report the performance of our generalizable model on a number of standard high-resolution NVS benchmarks, with and without source camera poses. Scores are reported for version 2 of the model in black, described in Sec. 3 and Sec. C.3 . Gray numbers represent scores for the previous version (v1) .
Figure A1 : Additional comparison vs LVSM. Our model better maintains consistent shape in regions where 2D matching across inputs is challenging (top 2 rows) and shows better monocular depth estimation (bottom row).
PSNR
SSIM
LPIPS
SVSM
30.01
0.910
0.096
Ours
31.39
0.928
0.078
Appendix
Table A2 : LagerNVS outperforms SVSM due to greater model capacity and 3D pre-training.
Figure A2 : Qualitative comparison vs Feed-forward 3DGS. Our model more accurately represents reflective areas ( mirror, top row and table, second row ) and thin structures ( metal rod, top row and chair arm, second row ). The global, latent representation leads to better surface alignment ( suitcase handle, third row and teddy bear, bottom row) and is more robust to occlusions ( floor, third row ).
Figure A3 : Occlusions. Our model can handle simple occlusions, such as the corner of the bathtub (top) or the bottom part of the black box (middle). In more difficult settings (bottom), completions are blurry (as expected), but still reasonable.
Figure A4 : Encoder. The encoder takes as source V images and, optionally, V cameras . The images are upsized (1) to the dimension expected by VGGT, and cameras g are projected (2) to conditioning tokens. Image and camera tokens are fed together to the encoder backbone , which is initialized with VGGT weights (3). The tokens from the last local attention layer and global attention layer of VGGT are concatenated, after discarding camera tokens. These tokens are then (4) projected to decoder channel dimension C to form the output latent 3D representation (z1,…,zV) .
Figure A5 : Decoder. The decoder tokenizes the target camera (1), in the form of a Plucker ray map [ 87 ] . The scene tokens (left) and target camera tokens s are fed to a Vision Transformer (2) [ 17 ] , with two variants of the attention mechanism (see text). The register tokens and scene tokens are discarded after the transformer, and output novel views are read out from the remaining tokens.
Max # imgs
Quality (PSNR)
Attention
Complexity
@ 512 real-time
2-view
4-view
6-view
Params
Full
O(V2)
6
21.30
24.17
24.84
85M
Unidirectional cross-attn. (not used)
O(V)
26
20.51
22.56
22.91
113M
Bidirectional cross-attn. (ours)
O(V)
9
21.02
23.65
24.54
170M
Appendix
Table A3 : Decoder attention variants. The bidirectional cross-attention mechanism used for the main model offers a good trade-off between real-time rendering speed and NVS quality.
Experiment
Iter.
LR
Batch
Data
Source Views
Res
Grad clip
LVSM comparison Tab 1. (a-c)
100k
4e−4
64
Re10k
2
256
1.0
LVSM comparison Tab 1. (d-f)
100k
4e−4
512
Re10k
2
256
1.0
Ablations Tab 2.
100k
1e−4
64
All
2–4
256
3.0
3DGS Tab 3. DL3DV posed
100k
4e−4
512
DL3DV
2–6
256
3.0
Final model, 3DGS Tab 3. Unposed and Tab 4.
250k
1e−4
512
All
1–10
512
3.0
Appendix
Table A4 : Hyperparameters. We detail the training hyperparameters for each experiment. The model used for comparison in “3DGS Tab 3. Unposed” can also accept camera poses as conditioning, and we use it as our final model.
We present BetterScene, an approach to enhance novel view synthesis (NVS) quality for diverse real-world scenes using extremely sparse, unconstrained photos. BetterScene leverages the production-ready Stable Video Diffusion (SVD) model pretrained on billions of frames as a strong backbone, aiming to mitigate artifacts and recover view-consistent details at inference time. Conventional methods have developed similar diffusion-based solutions to address these challenges of novel view synthesis. Despite significant improvements, these methods typically rely on off-the-shelf pretrained diffusion priors and fine-tune only the UNet module while keeping other components frozen, which still leads to inconsistent details and artifacts even when incorporating geometry-aware regularizations like depth or semantic conditions. To address this, we investigate the latent space of the diffusion model and introduce two components: (1) temporal equivariance regularization and (2) vision foundation model-aligned representation, both applied to the variational autoencoder (VAE) module within the SVD pipeline. BetterScene integrates a feed-forward 3D Gaussian Splatting (3DGS) model to render features as inputs for the SVD enhancer and generate continuous, artifact-free, consistent novel views. We evaluate on the challenging DL3DV-10K dataset and demonstrate superior performance compared to state-of-the-art methods.
Yuci Han, Charles Toth, John E. Anderson +2
Dept. of Electrical and Computer Engineering, The Ohio State University · USACE ERDC GRL
Recent Large View Synthesis Models (LVSMs) advocate an encoder-decoder architecture that separates reconstruction and rendering into distinct networks. We re-examine this design. Through controlled experiments, we show that a decoder-only architecture, which represents scenes implicitly as a KV-cache, outperforms encoder-decoder variants while using fewer parameters at identical rendering complexity. Further analysis shows that sharing weights between the color-input reconstruction network and the camera-only rendering network better aligns their features at the same viewpoint, facilitating image synthesis. Building on this finding, our model, dubbed DVSM, further incorporates foundation model priors and stage-wise patch sizing for an improved efficiency-quality tradeoff. Our results establish a new state of the art for novel-view synthesis across multiple benchmarks, in some cases even outperforming per-scene-optimized 3DGS under dense input views.
4D novel view synthesis (NVS) aims to render dynamic scenes across viewpoints and timestamps, which requires jointly inferring 3D structure and temporal dynamics. Existing monocular feed-forward methods often rely on explicit 4D Gaussian representations, which are prone to temporal inconsistency and severe rendering artifacts. We present FL4N, the first feed-forward latent 4D NVS model. Instead of relying on explicit 4D primitives, FL4N encodes a 4D scene prior into a unified latent representation across space and time. Our core insight is that learning temporal dynamics as residuals on top of a pretrained static latent prior enables effective 4D modeling with limited dynamic training data. To this end, we introduce a lightweight dynamic encoder that leverages static scene features to disentangle object motion from camera-induced parallax. Trained solely with an image synthesis objective, FL4N achieves state-of-the-art 4D NVS quality while rendering in real time. Its learned latent representation generalizes to real-world videos and disentangles viewpoint from time.