LagerNVS: Latent Geometry for Fully Neural Real-time Novel View Synthesis

Stanislaw Szymanowicz; Minghao Chen; Jianyuan Wang; Christian Rupprecht; Andrea Vedaldi

cs.CVMar 20, 2026

LagerNVS: Latent Geometry for Fully Neural Real-time Novel View Synthesis

Authors: Stanislaw Szymanowicz, Minghao Chen, Jianyuan Wang, Christian Rupprecht, Andrea Vedaldi

Organizations: Visual Geometry Group, University of Oxford · Meta AI

Abstract

Recent work has shown that neural networks can perform 3D tasks such as Novel View Synthesis (NVS) without explicit 3D reconstruction. Even so, we argue that strong 3D inductive biases are still helpful in the design of such networks. We show this point by introducing LagerNVS, an encoder-decoder neural network for NVS that builds on `3D-aware' latent features. The encoder is initialized from a 3D reconstruction network pre-trained using explicit 3D supervision. This is paired with a lightweight decoder, and trained end-to-end with photometric losses. LagerNVS achieves state-of-the-art deterministic feed-forward Novel View Synthesis (including 31.4 PSNR on Re10k), with and without known cameras, renders in real time, generalizes to in-the-wild data, and can be paired with a diffusion decoder for generative extrapolation.

Figures & tables

Figure 2: Method. The model takes any number of images and, optionally, their camera parameters as input. A large network initialized from a reconstruction model [ 70 ] outputs an intermediate feature representation with implicit 3D information. A lightweight network is queried on a target camera pose and renders a $512\times 512$ image at more than 30 FPS on a single H100 GPU when provided with up to 9 input images.

Figure 3: Generalization. Our model generalizes to a wide range of input types and applications, including (a) internet photos with unknown camera poses, (b) embodied applications (logos removed as postprocessing), (c) 360 ∘ scenes and (d) non-square aspect ratios.

Figure 4: Architectures. Our highway encoder-decoder (left) maximizes information flow from source images to the latent representation (R), making it more expressive than the bottleneck encoder-decoder (middle). Unlike decoder-only models (right), our design allows scaling the encoder without slowing decoding.

Figure 5: Comparison to LVSM. Our model estimates geometry better than LVSM in regions where 2D matching is challenging (top). It also uses monocular cues better for depth (bottom).

	Method	Batch	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
(a)	LVSM [ 34 ] Enc-dec. bottleneck	64	28.32	0.888	0.117
(b)	LVSM [ 34 ] Decoder-only	64	28.89	0.894	0.108
(c)	Ours Enc-dec. highway— x-attn.	64	30.11	0.912	0.089
(d)	Ours Enc-dec. highway— full	64	30.48	0.918	0.086
(e)	LVSM [ 34 ] Enc-dec. bottleneck	512	28.58	0.893	0.114
(f)	LVSM [ 34 ] Decoder-only	512	29.67	0.906	0.098

Table 1: Comparison to LVSM. Our model outperforms LVSM (SoTA) both at small (batch size 64) and large (batch size 512) training scales.

Figure 6: Single-view NVS is supported for small camera motion with LagerNVS.

	Encoder	E2E	X-attn	Pre-Tr.	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
(a)	Highway	✓	✓	3D	21.02	0.652	0.257
(b)	Highway	✓	✗	3D	21.30	0.667	0.248
(c)	Decoder only	✓	✗	—	18.39	0.533	0.407
(d)	Bottleneck	✓	✗	—	17.53	0.480	0.461
(e)	Highway	✓	✓	—	18.03	0.551	0.405
(f)	Highway	✓	✓	2D	18.17	0.515	0.388

Table 2: Ablations. Even when handicapped by using faster cross-attention, our highway encoder-decoder model outperforms the decoder-only and bottleneck encoder-decoder variants, which are analogous to the models introduced by LVSM [ 34 ] . Using 3D-aware pre-training (Pre-tr.) is crucial, but only effective if the features are fine-tuned end-to-end (“E2E”).

Figure 7: Ablations. Our architecture (1), pre-training (2) and end-to-end training (3) result in the best NVS quality.

	Method	Test data	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
w cameras single dataset	DepthSplat [ 80 ]	DL3DV 4-view	22.30	0.765	0.189
	Ours		27.56	0.869	0.095
	DepthSplat [ 80 ]	DL3DV 6-view	23.47	0.812	0.154
	Ours		29.45	0.904	0.068
w/o cameras generalizable	AnySplat [ 33 ]	Re10k 2-view	17.05	0.626	0.349
	Flare [ 90 ]		23.77	0.801	0.191

Table 3: Feed-forward 3DGS comparison. Our model outperforms state-of-the-art feed-forward 3DGS models across the board, both with and without input camera poses available, illustrating the strength of latent 3D representations. NoPoSplat only trained on RealEstate10k (a potentially unfair comparison), but we include it here for completeness. See Sec. C.3 for details of ‘Ours (v2)’.

Figure 8: Comparison to feed-forward 3DGS. Our model better handles thin parts, reflections, and occlusions, and supports inference with and without source camera poses.

Figure 9: Diffusion. Our decoder can be fine-tuned with a diffusion objective, enabling hallucination of unobserved regions.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Views	Posed	Split	PSNR	SSIM	LPIPS
Re10k	2	✓	[ 9 ]	28.99 (29.05)	0.900 (0.901)	0.149 (0.147)
Re10k	2	✗	[ 9 ]	27.88 (28.28)	0.875 (0.885)	0.161 (0.155)
Re10k	2	✓	[ 90 ]	26.36 (26.40)	0.866 (0.867)	0.190 (0.188)
Re10k	2	✗	[ 90 ]	25.11 (25.64)	0.833 (0.848)	0.210 (0.201)
DL3DV	2	✓	[ 80 ]	21.66 (21.77)	0.688 (0.692)	0.290 (0.287)
DL3DV	2	✗	[ 80 ]	21.27 (21.33)	0.666 (0.670)	0.303 (0.301)

Appendix

Table A1 : Generalizable model performance at

512\times 512

. To facilitate future comparisons, we report the performance of our generalizable model on a number of standard high-resolution NVS benchmarks, with and without source camera poses. Scores are reported for version 2 of the model in black, described in Sec. 3 and Sec. C.3 . Gray numbers represent scores for the previous version (v1) .

Figure A1 : Additional comparison vs LVSM. Our model better maintains consistent shape in regions where 2D matching across inputs is challenging (top 2 rows) and shows better monocular depth estimation (bottom row).

	PSNR	SSIM	LPIPS
SVSM	30.01	0.910	0.096
Ours	31.39	0.928	0.078

Appendix

Table A2 : LagerNVS outperforms SVSM due to greater model capacity and 3D pre-training.

Figure A2 : Qualitative comparison vs Feed-forward 3DGS. Our model more accurately represents reflective areas ( mirror, top row and table, second row ) and thin structures ( metal rod, top row and chair arm, second row ). The global, latent representation leads to better surface alignment ( suitcase handle, third row and teddy bear, bottom row) and is more robust to occlusions ( floor, third row ).

Figure A3 : Occlusions. Our model can handle simple occlusions, such as the corner of the bathtub (top) or the bottom part of the black box (middle). In more difficult settings (bottom), completions are blurry (as expected), but still reasonable.

Figure A4 : Encoder. The encoder takes as source

V

images and, optionally,

V

cameras . The images are upsized (1) to the dimension expected by VGGT, and cameras

\boldsymbol{g}

are projected (2) to conditioning tokens. Image and camera tokens are fed together to the encoder backbone , which is initialized with VGGT weights (3). The tokens from the last local attention layer and global attention layer of VGGT are concatenated, after discarding camera tokens. These tokens are then (4) projected to decoder channel dimension

C

to form the output latent 3D representation

\left(\boldsymbol{z}_{1},\dots,\boldsymbol{z}_{V}\right)

Figure A5 : Decoder. The decoder tokenizes the target camera (1), in the form of a Plucker ray map [ 87 ] . The scene tokens (left) and target camera tokens $\boldsymbol{s}$ are fed to a Vision Transformer (2) [ 17 ] , with two variants of the attention mechanism (see text). The register tokens and scene tokens are discarded after the transformer, and output novel views are read out from the remaining tokens.

		Max $\#$ imgs	Quality (PSNR)
Attention	Complexity	@ 512 real-time	2-view	4-view	6-view	Params
Full	$\mathcal{O}(V^{2})$	6	21.30	24.17	24.84	85M
Unidirectional cross-attn. (not used)	$\mathcal{O}(V)$	26	20.51	22.56	22.91	113M
Bidirectional cross-attn. (ours)	$\mathcal{O}(V)$	9	21.02	23.65	24.54	170M

Appendix

Table A3 : Decoder attention variants. The bidirectional cross-attention mechanism used for the main model offers a good trade-off between real-time rendering speed and NVS quality.

Experiment	Iter.	LR	Batch	Data	Source Views	Res	Grad clip
LVSM comparison Tab 1. (a-c)	100k	$4e-4$	$64$	Re10k	2	256	1.0
LVSM comparison Tab 1. (d-f)	100k	$4e-4$	$512$	Re10k	2	256	1.0
Ablations Tab 2.	100k	$1e-4$	$64$	All	2–4	256	3.0
3DGS Tab 3. DL3DV posed	100k	$4e-4$	$512$	DL3DV	2–6	256	3.0
Final model, 3DGS Tab 3. Unposed and Tab 4.	250k	$1e-4$	$512$	All	1–10	512	3.0

Appendix

Table A4 : Hyperparameters. We detail the training hyperparameters for each experiment. The model used for comparison in “3DGS Tab 3. Unposed” can also accept camera poses as conditioning, and we use it as our final model.

Novel View Synthesis

Novel View Synthesis creates images of a scene from camera viewpoints not present in the input, using learned geometry and appearance models.

See all 289 papers

BetterScene: 3D Scene Synthesis with Representation-Aligned Generative Model
Feb 26, 2026Yuci Han, Charles Toth, John E. Anderson +2Novel View Synthesis Multi-View Image Synthesis
Dept. of Electrical and Computer Engineering, The Ohio State University · USACE ERDC GRL
DVSM: Decoder-only View Synthesis Model Done Right
May 28, 2026Cheng Sun, Jaesung Choe, Min-Hung Chen +2Novel View Synthesis Multi-View Image Synthesis
NVIDIA · National Taiwan University
FL4N: Feed-forward Latent Model for 4D Novel View Synthesis
Oct 26, 2026Eunbeen Hong, Hyeonseo Yu, Honggyu An +8Dynamic Novel View Synthesis Novel View Synthesis
1KAIST AI · 2Google DeepMind

Feb 26, 2026J/K move · Enter open · S save

BetterScene: 3D Scene Synthesis with Representation-Aligned Generative Model

Yuci Han, Charles Toth, John E. Anderson, William J. Shuart+1

Dept. of Electrical and Computer Engineering, The Ohio State University · USACE ERDC GRL

Novel View Synthesis Multi-View Image Synthesis Sparse-View 3D Reconstruction Video Diffusion Models 3D Reconstruction

Open paper AskPDF arXiv

We present BetterScene, an approach to enhance novel view synthesis (NVS) quality for diverse real-world scenes using extremely sparse, unconstrained photos. BetterScene leverages the production-ready Stable Video Diffusion (SVD) model pretrained on billions of frames as a strong backbone, aiming to mitigate artifacts and recover view-consistent details at inference time. Conventional methods have developed similar diffusion-based solutions to address these challenges of novel view synthesis. Despite significant improvements, these methods typically rely on off-the-shelf pretrained diffusion priors and fine-tune only the UNet module while keeping other components frozen, which still leads to inconsistent details and artifacts even when incorporating geometry-aware regularizations like depth or semantic conditions. To address this, we investigate the latent space of the diffusion model and introduce two components: (1) temporal equivariance regularization and (2) vision foundation model-aligned representation, both applied to the variational autoencoder (VAE) module within the SVD pipeline. BetterScene integrates a feed-forward 3D Gaussian Splatting (3DGS) model to render features as inputs for the SVD enhancer and generate continuous, artifact-free, consistent novel views. We evaluate on the challenging DL3DV-10K dataset and demonstrate superior performance compared to state-of-the-art methods.

Abstract

Novel View Synthesis

Explore similar work

BetterScene: 3D Scene Synthesis with Representation-Aligned Generative Model

DVSM: Decoder-only View Synthesis Model Done Right

FL4N: Feed-forward Latent Model for 4D Novel View Synthesis