This paper examines the role of Novel View Synthesis (NVS) in geometric representation learning. In principle, NVS should reason about 3D scene structure, thereby enabling transferable multi-view geometric representations. Yet, existing encoder-based NVS methods yield poor representations. This is not because of a lack of supervisory signal, but rather due to inconspicuous architectural choices: spatially expressive decoders that dilute representational capabilities of the scene encoder, and low-level pixel-space targets that hinder feature learning. We present SNAP, a self-supervised encoder-decoder transformer that addresses both through a pose-conditioned local decoder and a latent-space reconstruction objective. SNAP is task agnostic, and we show that it is competitive with special-purpose geometry-supervised methods. SNAP also performs competitively against self-supervised representations across five tasks: visual localization, pose estimation, point correspondence, depth estimation, and robot manipulation. Remarkably, SNAP's patch features exhibit emergent viewpoint invariance that approaches heavily supervised models despite lower compute and data budgets. Under camera shifts where standard 2D representations collapse, SNAP degrades more gracefully, revealing that restricting decoder expressivity actively prevents the suppression of transferable geometric structure. https://snap-nvs.github.io
Figures & tables
Figure 1: Training and inference pathways. Top (Inference): A pose-free multi-view encoder processes reference frames via a frozen DINOv3 featurizer ( φ ) and cross-view self-attention to construct an implicit 3D scene latent Z . Bottom (Training): A pose-conditioned local decoder queries Z using PRoPE to predict the feature map of a novel target view. Supervision ( Lrecon ) is applied in a semantic latent space against a frozen DINOv3 teacher, completely bypassing pixel-level rendering and forcing geometric structure into the encoder.
Figure 2: Effect of Receptive field across three geometric tasks on RE10K. Normalized performance (Z-score) vs. RF on a log2 scale. Blue circles: reconstruction error (lower is better); dashed colored lines: downstream task performance (higher is better). Left: Point correspondence reveals the critical crossover: as RF expands, reconstruction error falls while correspondence error rises . The decoder’s improving reconstruction comes entirely at the cost of encoder spatial structure. Center & Right: Visual localization and pose estimation performance degrade monotonically as RF increases, implying that as decoder expressivity increases, downstream representational performance decreases.
Figure 3: Receptive field qualitative comparison. Each row tracks a query point (colored dot) and its feature similarity map across time. While a large receptive field ( RF=16×16 ) yields more scattered similarity maps, restricting the receptive field to RF=1×1 captures richer and more precise geometric correspondences albeit slightly more diffuse heatmaps, tracking with the Oracle across both indoor (kitchen, left) and object (peaches, right) scenes.
Figure 4: Performance landscape: initialization layer vs. prediction layer. Each cell represents a model where the encoder outputs from the “Input Layer” and the decoder predicts features from the “Prediction Layer” of a 12-layer DINOv3 VIT-B (layers 0-11). Color encodes performance across the evaluated metrics, scaled such that yellow always indicates better performance. For downstream geometric tasks like correspondence (middle), predicting mid-to-high-layer targets generally yields the strongest relative performance. Conversely, predicting raw RGB or early-layer targets (bottom rows) consistently degrades these downstream geometric metrics. Notably, downstream representational capacity does not perfectly align with intrinsic dimensionality (right) or feature reconstruction (left) which is the validation cosine similarity; while RGB targets yield poor geometric features, they maintain a lower intrinsic dimensionality than several deep-layer prediction configurations.
RealEstate10K
DL3DV
Hypersim
PCK ↑
APE ↓
PCK ↑
APE ↓
PCK ↑
APE ↓
Method
@5px
@10px
@5px
@10px
@5px
@10px
LVSM
5.51
14.76
41.04
3.53
11.05
45.75
5.62
13.41
42.15
RayZer
3.81
13.53
39.63
2.73
13.53
47.03
4.21
11.21
44.82
ERayZer
4.59
12.47
39.67
3.55
11.26
45.30
4.31
14.89
40.54
LagerNVS
9.61
27.36
26.54
3.81
11.31
42.41
8.94
24.30
31.02
Table 1: Point correspondence. Pixel-level matching precision (PCK) and average trajectory/pixel error (APE). Top results are highlighted in bold . SNAP outperforms baselines on RE10K and remains competitive on DL3DV and Hypersim without explicit 3D supervision.
Visual Localization (AUC / Acc ↑ )
Relative Pose Estimation (AUC / Acc ↑ )
RE10K
DL3DV
RE10K
DL3DV
Method
R 5
T 5
A 20
A 30
R 5
T 5
A 20
A 30
R 5
T 5
A 20
A 30
R 5
T 5
A 20
A 30
LVSM
69.87
8.48
20.62
29.39
20.39
3.55
6.02
11.82
70.31
38.98
43.03
46.97
54.41
32.37
27.65
30.18
RayZer
56.30
7.29
13.74
20.10
22.85
2.92
5.13
10.01
78.32
42.89
53.00
59.60
58.32
34.91
32.11
36.10
ERayZer
59.09
6.65
15.47
23.53
24.04
3.05
5.99
12.03
77.82
40.45
46.63
54.07
50.82
31.29
23.56
27.32
LagerNVS
58.65
9.42
17.07
25.01
22.81
7.35
5.79
12.05
70.81
30.74
35.67
42.17
57.81
36.91
34.96
44.13
Table 2: Visual localization and relative pose estimation performance. Metrics include Rotation Accuracy at 5∘ ( R5 ), Translation Accuracy at 5∘ ( T5 ), and AUC at 20∘ ( A20 ) and 30∘ ( A30 ). Top results are highlighted in bold . SNAP demonstrates superior geometric precision among self-supervised and NVS methods.
Figure 5: Emergent Viewpoint Invariance in Robot Manipulation. Success rate across three MimicGen tasks evaluated under camera shifts ( 0.0 to 1.0 radians). While standard 2D semantic priors (DINOv3, ResNet [ 33 ] ) collapse rapidly under viewpoint perturbation, SNAP (Ours) maintains highly robust spatial representations, tracking closely with the geometry-supervised VGGT ceiling and systematically outperforming methods explicitly designed for spatial robustness (Adapt3R).
Hypersim
7Scenes
Method
AbsRel ↓
δ1↑
AbsRel ↓
δ1↑
VGGT
0.2080
94.6036
1.9138
70.0987
LVSM
2.0648
38.5086
1.9474
32.5691
LagerNVS
0.7441
73.1069
2.3858
49.2679
ERayZer
1.1321
50.6075
1.3320
35.5672
SNAP (ours)
0.7093
64.6815
0.9257
52.0218
Table 3: Relative depth estimation. We evaluate relative depth prediction on a validation-split of Hypersim, and 7Scenes, a real-world held-out dataset. We report AbsRel and δ1 , the percentage of pixels satisfying max(d^/d,d/d^)<1.25 . Reporting for VGGT comes from the off-the-shelf depth estimation head.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Training and evaluation setting for robotics tasks : Left : Three MimicGen simulation task we train our downstream policies on. Right : To test the model’s emergent viewpoint invariance, we evaluate its performance on out-of-distribution camera shifts ranging from 0.2 to 1.0 radians, none of which were encountered during training.
RealEstate10K
DL3DV
Hypersim
Teacher
Features
@0.1
@0.2
@0.3
@0.1
@0.2
@0.3
@0.1
@0.2
@0.3
DINOv3 [ 15 ]
Base
45.83
80.13
93.71
28.00
62.53
84.45
29.29
63.67
84.22
SNAP
78.21
91.32
95.62
58.74
80.27
89.61
55.35
77.90
88.35
Base
64.60
81.61
89.99
49.17
70.75
82.71
40.82
64.80
78.74
SAM [ 55 ]
SNAP
75.58
87.40
92.81
60.09
77.53
86.58
52.27
72.62
83.64
MAE [ 9 ]
Base
17.71
50.33
76.96
13.50
42.67
70.15
11.01
35.31
60.81
Appendix
Table 4: Point correspondence with frozen teacher features. PCK (%) at thresholds 0.1 , 0.2 and 0.3 for frozen teacher features (Base) and the corresponding SNAP representations, using global soft-argmax matching ( τ=0.02 ). Hypersim is held out from SNAP’s training. SNAP improves over its teacher for every teacher, dataset and threshold.
Hypersim
7Scenes
Method
AbsRel ↓
δ1↑
AbsRel ↓
δ1↑
DINOv3 [ 15 ]
0.7147
61.8738
1.0991
52.8681
SNAP
0.7093
64.6815
0.9257
52.0218
Appendix
Table 5: Relative depth estimation on Hypersim and 7Scenes. AbsRel ( ↓ ) and δ ( ↑ ) for DINOv3 [ 15 ] and SNAP. Best results are highlighted in bold .
Backbone
Base
SNAP
Top-1 ↑
Top-5 ↑
Top-1 ↑
Top-5 ↑
DINOv3 [ 15 ]
68.45
87.38
68.81
87.41
SAM [ 55 ]
18.06
34.45
21.46
38.28
MAE [ 9 ]
30.78
49.59
28.64
46.98
Appendix
Table 6: ImageNet k -NN classification with frozen representations. Top-1 and Top-5 accuracy (%) are reported for the original teacher representations and their corresponding SNAP representations. Best results are highlighted in bold .
Representation
mIoU ↑
Pixel Acc. ↑
DINOv3 [ 15 ]
51.92
87.27
SNAP
49.87
84.31
Appendix
Table 7: ADE20K semantic segmentation with frozen representations. Linear-probe performance is reported in terms of mean Intersection-over-Union (mIoU) and pixel accuracy (%). Best results are highlighted in bold .
RealEstate10K
DL3DV
Hypersim
Method
@0.1
@0.2
@0.3
@0.1
@0.2
@0.3
@0.1
@0.2
@0.3
DINOv3 [ 15 ]
45.83 ± 0.80
80.13 ± 0.74
93.71 ± 0.47
28.00 ± 0.65
62.53 ± 0.87
84.45 ± 0.67
29.29 ± 0.73
63.67 ± 0.95
84.22 ± 0.79
[] V-JEPA 2.1-L [ 58 ]
35.85 ± 0.66
67.05 ± 0.77
85.05 ± 0.62
24.55 ± 0.55
54.94 ± 0.76
77.27 ± 0.67
21.45 ± 0.52
50.21 ± 0.80
72.44 ± 0.79
SNAP
78.21 ± 1.06
91.32 ± 0.78
95.62 ± 0.59
58.74 ± 1.14
80.27 ± 1.02
89.61 ± 0.81
55.35 ± 0.98
77.90 ± 0.95
88.35 ± 0.80
Appendix
Table 8: Point correspondence compared with V-JEPA 2.1. PCK (%) at thresholds 0.1 , 0.2 and 0.3 for DINOv3 [ 15 ] (ViT-B), SNAP (ViT-B), and V-JEPA 2.1-L [ 58 ] , using global soft-argmax matching ( τ=0.02 ). Values are mean ± CI. Hypersim is held out from SNAP’s training. Best results are highlighted in bold .
Method
Metric
V=1
V=2
V=4
V=8
V=10
V=20
V=50
V=100
Muskie [ 14 ]
Time (ms) ↓
12
13
13
16
16
25
67
180
Mem (GB) ↓
0.59
0.61
0.64
0.67
0.70
0.81
1.16
1.73
Params (M) ↓
303.10
303.10
303.10
303.10
303.10
303.10
303.10
303.10
VGGT [ 3 ]
Time (ms) ↓
60
55
41
71
89
123
294
712
Mem (GB) ↓
2.18
2.20
2.22
2.27
2.29
2.42
2.79
3.41
Params (M) ↓
909.11
909.11
909.11
909.11
909.11
909.11
909.11
909.11
Appendix
Table 9: Encoder forward-pass efficiency vs. number of input views. For each encoder we report forward-pass time (milliseconds), peak GPU memory, and number of parameters. All encoders are benchmarked under identical conditions: bfloat16, single forward call per measurement, and we utilized an NVIDIA RTX PRO 6000 Blackwell GPU.
Figure 7: Qualitative Depth Evaluation : Hypersim relative depth predictions from a sparse collection of four input images using a frozen backbone with trained DPT layers. SNAP features yield depth predictions comparable to LagerNVS, despite using a lower decoding resolution (256px vs. 518px for LagerNVS). The higher resolution of LagerNVS likely contributes to its stronger predictions on thin structures.
Figure 8: Extended Input vs. Prediction Layer Landscape. We observe a fundamental trade-off depending on the chosen input layer (yellow indicates superior performance across all metrics). While mid-layer inputs (e.g., Layer 4) optimize local discriminability (PCK, mAP/MRR), they create a distinct performance valley in holistic metrics. To maximize cycle consistency and scene clustering (Variance Ratio), initializing and predicting with the deepest layers (Layer 11) proves to demonstrate a robust optimal operating point.
Figure 9: Feature map visualizations across vision encoders. High-dimensional latents are projected to RGB space using PCA to visualize spatial abstraction. While the 2D teacher (DINO) produces flat semantic regions, SNAP introduces clear spatial gradients that track underlying 3D geometry and depth, all while preserving fine-grained, discriminative object boundaries. Baselines struggle with this geometric abstraction: Muskie heavily encodes raw 2D appearance, and LagerNVS exhibits severe spatial grid artifacts.
Geometry-free view synthesis transformers have recently achieved state-of-the-art performance in Novel View Synthesis (NVS), outperforming traditional approaches that rely on explicit geometry modeling. Yet the factors governing their scaling with compute remain unclear. We present a systematic study of scaling laws for view synthesis transformers and derive design principles for training compute-optimal NVS models. Contrary to prior findings, we show that encoder-decoder architectures can be compute-optimal; we trace earlier negative results to suboptimal architectural choices and comparisons across unequal training compute budgets. Across several compute levels, we demonstrate that our encoder-decoder architecture, which we call the Scalable View Synthesis Model (SVSM), scales as effectively as decoder-only models, achieves a superior performance-compute Pareto frontier, and surpasses the previous state-of-the-art on real-world NVS benchmarks with substantially reduced training compute.
Recent work has shown that neural networks can perform 3D tasks such as Novel View Synthesis (NVS) without explicit 3D reconstruction. Even so, we argue that strong 3D inductive biases are still helpful in the design of such networks. We show this point by introducing LagerNVS, an encoder-decoder neural network for NVS that builds on `3D-aware' latent features. The encoder is initialized from a 3D reconstruction network pre-trained using explicit 3D supervision. This is paired with a lightweight decoder, and trained end-to-end with photometric losses. LagerNVS achieves state-of-the-art deterministic feed-forward Novel View Synthesis (including 31.4 PSNR on Re10k), with and without known cameras, renders in real time, generalizes to in-the-wild data, and can be paired with a diffusion decoder for generative extrapolation.
Stanislaw Szymanowicz, Minghao Chen, Jianyuan Wang +2
Visual Geometry Group, University of Oxford · Meta AI
Transformer-based models have advanced feedforward novel view synthesis (NVS). Current architectures such as GS-LRM and LVSM mix semantic information (e.g., RGB) and spatial information (e.g., Plücker rays) into a shared feature space. Since Plücker rays naturally carry lattice-like spatial structure, these designs can make the spatial bias interfere with appearance representation and degrade rendering fidelity. To this end, we propose to decouple the representation of feedforward NVS transformers into separate semantic and spatial tokens. The decoupled design keeps semantic and spatial information explicit in their branches while preserving cross-branch interaction through shared attention routing. Built on this design, we introduce optional categorized supervision and bidirectional modulation: the former provides branch-specific training signals, while the latter improves interaction between the two branches. Notably, the base decoupled design introduces virtually zero additional inference latency due to its architectural design. The proposed designs achieve consistent improvements, demonstrating effectiveness across decoder-only and encoder-decoder feedforward NVS models.
Yihang Wu, Yihang Sun, Shaofeng Zhang +4
Institute of Trustworthy Embodied Artificial Intelligence (TEAI), Fudan University · 3Sch. of Artificial Intelligence & Sch. of Computer Science, Shanghai Jiao Tong University · University of Science and Technology of China +1