Less Decoder is More Encoder: Geometric Representation Learning from Novel View Synthesis

Keerthi Kaashyap; Dennis Anthony; Akshay Krishnan; Nhi Ngoc Nguyen; Jeremy Collins; James Hays; Shreyas Kousik; Animesh Garg

cs.CVOct 2, 2026

Less Decoder is More Encoder: Geometric Representation Learning from Novel View Synthesis

Authors: Keerthi Kaashyap, Dennis Anthony, Akshay Krishnan, Nhi Ngoc Nguyen, Jeremy Collins, James Hays, Shreyas Kousik, Animesh Garg

Organizations: Georgia Institute of Technology

Abstract

This paper examines the role of Novel View Synthesis (NVS) in geometric representation learning. In principle, NVS should reason about 3D scene structure, thereby enabling transferable multi-view geometric representations. Yet, existing encoder-based NVS methods yield poor representations. This is not because of a lack of supervisory signal, but rather due to inconspicuous architectural choices: spatially expressive decoders that dilute representational capabilities of the scene encoder, and low-level pixel-space targets that hinder feature learning. We present SNAP, a self-supervised encoder-decoder transformer that addresses both through a pose-conditioned local decoder and a latent-space reconstruction objective. SNAP is task agnostic, and we show that it is competitive with special-purpose geometry-supervised methods. SNAP also performs competitively against self-supervised representations across five tasks: visual localization, pose estimation, point correspondence, depth estimation, and robot manipulation. Remarkably, SNAP's patch features exhibit emergent viewpoint invariance that approaches heavily supervised models despite lower compute and data budgets. Under camera shifts where standard 2D representations collapse, SNAP degrades more gracefully, revealing that restricting decoder expressivity actively prevents the suppression of transferable geometric structure. https://snap-nvs.github.io

Figures & tables

Figure 1: Training and inference pathways. Top (Inference): A pose-free multi-view encoder processes reference frames via a frozen DINOv3 featurizer (

\varphi

) and cross-view self-attention to construct an implicit 3D scene latent

Z

. Bottom (Training): A pose-conditioned local decoder queries

Z

using PRoPE to predict the feature map of a novel target view. Supervision (

\mathcal{L}_{\mathrm{recon}}

) is applied in a semantic latent space against a frozen DINOv3 teacher, completely bypassing pixel-level rendering and forcing geometric structure into the encoder.

Figure 2: Effect of Receptive field across three geometric tasks on RE10K. Normalized performance (Z-score) vs. RF on a

\log_{2}

scale. Blue circles: reconstruction error (lower is better); dashed colored lines: downstream task performance (higher is better). Left: Point correspondence reveals the critical crossover: as RF expands, reconstruction error falls while correspondence error rises . The decoder’s improving reconstruction comes entirely at the cost of encoder spatial structure. Center & Right: Visual localization and pose estimation performance degrade monotonically as RF increases, implying that as decoder expressivity increases, downstream representational performance decreases.

Figure 3: Receptive field qualitative comparison. Each row tracks a query point (colored dot) and its feature similarity map across time. While a large receptive field ( $\text{RF}=16{\times}16$ ) yields more scattered similarity maps, restricting the receptive field to $\text{RF}=1{\times}1$ captures richer and more precise geometric correspondences albeit slightly more diffuse heatmaps, tracking with the Oracle across both indoor (kitchen, left) and object (peaches, right) scenes.

Figure 4: Performance landscape: initialization layer vs. prediction layer. Each cell represents a model where the encoder outputs from the “Input Layer” and the decoder predicts features from the “Prediction Layer” of a 12-layer DINOv3 VIT-B (layers 0-11). Color encodes performance across the evaluated metrics, scaled such that yellow always indicates better performance. For downstream geometric tasks like correspondence (middle), predicting mid-to-high-layer targets generally yields the strongest relative performance. Conversely, predicting raw RGB or early-layer targets (bottom rows) consistently degrades these downstream geometric metrics. Notably, downstream representational capacity does not perfectly align with intrinsic dimensionality (right) or feature reconstruction (left) which is the validation cosine similarity; while RGB targets yield poor geometric features, they maintain a lower intrinsic dimensionality than several deep-layer prediction configurations.

	RealEstate10K			DL3DV			Hypersim
	PCK $\uparrow$		APE $\downarrow$	PCK $\uparrow$		APE $\downarrow$	PCK $\uparrow$		APE $\downarrow$
Method	@5px	@10px		@5px	@10px		@5px	@10px
LVSM	5.51	14.76	41.04	3.53	11.05	45.75	5.62	13.41	42.15
RayZer	3.81	13.53	39.63	2.73	13.53	47.03	4.21	11.21	44.82
ERayZer	4.59	12.47	39.67	3.55	11.26	45.30	4.31	14.89	40.54
LagerNVS	9.61	27.36	26.54	3.81	11.31	42.41	8.94	24.30	31.02

Table 1: Point correspondence. Pixel-level matching precision (PCK) and average trajectory/pixel error (APE). Top results are highlighted in bold . SNAP outperforms baselines on RE10K and remains competitive on DL3DV and Hypersim without explicit 3D supervision.

	Visual Localization (AUC / Acc $\uparrow$ )								Relative Pose Estimation (AUC / Acc $\uparrow$ )
	RE10K				DL3DV				RE10K				DL3DV
Method	R 5	T 5	A 20	A 30	R 5	T 5	A 20	A 30	R 5	T 5	A 20	A 30	R 5	T 5	A 20	A 30
LVSM	69.87	8.48	20.62	29.39	20.39	3.55	6.02	11.82	70.31	38.98	43.03	46.97	54.41	32.37	27.65	30.18
RayZer	56.30	7.29	13.74	20.10	22.85	2.92	5.13	10.01	78.32	42.89	53.00	59.60	58.32	34.91	32.11	36.10
ERayZer	59.09	6.65	15.47	23.53	24.04	3.05	5.99	12.03	77.82	40.45	46.63	54.07	50.82	31.29	23.56	27.32
LagerNVS	58.65	9.42	17.07	25.01	22.81	7.35	5.79	12.05	70.81	30.74	35.67	42.17	57.81	36.91	34.96	44.13

Table 2: Visual localization and relative pose estimation performance. Metrics include Rotation Accuracy at

5^{\circ}

(

R_{5}

), Translation Accuracy at

5^{\circ}

(

T_{5}

), and AUC at

20^{\circ}

(

A_{20}

) and

30^{\circ}

(

A_{30}

). Top results are highlighted in bold . SNAP demonstrates superior geometric precision among self-supervised and NVS methods.

Figure 5: Emergent Viewpoint Invariance in Robot Manipulation. Success rate across three MimicGen tasks evaluated under camera shifts (

0.0

1.0

radians). While standard 2D semantic priors (DINOv3, ResNet [ 33 ] ) collapse rapidly under viewpoint perturbation, SNAP (Ours) maintains highly robust spatial representations, tracking closely with the geometry-supervised VGGT ceiling and systematically outperforming methods explicitly designed for spatial robustness (Adapt3R).

	Hypersim		7Scenes
Method	AbsRel $\downarrow$	$\delta_{1}$ $\uparrow$	AbsRel $\downarrow$	$\delta_{1}$ $\uparrow$
VGGT	0.2080	94.6036	1.9138	70.0987
LVSM	2.0648	38.5086	1.9474	32.5691
LagerNVS	0.7441	73.1069	2.3858	49.2679
ERayZer	1.1321	50.6075	1.3320	35.5672
SNAP (ours)	0.7093	64.6815	0.9257	52.0218

Table 3: Relative depth estimation. We evaluate relative depth prediction on a validation-split of Hypersim, and 7Scenes, a real-world held-out dataset. We report AbsRel and

\delta_{1}

, the percentage of pixels satisfying

\max(\hat{d}/d,d/\hat{d})<1.25

. Reporting for VGGT comes from the off-the-shelf depth estimation head.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Training and evaluation setting for robotics tasks : Left : Three MimicGen simulation task we train our downstream policies on. Right : To test the model’s emergent viewpoint invariance, we evaluate its performance on out-of-distribution camera shifts ranging from 0.2 to 1.0 radians, none of which were encountered during training.

		RealEstate10K			DL3DV			Hypersim
Teacher	Features	@0.1	@0.2	@0.3	@0.1	@0.2	@0.3	@0.1	@0.2	@0.3
DINOv3 [ 15 ]	Base	45.83	80.13	93.71	28.00	62.53	84.45	29.29	63.67	84.22
DINOv3 [ 15 ]	SNAP	78.21	91.32	95.62	58.74	80.27	89.61	55.35	77.90	88.35
	Base	64.60	81.61	89.99	49.17	70.75	82.71	40.82	64.80	78.74
SAM [ 55 ]	SNAP	75.58	87.40	92.81	60.09	77.53	86.58	52.27	72.62	83.64
MAE [ 9 ]	Base	17.71	50.33	76.96	13.50	42.67	70.15	11.01	35.31	60.81

Appendix

Table 4: Point correspondence with frozen teacher features. PCK (%) at thresholds

0.1

0.2

and

0.3

for frozen teacher features (Base) and the corresponding SNAP representations, using global soft-argmax matching (

\tau=0.02

). Hypersim is held out from SNAP’s training. SNAP improves over its teacher for every teacher, dataset and threshold.

	Hypersim		7Scenes
Method	AbsRel $\downarrow$	$\delta_{1}$ $\uparrow$	AbsRel $\downarrow$	$\delta_{1}$ $\uparrow$
DINOv3 [ 15 ]	0.7147	61.8738	1.0991	52.8681
SNAP	0.7093	64.6815	0.9257	52.0218

Appendix

Table 5: Relative depth estimation on Hypersim and 7Scenes. AbsRel (

\downarrow

) and

\delta

(

\uparrow

) for DINOv3 [ 15 ] and SNAP. Best results are highlighted in bold .

Backbone	Base		SNAP
	Top-1 $\uparrow$	Top-5 $\uparrow$	Top-1 $\uparrow$	Top-5 $\uparrow$
DINOv3 [ 15 ]	68.45	87.38	68.81	87.41
SAM [ 55 ]	18.06	34.45	21.46	38.28
MAE [ 9 ]	30.78	49.59	28.64	46.98

Appendix

Table 6: ImageNet

k

-NN classification with frozen representations. Top-1 and Top-5 accuracy (%) are reported for the original teacher representations and their corresponding SNAP representations. Best results are highlighted in bold .

Representation	mIoU $\uparrow$	Pixel Acc. $\uparrow$
DINOv3 [ 15 ]	51.92	87.27
SNAP	49.87	84.31

Appendix

Table 7: ADE20K semantic segmentation with frozen representations. Linear-probe performance is reported in terms of mean Intersection-over-Union (mIoU) and pixel accuracy (%). Best results are highlighted in bold .

	RealEstate10K			DL3DV			Hypersim
Method	@0.1	@0.2	@0.3	@0.1	@0.2	@0.3	@0.1	@0.2	@0.3
DINOv3 [ 15 ]	45.83 $\pm$ 0.80	80.13 $\pm$ 0.74	93.71 $\pm$ 0.47	28.00 $\pm$ 0.65	62.53 $\pm$ 0.87	84.45 $\pm$ 0.67	29.29 $\pm$ 0.73	63.67 $\pm$ 0.95	84.22 $\pm$ 0.79
[] V-JEPA 2.1-L [ 58 ]	35.85 $\pm$ 0.66	67.05 $\pm$ 0.77	85.05 $\pm$ 0.62	24.55 $\pm$ 0.55	54.94 $\pm$ 0.76	77.27 $\pm$ 0.67	21.45 $\pm$ 0.52	50.21 $\pm$ 0.80	72.44 $\pm$ 0.79
SNAP	78.21 $\pm$ 1.06	91.32 $\pm$ 0.78	95.62 $\pm$ 0.59	58.74 $\pm$ 1.14	80.27 $\pm$ 1.02	89.61 $\pm$ 0.81	55.35 $\pm$ 0.98	77.90 $\pm$ 0.95	88.35 $\pm$ 0.80

Appendix

Table 8: Point correspondence compared with V-JEPA 2.1. PCK (%) at thresholds

0.1

0.2

and

0.3

for DINOv3 [ 15 ] (ViT-B), SNAP (ViT-B), and V-JEPA 2.1-L [ 58 ] , using global soft-argmax matching (

\tau=0.02

). Values are mean

\pm

CI. Hypersim is held out from SNAP’s training. Best results are highlighted in bold .

Method	Metric	$V{=}1$	$V{=}2$	$V{=}4$	$V{=}8$	$V{=}10$	$V{=}20$	$V{=}50$	$V{=}100$
Muskie [ 14 ]	Time (ms) $\downarrow$	12	13	13	16	16	25	67	180
	Mem (GB) $\downarrow$	0.59	0.61	0.64	0.67	0.70	0.81	1.16	1.73
	Params (M) $\downarrow$	303.10	303.10	303.10	303.10	303.10	303.10	303.10	303.10
VGGT [ 3 ]	Time (ms) $\downarrow$	60	55	41	71	89	123	294	712
	Mem (GB) $\downarrow$	2.18	2.20	2.22	2.27	2.29	2.42	2.79	3.41
	Params (M) $\downarrow$	909.11	909.11	909.11	909.11	909.11	909.11	909.11	909.11

Appendix

Table 9: Encoder forward-pass efficiency vs. number of input views. For each encoder we report forward-pass time (milliseconds), peak GPU memory, and number of parameters. All encoders are benchmarked under identical conditions: bfloat16, single forward call per measurement, and we utilized an NVIDIA RTX PRO 6000 Blackwell GPU.

Figure 7: Qualitative Depth Evaluation : Hypersim relative depth predictions from a sparse collection of four input images using a frozen backbone with trained DPT layers. SNAP features yield depth predictions comparable to LagerNVS, despite using a lower decoding resolution (256px vs. 518px for LagerNVS). The higher resolution of LagerNVS likely contributes to its stronger predictions on thin structures.

Figure 8: Extended Input vs. Prediction Layer Landscape. We observe a fundamental trade-off depending on the chosen input layer (yellow indicates superior performance across all metrics). While mid-layer inputs (e.g., Layer 4) optimize local discriminability (PCK, mAP/MRR), they create a distinct performance valley in holistic metrics. To maximize cycle consistency and scene clustering (Variance Ratio), initializing and predicting with the deepest layers (Layer 11) proves to demonstrate a robust optimal operating point.

Figure 9: Feature map visualizations across vision encoders. High-dimensional latents are projected to RGB space using PCA to visualize spatial abstraction. While the 2D teacher (DINO) produces flat semantic regions, SNAP introduces clear spatial gradients that track underlying 3D geometry and depth, all while preserving fine-grained, discriminative object boundaries. Baselines struggle with this geometric abstraction: Muskie heavily encodes raw 2D appearance, and LagerNVS exhibits severe spatial grid artifacts.

Geometric Representation Learning

Geometric representation learning encodes data to preserve its intrinsic shape, distances, symmetries, and relationships.

See all 256 papers

Scaling View Synthesis Transformers
Feb 24, 2026Evan Kim, Hyunwoo Ryu, Thomas W. Mitchel +1Novel View Synthesis Neural Scaling Laws
MIT · Adobe
LagerNVS: Latent Geometry for Fully Neural Real-time Novel View Synthesis
Mar 20, 2026Stanislaw Szymanowicz, Minghao Chen, Jianyuan Wang +2Novel View Synthesis 3D Representation Learning
Visual Geometry Group, University of Oxford · Meta AI
Resolving Representation Ambiguity in Feedforward Novel View Synthesis Transformer via Semantic-Spatial Decoupling
May 18, 2026Yihang Wu, Yihang Sun, Shaofeng Zhang +4Disentangled Representation Learning Vision Transformer
Institute of Trustworthy Embodied Artificial Intelligence (TEAI), Fudan University · 3Sch. of Artificial Intelligence & Sch. of Computer Science, Shanghai Jiao Tong University · University of Science and Technology of China +1

Feb 24, 2026J/K move · Enter open · S save

Scaling View Synthesis Transformers

Evan Kim, Hyunwoo Ryu, Thomas W. Mitchel, Vincent Sitzmann

MIT · Adobe

Novel View Synthesis Neural Scaling Laws

Open paper AskPDF arXiv

Geometry-free view synthesis transformers have recently achieved state-of-the-art performance in Novel View Synthesis (NVS), outperforming traditional approaches that rely on explicit geometry modeling. Yet the factors governing their scaling with compute remain unclear. We present a systematic study of scaling laws for view synthesis transformers and derive design principles for training compute-optimal NVS models. Contrary to prior findings, we show that encoder-decoder architectures can be compute-optimal; we trace earlier negative results to suboptimal architectural choices and comparisons across unequal training compute budgets. Across several compute levels, we demonstrate that our encoder-decoder architecture, which we call the Scalable View Synthesis Model (SVSM), scales as effectively as decoder-only models, achieves a superior performance-compute Pareto frontier, and surpasses the previous state-of-the-art on real-world NVS benchmarks with substantially reduced training compute.

Abstract

Geometric Representation Learning

Explore similar work

Scaling View Synthesis Transformers

LagerNVS: Latent Geometry for Fully Neural Real-time Novel View Synthesis

Resolving Representation Ambiguity in Feedforward Novel View Synthesis Transformer via Semantic-Spatial Decoupling