Decoder-only Large View Synthesis Models (LVSMs) with KV-cache have recently achieved state-of-the-art quality by regressing novel views from neural networks without reconstructing 3D geometry, while Test-Time Training (TTT) layers save computation but sacrifice quality. We propose Test-Time Baking (TTB), a strategy that learns to bake global scene information from cross-view attention into a lightweight MLP, enabling fast and high-fidelity novel-view decoding. The first key is to repurpose the fast-weight MLP of TTT to learn the cross-view attention mapping explicitly from its input-output pairs, rather than replacing attention entirely, achieving the same rendering efficiency as TTT while delivering substantially higher quality. The second key addresses a modality misalignment between context prefilling and novel-view rendering: we disentangle context tokens into camera-only context rendering tokens to serve as TTB input, and context ground-truth tokens with image content to serve as the TTB mapping target, ensuring the baked MLPs are relieved of the modality generalization challenge between context and novel-view processing. Our method achieves up to 23x rendering speedup with quality on par with the state-of-the-art across several benchmarks. Project page: https://test-time-baking.github.io
Figures & tables
Figure 1: T est- T ime B aking ( TTB ) is a feed-forward, generalizable view synthesis model that bakes scene information from cross-view attention into a fast-weight MLP, achieving constant-complexity rendering with quality on par with decoder-only architectures. TTB sits on the Pareto frontier of PSNR vs. FPS, rendering up to 23× faster than cross-attention baseline at comparable quality.
Figure 2: Motivation. We show three variants of the inner-loop optimization in TTT-style fast-weight MLPs. (a) TTT with KV binding [ 77 ] learns from raw value tokens projected from X(ctx) . (b) TTB without context pairing targets the cross-view attention output instead, collaborating with attention rather than replacing it. (c) Our full TTB additionally pairs each context view: X(ctx-gt) carries image content and supplies the cross-view attended target, while X(ctx-rend) is camera-only and matches the modality of X(nv) . The t-SNE plots show that only (c) aligns the modalities from the first block.
Figure 3: Pipeline. Our model stacks L identical blocks of Intra-view Attention, FFN, Test-Time Baking, and FFN. In the reconstruction stage, each context view is duplicated into a context pair, where the context ground-truth token X(ctx-gt) carries RGB and pose while the context rendering token X(ctx-rend) is pose-only and matches the modality of the novel rendering token X(nv) . At each TTB layer, the inner-loop update bakes the fast-weight MLP via MSE loss to map X(ctx-rend) toward the cross-view attention output of X(ctx-gt) . The baked MLP is applied to serve three token streams: it propagates both X(ctx-rend) and X(nv) to the next layer, and provides feedback to X(ctx-gt) as an auxiliary signal. In the novel-view rendering stage, each token retrieves global scene information in O(1) from the baked MLP, instead of O(N) via cross-attention. The rendering loss is backpropagated through the inner-loop baking process to train the model end-to-end.
Figure 4: Per-rank context contribution.
Figure 5: Qualitative comparison on DL3DV. TTB preserves fine details on par with DVSM [ 58 ] , while LaCT [ 77 ] and LVSM [ 25 ] produce visible blur.
Figure 6: Qualitative comparison on in-the-wild datasets. TTB generalizes to out-of-domain scenes and preserves fine details consistent with the in-domain results.
Method
Quality ( V=32 )
Recon-time ↓
Rend-fps ↑
PSNR ↑
SSIM ↑
LPIPS ↓
V=16
V=32
V=64
V=16
V=32
V=64
(a) = DVSM [ 58 ]
24.68
0.776
0.220
1.300
4.907
18.396
12.1
6.8
3.7
(b) = (a) + TTT [ 77 ]
23.12
0.704
0.286
0.439
0.862
1.706
49.5
49.1
49.1
TTB (Ours):
(c) = (b) + context pairing
23.78
0.728
0.264
0.626
1.237
2.458
50.8
50.8
50.7
(d) = (b) + cross-view attn.
24.13
0.742
0.252
1.267
5.178
17.745
50.4
50.4
50.5
Table 4: Ablation study from TTT to TTB on DL3DV. ( light =worst, dark =best).
Table 8
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Training data
GPUs
Pretrain source
LVSM [ 25 ]
Objaverse / Re10K
64 × A100
from scratch
Efficient-LVSM [ 22 ]
Re10K
64 × A100
from scratch
DVSM [ 58 ]
DL3DV / Re10K
64 × A100
from scratch
LaCT [ 77 ]
Objaverse / DL3DV
64 × A100
from scratch
LongLRM [ 12 ]
DL3DV
64 × A100
from scratch
tttLRM [ 63 ]
Objaverse / DL3DV
64 × A100
LaCT [ 77 ] ckpt
Appendix
Table 7: Training settings across methods. “Multi-dataset” indicates training on a curated mixture beyond a single benchmark. “DVSM ckpt” refers to initializing from the corresponding DVSM [ 58 ] checkpoint released for that benchmark. “–” marks compute resources that are not explicitly reported by the corresponding paper.
Method
Reconstruction
Rendering (per token)
Scene representation
LVSM (decoder-only) [ 25 ]
—
O(V⋅L)
context tokens
LVSM (enc.-dec.) [ 25 ]
O(V2⋅L2)
O(Ls)
latent scene tokens
Efficient-LVSM [ 22 ]
O(V⋅L2)
O(V⋅L)
KV-cache
DVSM [ 58 ]
O(V2⋅L2)
O(V⋅L)
KV-cache
LaCT [ 77 ]
O(V⋅L)
O(1)
fast weights
LongLRM [ 12 ]
O(V⋅L)
GS rasterizer
3D Gaussians
Appendix
Table 8: Asymptotic complexity comparison. V denotes the number of context views and L the number of tokens per view. Reconstruction is reported per scene, rendering per novel-view token. “GS rasterizer” denotes that rendering is handled by a Gaussian splatting rasterizer rather than a neural module, which is not directly comparable in O(⋅) form. We only list the complexity of the main model layer for global scene understanding. LongLRM still employs a few cross-view attention layers with O(V2⋅L2) complexity. LaCT, tttLRM, LagerNVS, and ours also employ intra-view attention with O(V⋅L2) reconstruction complexity and O(L) per-token rendering complexity.
Method
# blocks
Latent dim
Cross-view sublayer
LVSM [ 25 ]
24
768
cross-view attention
Efficient-LVSM [ 22 ]
24
1024
cached KV cross-attention
DVSM [ 58 ]
12
768
cross-view attention (cached KV)
LaCT [ 77 ]
24
768
large-chunk TTT layer
LongLRM [ 12 ]
24
768
Mamba2 + attention hybrid
tttLRM [ 63 ]
24
768
large-chunk TTT layer
Appendix
Table 9: Architectural composition. “Cross-view sublayer” indicates the module that exchanges information across views in each block. The depth column is the number of decoder blocks, with the encoder depth listed in parentheses for encoder-decoder methods. The latent-dim column reports the decoder width, with the encoder width also given for encoder-decoder methods.
Method
V=16
V=32
V=64
V=128
V=256
PSNR ↑
DVSM
22.83
24.68
26.23
27.20
27.56
TTB-single
23.26
24.84
25.60
25.82
25.78
TTB-duplicate
23.26
24.84
26.13
27.14
27.74
Recon-time ↓
DVSM
0.136
0.389
1.258
4.815
18.106
TTB-single
0.337
0.740
1.911
6.065
20.549
TTB-duplicate
0.333
1.048
3.578
13.557
52.064
Appendix
Table 11: Inference at scale on DL3DV. As the number of context views V grows, TTB-single —a single fast-weight MLP per TTB layer that absorbs all V views—suffers capacity saturation : its PSNR plateaus around V=64 and even regresses thereafter. TTB-duplicate , which maintains K local copies of the per-layer fast-weight state, each baked from a spatially-local group of views, prevents this saturation and tracks DVSM’s quality scaling while preserving O(1) per-frame rendering. Best per column is shaded red, second-best orange.
Method
Venue
PSNR ↑
SSIM ↑
LPIPS ↓
#params ↓
Recon-time ↓
Rend-fps ↑
Infer-mem ↓
Train-time/it ↓
Train-mem ↓
LVSM [ 25 ]
ICLR’25
22.24
0.691
0.352
165M
15s
20
9G
73s
42G
LaCT [ 77 ]
ICLR’26
26.90
0.837
0.185
379M
3.7s
10
23G
36s
72G
DVSM ps8
arXiv’26
29.13
0.893
0.129
163M
15s
2
20G
133s
65G
TTB– (ours)
–
28.12
0.867
0.158
244M
3.7s
23
18G
26s
68G
TTB (ours)
–
28.92
0.882
0.145
271M
18s
23
18G
90s
76G
Appendix
Table 14: Efficiency comparison on DL3DV at 544×960 resolution with 32 input views. Recon-time is the processing time of a single scene. Rend-fps indicates the number of frames rendered per second without batching. Infer-mem is the peak GPU memory usage when measuring the reconstruction and rendering. Train-time/it is the sum of model forward and backward time per iteration under training mode with a batch containing 32 context views. Train-mem is the peak memory usage when measuring training time. The actual training randomly samples different number of input views per scene rather than a fixed number. We highlight cells with numbers similar to the most efficient one under each column.
V
Method
Recon
FPS
N=1
N=2
N=5
N=10
N=30
N=100
16
DVSM
4.4
4
4.7
4.9
5.7
6.9
11.9
29.4
TTB–
1.9
23
1.9
2.0
2.1
2.3
3.2
6.2
TTB
5.4
23
5.4
5.5
5.6
5.8
6.7
9.7
32
DVSM
15.0
2
15.5
16.0
17.5
20.0
30.0
65.0
TTB–
3.7
23
3.7
3.8
3.9
4.1
5.0
8.0
TTB
18.0
23
18.0
18.1
18.2
18.4
19.3
22.3
Appendix
Table 15: End-to-end time for rendering N novel views (seconds, DL3DV at 960p), computed as reconstruction time plus N/FPS from Tab. 1 . TTB overtakes DVSM after 5–7 views at every V , while TTB– leads from N=1 .
Figure 7: Scene representation size as V grows at 544×960 resolution, measured on a single A100. The DVSM [ 58 ] KV-cache grows linearly in V (hollow points: linear extrapolation where DVSM ran out of 80 GB memory), TTB’s baked fast weights stay constant, and TTB-duplicate grows with the number of copies K=V/16 while remaining 21× smaller than the KV-cache.
Figure 8: Additional qualitative comparison on DL3DV. 64 context views.
Figure 9: Qualitative comparison on Re10K. Two-view setting with sparse input.
Figure 10: Zero-shot qualitative results on MipNeRF360. 64 context views.
Figure 11: Zero-shot qualitative results on Free. 64 context views.
Figure 12: Zero-shot qualitative results on Hike. 64 context views.
Recent Large View Synthesis Models (LVSMs) advocate an encoder-decoder architecture that separates reconstruction and rendering into distinct networks. We re-examine this design. Through controlled experiments, we show that a decoder-only architecture, which represents scenes implicitly as a KV-cache, outperforms encoder-decoder variants while using fewer parameters at identical rendering complexity. Further analysis shows that sharing weights between the color-input reconstruction network and the camera-only rendering network better aligns their features at the same viewpoint, facilitating image synthesis. Building on this finding, our model, dubbed DVSM, further incorporates foundation model priors and stage-wise patch sizing for an improved efficiency-quality tradeoff. Our results establish a new state of the art for novel-view synthesis across multiple benchmarks, in some cases even outperforming per-scene-optimized 3DGS under dense input views.
Geometry-free view synthesis transformers have recently achieved state-of-the-art performance in Novel View Synthesis (NVS), outperforming traditional approaches that rely on explicit geometry modeling. Yet the factors governing their scaling with compute remain unclear. We present a systematic study of scaling laws for view synthesis transformers and derive design principles for training compute-optimal NVS models. Contrary to prior findings, we show that encoder-decoder architectures can be compute-optimal; we trace earlier negative results to suboptimal architectural choices and comparisons across unequal training compute budgets. Across several compute levels, we demonstrate that our encoder-decoder architecture, which we call the Scalable View Synthesis Model (SVSM), scales as effectively as decoder-only models, achieves a superior performance-compute Pareto frontier, and surpasses the previous state-of-the-art on real-world NVS benchmarks with substantially reduced training compute.
Large view synthesis models (LVSMs) reconstruct scenes from sparse views and render novel views quickly, but they incur unnecessary storage and rendering costs due to token redundancy. We present GLoD-LVSM, a compact, geometry-grounded scene representation for LVSMs with continuous levels of detail. First, instead of learning a fixed set of scene anchor initializations, GLoD-LVSM learns a continuous field that initializes a flexible number of anchors to match the desired budget. A condenser then aggregates scene information into these anchors, with both the field and condenser trained end-to-end. Second, the condenser infers reference-token and anchor geometry to guide compression through geometry-grounded attention, improving rendering quality and enabling frustum culling during decoding. Compared with existing LVSM scene-compression baselines, GLoD-LVSM achieves the best trade-off between rendering quality and scene token count. Compared with the uncompressed base model, GLoD-LVSM reduces the scene-token count by 33.4% and accelerates rendering by 1.23× with no quality loss. Under a relaxed quality-loss budget, the reduction increases to 85.5%, with 3.00× rendering speedup. Further analysis reveals that view-synthesis supervision yields emergent scene depth and self-organized anchors that align with surfaces and adapt their density to local texture complexity. Beyond rendering, we further show that our scene representation also supports scene-semantic understanding, highlighting its potential for broader downstream tasks.