TTB: Test-time Baking for Efficient Rendering of Decoder-only View Synthesis Models

Chin-Yang Lin; Ryo Hachiuma; Jaesung Choe; Min-Hung Chen; Yu-Chiang Frank Wang; Yen-Yu Lin; Wei-Chen Chiu; Yu-Lun Liu

cs.CVNov 12, 2026

TTB: Test-time Baking for Efficient Rendering of Decoder-only View Synthesis Models

Authors: Chin-Yang Lin, Ryo Hachiuma, Jaesung Choe, Min-Hung Chen, Yu-Chiang Frank Wang, Yen-Yu Lin, Wei-Chen Chiu, Yu-Lun Liu,

Organizations: NVIDIA · Department of Computer Science, National Yang Ming Chiao Tung University

Abstract

Decoder-only Large View Synthesis Models (LVSMs) with KV-cache have recently achieved state-of-the-art quality by regressing novel views from neural networks without reconstructing 3D geometry, while Test-Time Training (TTT) layers save computation but sacrifice quality. We propose Test-Time Baking (TTB), a strategy that learns to bake global scene information from cross-view attention into a lightweight MLP, enabling fast and high-fidelity novel-view decoding. The first key is to repurpose the fast-weight MLP of TTT to learn the cross-view attention mapping explicitly from its input-output pairs, rather than replacing attention entirely, achieving the same rendering efficiency as TTT while delivering substantially higher quality. The second key addresses a modality misalignment between context prefilling and novel-view rendering: we disentangle context tokens into camera-only context rendering tokens to serve as TTB input, and context ground-truth tokens with image content to serve as the TTB mapping target, ensuring the baked MLPs are relieved of the modality generalization challenge between context and novel-view processing. Our method achieves up to 23x rendering speedup with quality on par with the state-of-the-art across several benchmarks. Project page: https://test-time-baking.github.io

Figures & tables

Figure 1: T est- T ime B aking ( TTB ) is a feed-forward, generalizable view synthesis model that bakes scene information from cross-view attention into a fast-weight MLP, achieving constant-complexity rendering with quality on par with decoder-only architectures. TTB sits on the Pareto frontier of PSNR vs. FPS, rendering up to $23\times$ faster than cross-attention baseline at comparable quality.

Figure 2: Motivation. We show three variants of the inner-loop optimization in TTT-style fast-weight MLPs. (a) TTT with KV binding [ 77 ] learns from raw value tokens projected from $X^{(\text{ctx})}$ . (b) TTB without context pairing targets the cross-view attention output instead, collaborating with attention rather than replacing it. (c) Our full TTB additionally pairs each context view: $X^{(\text{ctx-gt})}$ carries image content and supplies the cross-view attended target, while $X^{(\text{ctx-rend})}$ is camera-only and matches the modality of $X^{(\text{nv})}$ . The t-SNE plots show that only (c) aligns the modalities from the first block.

Figure 3: Pipeline. Our model stacks $L$ identical blocks of Intra-view Attention, FFN, Test-Time Baking, and FFN. In the reconstruction stage, each context view is duplicated into a context pair, where the context ground-truth token $X^{(\text{ctx-gt})}$ carries RGB and pose while the context rendering token $X^{(\text{ctx-rend})}$ is pose-only and matches the modality of the novel rendering token $X^{(\text{nv})}$ . At each TTB layer, the inner-loop update bakes the fast-weight MLP via MSE loss to map $X^{(\text{ctx-rend})}$ toward the cross-view attention output of $X^{(\text{ctx-gt})}$ . The baked MLP is applied to serve three token streams: it propagates both $X^{(\text{ctx-rend})}$ and $X^{(\text{nv})}$ to the next layer, and provides feedback to $X^{(\text{ctx-gt})}$ as an auxiliary signal. In the novel-view rendering stage, each token retrieves global scene information in $O(1)$ from the baked MLP, instead of $O(N)$ via cross-attention. The rendering loss is backpropagated through the inner-loop baking process to train the model end-to-end.

Figure 4: Per-rank context contribution.

Figure 5: Qualitative comparison on DL3DV. TTB preserves fine details on par with DVSM [ 58 ] , while LaCT [ 77 ] and LVSM [ 25 ] produce visible blur.

Figure 6: Qualitative comparison on in-the-wild datasets. TTB generalizes to out-of-domain scenes and preserves fine details consistent with the in-domain results.

TTB (Ours):
Method	Quality ( $V{=}32$ )			Recon-time $\downarrow$			Rend-fps $\uparrow$
Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	$V{=}16$	$V{=}32$	$V{=}64$	$V{=}16$	$V{=}32$	$V{=}64$
(a) = DVSM [ 58 ]	24.68	0.776	0.220	1.300	4.907	18.396	12.1	6.8	3.7
(b) = (a) + TTT [ 77 ]	23.12	0.704	0.286	0.439	0.862	1.706	49.5	49.1	49.1
(c) = (b) + context pairing	23.78	0.728	0.264	0.626	1.237	2.458	50.8	50.8	50.7
(d) = (b) + cross-view attn.	24.13	0.742	0.252	1.267	5.178	17.745	50.4	50.4	50.5

Table 4: Ablation study from TTT to TTB on DL3DV. ( light =worst, dark =best).

Table 8

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Training data	GPUs	Pretrain source
LVSM [ 25 ]	Objaverse / Re10K	64 $\times$ A100	from scratch
Efficient-LVSM [ 22 ]	Re10K	64 $\times$ A100	from scratch
DVSM [ 58 ]	DL3DV / Re10K	64 $\times$ A100	from scratch
LaCT [ 77 ]	Objaverse / DL3DV	64 $\times$ A100	from scratch
LongLRM [ 12 ]	DL3DV	64 $\times$ A100	from scratch
tttLRM [ 63 ]	Objaverse / DL3DV	64 $\times$ A100	LaCT [ 77 ] ckpt

Appendix

Table 7: Training settings across methods. “Multi-dataset” indicates training on a curated mixture beyond a single benchmark. “DVSM ckpt” refers to initializing from the corresponding DVSM [ 58 ] checkpoint released for that benchmark. “–” marks compute resources that are not explicitly reported by the corresponding paper.

Method	Reconstruction	Rendering (per token)	Scene representation
LVSM (decoder-only) [ 25 ]	—	$\mathcal{O}(V\cdot L)$	context tokens
LVSM (enc.-dec.) [ 25 ]	$\mathcal{O}(V^{2}\cdot L^{2})$	$\mathcal{O}(L_{s})$	latent scene tokens
Efficient-LVSM [ 22 ]	$\mathcal{O}(V\cdot L^{2})$	$\mathcal{O}(V\cdot L)$	KV-cache
DVSM [ 58 ]	$\mathcal{O}(V^{2}\cdot L^{2})$	$\mathcal{O}(V\cdot L)$	KV-cache
LaCT [ 77 ]	$\mathcal{O}(V\cdot L)$	$\mathcal{O}(1)$	fast weights
LongLRM [ 12 ]	$\mathcal{O}(V\cdot L)$	GS rasterizer	3D Gaussians

Appendix

Table 8: Asymptotic complexity comparison.

V

denotes the number of context views and

L

the number of tokens per view. Reconstruction is reported per scene, rendering per novel-view token. “GS rasterizer” denotes that rendering is handled by a Gaussian splatting rasterizer rather than a neural module, which is not directly comparable in

\mathcal{O}(\cdot)

form. We only list the complexity of the main model layer for global scene understanding. LongLRM still employs a few cross-view attention layers with

\mathcal{O}(V^{2}\cdot L^{2})

complexity. LaCT, tttLRM, LagerNVS, and ours also employ intra-view attention with

\mathcal{O}(V\cdot L^{2})

reconstruction complexity and

\mathcal{O}(L)

per-token rendering complexity.

Method	# blocks	Latent dim	Cross-view sublayer
LVSM [ 25 ]	24	768	cross-view attention
Efficient-LVSM [ 22 ]	24	1024	cached KV cross-attention
DVSM [ 58 ]	12	768	cross-view attention (cached KV)
LaCT [ 77 ]	24	768	large-chunk TTT layer
LongLRM [ 12 ]	24	768	Mamba2 + attention hybrid
tttLRM [ 63 ]	24	768	large-chunk TTT layer

Appendix

Table 9: Architectural composition. “Cross-view sublayer” indicates the module that exchanges information across views in each block. The depth column is the number of decoder blocks, with the encoder depth listed in parentheses for encoder-decoder methods. The latent-dim column reports the decoder width, with the encoder width also given for encoder-decoder methods.

	Method	$V{=}16$	$V{=}32$	$V{=}64$	$V{=}128$	$V{=}256$
PSNR $\uparrow$	DVSM	22.83	24.68	26.23	27.20	27.56
	TTB-single	23.26	24.84	25.60	25.82	25.78
	TTB-duplicate	23.26	24.84	26.13	27.14	27.74
Recon-time $\downarrow$	DVSM	0.136	0.389	1.258	4.815	18.106
	TTB-single	0.337	0.740	1.911	6.065	20.549
	TTB-duplicate	0.333	1.048	3.578	13.557	52.064

Appendix

Table 11: Inference at scale on DL3DV. As the number of context views

V

grows, TTB-single —a single fast-weight MLP per TTB layer that absorbs all

V

views—suffers capacity saturation : its PSNR plateaus around

V{=}64

and even regresses thereafter. TTB-duplicate , which maintains

K

local copies of the per-layer fast-weight state, each baked from a spatially-local group of views, prevents this saturation and tracks DVSM’s quality scaling while preserving

\mathcal{O}(1)

per-frame rendering. Best per column is shaded red, second-best orange.

Method	Venue	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	#params $\downarrow$	Recon-time $\downarrow$	Rend-fps $\uparrow$	Infer-mem $\downarrow$	Train-time/it $\downarrow$	Train-mem $\downarrow$
LVSM [ 25 ]	ICLR’25	22.24	0.691	0.352	165M	15s	20	9G	73s	42G
LaCT [ 77 ]	ICLR’26	26.90	0.837	0.185	379M	3.7s	10	23G	36s	72G
DVSM ${}_{\text{ps8}}$	arXiv’26	29.13	0.893	0.129	163M	15s	2	20G	133s	65G
TTB– (ours)	–	28.12	0.867	0.158	244M	3.7s	23	18G	26s	68G
TTB (ours)	–	28.92	0.882	0.145	271M	18s	23	18G	90s	76G

Appendix

Table 14: Efficiency comparison on DL3DV at

544{\times}960

resolution with 32 input views. Recon-time is the processing time of a single scene. Rend-fps indicates the number of frames rendered per second without batching. Infer-mem is the peak GPU memory usage when measuring the reconstruction and rendering. Train-time/it is the sum of model forward and backward time per iteration under training mode with a batch containing 32 context views. Train-mem is the peak memory usage when measuring training time. The actual training randomly samples different number of input views per scene rather than a fixed number. We highlight cells with numbers similar to the most efficient one under each column.

$V$	Method	Recon	FPS	$N{=}1$	$N{=}2$	$N{=}5$	$N{=}10$	$N{=}30$	$N{=}100$
16	DVSM	4.4	4	4.7	4.9	5.7	6.9	11.9	29.4
	TTB–	1.9	23	1.9	2.0	2.1	2.3	3.2	6.2
	TTB	5.4	23	5.4	5.5	5.6	5.8	6.7	9.7
32	DVSM	15.0	2	15.5	16.0	17.5	20.0	30.0	65.0
	TTB–	3.7	23	3.7	3.8	3.9	4.1	5.0	8.0
	TTB	18.0	23	18.0	18.1	18.2	18.4	19.3	22.3

Appendix

Table 15: End-to-end time for rendering

N

novel views (seconds, DL3DV at 960p), computed as reconstruction time plus

N/\mathrm{FPS}

from Tab. 1 . TTB overtakes DVSM after 5–7 views at every

V

, while TTB– leads from

N{=}1

Figure 7: Scene representation size as

V

grows at

544{\times}960

resolution, measured on a single A100. The DVSM [ 58 ] KV-cache grows linearly in

V

(hollow points: linear extrapolation where DVSM ran out of 80 GB memory), TTB’s baked fast weights stay constant, and TTB-duplicate grows with the number of copies

K{=}V/16

while remaining

21{\times}

smaller than the KV-cache.

Figure 8: Additional qualitative comparison on DL3DV. 64 context views.

Figure 9: Qualitative comparison on Re10K. Two-view setting with sparse input.

Figure 10: Zero-shot qualitative results on MipNeRF360. 64 context views.

Figure 11: Zero-shot qualitative results on Free. 64 context views.

Figure 12: Zero-shot qualitative results on Hike. 64 context views.

Novel View Synthesis

Novel View Synthesis creates images of a scene from camera viewpoints not present in the input, using learned geometry and appearance models.

See all 306 papers

DVSM: Decoder-only View Synthesis Model Done Right
May 28, 2026Cheng Sun, Jaesung Choe, Min-Hung Chen +2Novel View Synthesis Multi-View Image Synthesis
NVIDIA · National Taiwan University
Scaling View Synthesis Transformers
Feb 24, 2026Evan Kim, Hyunwoo Ryu, Thomas W. Mitchel +1Novel View Synthesis Neural Scaling Laws
MIT · Adobe
GLoD-LVSM: Learning Geometry-Grounded Continuous-LoD Scene Representations for Large View Synthesis Models
Oct 14, 2026Weikai Lin, Nicholas Milef, Carl S Marshall +33D Scene Representation Novel View Synthesis
University of Rochester · Reality Labs, Meta

May 28, 2026J/K move · Enter open · S save

DVSM: Decoder-only View Synthesis Model Done Right

Cheng Sun, Jaesung Choe, Min-Hung Chen, Ryo Hachiuma+1

NVIDIA · National Taiwan University

Novel View Synthesis Multi-View Image Synthesis

Open paper AskPDF arXiv

Recent Large View Synthesis Models (LVSMs) advocate an encoder-decoder architecture that separates reconstruction and rendering into distinct networks. We re-examine this design. Through controlled experiments, we show that a decoder-only architecture, which represents scenes implicitly as a KV-cache, outperforms encoder-decoder variants while using fewer parameters at identical rendering complexity. Further analysis shows that sharing weights between the color-input reconstruction network and the camera-only rendering network better aligns their features at the same viewpoint, facilitating image synthesis. Building on this finding, our model, dubbed DVSM, further incorporates foundation model priors and stage-wise patch sizing for an improved efficiency-quality tradeoff. Our results establish a new state of the art for novel-view synthesis across multiple benchmarks, in some cases even outperforming per-scene-optimized 3DGS under dense input views.

Abstract

Novel View Synthesis

Explore similar work

DVSM: Decoder-only View Synthesis Model Done Right

Scaling View Synthesis Transformers

GLoD-LVSM: Learning Geometry-Grounded Continuous-LoD Scene Representations for Large View Synthesis Models