tttLRM: Test-Time Training for Long Context and Autoregressive 3D Reconstruction

Chen Wang; Hao Tan; Wang Yifan; Zhiqin Chen; Yuheng Liu; Kalyan Sunkavalli; Sai Bi; Lingjie Liu

cs.CVFeb 23, 2026

tttLRM: Test-Time Training for Long Context and Autoregressive 3D Reconstruction

Authors: Chen Wang, Hao Tan, Wang Yifan, Zhiqin Chen, Yuheng Liu, Kalyan Sunkavalli, Sai Bi, Lingjie Liu, +1 more

Organizations: University of Pennsylvania · Adobe Research · UCI

Abstract

We propose tttLRM, a novel large 3D reconstruction model that leverages a Test-Time Training (TTT) layer to enable long-context, autoregressive 3D reconstruction with linear computational complexity, further scaling the model's capability. Our framework efficiently compresses multiple image observations into the fast weights of the TTT layer, forming an implicit 3D representation in the latent space that can be decoded into various explicit formats, such as Gaussian Splats (GS) for downstream applications. The online learning variant of our model supports progressive 3D reconstruction and refinement from streaming observations. We demonstrate that pretraining on novel view synthesis tasks effectively transfers to explicit 3D modeling, resulting in improved reconstruction quality and faster convergence. Extensive experiments show that our method achieves superior performance in feedforward 3D Gaussian reconstruction compared to state-of-the-art approaches on both objects and scenes.

Figures & tables

Figure 2: Given a set of posed input images, tttLRM encodes them into tokens (green boxes) after patchifying. The input tokens are fed into the LaCT block (shown in the blue frame) where fast weights are updated accordingly. Another set of virtual tokens (blue boxes) are used to query the updated fast weights, and decoded into 3D representations like 3DGS for high-quality novel view synthesis.

Figure 3: Illustration of distributed feedforward reconstruction training. First, image tokens are sharded across GPUs, and each GPU predicts Gaussians for its assigned virtual views after the fast weights are synchronized. The predicted Gaussians are then gathered to construct the full scene, after which each GPU renders a subset of novel views and computes its respective losses. Gradients are finally all reduced and backpropagated across all devices.

Figure 4: Qualitative comparison between our method and baseline approaches. Our model reconstructs the 3DGS scene with higher fidelity than both optimization-based and feedforward baselines, as also reflected in the PSNR metrics. Please zoom in for a better comparison.

Figure 5: We demonstrate that our high-resolution $1024\times 1024$ 3DGS tttLRM can be effectively used for image-to-3D generation when combined with a multi-view generator. Our model enables the reconstruction of fine-grained, photorealistic details e.g . , hair, fur, and text, from the input images. Video results are provided in the supplemental material.

Figure 6: We show that tttLRM, as a general framework, can also interpret the latent 3D memory into formats besides 3DGS. In this experiment, we use a set of triplane tokens to query the fast weights and then fine-tune the model for triplane-based NeRF reconstruction. We visualize the resulting triplanes and present the corresponding renderings and depth maps for 4 views at a resolution of $512\times 512$ .

Method	Resolution	Views	Time (s) $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
GS-LRM [ 67 ]	256 $\times$ 256	8	0.1	31.55	0.964	0.028
Ours	256 $\times$ 256	8	0.1	33.14	0.972	0.024
GS-LRM [ 67 ]	512 $\times$ 512	8	0.7	32.83	0.969	0.029
Ours		8	0.3	34.02	0.974	0.025
GS-LRM [ 67 ]		16	2.5	33.55	0.976	0.023
Ours		16 (10 V.)	0.8	34.67	0.978	0.022

Table 1: Comparison between our method and GS-LRM [ 67 ] on the GSO dataset under different resolutions and numbers of input views. Our method consistently outperforms GS-LRM in both inference speed and reconstruction quality, and also shows strong generalization ability. V. denotes the number of virtual views used to query the fast weight, which equals input views unless noted.

Views	Method	Time $\downarrow$	DL3DV-140			Tanks&Temples
Views	Method	Time $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
16	3D GS 30k	13m	21.20	0.708	0.264	16.76	0.598	0.334
	Mip-Splatting 30k	13m	20.88	0.712	0.274	16.82	0.616	0.332
	Scaffold-GS 30k	16m	22.13	0.738	0.250	17.02	0.634	0.321
	Long-LRM (16v model)	0.4s	22.66	0.740	0.292	17.51	0.555	0.408
	Ours (single model)	3.6s	23.60	0.784	0.255	18.15	0.613	0.360

Table 2: Quantitative comparison on both DL3DV-140 and Tanks&Temples datasets under different numbers of input views. Our method surpasses previous feedforward methods and is comparable with optimization-based methods. Note that Long-LRM trains a separate model for each input view, while we are a single model across all input views. Our model can be linearly accelerated with multiple GPUs , here we report time on 1 A100.

3D Rep.	Type	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
GS	w/o Pretrain	32.77	0.026	0.969
GS	w Pretrain	33.14	0.024	0.972
Triplane	w/o Pretrain	26.40	0.903	0.093
Triplane	w Pretrain	27.87	0.925	0.075

Table 3: Leveraging pretrained knowledge from novel view synthesis tasks improves the final 3D reconstruction quality across different 3D representations.

Figure 7: Our 3DGS reconstruction model leverages pretraining with LVSM on novel view synthesis tasks, which significantly accelerates learning and leads to better performance, compared to training from scratch.

	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Predict & Merge	21.50	0.891	0.318
Ours	23.63	0.904	0.259

Table 4: Although progressive GS prediction with merging provides more efficient computation, the reconstruction quality is degraded due to accumulated errors (compared on 32 views under 1K iterations finetuning).

Muon	Opacity+Depth	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	Opacity $>0.001$
✗	✗	20.44	0.649	0.295	96%
✓	✗	20.68	0.661	0.290	97%
✓	✓	20.76	0.666	0.285	47%

Table 5: Ablation on

32

view

256\times 144

input with the same iterations across settings.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
w/o selective	24.81	0.814	0.225
w selective	24.95	0.818	0.223

Appendix

Table 6: Training-free selective update considering history fast weights can further enhance our AR model.

Figure 8: Time comparison of 3 Attention layers vs 24 layers of LaCT blocks under different numbers of tokens.

Views	Method	Time $\downarrow$	DL3DV-140			Tanks&Temples
Views	Method	Time $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
32	3D GS 30k	13m	23.60	0.779	0.213	18.10	0.688	0.269
	Mip-Splatting 30k	13m	23.32	0.784	0.217	18.39	0.700	0.262
	Scaffold-GS 30k	16m	24.77	0.805	0.205	18.41	0.691	0.290
	Long-LRM (32v model)	1s	24.10	0.783	0.254	18.38	0.601	0.363
	Long-LRM (w/ 3-step optim)	12s	24.99	0.809	0.243	18.69	0.623	0.360

Appendix

Table 7: More quantitative comparison on both DL3DV-140 and Tanks&Temples datasets under 32/64 input views. Our method surpasses previous feedforward methods and can further surpass optimization-based methods with a few steps post-optimization. Note that Long-LRM trains a separate model for each input view, while we are a single model across all input views. Our model can be linearly accelerated with multiple GPUs , here we report time on a single Nvidia A100 80GB GPU.

Explore similar work

VGG-T $^3$ : Offline Feed-Forward 3D Reconstruction at Scale
Feb 26, 2026Sven Elflein, Ruilong Li, Sérgio Agostinho +43D Reconstruction Feed-Forward 3D Reconstruction
NVIDIA · Vector Institute · University of Toronto
TTSA3R: Training-Free Temporal-Spatial Adaptive Persistent State for Streaming 3D Reconstruction
Jan 30, 2026Zhijie Zheng, Xinhao Xiang, Jiawei ZhangStreaming 3D Reconstruction 3D Reconstruction
University of California, Davis · CA 95616, USA
Scal3R: Scalable Test-Time Training for Large-Scale 3D Reconstruction
Apr 9, 2026Tao Xie, Peishan Yang, Yudong Jin +83D Reconstruction Feed-Forward 3D Reconstruction
Zhejiang University · State Key Lab of CAD&CG · Horizon Robotics +1

Feb 26, 2026J/K move · Enter open · S save

VGG-T $^3$ : Offline Feed-Forward 3D Reconstruction at Scale

Sven Elflein, Ruilong Li, Sérgio Agostinho, Zan Gojcic+3

NVIDIA · Vector Institute · University of Toronto

3D Reconstruction Feed-Forward 3D Reconstruction 3D Scene Representation

Open paper AskPDF arXiv

We present a scalable 3D reconstruction model that addresses a critical limitation in offline feed-forward methods: their computational and memory requirements grow quadratically w.r.t. the number of input images. Our approach is built on the key insight that this bottleneck stems from the varying-length Key-Value (KV) space representation of scene geometry, which we distill into a fixed-size Multi-Layer Perceptron (MLP) via test-time training. VGG-T

^3

(Visual Geometry Grounded Test Time Training) scales linearly w.r.t. the number of input views, similar to online models, and reconstructs a

1k

image collection in just

54

seconds, achieving a

11.6\times

speed-up over baselines that rely on softmax attention. Since our method retains global scene aggregation capability, our point map reconstruction error outperforming other linear-time methods by large margins. Finally, we demonstrate visual localization capabilities of our model by querying the scene representation with unseen images.

Abstract

Explore similar work

VGG-T3^33: Offline Feed-Forward 3D Reconstruction at Scale

TTSA3R: Training-Free Temporal-Spatial Adaptive Persistent State for Streaming 3D Reconstruction

Scal3R: Scalable Test-Time Training for Large-Scale 3D Reconstruction

VGG-T $^3$ : Offline Feed-Forward 3D Reconstruction at Scale