ZipMap: Linear-Time Stateful 3D Reconstruction via Test-Time Training

Haian Jin; Rundi Wu; Tianyuan Zhang; Ruiqi Gao; Jonathan T. Barron; Noah Snavely; Aleksander Holynski

cs.CVMar 4, 2026

ZipMap: Linear-Time Stateful 3D Reconstruction via Test-Time Training

Authors: Haian Jin, Rundi Wu, Tianyuan Zhang, Ruiqi Gao, Jonathan T. Barron, Noah Snavely, Aleksander Holynski

Organizations: Google DeepMind · Cornell University · Massachusetts Institute of Technology

Abstract

Feed-forward transformer models have driven rapid progress in 3D vision, but state-of-the-art methods such as VGGT and $π^3$ have a computational cost that scales quadratically with the number of input images, making them inefficient when applied to large image collections. Sequential-reconstruction approaches reduce this cost but sacrifice reconstruction quality. We introduce ZipMap, a stateful feed-forward model that achieves linear-time, bidirectional 3D reconstruction while matching or surpassing the accuracy of quadratic-time methods. ZipMap employs test-time training layers to zip an entire image collection into a compact hidden scene state in a single forward pass, enabling reconstruction of over 700 frames in under 10 seconds on a single H100 GPU, more than $20\times$ faster than state-of-the-art methods such as VGGT. Moreover, we demonstrate the benefits of having a stateful representation in real-time scene-state querying and its extension to sequential streaming reconstruction.

Figures & tables

Figure 2: Method Overview. ZipMap is a stateful feed-forward model with local window attention and large-chunk TTT layers [ 65 , 86 ] . Given $N$ input images, a single linear-time pass predicts camera poses, depth maps, and point maps while storing a compact scene representation in TTT fast weights, which can be queried in real time at novel cameras to synthesize new-view point maps.

Figure 3: Example reconstruction results A sparse subset of input images are shown on the left, and a visualization of the output 3D reconstructions are shown on the right. Note that our method performs well on challenging cases like long sequence inputs, dynamic scenes and internet photo collections.

Method	RealEstate10K			Co3Dv2
Method	AUC@5 $\uparrow$	AUC@15 $\uparrow$	AUC@30 $\uparrow$	AUC@5 $\uparrow$	AUC@15 $\uparrow$	AUC@30 $\uparrow$
$\mathcal{O}(N^{2})$ Inference Speed
Fast3R [ 79 ]	22.36	46.71	61.68	31.05	59.63	73.43
FLARE [ 85 ]	38.47	67.02	80.01	23.84	57.78	73.99
VGGT [ 68 ]	38.71	66.46	78.89	67.84	83.95	89.99
$\pi^{3}$ [ 75 ]	63.10	80.31	87.40	57.12	79.86	87.93

Table 1: Camera Pose Estimation: RealEstate10K [ 89 ] and Co3Dv2 [ 48 ] . We report pose AUC under angular error thresholds of 5/15/30 degrees. All methods have seen Co3Dv2 during training. CUT3R and TTT3R additionally use RealEstate10K for training, while the remaining methods do not.

Method	Sintel			TUM-dynamics			ScanNet (seen)
Method	ATE $\downarrow$	RPE trans $\downarrow$	RPE rot $\downarrow$	ATE $\downarrow$	RPE trans $\downarrow$	RPE rot $\downarrow$	ATE $\downarrow$	RPE trans $\downarrow$	RPE rot $\downarrow$
$\mathcal{O}(N^{2})$ Inference Speed
Fast3R [ 79 ]	0.371	0.298	13.750	0.090	0.101	1.425	0.155	0.123	3.491
FLARE [ 85 ]	0.207	0.090	3.015	0.026	0.013	0.475	0.064	0.023	0.971
VGGT [ 68 ]	0.172	0.061	0.471	0.012	0.010	0.309	0.035	0.015	0.376
$\pi^{3}$ [ 75 ]	0.073	0.038	0.288	0.014	0.009	0.307	0.030	0.013	0.345

Table 2: Camera Pose Estimation: Sintel [ 9 ] , TUM-dynamics [ 61 ] and ScanNet [ 14 ] . We measure the distance error of rotation/translation. All methods have seen ScanNet or ScanNet++ [ 83 ] in training. None has seen Sintel or TUM-dynamics.

Method	7-Scenes						NRGBD
	Acc. $\downarrow$		Comp. $\downarrow$		NC. $\uparrow$		Acc. $\downarrow$		Comp. $\downarrow$		NC. $\uparrow$
	Mean	Med.	Mean	Med.	Mean	Med.	Mean	Med.	Mean	Med.	Mean	Med.
Sparse View
Fast3R [ 79 ]	0.095	0.065	0.144	0.089	0.673	0.759	0.135	0.091	0.163	0.104	0.759	0.877
CUT3R [ 76 ]	0.080	0.055	0.102	0.066	0.711	0.811	0.098	0.038	0.075	0.029	0.830	0.974
TTT3R [ 13 ]	0.098	0.062	0.159	0.107	0.681	0.768	0.101	0.039	0.076	0.029	0.826	0.973

Table 3: Point Map Estimation: 7-Scenes [ 58 ] and NRGBD [ 3 ] . Metrics are the same as in Table 4 . Keyframes are selected every 200 (sparse) and 40 (dense) frames for 7-Scenes, and every 500 (sparse) and 100 (dense) frames for NRGBD.

Method	DTU						ETH3D
	Acc. $\downarrow$		Comp. $\downarrow$		N.C. $\uparrow$		Acc. $\downarrow$		Comp. $\downarrow$		N.C. $\uparrow$
	Mean	Med.	Mean	Med.	Mean	Med.	Mean	Med.	Mean	Med.	Mean	Med.
$\mathcal{O}(N^{2})$ Inference Speed
Fast3R [ 79 ]	3.340	1.919	2.929	1.125	0.671	0.755	0.832	0.691	0.978	0.683	0.667	0.766
FLARE [ 85 ]	2.541	1.468	3.174	1.420	0.684	0.774	0.464	0.338	0.664	0.395	0.744	0.864
VGGT [ 68 ]	1.308	0.761	1.929	1.015	0.665	0.751	0.270	0.174	0.304	0.180	0.841	0.942

Table 4: Point Map Estimation: DTU [ 26 ] and ETH3D [ 55 ] . Following

\pi^{3}

[ 75 ] , we report the mean and median of accuracy (Acc.), completeness (Comp.), and normal-consistency (NC.) for keyframes selected every 5 images.

		Sintel		Bonn		KITTI
Method	Params	AbsRel $\downarrow$	$\delta<1.25$ $\uparrow$	AbsRel $\downarrow$	$\delta<1.25$ $\uparrow$	AbsRel $\downarrow$	$\delta<1.25$ $\uparrow$
$\mathcal{O}(N^{2})$ Inference Speed
Fast3R [ 79 ]	648M	0.638	0.422	0.194	0.772	0.138	0.834
FLARE [ 85 ]	1.40B	0.729	0.336	0.152	0.790	0.356	0.570
VGGT [ 68 ]	1.26B	0.298	0.643	0.055	0.971	0.073	0.965
$\pi^{3}$ [ 75 ]	959M	0.228	0.672	0.051	0.975	0.038	0.986

Table 5: Video Depth Estimation: Sintel [ 9 ] , Bonn [ 41 ] and KITTI [ 20 ] . We report results under scale-only alignment; results using joint scale-and-shift alignment are provided in the Appendix.

Figure 4: Long-sequence camera evaluation on DL3DV. We evaluate camera pose accuracy (ATE $\downarrow$ ) on the DL3DV test set [ 37 ] under two protocols: Left : increasing scene scale by using the first $N$ frames of each sequence; Right : increasing view density by uniformly subsampling $N$ frames along a fixed trajectory. Our method maintains low error and matches quadratic-time baselines ( $\pi^{3}$ , VGGT) while other linear-time methods (CUT3R, TTT3R) degrade significantly as $N$ grows.

Method	Acc. $\downarrow$		Comp. $\downarrow$		N.C. $\uparrow$
Method	Mean	Med.	Mean	Med.	Mean	Med.
Ours	0.337	0.224	0.357	0.217	0.810	0.918
Ours w/o gated unit	0.354	0.251	0.381	0.234	0.802	0.901
Ours w/o Newton–Schulz	0.408	0.283	0.430	0.249	0.787	0.898
Ours w/ global TTT lr (0.1)	0.411	0.303	0.490	0.317	0.779	0.886
Ours w/ global TTT lr (1.0)	0.464	0.366	0.537	0.343	0.782	0.890

Table 6: Ablation Study for TTT key components. We evaluate point-map estimation on ETH3D [ 55 ] . All variants are trained with reduced compute and a smaller data scale.

Figure 5: Querying Unseen Structure. Left: input images (a), GT images at query poses (b), and our predicted depth at those poses (c). Middle: point cloud reconstructed from input images only. Right: point cloud after querying (right column), where the queried point cloud is merged with the input image point cloud. This demonstrates our model’s ability to infer common 3D structure (e.g., walls, floors, and ground) in the unseen regions, thereby indicating an understanding of basic 3D scene priors.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Complexity	Params	Time to Reconstruct $N$ Frames (seconds)
			$N=$ 5	10	25	50	100	200	300	400	500	750
VGGT [ 68 ]	$\mathcal{O}(N^{2})$	1.26B	0.102	0.194	0.569	1.524	4.689	16.040	34.022	58.842	90.389	200.364
$\pi^{3}$ [ 75 ]	$\mathcal{O}(N^{2})$	959M	0.087	0.157	0.450	1.186	3.604	12.190	25.765	44.464	68.255	151.159
CUT3R [ 76 ]	$\mathcal{O}(N)$	793M	0.206	0.413	1.018	2.056	4.088	8.222	12.430	16.618	21.025	31.246
TTT3R [ 13 ]	$\mathcal{O}(N)$	793M	0.206	0.411	1.033	2.036	4.128	8.267	12.435	16.511	20.767	31.197
Ours	$\mathcal{O}(N)$	1.40B	0.125	0.183	0.383	0.712	1.362	2.681	4.017	5.348	6.671	9.999

Appendix

Table 7: Runtime evaluation. Inference time (in seconds) as a function of the number of input images

N

. VGGT and

\pi^{3}

scale quadratically with

N

, resulting in slow speeds when

N

is large. CUT3R, TTT3R, and our method scale linearly with

N

, with ours being the fastest for dense input frames.

Figure 6: Qualitative comparison. Point cloud reconstructions of scenes from the ETH3D and DTU datasets.

Figure 7: Querying the Scene State. The left panels show: input images (a), GT RGB at query poses (b), our RGB predictions (c), GT depth (d), and predicted depth (e). The middle panels visualize the 3D point clouds reconstructed from the input images. The right panels show point clouds attained solely by querying the scene state . The close visual match between these two point clouds indicates that the learned scene state faithfully captures the geometry and appearance of the input scene.

	Sintel		Bonn		KITTI		NYU-v2
Method	AbsRel $\downarrow$	$\delta<1.25$ $\uparrow$	AbsRel $\downarrow$	$\delta<1.25$ $\uparrow$	AbsRel $\downarrow$	$\delta<1.25$ $\uparrow$	AbsRel $\downarrow$	$\delta<1.25$ $\uparrow$
MASt3R [ 34 ]	0.413	0.569	0.123	0.833	0.077	0.948	0.110	0.865
MonST3R [ 84 ]	0.402	0.525	0.069	0.954	0.098	0.895	0.094	0.887
Fast3R [ 79 ]	0.544	0.509	0.169	0.796	0.120	0.861	0.093	0.898
CUT3R [ 76 ]	0.418	0.520	0.058	0.967	0.097	0.914	0.081	0.914
FLARE [ 85 ]	0.606	0.402	0.130	0.836	0.312	0.513	0.089	0.898

Appendix

Table 8: Monocular Depth Estimation. We evalute the frame-independent monocular depth estimation on the Sintel [ 9 ] , Bonn [ 41 ] , KITTI [ 20 ] and NYU-v2 [ 59 ] datasets.

		Sintel		Bonn		KITTI
Method	Params	AbsRel $\downarrow$	$\delta<1.25$ $\uparrow$	AbsRel $\downarrow$	$\delta<1.25$ $\uparrow$	AbsRel $\downarrow$	$\delta<1.25$ $\uparrow$
$\mathcal{O}(N^{2})$ Inference Speed
Fast3R [ 79 ]	648M	0.518	0.486	0.196	0.768	0.139	0.808
FLARE [ 85 ]	1.40B	0.791	0.358	0.142	0.797	0.357	0.579
VGGT [ 68 ]	1.26B	0.226	0.683	0.049	0.974	0.059	0.961
$\pi^{3}$ [ 75 ]	959M	0.206	0.735	0.045	0.976	0.036	0.986

Appendix

Table 9: Video Depth Estimation: Sintel [ 9 ] , Bonn [ 41 ] and KITTI [ 20 ] . We have reported results under scale-only alignment in the main paper. Here we further report results using joint scale-and-shift alignment.

Method	Sintel			TUM-dynamics			ScanNet (seen)
Method	ATE $\downarrow$	RPE trans $\downarrow$	RPE rot $\downarrow$	ATE $\downarrow$	RPE trans $\downarrow$	RPE rot $\downarrow$	ATE $\downarrow$	RPE trans $\downarrow$	RPE rot $\downarrow$
Ours w/ ref	0.125	0.058	0.420	0.012	0.009	0.309	0.034	0.015	0.398
Ours w/o ref	0.132	0.066	0.438	0.012	0.010	0.310	0.034	0.015	0.385

Appendix

Table 10: Ablation: Removing the Reference View (Camera pose estimation).

Figure 8: Long-sequence camera estimation. We evaluate camera ATE on the ScanNet-v2 and DL3DV datasets by taking the first $N$ frames of each test sequence and gradually increasing $N$ . We see that, when the input sequence length becomes long, removing the reference view and fine-tuning with the affine-invariant camera loss from $\pi^{3}$ [ 75 ] (“Ours w/o ref”) improves the camera pose estimation accuracy compared to the reference-based variant (“Ours w/ ref”).

	Sintel		Bonn		KITTI
Method	AbsRel $\downarrow$	$\delta<1.25$ $\uparrow$	AbsRel $\downarrow$	$\delta<1.25$ $\uparrow$	AbsRel $\downarrow$	$\delta<1.25$ $\uparrow$
CUT3R [ 76 ]	0.432	0.510	0.072	0.951	0.152	0.805
TTT3R [ 13 ]	0.426	0.522	0.061	0.970	0.149	0.812
Ours-streaming	0.273	0.638	0.067	0.965	0.100	0.903

Appendix

Table 13: Streaming Video Depth Estimation. We report results under scale-only alignment on Sintel [ 9 ] , Bonn [ 41 ] and KITTI [ 20 ] .

Method	Sintel			Co3Dv2
Method	ATE $\downarrow$	RPE trans $\downarrow$	RPE rot $\downarrow$	AUC@5 $\uparrow$	AUC@15 $\uparrow$	AUC@30 $\uparrow$
CUT3R [ 76 ]	0.2160	0.0710	0.6220	24.88	56.28	71.72
TTT3R [ 13 ]	0.2040	0.0850	0.6900	22.61	53.49	69.46
Ours-Streaming	0.1593	0.0655	0.7508	45.38	72.58	83.12

Appendix

Table 14: Streaming Camera Pose Estimation: Sintel [ 9 ] and Co3Dv2 [ 48 ] . For Sintel, we report ATE and RPE translation/rotation errors. For Co3Dv2, we report pose AUC under angular error thresholds of 5/15/30 degrees.

Method	DTU			ETH3D			NRGBD-dense
Method	Acc. $\downarrow$	Comp. $\downarrow$	N.C. $\uparrow$	Acc. $\downarrow$	Comp. $\downarrow$	N.C. $\uparrow$	Acc. $\downarrow$	Comp. $\downarrow$	N.C. $\uparrow$
CUT3R [ 76 ]	5.045	6.437	0.666	0.593	0.747	0.754	0.065	0.036	0.812
TTT3R [ 13 ]	5.337	6.593	0.666	0.763	0.881	0.739	0.074	0.037	0.803
Ours-streaming	4.091	3.495	0.693	0.614	0.941	0.750	0.038	0.028	0.836

Appendix

Table 15: Streaming Point Map Reconstruction Comparison. We report the results on DTU [ 26 ] , ETH3D [ 55 ] , and NRGBD-dense.

Figure 9: Long-sequence video depth estimation. We evaluate on the ScanNet-v2 dataset by taking the first $N$ frames of each test sequence and gradually increasing $N$ .

Figure 10: Long-sequence 3D point estimation. We evaluate on the 7-Scenes [ 58 ] dataset under two cases. Left : increasing scene scale by using the first $N$ frames of each sequence; Right : increasing view density by uniformly subsampling $N$ frames.

Method	6 Views			16 Views
Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
NoPoSplat [ 82 ]	15.92	0.416	0.541	15.47	0.361	0.606
Flare [ 85 ]	15.35	0.407	0.573	13.21	0.348	0.695
AnySplat [ 28 ]	18.32	0.524	0.329	21.85	0.670	0.250
Ours	15.60	0.414	0.486	17.65	0.459	0.374

Appendix

Table 16: NVS Evaluation on Mip-NeRF360 [ 4 ] . Baseline results are taken from Tab. 1 of the AnySplat paper. ZipMap is competitive with several baselines. However, as noted in our Limitations, our query results are primarily designed to generate colored point clouds rather than high-fidelity novel view synthesis. Accordingly, NVS is not our main focus, and we do not claim state-of-the-art performance compared with explicit 3DGS-based methods.

Explore similar work

VGG-T $^3$ : Offline Feed-Forward 3D Reconstruction at Scale
Feb 26, 2026Sven Elflein, Ruilong Li, Sérgio Agostinho +43D Reconstruction Feed-Forward 3D Reconstruction
NVIDIA · Vector Institute · University of Toronto
OTT3R: Multi-View 3D Reconstruction and Fast Dataset Generation at 1% Compute
Sep 28, 2026Brandon Leblanc, Charalambos PoullisMulti-View 3D Reconstruction Pseudo-Labeling
Immersive and Creative Technologies Lab, Concordia University, Montreal, Canada
Spark3R: Asymmetric Token Reduction Makes Fast Feed-Forward 3D Reconstruction
May 7, 2026Zecheng Tang, Jiaye Fu, Qiankun Gao +53D Reconstruction Efficient ViTs
School of Electronic and Computer Engineering, Peking University, Shenzhen 518055, China

Feb 26, 2026J/K move · Enter open · S save

VGG-T $^3$ : Offline Feed-Forward 3D Reconstruction at Scale

Sven Elflein, Ruilong Li, Sérgio Agostinho, Zan Gojcic+3

NVIDIA · Vector Institute · University of Toronto

3D Reconstruction Feed-Forward 3D Reconstruction 3D Scene Representation

Open paper AskPDF arXiv

We present a scalable 3D reconstruction model that addresses a critical limitation in offline feed-forward methods: their computational and memory requirements grow quadratically w.r.t. the number of input images. Our approach is built on the key insight that this bottleneck stems from the varying-length Key-Value (KV) space representation of scene geometry, which we distill into a fixed-size Multi-Layer Perceptron (MLP) via test-time training. VGG-T

^3

(Visual Geometry Grounded Test Time Training) scales linearly w.r.t. the number of input views, similar to online models, and reconstructs a

1k

image collection in just

54

seconds, achieving a

11.6\times

speed-up over baselines that rely on softmax attention. Since our method retains global scene aggregation capability, our point map reconstruction error outperforming other linear-time methods by large margins. Finally, we demonstrate visual localization capabilities of our model by querying the scene representation with unseen images.

Abstract

Explore similar work

VGG-T3^33: Offline Feed-Forward 3D Reconstruction at Scale

OTT3R: Multi-View 3D Reconstruction and Fast Dataset Generation at 1% Compute

Spark3R: Asymmetric Token Reduction Makes Fast Feed-Forward 3D Reconstruction

VGG-T $^3$ : Offline Feed-Forward 3D Reconstruction at Scale