ZipMap: Linear-Time Stateful 3D Reconstruction via Test-Time Training

Haian Jin; Rundi Wu; Tianyuan Zhang; Ruiqi Gao; Jonathan T. Barron; Noah Snavely; Aleksander Holynski

cs.CVMar 4, 2026

ZipMap: Linear-Time Stateful 3D Reconstruction via Test-Time Training

Authors: Haian Jin, Rundi Wu, Tianyuan Zhang, Ruiqi Gao, Jonathan T. Barron, Noah Snavely, Aleksander Holynski

Organizations: Google DeepMind · Cornell University · Massachusetts Institute of Technology

Abstract

Feed-forward transformer models have driven rapid progress in 3D vision, but state-of-the-art methods such as VGGT and $π^3$ have a computational cost that scales quadratically with the number of input images, making them inefficient when applied to large image collections. Sequential-reconstruction approaches reduce this cost but sacrifice reconstruction quality. We introduce ZipMap, a stateful feed-forward model that achieves linear-time, bidirectional 3D reconstruction while matching or surpassing the accuracy of quadratic-time methods. ZipMap employs test-time training layers to zip an entire image collection into a compact hidden scene state in a single forward pass, enabling reconstruction of over 700 frames in under 10 seconds on a single H100 GPU, more than $20\times$ faster than state-of-the-art methods such as VGGT. Moreover, we demonstrate the benefits of having a stateful representation in real-time scene-state querying and its extension to sequential streaming reconstruction.

Figures & tables

Figure 2: Method Overview. ZipMap is a stateful feed-forward model with local window attention and large-chunk TTT layers [ 65 , 86 ] . Given $N$ input images, a single linear-time pass predicts camera poses, depth maps, and point maps while storing a compact scene representation in TTT fast weights, which can be queried in real time at novel cameras to synthesize new-view point maps.

Figure 3: Example reconstruction results A sparse subset of input images are shown on the left, and a visualization of the output 3D reconstructions are shown on the right. Note that our method performs well on challenging cases like long sequence inputs, dynamic scenes and internet photo collections.

Method	RealEstate10K			Co3Dv2
Method	AUC@5 $\uparrow$	AUC@15 $\uparrow$	AUC@30 $\uparrow$	AUC@5 $\uparrow$	AUC@15 $\uparrow$	AUC@30 $\uparrow$
$\mathcal{O}(N^{2})$ Inference Speed
Fast3R [ 79 ]	22.36	46.71	61.68	31.05	59.63	73.43
FLARE [ 85 ]	38.47	67.02	80.01	23.84	57.78	73.99
VGGT [ 68 ]	38.71	66.46	78.89	67.84	83.95	89.99
$\pi^{3}$ [ 75 ]	63.10	80.31	87.40	57.12	79.86	87.93

Table 1: Camera Pose Estimation: RealEstate10K [ 89 ] and Co3Dv2 [ 48 ] . We report pose AUC under angular error thresholds of 5/15/30 degrees. All methods have seen Co3Dv2 during training. CUT3R and TTT3R additionally use RealEstate10K for training, while the remaining methods do not.

Method	Sintel			TUM-dynamics			ScanNet (seen)
Method	ATE $\downarrow$	RPE trans $\downarrow$	RPE rot $\downarrow$	ATE $\downarrow$	RPE trans $\downarrow$	RPE rot $\downarrow$	ATE $\downarrow$	RPE trans $\downarrow$	RPE rot $\downarrow$
$\mathcal{O}(N^{2})$ Inference Speed
Fast3R [ 79 ]	0.371	0.298	13.750	0.090	0.101	1.425	0.155	0.123	3.491
FLARE [ 85 ]	0.207	0.090	3.015	0.026	0.013	0.475	0.064	0.023	0.971
VGGT [ 68 ]	0.172	0.061	0.471	0.012	0.010	0.309	0.035	0.015	0.376
$\pi^{3}$ [ 75 ]	0.073	0.038	0.288	0.014	0.009	0.307	0.030	0.013	0.345

Table 2: Camera Pose Estimation: Sintel [ 9 ] , TUM-dynamics [ 61 ] and ScanNet [ 14 ] . We measure the distance error of rotation/translation. All methods have seen ScanNet or ScanNet++ [ 83 ] in training. None has seen Sintel or TUM-dynamics.

Method	7-Scenes						NRGBD
	Acc. $\downarrow$		Comp. $\downarrow$		NC. $\uparrow$		Acc. $\downarrow$		Comp. $\downarrow$		NC. $\uparrow$
	Mean	Med.	Mean	Med.	Mean	Med.	Mean	Med.	Mean	Med.	Mean	Med.
Sparse View
Fast3R [ 79 ]	0.095	0.065	0.144	0.089	0.673	0.759	0.135	0.091	0.163	0.104	0.759	0.877
CUT3R [ 76 ]	0.080	0.055	0.102	0.066	0.711	0.811	0.098	0.038	0.075	0.029	0.830	0.974
TTT3R [ 13 ]	0.098	0.062	0.159	0.107	0.681	0.768	0.101	0.039	0.076	0.029	0.826	0.973

Table 3: Point Map Estimation: 7-Scenes [ 58 ] and NRGBD [ 3 ] . Metrics are the same as in Table 4 . Keyframes are selected every 200 (sparse) and 40 (dense) frames for 7-Scenes, and every 500 (sparse) and 100 (dense) frames for NRGBD.

Method	DTU						ETH3D
	Acc. $\downarrow$		Comp. $\downarrow$		N.C. $\uparrow$		Acc. $\downarrow$		Comp. $\downarrow$		N.C. $\uparrow$
	Mean	Med.	Mean	Med.	Mean	Med.	Mean	Med.	Mean	Med.	Mean	Med.
$\mathcal{O}(N^{2})$ Inference Speed
Fast3R [ 79 ]	3.340	1.919	2.929	1.125	0.671	0.755	0.832	0.691	0.978	0.683	0.667	0.766
FLARE [ 85 ]	2.541	1.468	3.174	1.420	0.684	0.774	0.464	0.338	0.664	0.395	0.744	0.864
VGGT [ 68 ]	1.308	0.761	1.929	1.015	0.665	0.751	0.270	0.174	0.304	0.180	0.841	0.942

Table 4: Point Map Estimation: DTU [ 26 ] and ETH3D [ 55 ] . Following

\pi^{3}

[ 75 ] , we report the mean and median of accuracy (Acc.), completeness (Comp.), and normal-consistency (NC.) for keyframes selected every 5 images.

		Sintel		Bonn		KITTI
Method	Params	AbsRel $\downarrow$	$\delta<1.25$ $\uparrow$	AbsRel $\downarrow$	$\delta<1.25$ $\uparrow$	AbsRel $\downarrow$	$\delta<1.25$ $\uparrow$
$\mathcal{O}(N^{2})$ Inference Speed
Fast3R [ 79 ]	648M	0.638	0.422	0.194	0.772	0.138	0.834
FLARE [ 85 ]	1.40B	0.729	0.336	0.152	0.790	0.356	0.570
VGGT [ 68 ]	1.26B	0.298	0.643	0.055	0.971	0.073	0.965
$\pi^{3}$ [ 75 ]	959M	0.228	0.672	0.051	0.975	0.038	0.986

Table 5: Video Depth Estimation: Sintel [ 9 ] , Bonn [ 41 ] and KITTI [ 20 ] . We report results under scale-only alignment; results using joint scale-and-shift alignment are provided in the Appendix.

Figure 4: Long-sequence camera evaluation on DL3DV. We evaluate camera pose accuracy (ATE $\downarrow$ ) on the DL3DV test set [ 37 ] under two protocols: Left : increasing scene scale by using the first $N$ frames of each sequence; Right : increasing view density by uniformly subsampling $N$ frames along a fixed trajectory. Our method maintains low error and matches quadratic-time baselines ( $\pi^{3}$ , VGGT) while other linear-time methods (CUT3R, TTT3R) degrade significantly as $N$ grows.

Method	Acc. $\downarrow$		Comp. $\downarrow$		N.C. $\uparrow$
Method	Mean	Med.	Mean	Med.	Mean	Med.
Ours	0.337	0.224	0.357	0.217	0.810	0.918
Ours w/o gated unit	0.354	0.251	0.381	0.234	0.802	0.901
Ours w/o Newton–Schulz	0.408	0.283	0.430	0.249	0.787	0.898
Ours w/ global TTT lr (0.1)	0.411	0.303	0.490	0.317	0.779	0.886
Ours w/ global TTT lr (1.0)	0.464	0.366	0.537	0.343	0.782	0.890

Table 6: Ablation Study for TTT key components. We evaluate point-map estimation on ETH3D [ 55 ] . All variants are trained with reduced compute and a smaller data scale.

Figure 5: Querying Unseen Structure. Left: input images (a), GT images at query poses (b), and our predicted depth at those poses (c). Middle: point cloud reconstructed from input images only. Right: point cloud after querying (right column), where the queried point cloud is merged with the input image point cloud. This demonstrates our model’s ability to infer common 3D structure (e.g., walls, floors, and ground) in the unseen regions, thereby indicating an understanding of basic 3D scene priors.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Complexity	Params	Time to Reconstruct $N$ Frames (seconds)
			$N=$ 5	10	25	50	100	200	300	400	500	750
VGGT [ 68 ]	$\mathcal{O}(N^{2})$	1.26B	0.102	0.194	0.569	1.524	4.689	16.040	34.022	58.842	90.389	200.364
$\pi^{3}$ [ 75 ]	$\mathcal{O}(N^{2})$	959M	0.087	0.157	0.450	1.186	3.604	12.190	25.765	44.464	68.255	151.159
CUT3R [ 76 ]	$\mathcal{O}(N)$	793M	0.206	0.413	1.018	2.056	4.088	8.222	12.430	16.618	21.025	31.246
TTT3R [ 13 ]	$\mathcal{O}(N)$	793M	0.206	0.411	1.033	2.036	4.128	8.267	12.435	16.511	20.767	31.197
Ours	$\mathcal{O}(N)$	1.40B	0.125	0.183	0.383	0.712	1.362	2.681	4.017	5.348	6.671	9.999

Appendix

Table 7: Runtime evaluation. Inference time (in seconds) as a function of the number of input images

N

. VGGT and

\pi^{3}

scale quadratically with

N

, resulting in slow speeds when

N

is large. CUT3R, TTT3R, and our method scale linearly with

N

, with ours being the fastest for dense input frames.

Figure 6: Qualitative comparison. Point cloud reconstructions of scenes from the ETH3D and DTU datasets.

Figure 7: Querying the Scene State. The left panels show: input images (a), GT RGB at query poses (b), our RGB predictions (c), GT depth (d), and predicted depth (e). The middle panels visualize the 3D point clouds reconstructed from the input images. The right panels show point clouds attained solely by querying the scene state . The close visual match between these two point clouds indicates that the learned scene state faithfully captures the geometry and appearance of the input scene.

	Sintel		Bonn		KITTI		NYU-v2
Method	AbsRel $\downarrow$	$\delta<1.25$ $\uparrow$	AbsRel $\downarrow$	$\delta<1.25$ $\uparrow$	AbsRel $\downarrow$	$\delta<1.25$ $\uparrow$	AbsRel $\downarrow$	$\delta<1.25$ $\uparrow$
MASt3R [ 34 ]	0.413	0.569	0.123	0.833	0.077	0.948	0.110	0.865
MonST3R [ 84 ]	0.402	0.525	0.069	0.954	0.098	0.895	0.094	0.887
Fast3R [ 79 ]	0.544	0.509	0.169	0.796	0.120	0.861	0.093	0.898
CUT3R [ 76 ]	0.418	0.520	0.058	0.967	0.097	0.914	0.081	0.914
FLARE [ 85 ]	0.606	0.402	0.130	0.836	0.312	0.513	0.089	0.898

Appendix

Table 8: Monocular Depth Estimation. We evalute the frame-independent monocular depth estimation on the Sintel [ 9 ] , Bonn [ 41 ] , KITTI [ 20 ] and NYU-v2 [ 59 ] datasets.

		Sintel		Bonn		KITTI
Method	Params	AbsRel $\downarrow$	$\delta<1.25$ $\uparrow$	AbsRel $\downarrow$	$\delta<1.25$ $\uparrow$	AbsRel $\downarrow$	$\delta<1.25$ $\uparrow$
$\mathcal{O}(N^{2})$ Inference Speed
Fast3R [ 79 ]	648M	0.518	0.486	0.196	0.768	0.139	0.808
FLARE [ 85 ]	1.40B	0.791	0.358	0.142	0.797	0.357	0.579
VGGT [ 68 ]	1.26B	0.226	0.683	0.049	0.974	0.059	0.961
$\pi^{3}$ [ 75 ]	959M	0.206	0.735	0.045	0.976	0.036	0.986

Appendix

Table 9: Video Depth Estimation: Sintel [ 9 ] , Bonn [ 41 ] and KITTI [ 20 ] . We have reported results under scale-only alignment in the main paper. Here we further report results using joint scale-and-shift alignment.

Method	Sintel			TUM-dynamics			ScanNet (seen)
Method	ATE $\downarrow$	RPE trans $\downarrow$	RPE rot $\downarrow$	ATE $\downarrow$	RPE trans $\downarrow$	RPE rot $\downarrow$	ATE $\downarrow$	RPE trans $\downarrow$	RPE rot $\downarrow$
Ours w/ ref	0.125	0.058	0.420	0.012	0.009	0.309	0.034	0.015	0.398
Ours w/o ref	0.132	0.066	0.438	0.012	0.010	0.310	0.034	0.015	0.385

Appendix

Table 10: Ablation: Removing the Reference View (Camera pose estimation).

Figure 8: Long-sequence camera estimation. We evaluate camera ATE on the ScanNet-v2 and DL3DV datasets by taking the first $N$ frames of each test sequence and gradually increasing $N$ . We see that, when the input sequence length becomes long, removing the reference view and fine-tuning with the affine-invariant camera loss from $\pi^{3}$ [ 75 ] (“Ours w/o ref”) improves the camera pose estimation accuracy compared to the reference-based variant (“Ours w/ ref”).

	Sintel		Bonn		KITTI
Method	AbsRel $\downarrow$	$\delta<1.25$ $\uparrow$	AbsRel $\downarrow$	$\delta<1.25$ $\uparrow$	AbsRel $\downarrow$	$\delta<1.25$ $\uparrow$
CUT3R [ 76 ]	0.432	0.510	0.072	0.951	0.152	0.805
TTT3R [ 13 ]	0.426	0.522	0.061	0.970	0.149	0.812
Ours-streaming	0.273	0.638	0.067	0.965	0.100	0.903

Appendix

Table 13: Streaming Video Depth Estimation. We report results under scale-only alignment on Sintel [ 9 ] , Bonn [ 41 ] and KITTI [ 20 ] .

Method	Sintel			Co3Dv2
Method	ATE $\downarrow$	RPE trans $\downarrow$	RPE rot $\downarrow$	AUC@5 $\uparrow$	AUC@15 $\uparrow$	AUC@30 $\uparrow$
CUT3R [ 76 ]	0.2160	0.0710	0.6220	24.88	56.28	71.72
TTT3R [ 13 ]	0.2040	0.0850	0.6900	22.61	53.49	69.46
Ours-Streaming	0.1593	0.0655	0.7508	45.38	72.58	83.12

Appendix

Table 14: Streaming Camera Pose Estimation: Sintel [ 9 ] and Co3Dv2 [ 48 ] . For Sintel, we report ATE and RPE translation/rotation errors. For Co3Dv2, we report pose AUC under angular error thresholds of 5/15/30 degrees.

Method	DTU			ETH3D			NRGBD-dense
Method	Acc. $\downarrow$	Comp. $\downarrow$	N.C. $\uparrow$	Acc. $\downarrow$	Comp. $\downarrow$	N.C. $\uparrow$	Acc. $\downarrow$	Comp. $\downarrow$	N.C. $\uparrow$
CUT3R [ 76 ]	5.045	6.437	0.666	0.593	0.747	0.754	0.065	0.036	0.812
TTT3R [ 13 ]	5.337	6.593	0.666	0.763	0.881	0.739	0.074	0.037	0.803
Ours-streaming	4.091	3.495	0.693	0.614	0.941	0.750	0.038	0.028	0.836

Appendix

Table 15: Streaming Point Map Reconstruction Comparison. We report the results on DTU [ 26 ] , ETH3D [ 55 ] , and NRGBD-dense.

Figure 9: Long-sequence video depth estimation. We evaluate on the ScanNet-v2 dataset by taking the first $N$ frames of each test sequence and gradually increasing $N$ .

Figure 10: Long-sequence 3D point estimation. We evaluate on the 7-Scenes [ 58 ] dataset under two cases. Left : increasing scene scale by using the first $N$ frames of each sequence; Right : increasing view density by uniformly subsampling $N$ frames.

Method	6 Views			16 Views
Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
NoPoSplat [ 82 ]	15.92	0.416	0.541	15.47	0.361	0.606
Flare [ 85 ]	15.35	0.407	0.573	13.21	0.348	0.695
AnySplat [ 28 ]	18.32	0.524	0.329	21.85	0.670	0.250
Ours	15.60	0.414	0.486	17.65	0.459	0.374

Appendix

Table 16: NVS Evaluation on Mip-NeRF360 [ 4 ] . Baseline results are taken from Tab. 1 of the AnySplat paper. ZipMap is competitive with several baselines. However, as noted in our Limitations, our query results are primarily designed to generate colored point clouds rather than high-fidelity novel view synthesis. Accordingly, NVS is not our main focus, and we do not claim state-of-the-art performance compared with explicit 3DGS-based methods.

Explore similar work

Feb 26, 2026

VGG-T $^3$ : Offline Feed-Forward 3D Reconstruction at Scale

We present a scalable 3D reconstruction model that addresses a critical limitation in offline feed-forward methods: their computational and memory requirements grow quadratically w.r.t. the number of input images. Our approach is built on the key insight that this bottleneck stems from the varying-length Key-Value (KV) space representation of scene geometry, which we distill into a fixed-size Multi-Layer Perceptron (MLP) via test-time training. VGG-T

^3

(Visual Geometry Grounded Test Time Training) scales linearly w.r.t. the number of input views, similar to online models, and reconstructs a

1k

image collection in just

54

seconds, achieving a

11.6\times

speed-up over baselines that rely on softmax attention. Since our method retains global scene aggregation capability, our point map reconstruction error outperforming other linear-time methods by large margins. Finally, we demonstrate visual localization capabilities of our model by querying the scene representation with unseen images.

Sven Elflein, Ruilong Li, Sérgio Agostinho +4

NVIDIA · Vector Institute · University of Toronto

3D Reconstruction Feed-Forward 3D Reconstruction 3D Scene Representation

Ask PDF

Sep 28, 2026

OTT3R: Multi-View 3D Reconstruction and Fast Dataset Generation at 1% Compute

Feed-forward 3D reconstruction models have achieved impressive performance by scaling model and dataset size, but their cost excludes most research groups and precludes edge deployment. Additionally, generating 3D supervision without sensors still relies on slow, unreliable Structure-from-Motion, as the community lacks a COLMAP-like system for neural 3D pseudo-label generation. We present OTT3R (RGB-Only Tiny Transformer for 3D Reconstruction), a knowledge distillation framework that addresses both problems on a single workstation equipped with 2 GPUs. Distilling

π^3

(959M parameters) into a 102M-parameter student yields 9.4

\times

compression and up to 7

\times

faster inference, trained at 1.6% of VGGT's training compute. An integrated pseudo-label pipeline offers a reliable, high-throughput alternative to COLMAP, generating dense per-pixel point maps and SE(3) camera poses for a 667K-image corpus in 3.5 hours on two commodity GPUs and succeeding on every sequence we tested, including those where COLMAP fails. The general student tracks the teacher on in-distribution monocular depth and, zero-shot, outperforms COLMAP on 7-Scenes and on DTU completion, but it does not replace the teacher on out-of-distribution multi-view geometry. The deployable artifact is the domain-specialized student: after specialization at 0.2% compute, it is 4

\times

more accurate than COLMAP on 7-Scenes at 980

\times

throughput, with near-teacher completion. Code is available at https://github.com/TheFourthKaramazov/OTT3R

Brandon Leblanc, Charalambos Poullis

Immersive and Creative Technologies Lab, Concordia University, Montreal, Canada

Multi-View 3D Reconstruction Pseudo-Labeling Feed-Forward 3D Reconstruction

Ask PDF

May 7, 2026

Spark3R: Asymmetric Token Reduction Makes Fast Feed-Forward 3D Reconstruction

Feed-forward 3D reconstruction models based on Vision Transformers can directly estimate scene geometry and camera poses from a small set of input images, but scaling them to video inputs with hundreds or thousands of frames remains challenging due to the quadratic cost of global attention layers. Recent token-merging methods accelerate these models by compressing the token sequence within the global attention layers, but they apply a uniform reduction to query tokens and key-value tokens, ignoring their functionally distinct roles in 3D reconstruction. In this work, we identify a key property of feed-forward 3D reconstruction models: query tokens encode view-specific geometric requests and are sensitive to compression, while key-value tokens represent shared scene context and tolerate aggressive compression. Guided by this insight, we propose Spark3R, a training-free acceleration framework that decouples the compression of query tokens and key-value tokens by assigning distinct reduction factors, with intra-group token merging applied to query tokens and lightweight token pruning to key-value tokens. Additionally, Spark3R adaptively adjusts the key-value reduction factor across layers, further improving the quality-efficiency trade-off. As a plug-and-play framework requiring no retraining, Spark3R integrates directly into multiple pretrained feed-forward 3D reconstruction models, including VGGT,

π^3

, Depth-Anything-3, and VGGT-

Ω

, and achieves up to

28\times

speedup on 1,000-frame inputs while maintaining competitive reconstruction quality.

Zecheng Tang, Jiaye Fu, Qiankun Gao +5

School of Electronic and Computer Engineering, Peking University, Shenzhen 518055, China

3D Reconstruction Efficient ViTs Feed-Forward 3D Reconstruction Visual Token Pruning Visual Token Compression

Ask PDF

Abstract

Explore similar work

VGG-T3^33: Offline Feed-Forward 3D Reconstruction at Scale

OTT3R: Multi-View 3D Reconstruction and Fast Dataset Generation at 1% Compute

Spark3R: Asymmetric Token Reduction Makes Fast Feed-Forward 3D Reconstruction

VGG-T $^3$ : Offline Feed-Forward 3D Reconstruction at Scale