This paper addresses the task of large-scale 3D scene reconstruction from long video sequences. Recent feed-forward reconstruction models have shown promising results by directly regressing 3D geometry from RGB images without explicit 3D priors or geometric constraints. However, these methods often struggle to maintain reconstruction accuracy and consistency over long sequences due to limited memory capacity and the inability to effectively capture global contextual cues. In contrast, humans can naturally exploit the global understanding of the scene to inform local perception. Motivated by this, we propose a novel neural global context representation that efficiently compresses and retains long-range scene information, enabling the model to leverage extensive contextual cues for enhanced reconstruction accuracy and consistency. The context representation is realized through a set of lightweight neural sub-networks that are rapidly adapted during test time via self-supervised objectives, which substantially increases memory capacity without incurring significant computational overhead. The experiments on multiple large-scale benchmarks, including the KITTI Odometry~\cite{Geiger2012CVPR} and Oxford Spires~\cite{tao2025spires} datasets, demonstrate the effectiveness of our approach in handling ultra-large scenes, achieving leading pose accuracy and state-of-the-art 3D reconstruction accuracy while maintaining efficiency. Code is available at https://zju3dv.github.io/scal3r.
Figures & tables
Figure 2: Overview of Scal3R . Our model takes a long sequence of RGB images as input and reconstructs the 3D scene within a unified inference pipeline. Specifically, the input sequence is divided into overlapping chunks that are processed in parallel across multiple GPUs. Each chunk is processed by our Scal3R backbone, which incorporates our proposed neural global context representation and aggregation mechanism to capture and share global context across the entire sequence. The resulting camera poses and depth maps from all chunks are then aligned and fused to generate the final 3D reconstruction of the scene.
Method
VKITTI2
KITTI
Oxford Spires
Resources
223–837 frames, 52–711 meters
271–4661 frames, 394–5067 meters
351–787 frames, 280–773 meters
KITTI 03/04/10, avg. 758 frames
RRE ↓
RTE ↓
ATE ↓
RRE ↓
RTE ↓
ATE ↓
RRE ↓
RTE ↓
ATE ↓
Memory ↓
Time ↓
FPS ↑
MASt3R-SLAM [ 44 ]
15.81
70.48
78.33
22.42
67.72
191.71
59.67
29.82
29.22
6.74
99.30
7.37
VGGT-SLAM [ 41 ]
12.92
21.27
17.18
33.27
78.95
214.88
55.60
32.14
26.85
10.67
39.72
19.85
StreamVGGT [ 101 ]
13.47
58.07
68.97
24.06
84.46
226.15
71.28
37.14
34.35
6.66
32.61
23.14
STream3R [ 33 ]
13.46
76.06
70.87
24.06
81.63
227.77
71.29
36.65
34.65
4.70
111.23
8.19
Table 1: Camera pose and resource evaluation on Virtual KITTI [ 7 ] , KITTI Odometry [ 22 ] , and Oxford Spires [ 70 ] . We report RRE ( ∘ /100m), RTE (m/100m), and ATE (m). Failed scenes are assigned the worst valid score when computing dataset averages. Methods marked with † require known camera intrinsics. Best and second-best results are shown in bold and underlined .
Figure 3: Camera trajectory comparison on KITTI Odometry [ 22 ] and Oxford Spires [ 70 ] . Scal3R preserves global structure with lower drift, whereas baselines often lose tracking or diverge on long sequences.
Figure 4: Qualitative comparison of point-cloud reconstruction on outdoor and indoor scenes. Scal3R reconstructs large-scale outdoor scenes more reliably and preserves more consistent local geometry indoors.
Method
ETH3D
Oxford Spires
VKITTI2
CD ↓
F1 ↑
CD ↓
F1 ↑
CD ↓
F1 ↑
MASt3R-SLAM [ 44 ]
0.89
0.31
7.78
0.53
17.08
0.33
VGGT-SLAM [ 41 ]
0.78
0.72
10.16
0.22
9.74
0.57
StreamVGGT [ 101 ]
1.86
0.14
12.23
0.25
20.45
0.35
STream3R [ 33 ]
1.81
0.14
12.20
0.25
18.77
0.36
CUT3R [ 80 ]
0.41
0.60
6.93
0.45
5.67
0.39
Table 2: 3D reconstruction evaluation on ETH3D [ 60 ] , Oxford Spires [ 70 ] , and Virtual KITTI [ 7 ] . We report Chamfer Distance (CD) and F1 score. Best and second-best results are shown in bold and underlined .
State Size
Global Context
Variants
RRE ↓
RTE ↓
ATE ↓
Variants
RRE ↓
RTE ↓
ATE ↓
1M size
1.01
1.01
0.99
w/o GCM
1.30
7.03
19.00
2M size
0.95
0.91
0.93
w/o GCS
1.28
7.01
15.80
4M size
0.87
0.84
0.85
Full model
1.17
5.99
13.70
Table 3: Ablation studies. Left: varying GCM state size. Right: ablating global context on a complementary long-sequence setting. The two blocks are not directly comparable. Best and second-best results are shown in bold and underlined .
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Camera trajectory comparison. Scal3R preserves global structure with substantially lower drift, whereas baselines frequently lose tracking or diverge, demonstrating our capability of reconstructing large-scale scenarios with high accuracy.
Figure 6: Point-cloud reconstruction comparison. Scal3R produces more accurate large-scale reconstructions for large-scale outdoor environments where baselines often fail, and achieves higher local geometric accuracy and consistency in indoor scenes.
Method
ScanNet++
TUM-RGBD
Waymo
avg. 924 fr.
avg. 926 fr.
avg. 198 fr.
MASt3R-SLAM [ 44 ]
0.47
0.08
7.63
VGGT-SLAM [ 41 ]
0.29
0.12
7.43
StreamVGGT [ 101 ]
1.70
0.63
45.10
STream3R [ 33 ]
1.75
0.63
42.20
CUT3R [ 80 ]
1.27
0.54
9.40
Appendix
Table 4: Additional pose benchmark comparisons. We report ATE (m, lower is better) on three supplementary benchmarks. The best results are in bold , and the second best are underlined .
Frames
150
270
510
990
RPE (m) ↓
0.08
0.08
0.07
0.08
Time (s) ↓
51.19
98.81
195.24
382.80
FPS ↑
2.93
2.73
2.61
2.59
Appendix
Table 5: Runtime scaling with sequence length. Using the same single-GPU evaluation setting as the main-paper resource comparison, we report relative pose error (RPE, m), total inference time, and FPS as the input length increases. Runtime grows smoothly with sequence length while RPE remains stable.
Figure 7: Failure case under abrupt illumination changes. Large appearance shifts within a sequence weaken cross-chunk correspondences and can lead to inaccurate global alignment.
We propose tttLRM, a novel large 3D reconstruction model that leverages a Test-Time Training (TTT) layer to enable long-context, autoregressive 3D reconstruction with linear computational complexity, further scaling the model's capability. Our framework efficiently compresses multiple image observations into the fast weights of the TTT layer, forming an implicit 3D representation in the latent space that can be decoded into various explicit formats, such as Gaussian Splats (GS) for downstream applications. The online learning variant of our model supports progressive 3D reconstruction and refinement from streaming observations. We demonstrate that pretraining on novel view synthesis tasks effectively transfers to explicit 3D modeling, resulting in improved reconstruction quality and faster convergence. Extensive experiments show that our method achieves superior performance in feedforward 3D Gaussian reconstruction compared to state-of-the-art approaches on both objects and scenes.
We present a scalable 3D reconstruction model that addresses a critical limitation in offline feed-forward methods: their computational and memory requirements grow quadratically w.r.t. the number of input images. Our approach is built on the key insight that this bottleneck stems from the varying-length Key-Value (KV) space representation of scene geometry, which we distill into a fixed-size Multi-Layer Perceptron (MLP) via test-time training. VGG-T3 (Visual Geometry Grounded Test Time Training) scales linearly w.r.t. the number of input views, similar to online models, and reconstructs a 1k image collection in just 54 seconds, achieving a 11.6× speed-up over baselines that rely on softmax attention. Since our method retains global scene aggregation capability, our point map reconstruction error outperforming other linear-time methods by large margins. Finally, we demonstrate visual localization capabilities of our model by querying the scene representation with unseen images.
Online 3D reconstruction models perform poorly on long videos. This happens because regressing poses relative to a fixed first-frame anchor forces extrapolation far beyond the training distribution. Small drifts accumulate and amplify into significant geometric collapse. However, we observe that per-frame depth remains stable throughout this failure. The backbone's local geometry remains intact; only the global pose head breaks down. Motivated by this decoupling, we introduce Scal3R. This approach reformulates online reconstruction as multi-reference relative pose querying. We use lightweight learnable tokens, which make up about ~1% of the parameters, and inject them into a completely frozen backbone via asymmetric attention. This setup queries poses relative to multiple past keyframes. An online pose-graph optimization system with loop closure suppresses long-range drift. Scal3R reaches convergence in 8 hours on a single GPU. It reduces the average ATE by over 60% on KITTI compared to the online baseline. It also achieves state-of-the-art performance across Virtual KITTI, Sintel, TUM-Dynamic, ScanNet, and 7-Scenes. Project page: https://linjohnss.github.io/scal3r/
Chin-Yang Lin, Yang-Che Sun, Cheng Sun +5
Department of Computer Science, National Yang Ming Chiao Tung University · NVIDIA