While recent feed-forward 3D reconstruction models accelerate 3D reconstruction by jointly inferring dense geometry and camera poses in a single pass, their reliance on dense attention imposes a quadratic complexity, creating a prohibitive computational bottleneck that severely limits inference speed. To resolve this, we introduce Speed3R, an end-to-end trainable model inspired by the core principle of Structure-from-Motion: that a sparse set of keypoints is sufficient for robust pose estimation. Speed3R features a dual-branch attention mechanism where a compression branch creates a coarse contextual prior to guide a selection branch, which performs fine-grained attention only on the most informative image tokens. This strategy mimics the efficiency of traditional keypoint matching, achieving a remarkable 12.4x inference speedup on 1000-view sequences, while introducing a minimal, controlled trade-off in geometric accuracy. Validated on standard benchmarks with both VGGT and π3 backbones, our method delivers high-quality reconstructions at a fraction of computational cost, paving the way for efficient large-scale scene modeling.
Figures & tables
Figure 1: Performance comparison of different methods . CO3Dv2 pose estimation accuracy vs. sparsity ratio, highlighting the trade-off between sparsity and accuracy. The Ideal Region marks the desired balance of high accuracy and high sparsity.
Figure 2: Architecture. Our model processes a sequence of input images through a shared feature encoder. The resulting tokens are then processed by a series of transformer blocks that alternate between local Frame Attention (within each view) and our proposed Global Sparse Attention (across all views). The GSA module efficiently integrates global information by decomposing attention into a Compression Branch for coarse context and a Selection Branch for fine-grained details, guided by a Top-k selection mechanism. Finally, updated tokens are passed to task-specific heads to predict camera pose and dense depth maps.
Method
DTU [ 16 ]
ETH3D [ 32 ]
Acc. ↓
Comp. ↓
N.C. ↑
Acc. ↓
Comp. ↓
N.C. ↑
Mean
Med.
Mean
Med.
Mean
Med.
Mean
Med.
Mean
Med.
Mean
Med.
VGGT [ 40 ]
1.403
0.802
2.566
1.307
0.658
0.742
0.289
0.192
0.294
0.173
0.847
0.953
Block Sparse-VGGT [ 37 ]
1.966
1.052
2.311
1.135
0.647
0.715
0.861
0.754
1.171
0.812
0.681
0.772
FastVGGT-VGGT [ 33 ]
1.466
0.786
2.385
1.188
0.654
0.736
0.510
0.379
0.580
0.354
0.788
0.913
Speed3R-VGGT
1.426
0.827
2.179
1.101
0.657
0.740
0.295
0.190
0.289
0.168
0.853
0.953
Table 4: Pointmap Estimation on the DTU [ 16 ] and ETH3D [ 32 ] datasets . The arrows ( ↓ / ↑ ) indicate whether lower or higher values are better. Best results are highlighted in bold .
Figure 3: Qualitative comparison of predicted point clouds from our method and baselines. Compared with sparse baselines, our method produces visually more accurate reconstructions and reduces the multi-layer wall phenomenon across both architectures.
Figure 4: Inference Time for Different Models. Mean latency (in seconds) for varying sequence lengths. Our method achieves a 12.4× speedup on sequences of 1024 images.
Figure 5: Training loss curve of dense attention and our GSA .
Seq Length
32
64
128
256
512
1024
Full Attn.(VGGT)
0.79
1.81
5.44
18.21
67.27
271.21
FastVGGT [ 37 ]
0.63
1.25
2.59
6.48
18.30
61.33
Block Sparse [ 33 ]
0.53
1.16
2.89
8.16
26.15
75.42
Ours
0.46
0.87
1.78
3.88
9.16
24.85
Table 8: Inference Time for Different Models. Mean latency (in seconds) for varying sequence lengths on VGGT backbone. Our method achieves a 10.9× speedup on sequences of 1024 images.
Dataset Name
Scene/Sequence Count
Downsample Ratio
Co3Dv2 [ 26 ]
10764
2x
Hypersim [ 27 ]
461
1x
Scannetpp [ 50 ]
996
2x
DL3DV [ 20 ]
6,378
4x
ARKitScenes [ 1 ]
3,344
2x
WildRGBD [ 46 ]
2,753
5x
Table 9: Summary of Datasets Used in Training and Testing
Feed-forward 3D reconstruction models based on Vision Transformers can directly estimate scene geometry and camera poses from a small set of input images, but scaling them to video inputs with hundreds or thousands of frames remains challenging due to the quadratic cost of global attention layers. Recent token-merging methods accelerate these models by compressing the token sequence within the global attention layers, but they apply a uniform reduction to query tokens and key-value tokens, ignoring their functionally distinct roles in 3D reconstruction. In this work, we identify a key property of feed-forward 3D reconstruction models: query tokens encode view-specific geometric requests and are sensitive to compression, while key-value tokens represent shared scene context and tolerate aggressive compression. Guided by this insight, we propose Spark3R, a training-free acceleration framework that decouples the compression of query tokens and key-value tokens by assigning distinct reduction factors, with intra-group token merging applied to query tokens and lightweight token pruning to key-value tokens. Additionally, Spark3R adaptively adjusts the key-value reduction factor across layers, further improving the quality-efficiency trade-off. As a plug-and-play framework requiring no retraining, Spark3R integrates directly into multiple pretrained feed-forward 3D reconstruction models, including VGGT, π3, Depth-Anything-3, and VGGT-Ω, and achieves up to 28× speedup on 1,000-frame inputs while maintaining competitive reconstruction quality.
Zecheng Tang, Jiaye Fu, Qiankun Gao +5
School of Electronic and Computer Engineering, Peking University, Shenzhen 518055, China
We present a scalable 3D reconstruction model that addresses a critical limitation in offline feed-forward methods: their computational and memory requirements grow quadratically w.r.t. the number of input images. Our approach is built on the key insight that this bottleneck stems from the varying-length Key-Value (KV) space representation of scene geometry, which we distill into a fixed-size Multi-Layer Perceptron (MLP) via test-time training. VGG-T3 (Visual Geometry Grounded Test Time Training) scales linearly w.r.t. the number of input views, similar to online models, and reconstructs a 1k image collection in just 54 seconds, achieving a 11.6× speed-up over baselines that rely on softmax attention. Since our method retains global scene aggregation capability, our point map reconstruction error outperforming other linear-time methods by large margins. Finally, we demonstrate visual localization capabilities of our model by querying the scene representation with unseen images.
Transformer-based 3D reconstruction has emerged as a powerful paradigm for recovering geometry and appearance from multi-view observations, offering strong performance across challenging visual conditions. As these models scale to larger backbones and higher-resolution inputs, improving their efficiency becomes increasingly important for practical deployment. However, modern 3D transformer pipelines face two coupled challenges: dense multi-view attention creates substantial token-mixing overhead, and low-precision execution can destabilize geometry-sensitive representations and degrade depth, pose, and 3D consistency. To address the first challenge, we propose Lite3R, a model-agnostic teacher-student framework that replaces dense attention with Sparse Linear Attention to preserve important geometric interactions while reducing attention cost. To address the second challenge, we introduce a parameter-efficient FP8-aware quantization-aware training (FP8-aware QAT) strategy with partial attention distillation, which freezes the vast majority of pretrained backbone parameters and trains only lightweight linear-branch projection layers, enabling stable low-precision deployment while retaining pretrained geometric priors. We further evaluate Lite3R on two representative backbones, VGGT and DA3-Large, over BlendedMVS and DTU64, showing that it substantially reduces latency (1.7-2.0x) and memory usage (1.9-2.4x) while preserving competitive reconstruction quality overall. These results demonstrate that Lite3R provides an effective algorithm-system co-design approach for practical transformer-based 3D reconstruction. Code: https://github.com/AIGeeksGroup/Lite3R. Website: https://aigeeksgroup.github.io/Lite3R.