Feed-forward transformer models have recently achieved strong performance in multi-view 3D reconstruction, but they still rely primarily on appearance-based attention and often struggle in dynamic scenes with weak geometric cues. In this work, we introduce a motion- and gravity-aware framework under the Learning Using Privileged Information (LUPI) paradigm. During training, residual optical flow and gravity direction are provided as privileged signals to guide geometric representation learning; during inference, the model uses standard visual inputs only. We integrate these priors into transformer backbones with lightweight architectural changes, preserving feed-forward efficiency. Experiments on standard benchmarks show consistent gains in reconstruction accuracy and robustness, especially in scenes with dynamic objects, ambiguous structure, and challenging camera motion. Our analysis further shows that the optimal injection stage is backbone-dependent: middle-layer integration is generally preferable for VGGT, while late-stage integration is more effective for π3. These results suggest that LUPI offers a practical way to inject motion and physical priors into feed-forward 3D reconstruction without increasing test-time complexity.
Figures & tables
Figure 1: Optical flow decomposition.
Figure 2: Architecture Overview.
Dataset
#images
#images w OF
%images w OF
Datasets for training
BlendedMVS [ 30 ]
230,284
0
0
Mapillary Planet-Scale Depth [ 31 ]
1,688,784
0
0
Spring [ 32 ]
34,000
11,853
34.9
UnrealStereo4K [ 33 ]
16,400
0
0
Aria Synthetic Environments [ 34 ]
11,765,050
10,647,370
90.5
Table 1: Datasets used for training and testing.
Figure 3: π3 injection results
Figure 4: VGGT injection results
Backbone
Flow Type
Depth rel ↓
Rays err ∘ ↓
Pose AUC ↑
Depth Inliers ↑
π3
Residual flow
0.107
2.055
40.94
40.87
Full flow
0.113
2.235
42.19
38.76
No injection
0.116
2.242
42.14
39.55
VGGT
Residual flow
0.114
2.459
36.69
36.94
Full flow
0.124
2.491
35.46
35.44
No injection
0.121
2.478
35.80
36.11
Table 2: Residual vs. Full Flow Injection.
Backbone
Injection Layer
Depth rel ↓
Rays err ∘ ↓
Pose AUC ↑
Depth Inliers ↑
π3(ResFlow)
Early (Input)
0.115
1.714
46.19
44.43
Middle (Layer 17)
0.110
1.621
47.87
45.85
Late (Output)
0.102
1.539
48.62
46.13
π3(Grav)
Early (Input)
0.104
1.707
43.03
44.08
Middle (Layer 17)
0.096
1.561
44.38
45.67
Late (Output)
0.095
1.478
49.39
46.39
Table 3: Layer-wise injection analysis on π3 and VGGT.
Figure 5: Qualitative comparison with and without gravity injection. In each pair, the upper image shows reconstruction without gravity injection, and the lower image shows reconstruction with gravity injection.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Performance of the π3 model fine-tuned with varying effective dataset scales. PI3_10% denotes training with 10% of scenes sampled per epoch over a 10× total epoch duration, while PI3_50% utilizes 50% of scenes over a 2× duration to maintain constant total training steps.
Figure 7: Comparison of π3 performance under default fine-tuning settings versus fine-tuning with a 10% effective dataset size augmented by residual flow injection.
Figure 8: Comparison of VGGT performance under default fine-tuning settings versus fine-tuning with a 10% effective dataset size augmented by residual flow injection.
Figure 9: Ablation study of residual flow injection locations within the π3 model. Performance of default injection point is evaluated against injection occurring both prior to and within the middle layers of the aggregator module.
Figure 10: Ablation study of gravity injection locations within the π3 model. Performance of default injection point is evaluated against injection occurring both prior to and within the middle layers of the aggregator module.
Figure 11: Ablation study of residual flow injection locations within the VGGT model. Performance of default injection point is evaluated against injection occurring both before and after the aggregator module.
Figure 12: Ablation study of gravity injection locations within the VGGT model. Performance of the default injection point is evaluated against injection occurring both before and after the aggregator module.
Modern feed-forward 3D reconstruction methods like VGGT predict pixel-aligned pointmaps in camera-centric coordinate frames. However, this choice of coordinate frame is not always optimal. We propose instead to predict pointmaps in upright, gravity-aligned frames that exploit strong structural cues present in many real-world scenes. Unlike camera-centric frames, gravity-aligned frames share a common vertical axis across viewpoints, reducing the rotational degrees of freedom needed to relate pointmaps to one another. To this end, we introduce the Gravity Grounded Geometry Transformer (G3T), fine-tuned from existing models on gravity-aligned 3D data. G3T produces highly accurate gravity-aware predictions, including upright pointmaps and camera-to-gravity poses. We further introduce G3T-Long, a submap-based incremental 3D reconstruction pipeline that leverages the reduced rotational degrees of freedom afforded by upright frames to achieve significantly improved reconstruction accuracy.
Transformer-based 3D reconstruction has emerged as a powerful paradigm for recovering geometry and appearance from multi-view observations, offering strong performance across challenging visual conditions. As these models scale to larger backbones and higher-resolution inputs, improving their efficiency becomes increasingly important for practical deployment. However, modern 3D transformer pipelines face two coupled challenges: dense multi-view attention creates substantial token-mixing overhead, and low-precision execution can destabilize geometry-sensitive representations and degrade depth, pose, and 3D consistency. To address the first challenge, we propose Lite3R, a model-agnostic teacher-student framework that replaces dense attention with Sparse Linear Attention to preserve important geometric interactions while reducing attention cost. To address the second challenge, we introduce a parameter-efficient FP8-aware quantization-aware training (FP8-aware QAT) strategy with partial attention distillation, which freezes the vast majority of pretrained backbone parameters and trains only lightweight linear-branch projection layers, enabling stable low-precision deployment while retaining pretrained geometric priors. We further evaluate Lite3R on two representative backbones, VGGT and DA3-Large, over BlendedMVS and DTU64, showing that it substantially reduces latency (1.7-2.0x) and memory usage (1.9-2.4x) while preserving competitive reconstruction quality overall. These results demonstrate that Lite3R provides an effective algorithm-system co-design approach for practical transformer-based 3D reconstruction. Code: https://github.com/AIGeeksGroup/Lite3R. Website: https://aigeeksgroup.github.io/Lite3R.
Feed-forward 3D reconstruction models are efficient but rigid: once trained, they perform inference in a zero-shot manner and cannot adapt to the test scene. As a result, visually plausible reconstructions often contain errors, particularly under occlusions, specularities, and ambiguous cues. To address this, we introduce Free Geometry, a framework that enables feed-forward 3D reconstruction models to self-evolve at test time without any 3D ground truth. Our key insight is that, when the model receives more views, it produces more reliable and view-consistent reconstructions. Leveraging this property, given a testing sequence, we mask a subset of frames to construct a self-supervised task. Free Geometry enforces cross-view feature consistency between representations from full and partial observations, while maintaining the pairwise relations implied by the held-out frames. This self-supervision allows for fast recalibration via lightweight LoRA updates, taking less than 2 minutes per dataset on a single GPU. Our approach consistently improves state-of-the-art foundation models, including Depth Anything 3 and VGGT, across 4 benchmark datasets, yielding an average improvement of 3.73% in camera pose accuracy and 2.88% in point map prediction. Code is available at https://github.com/hiteacherIamhumble/Free-Geometry .
Yuhang Dai, Xingyi Yang
The Hong Kong Polytechnic University, Hong Kong SAR