cs.CVOct 15, 2026

Motion- and Gravity-Aware Feedforward 3D Reconstruction

Authors: Shiyao Qian, Yuan Ren, Dongfeng Bai

Organizations: Huawei Noah’s Ark Lab, Markham, ON, Canada. · University of Toronto, Toronto, ON, Canada.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Feed-forward transformer models have recently achieved strong performance in multi-view 3D reconstruction, but they still rely primarily on appearance-based attention and often struggle in dynamic scenes with weak geometric cues. In this work, we introduce a motion- and gravity-aware framework under the Learning Using Privileged Information (LUPI) paradigm. During training, residual optical flow and gravity direction are provided as privileged signals to guide geometric representation learning; during inference, the model uses standard visual inputs only. We integrate these priors into transformer backbones with lightweight architectural changes, preserving feed-forward efficiency. Experiments on standard benchmarks show consistent gains in reconstruction accuracy and robustness, especially in scenes with dynamic objects, ambiguous structure, and challenging camera motion. Our analysis further shows that the optimal injection stage is backbone-dependent: middle-layer integration is generally preferable for VGGT, while late-stage integration is more effective for π3. These results suggest that LUPI offers a practical way to inject motion and physical priors into feed-forward 3D reconstruction without increasing test-time complexity.

Figures & tables

Figure 1: Optical flow decomposition.
Figure 2: Architecture Overview.
Dataset#images#images w OF%images w OF
Datasets for training
BlendedMVS [ 30 ]230,28400
Mapillary Planet-Scale Depth [ 31 ]1,688,78400
Spring [ 32 ]34,00011,85334.9
UnrealStereo4K [ 33 ]16,40000
Aria Synthetic Environments [ 34 ]11,765,05010,647,37090.5
Table 1: Datasets used for training and testing.
Figure 3: π3\pi^{3} injection results
Figure 4: VGGT injection results
BackboneFlow TypeDepth rel ↓\downarrowRays err ∘ ↓\downarrowPose AUC ↑\uparrowDepth Inliers ↑\uparrow
π3\pi^{3}Residual flow0.1072.05540.9440.87
Full flow0.1132.23542.1938.76
 No injection0.1162.24242.1439.55
VGGTResidual flow0.1142.45936.6936.94
Full flow0.1242.49135.4635.44
 No injection0.1212.47835.8036.11
Table 2: Residual vs. Full Flow Injection.
BackboneInjection LayerDepth rel ↓\downarrowRays err ∘ ↓\downarrowPose AUC ↑\uparrowDepth Inliers ↑\uparrow
π3(ResFlow)\pi^{3}(ResFlow)Early (Input)0.1151.71446.1944.43
Middle (Layer 17)0.1101.62147.8745.85
Late (Output)0.1021.53948.6246.13
π3(Grav)\pi^{3}(Grav)Early (Input)0.1041.70743.0344.08
Middle (Layer 17)0.0961.56144.3845.67
Late (Output)0.0951.47849.3946.39
Table 3: Layer-wise injection analysis on π3\pi^{3} and VGGT.
Figure 5: Qualitative comparison with and without gravity injection. In each pair, the upper image shows reconstruction without gravity injection, and the lower image shows reconstruction with gravity injection.
Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Performance of the π3\pi^{3} model fine-tuned with varying effective dataset scales. PI3_10% denotes training with 10% of scenes sampled per epoch over a 10×10\times total epoch duration, while PI3_50% utilizes 50% of scenes over a 2×2\times duration to maintain constant total training steps.
Figure 7: Comparison of π3\pi^{3} performance under default fine-tuning settings versus fine-tuning with a 10% effective dataset size augmented by residual flow injection.
Figure 8: Comparison of VGGT performance under default fine-tuning settings versus fine-tuning with a 10% effective dataset size augmented by residual flow injection.
Figure 9: Ablation study of residual flow injection locations within the π3\pi^{3} model. Performance of default injection point is evaluated against injection occurring both prior to and within the middle layers of the aggregator module.
Figure 10: Ablation study of gravity injection locations within the π3\pi^{3} model. Performance of default injection point is evaluated against injection occurring both prior to and within the middle layers of the aggregator module.
Figure 11: Ablation study of residual flow injection locations within the VGGT model. Performance of default injection point is evaluated against injection occurring both before and after the aggregator module.
Figure 12: Ablation study of gravity injection locations within the VGGT model. Performance of the default injection point is evaluated against injection occurring both before and after the aggregator module.
More in this topic

Multi-View 3D Reconstruction

Multi-View 3D Reconstruction recovers scene geometry and appearance from images captured at multiple viewpoints.

See all 277 papers

Explore similar work

CardsList

  1. G3T Up! Gravity Aligned Coordinate Frames Simplify Pointmap Processing

    May 26, 2026Bharath Raj Nagoor Kani, Noah Snavely3D ReconstructionCamera Pose Estimation

    Cornell University

  2. Lite3R: A Model-Agnostic Framework for Efficient Feed-Forward 3D Reconstruction

    May 12, 2026Haoyu Zhang, Zeyu Zhang, Zedong Zhou +2Quantization-Aware Training3D Reconstruction

    1Peking University · 2La Trobe University

  3. Free Geometry: Refining 3D Reconstruction from Longer Versions of Itself

    Apr 15, 2026Yuhang Dai, Xingyi YangFeed-Forward 3D ReconstructionMulti-View 3D Reconstruction

    The Hong Kong Polytechnic University, Hong Kong SAR