Feed-forward 3D foundation models have recently reshaped multi-view reconstruction, yet dynamic 4D understanding still often relies on large-scale motion supervision or hand-crafted, architecture-specific heuristics. We ask whether frozen 3D foundation model tokens already encode motion-relevant information that can be exposed by lightweight readout functions, rather than re-learning motion-specific features with high-capacity decoders. Through probing on three representative backbones, we find that static--dynamic separation is decodable from frozen token representations. Our rank-restricted and spectral analyses on π3X reveal that the recovered motion cues are concentrated in a low-dimensional subspace. Building on this, we propose Probing-based Motion Distillation (PMD), which converts pairwise motion-state compatibility scores against a fixed anchor token into dense motion masks. Trained on only 160 YouTube-VOS videos using a single 16 GB GPU, PMD transfers zero-shot to DAVIS, SegTrackv2, and FBMS-59 without target-benchmark fine-tuning. With multi-view pretrained geometric backbones, PMD outperforms training-free heuristics and approaches substantially heavier supervised and self-supervised methods, suggesting that motion decoding from frozen 3D foundation models can be lightweight and data-efficient.
Figures & tables
Figure 1: Overview of Probing-based Motion Distillation (PMD). Given an input frame group, a frozen feed-forward 3D foundation model such as π3 X [ 36 ] , VGGT [ 34 ] , or Depth Anything 3 (DA3) [ 21 ] produces token representations. A lightweight probe trained on only 160 videos maps these features into a probe-induced static–dynamic discriminative representation. PMD then converts anchor-based pairwise probe scores into dense dynamic score maps, which are thresholded into coarse masks and optionally refined with SAM2 [ 29 ] . It transfers zero-shot to DAVIS [ 26 , 27 ] , SegTrackv2 [ 17 ] , and FBMS-59 [ 14 ] , while requiring substantially less training data and trainable capacity than high-capacity motion decoders such as GeoMotion [ 8 ] .
Backbone
Selected Blocks
Probe
Rank k
π3 / π3 X [ 36 ]
18 × inter-frame blocks
low-rank quad.
8
DA3 [ 21 ]
14 × inter-frame blocks
low-rank quad.
8
DUSt3R [ 35 ]
12 × decoder-view-1 + 4 × head-in-view-1 blocks
low-rank quad.
8
Table 1: Default PMD settings. All probes are trained on a subset of 160 YouTube-VOS [ 41 ] videos. For Depth Anything 3 (DA3), we use the DA3-GIANT-1.1 model.
Figure 2: Layer-wise motion decodability on DAVIS-all [ 27 ] . Low-rank quadratic probes expose static–dynamic discriminative cues across the multi-view geometric modules of all three backbones, outperforming linear and diagonal probes.
Method
Motion-mask supervision; inference components
DAVIS17 J-M / J-R
SegTrackv2 J-M / J-R
FBMS-59 J-M / J-R
Baselines without motion-mask supervision
VGGT4D [ 9 ]
None; VGGT geometry
56.5/65.6
–
–
Easi3R dust3r [ 5 ]
None; DUSt3R geometry
46.9/50.5
–
–
Easi3R dust3r +SAM2
None; DUSt3R geometry, SAM2
60.1/65.3
–
–
Baselines with motion-mask supervision
GeoMotion [ 8 ]
Five datasets; π3 , RAFT, DINOv2
75.4 /87.2
–
–
Table 2: Motion segmentation benchmarking. Cells report J-M/J-R ( ↑ ). First , second , and third mark the top three distinct reported scores for each dataset and metric across all rows; ties share a color. Non-PMD scores are quoted from the cited papers; PMD scores are evaluated by us. Gray text marks target-benchmark fine-tuning; “–” denotes an unreported score.
Method
Flow
w/o SAM2
w/ SAM2
Δ J-R ↓
J-M ↑
J-R ↑
J-M ↑
J-R ↑
Easi3R dust3r [ 5 ]
×
46.9
50.5
60.1
65.3
14.8
Easi3R monst3r [ 5 ]
×
54.8
66.2
67.9
76.1
9.9
GeoMotion [ 8 ]
✓
75.4
87.2
81.1
92.6
5.4
π3 -PMD (ours)
×
72.7
89.9
80.2
91.1
1.2
π3 X-PMD (ours)
×
73.7
91.0
81.9
92.3
1.3
Table 3: SAM2 [ 29 ] refinement on DAVIS2017 [ 27 ] . Δ J-R is the percentage-point increase in the fraction of frames with Jaccard above 0.5. Additional backbones and methods are reported in Appendices G.2 and G.4 .
Figure 3: Qualitative comparison on DAVIS [ 26 ] and SegTrackv2 [ 17 ] against training-free methods based on feed-forward 3D foundation models. The semantic classes shown here are absent from the 160-video training subset of π3 X-PMD. All binary masks shown here are generated without SAM2 [ 29 ] refinement. PMD produces cleaner dynamic score maps and more coherent coarse masks.
Method
Training Data
DAVIS17 (J-M ↑ )
SegTrackv2 (J-M ↑ )
FBMS-59 (J-M ↑ )
Trainable Params (M) ↓
Added FLOPs (G) ↓
Large-scale supervised
SegAnyMo [ 10 ]
Kubric + Dynamic Replica + HOI4D + etc.
–
76.3
78.3
3.44
> 2.08
Flow-SAM [ 39 ]
Synthetic pretraining + DAVIS fine tuning + etc.
71.6
84.2
88.7
9.83
> 3.63\times104
GeoMotion [ 8 ]
GOT-Motion + HOI4D + Dynamic Replica + etc.
81.1
77.3
72.5
80.3
> 3.09\times102
Ours (data-efficient probing)
π3 -PMD
160 YouTube-VOS 2018 videos
80.2
84.5
81.1
0.295
1.62
Table 4: Comparison of data efficiency, performance, and inference overhead. Mask scores use the complete pipelines, including SAM/SAM2 where applicable; PMD and GeoMotion scores include SAM2. Trainable Params count optimized parameters only. Added FLOPs exclude SAM/SAM2 and, for PMD, frozen-backbone inference; end-to-end latency and measurement details are reported in Appendix E .
Figure 4: Low-rank structure of the probe-induced static–dynamic discriminative representation on π3 X-PMD, illustrated for DAVIS-all [ 27 ] . (a) Principal Component Analysis (PCA) illustrates clear separation between static and dynamic tokens. (b) The singular spectrum decays steeply in the leading components, especially in earlier layers, revealing strong rank concentration. (c) Performance saturates at a very low effective rank, indicating that the probe-induced motion cues are compact.
Figure 5: Scene-diversity starvation analysis. (a) Low-rank probing ( k=8 ) matches or exceeds full-rank probing under reduced training data, indicating a compact static–dynamic discriminative structure. (b) Downstream mask quality degrades gradually rather than collapsing.
Probe / labels
Params
Pair accuracy (%)
Gain (pp)
Balanced accuracy (%)
k=4 , correct
0.147M
95.70
+11.23
91.60
k=8 , correct
0.295M
95.72
+11.25
91.67
k=8 , shuffled
0.295M
84.48
+0.01
50.34
Always-same baseline
0
84.47
0.00
50.00
Table 5: Selectivity of low-rank quadratic probes on frozen π3 X tokens. Gain is the percentage-point improvement in pair accuracy over the always-same-state baseline.
Figure 6: Semantic leakage in π3 X-PMD. The two static background flamingos receive elevated dynamic scores alongside the moving foreground instance, illustrating imperfect separation of motion and semantic cues.
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
Rule
DAVIS-all
SegTrackv2
FBMS-59
Mean
Std.
Fixed 0.30
64.5
69.4
66.6
66.8
1.98
Fixed 0.40
68.1
73.3
71.1
70.8
2.15
Fixed 0.50
69.9
73.3
73.9
72.4
1.75
Fixed 0.60
69.7
69.4
74.2
71.1
2.20
Fixed 0.70
66.5
61.6
71.2
66.4
3.93
Otsu
69.1
72.2
70.8
70.7
1.28
Appendix
Table 6: Threshold robustness of coarse π3 X-PMD without SAM2. Mean and Std. summarize J-M across the three benchmarks. The last two rows instead report across-rule standard deviations for each benchmark and for the three-benchmark mean.
Split
# Seq.
# GT Frames
Static Pixels (%)
Static Upper-left 14×14 Patch (%)
Full official split
Train
2,985
89,750
82.75
97.03
Valid
492
16,563
83.08
96.61
Our subset
Train
160
4,933
87.03
97.73
Valid
80
2,893
84.79
96.58
Appendix
Table 7: Fixed-reference staticness diagnostic on YouTube-VOS 2018 [ 41 ] . The last column reports the percentage of frames in which the upper-left 14×14 patch corresponding to the fixed reference token is static under the projected-mask labeling rule.
Figure 7: Stress test of the fixed anchor token a=0 convention on a DAVIS-all [ 27 ] evaluation example for the two non-padded backbones in the main analysis. The upper-left reference patch is marked in green in the input frame. For each backbone, we compare a standard case where the fixed a=0 patch is static with a rare violation where the same patch is fully dynamic according to the ground-truth mask. For both DA3-PMD and DUSt3R-PMD, the violation moderately increases the background response, but it does not invert the prediction: the moving foreground remains separable and is recovered by both the coarse and refined masks.
Dataset
# Valid Tokens
Static Tokens (%)
Dynamic Tokens (%)
DAVIS-all [ 27 ]
392,512
92.0
8.0
SegTrackv2 [ 17 ]
68,224
95.3
4.7
FBMS-59 [ 14 ]
46,080
88.4
11.6
Appendix
Table 8: Representative token-level class imbalance on the evaluation benchmarks. Static tokens dominate across all test sets. Ratios are computed from the π3 X [ 36 ] tokenization of the probed features; normalized-gain baselines are computed separately for each backbone using its own tokenization.
Rank k
PPMD(k)
PPMD(k)/Pfull
Accuracy (%) ↑
NG ×102↑
4
147,474
0.78%
96.10
60.60
8
294,930
1.56%
96.12
60.77
16
589,842
3.12%
96.10
60.55
32
1,179,666
6.25%
96.11
60.63
64
2,359,314
12.50%
96.08
60.38
128
4,718,610
25.00%
96.08
60.38
Appendix
Table 9: Validation-based rank selection for PMD. Parameter counts are reported across candidate rank hyperparameters k , compared with the implemented full-rank quadratic probe at d=1024 and Lagg=18 aggregated blocks. For π3 X-PMD, we report the best token-level Accuracy and Normalized Gain on the shared YouTube-VOS validation subset (80 sequences) over all inter-frame blocks. This validation subset, not any target evaluation benchmark, is used to select the default rank k=8 .
Rank k
PPMD(k)
PPMD(k)/Pfull
Accuracy (%) ↑
NG ×102↑
4
147,474
0.78%
95.54
66.48
8
294,930
1.56%
95.57
66.70
16
589,842
3.12%
95.54
66.49
32
1,179,666
6.25%
95.60
66.95
64
2,359,314
12.50%
95.48
66.01
128
4,718,610
25.00%
95.36
65.09
Appendix
Table 10: Target-benchmark diagnostic rank sweep on DAVIS-all [ 27 ] . Parameter counts are reported across candidate rank hyperparameters k , compared with the implemented full-rank quadratic probe at d=1024 and Lagg=18 aggregated blocks. For π3 X-PMD, we report the best token-level Accuracy and Normalized Gain over all inter-frame blocks. This analysis is used only to diagnose whether the compact-rank behavior transfers to DAVIS-all, not for selecting k .
PMD rank k
PGeoMotion/PPMD(k)
Laggkd/(Lddec2)↓
4
5.45\times102
1.41\times10−2
8
2.72\times102
2.81\times10−2
16
1.36\times102
5.63\times10−2
32
6.81\times101
1.13\times10−1
64
3.40\times101
2.25\times10−1
128
1.70\times101
4.50\times10−1
Appendix
Table 11: Comparison between PMD and the trainable GeoMotion [ 8 ] motion head. The second column reports the concrete parameter ratio. The third column reports a simple degree-of-freedom proxy Laggkd/(Lddec2) , ignoring constant factors, with Lagg=18 , d=ddec=1024 , and L=5 ; it is not a formal sample-complexity bound.
Method
Backbone
Trainable Module
Trainable Params ↓
Added FLOPs / Frame ↓
Large-scale supervised
SegAnyMo [ 10 ]
DINOv2 + Depth Anything v2 + TAPIR
Trajectory-based motion segmentation network
3.44 M
> 2.08 G
Flow-SAM [ 39 ]
SAM + RAFT
FlowP-SAM decoder + FlowI-SAM decoder
9.83 M
> 3.63\times104 G
GeoMotion [ 8 ]
π3 + RAFT
Confidence decoder + confidence head + motion-aware decoder
80.3 M
> 3.09\times102 G
Ours (PMD, k=8 )
PMD
π3
Low-rank quadratic probe
0.295 M
1.62 G
Appendix
Table 12: Inference efficiency comparison.
Pipeline
Included inference scope
Latency (ms/frame)
Peak VRAM (GiB)
π3 X backbone
Preprocessing and frozen-backbone forward
480.6
4.47
π3 X+coarse PMD
Above, plus probe, aggregation, upsampling, thresholding
484.8
4.47
π3 X+PMD+SAM2
Above, plus sequence refinement
952.8
4.47
GeoMotion coarse [ 8 ]
Released π3 + RAFT + motion-decoder pipeline
371.3
8.83
GeoMotion+SAM2
Above, plus released sequence refinement
526.7
9.80
Flow-SAM [ 39 ]
Released RAFT + FlowP/FlowI + SAM pipeline
3,104.5
12.33
Appendix
Table 13: End-to-end inference measurements. SAM2 is included only in the corresponding refined pipelines; native SAM/SAM2 components are included for Flow-SAM and SegAnyMo.
Table 14: Additional PMD benchmarking of different feed-forward 3D foundation model backbones. Each benchmark cell reports J-Mean / J-Recall. Δ J-R denotes the J-Recall gain from SAM2 [ 29 ] refinement. Avg. Δ J-R and Std. Δ J-R are computed over DAVIS2017 [ 27 ] , SegTrackv2 [ 17 ] , and FBMS-59 [ 14 ] ; smaller gains indicate less change in the fraction of frames exceeding 0.5 Jaccard, not necessarily smaller changes in mask quality.
Backbone
Architecture
Source-val. NG mean ± std.
Target J-M ( ↑ )
Multi-view backbones (probe group: inter-frame)
π3 X [ 36 ]
DINOv2-L + 36 alternating frame/global blocks
72.46 ±
2.06
73.35
π3 [ 36 ]
DINOv2-L + 36 alternating frame/global blocks
76.76 ±
0.78
72.77
VGGT [ 34 ]
DINOv2-L + 24 paired frame/global stages
73.60 ±
4.18
72.31
DA3 [ 21 ]
40-block ViT-G; local/global alternation from block 13
61.87 ±
6.60
68.58
MapAnything [ 13 ]
DINOv2-G + separate 16-block UniCeption stack
67.20 ±
2.10
66.86
Appendix
Table 15: Source-validation pairwise decodability and target coarse-mask performance. Source NG is NGpair×102 (mean ± standard deviation across blocks in each selected group); target J-M is averaged over DAVIS17, SegTrackv2, and FBMS-59.
Broad design regime
Backbones
Mean source NG ×102
Mean target J-M
Separate ViT-L encoder + alternating frame/global multi-view stack
Pairwise asymmetric encoder–decoder with reciprocal cross-attention
DUSt3R
33.73
61.59
Appendix
Table 16: Descriptive grouping by broad architectural design regime. Source-validation NG and target J-M are averaged over the backbones in each row.
Comparison
Spearman ρ
Pearson r
All six backbones
0.77
0.93
Five multi-view backbones
0.60
0.80
Three design-regime means
1.00
0.97
Appendix
Table 17: Descriptive associations between source-validation pairwise NG and mean target coarse-mask J-M.
Method
Flow
w/o SAM2
w/ SAM2
Δ J-R ↓
J-M ↑
J-R ↑
J-M ↑
J-R ↑
Easi3R dust3r [ 5 ]
×
46.9
50.5
60.1
65.3
14.8
Easi3R monst3r [ 5 ]
×
54.8
66.2
67.9
76.1
9.9
MonST3R [ 46 ]
✓
38.1
36.1
56.4
59.6
23.5
DAS3R [ 40 ]
×
44.5
44.0
57.4
61.3
17.3
VGGT4D [ 9 ]
×
56.5
65.6
–
–
–
Appendix
Table 18: Feed-forward geometry-based motion segmentation comparison on DAVIS2017 [ 27 ] . We report J-Mean (J-M), J-Recall (J-R), and the J-Recall gain from SAM2 refinement. Δ J-R measures changes in the fraction of frames above 0.5 Jaccard; J-M separately captures changes in average mask overlap. Note that PMD introduces substantially fewer additional parameters and lower FLOP overhead than GeoMotion; see Table 12 .
Figure 9: Additional scene-diversity starvation analysis on DAVIS-all [ 27 ] . We report token-level Normalized Gain for the default low-rank quadratic probe ( k=8 ) across π3 X [ 36 ] , DA3 [ 21 ] , and DUSt3R [ 35 ] while reducing the number of YouTube-VOS training videos. Solid lines denote the mean over available random subset seeds, and shaded bands denote one standard deviation when multiple seeds are available. Performance degrades gradually as training diversity is reduced, while the uncertainty bands remain moderate overall and widen mainly in the most data-starved regimes.
Method
DAVIS16 (J-M ↑ )
DAVIS17 (J-M ↑ )
DAVIS-all (J-M ↑ )
SegTrackv2 (J-M ↑ )
FBMS-59 (J-M ↑ )
Unsupervised / Self-supervised
CIS [ 43 ]
71.5
–
–
62.0
63.5
OCLR [ 37 ]
80.9
–
–
72.3
69.8
DyStaB [ 44 ]
82.8
–
–
72.7
68.6
RCF [ 20 ]
80.9
–
–
76.7
69.9
OCMOS [ 38 ]
81.1
–
–
76.6
81.9
Appendix
Table 19: Extended motion segmentation benchmarking. All non-PMD scores are quoted from the cited papers; only PMD rows are evaluated with our evaluator. These complete-pipeline results have differing supervision and inference priors, separated in Table 2 . Trained on only 160 YouTube-VOS 2018 [ 41 ] videos, PMD transfers across DAVIS [ 26 , 27 ] , SegTrackv2 [ 17 ] , and FBMS-59 [ 14 ] ; its strongest multi-view pretrained variants approach supervised and self-supervised methods. Gray entries denote in-domain training, fine-tuning, or test-time optimization on the target benchmark. Methods using SAM/SAM2 [ 15 , 29 ] , either for external refinement or as a native pipeline component, are marked with † .
Figure 10: Scene-diversity starvation and layer-wise semantic leakage on π3 X [ 36 ] . Visualization of PMD dynamic score maps across network depth under varying amounts of training data. While deeper representations entangle motion with semantics, reducing the training subset to only 10 videos exacerbates this leakage across all layers. The starved probe incorrectly highlights static foreground objects, whereas the 160-video model maintains a focus on actual motion, particularly in its shallow and intermediate layers.
Figure 11: Limitation: Fine-grained Motion vs. Binary Detection. PMD highlights the breakdancer with the most prominent motion and assigns moderately elevated scores to nearby people with subtler in-place motion relative to the static background. However, the pipeline ultimately produces a binary static–dynamic mask rather than distinguishing different motion types or motion magnitudes.
Asset
Use in this work
License / terms
SAM2 [ 29 ]
Optional mask refinement
SAM2 code and model weights are released under Apache-2.0; the SAM2 evaluation code is released under BSD-3-Clause. We use SAM2 only as an optional refinement module and follow the official license terms.
π3 / π3 X [ 36 ]
Frozen backbone
Code is released under BSD-3-Clause; π3 and π3 X model weights are released under CC BY-NC 4.0 and are restricted to non-commercial research and educational use.
Depth Anything 3 [ 21 ]
Frozen backbone
Code is released under Apache-2.0. The DA3-GIANT-1.1 model weights used in this work are released under CC BY-NC 4.0 and are restricted to non-commercial use.
DUSt3R [ 35 ]
Frozen backbone
Code and released materials are distributed under CC BY-NC-SA 4.0, with non-commercial use only. We use DUSt3R only for research evaluation.
VGGT [ 34 ]
Frozen backbone
The VGGT repository is distributed under the custom VGGT License. According to the official repository, only the newer VGGT-1B-Commercial checkpoint is licensed for commercial usage, whereas the original VGGT-1B checkpoint remains non-commercial. Our experiments use the original VGGT-1B checkpoint only for non-commercial research evaluation and follow the corresponding license terms.
MapAnything [ 13 ]
Frozen backbone
Code is released under Apache-2.0. The official pretrained model variants have separate licenses: facebook/map-anything is CC BY-NC 4.0, while facebook/map-anything-apache is Apache-2.0. We follow the license of the specific checkpoint used in each experiment.
Appendix
Table 20: Existing pretrained models and tools used in this work. We obtain all models and tools from their official sources and follow their respective licenses and terms of use.
Asset
Use in this work
License / terms
YouTube-VOS 2018 [ 41 ]
Probe training and validation source
Annotations are licensed under CC BY 4.0; data are released for non-commercial research purposes only. We use the dataset only for research and obtain it from the official source.
DAVIS 2016/2017 [ 26 , 27 ]
Evaluation benchmark
DAVIS 2017 challenge annotations are licensed under CC BY 4.0; the official DAVIS 2017 evaluation code is released under BSD-3-Clause. We use the dataset only for benchmark evaluation and cite the official DAVIS papers.
SegTrackv2 [ 17 ]
Evaluation benchmark
The official dataset page does not explicitly state a standard license. We use SegTrackv2 only for non-commercial research evaluation, obtain it from the official project page, and cite the original dataset paper.
FBMS-59 [ 14 ]
Evaluation benchmark
Custom research-only terms: the dataset is provided only for research purposes and without warranty; commercial use is prohibited. We use FBMS-59 only for research evaluation and cite the corresponding dataset paper.
Appendix
Table 21: Existing datasets and benchmarks used in this work. We obtain all datasets from their official sources and follow their respective licenses and terms of use.
Recent depth foundation models like Depth Anything 3 (DA3) achieve remarkable multi-view depth estimation but assume static 3D scenes, limiting their applicability to real-world dynamic environments. Existing training-free 4D methods like Easi3R and VGGT4D rely on correspondence-trained backbones whose attention encodes cross-frame matching, a property absent in depth-only models like DA3. We present Dyna3, a training-free framework that extends DA3 for 4D dynamic scene reconstruction without any fine-tuning. Our key insight is that DA3's cross-view features, though trained only for depth consistency, implicitly encode motion-discriminative signals when combined with best-match feature search across frames. Its static surfaces find consistent matches globally, while dynamic objects cannot. We further adopt vision-language models (VLM) to automatically generate scene-specific semantic prompts for SAM 3, enabling precise instance-level segmentation that distinguishes which objects move from what objects exist. For reconstruction, we decouple the scene into a cross-frame aligned static background and per-frame dynamic point clouds. Experiments on four datasets demonstrate that Dyna3 surpasses correspondence-trained methods with +5.5pp J-Mean over state-of-the-art VGGT4D on dynamic object segmentation, while achieving up to 13x faster pose estimation and 3x faster 4D reconstruction with 4 to 8x lower memory. Dyna3 could therefore enable much denser temporal sampling that prior methods cannot support.
Xinhao Xiang, Weiyang Li, Zhijie Zheng +2
IFM Lab, University of California, Davis, CA, USA · Genies Inc., CA, USA
Recent progress in 3D foundation models has enabled rapid 3D reconstruction and camera calibration by leveraging learned 3D priors from vast amount of spatial data. However, the all-to-all global attention design leads to quadratic complexity and limits long-sequence inference; unconstrained cross-view interactions also can propagate unreliable evidence from occluded or visually similar but geometrically distant views. In this paper, We introduce a Masked Geometric Encoder (MGE), which promotes the learning of robust geometric representations under incomplete cross-view context. During training, MGE strategically drops frame tokens from global attention and distills from a pretrained full-context teacher model. This allows the model to learn an intrinsically richer per-frame representation while providing sufficient intermediate supervision to avoid performance degradation. Through extensive experiments, we show that MGE leads to much stronger performance under occlusion and doppelganger views while retaining high performance on standard benchmarks. Such a richer frame representation also leads to more effective token reduction during inference. To this end, we develop a novel Anchor-Guided Adaptive token merging technique that preserves representative anchor frames while jointly merging redundant tokens from the remaining views. Compared to other efficient inference approaches, we can achieve inference speedup while consistently maintaining higher reconstruction quality, particularly in limited-view settings.
Zhimin Shao, Xijun Liu, Zhaoliang Zhang +4
Department of Electrical and Computer Engineering Johns Hopkins University · Department of Data Science University of Virginia
3D foundation models (3DFMs) excel at predicting camera poses and dense depth from multiple views of a scene, showcasing strong zero-shot generalization. However, as metric scale is not observable from monocular images, their absolute scale predictions are typically inaccurate. Inertial measurement units (IMUs), present in most devices, naturally complement monocular cameras by observing scaled motion. We introduce VI3, a model-agnostic framework that metrically anchors a pretrained 3DFM using only IMU readings. VI3 initializes and preintegrates the IMU to obtain a metric motion reference, which is then used to recover the scale of the 3DFM outputs. Our method includes adaptable anchoring strategies tailored to diverse 3DFM architectures. Experiments on synthetic and real aerial datasets demonstrate that VI3 recovers metric scale without ground-truth supervision while preserving geometric consistency, acting as a fine refinement under well-conditioned motion and as a strong prior when motion is less informative.