ProMo3D: Probing Motion Cues in Frozen 3D Foundation Models

Xiaoang Zhang; Mert Kiray; Yordanka Velikova; Manoj Biswanath; Benjamin Busam

cs.CVOct 24, 2026

ProMo3D: Probing Motion Cues in Frozen 3D Foundation Models

Authors: Xiaoang Zhang, Mert Kiray, Yordanka Velikova, Manoj Biswanath, Benjamin Busam

Organizations: Technical University of Munich (TUM), Germany · Munich Center for Machine Learning (MCML), Germany · Obsphera, Munich, Germany

Abstract

Feed-forward 3D foundation models have recently reshaped multi-view reconstruction, yet dynamic 4D understanding still often relies on large-scale motion supervision or hand-crafted, architecture-specific heuristics. We ask whether frozen 3D foundation model tokens already encode motion-relevant information that can be exposed by lightweight readout functions, rather than re-learning motion-specific features with high-capacity decoders. Through probing on three representative backbones, we find that static--dynamic separation is decodable from frozen token representations. Our rank-restricted and spectral analyses on $π^3$ X reveal that the recovered motion cues are concentrated in a low-dimensional subspace. Building on this, we propose Probing-based Motion Distillation (PMD), which converts pairwise motion-state compatibility scores against a fixed anchor token into dense motion masks. Trained on only 160 YouTube-VOS videos using a single 16 GB GPU, PMD transfers zero-shot to DAVIS, SegTrackv2, and FBMS-59 without target-benchmark fine-tuning. With multi-view pretrained geometric backbones, PMD outperforms training-free heuristics and approaches substantially heavier supervised and self-supervised methods, suggesting that motion decoding from frozen 3D foundation models can be lightweight and data-efficient.

Figures & tables

Figure 1: Overview of Probing-based Motion Distillation (PMD). Given an input frame group, a frozen feed-forward 3D foundation model such as $\pi^{3}$ X [ 36 ] , VGGT [ 34 ] , or Depth Anything 3 (DA3) [ 21 ] produces token representations. A lightweight probe trained on only 160 videos maps these features into a probe-induced static–dynamic discriminative representation. PMD then converts anchor-based pairwise probe scores into dense dynamic score maps, which are thresholded into coarse masks and optionally refined with SAM2 [ 29 ] . It transfers zero-shot to DAVIS [ 26 , 27 ] , SegTrackv2 [ 17 ] , and FBMS-59 [ 14 ] , while requiring substantially less training data and trainable capacity than high-capacity motion decoders such as GeoMotion [ 8 ] .

Backbone	Selected Blocks	Probe	Rank $k$
$\pi^{3}$ / $\pi^{3}$ X [ 36 ]	$18$ × inter-frame blocks	low-rank quad.	8
DA3 [ 21 ]	$14$ × inter-frame blocks	low-rank quad.	8
DUSt3R [ 35 ]	$12$ × decoder-view-1 + $4$ × head-in-view-1 blocks	low-rank quad.	8

Table 1: Default PMD settings. All probes are trained on a subset of

160

YouTube-VOS [ 41 ] videos. For Depth Anything 3 (DA3), we use the DA3-GIANT-1.1 model.

Figure 2: Layer-wise motion decodability on DAVIS-all [ 27 ] . Low-rank quadratic probes expose static–dynamic discriminative cues across the multi-view geometric modules of all three backbones, outperforming linear and diagonal probes.

Method	Motion-mask supervision; inference components	DAVIS17 J-M / J-R	SegTrackv2 J-M / J-R	FBMS-59 J-M / J-R
Baselines without motion-mask supervision
VGGT4D [ 9 ]	None; VGGT geometry	56.5/65.6	–	–
Easi3R dust3r [ 5 ]	None; DUSt3R geometry	46.9/50.5	–	–
Easi3R dust3r +SAM2	None; DUSt3R geometry, SAM2	60.1/65.3	–	–
Baselines with motion-mask supervision
GeoMotion [ 8 ]	Five datasets; $\pi^{3}$ , RAFT, DINOv2	75.4 /87.2	–	–

Table 2: Motion segmentation benchmarking. Cells report J-M/J-R (

\uparrow

). First , second , and third mark the top three distinct reported scores for each dataset and metric across all rows; ties share a color. Non-PMD scores are quoted from the cited papers; PMD scores are evaluated by us. Gray text marks target-benchmark fine-tuning; “–” denotes an unreported score.

Method	Flow	w/o SAM2		w/ SAM2		$\Delta$ J-R $\downarrow$
Method	Flow	J-M $\uparrow$	J-R $\uparrow$	J-M $\uparrow$	J-R $\uparrow$	$\Delta$ J-R $\downarrow$
Easi3R dust3r [ 5 ]	$\times$	46.9	50.5	60.1	65.3	14.8
Easi3R monst3r [ 5 ]	$\times$	54.8	66.2	67.9	76.1	9.9
GeoMotion [ 8 ]	$\checkmark$	75.4	87.2	81.1	92.6	5.4
$\pi^{3}$ -PMD (ours)	$\times$	72.7	89.9	80.2	91.1	1.2
$\pi^{3}$ X-PMD (ours)	$\times$	73.7	91.0	81.9	92.3	1.3

Table 3: SAM2 [ 29 ] refinement on DAVIS2017 [ 27 ] .

\Delta

J-R is the percentage-point increase in the fraction of frames with Jaccard above 0.5. Additional backbones and methods are reported in Appendices G.2 and G.4 .

Figure 3: Qualitative comparison on DAVIS [ 26 ] and SegTrackv2 [ 17 ] against training-free methods based on feed-forward 3D foundation models. The semantic classes shown here are absent from the 160-video training subset of $\pi^{3}$ X-PMD. All binary masks shown here are generated without SAM2 [ 29 ] refinement. PMD produces cleaner dynamic score maps and more coherent coarse masks.

Method	Training Data	DAVIS17 (J-M $\uparrow$ )	SegTrackv2 (J-M $\uparrow$ )	FBMS-59 (J-M $\uparrow$ )	Trainable Params (M) $\downarrow$	Added FLOPs (G) $\downarrow$
Large-scale supervised
SegAnyMo [ 10 ]	Kubric + Dynamic Replica + HOI4D + etc.	–	$76.3$	$78.3$	$3.44$	> $2.08$
Flow-SAM [ 39 ]	Synthetic pretraining + DAVIS fine tuning + etc.	$71.6$	$84.2$	$88.7$	$9.83$	> $3.63\text{\times}{10}^{4}$
GeoMotion [ 8 ]	GOT-Motion + HOI4D + Dynamic Replica + etc.	$81.1$	$77.3$	$72.5$	$80.3$	> $3.09\text{\times}{10}^{2}$
Ours (data-efficient probing)
$\pi^{3}$ -PMD	$160$ YouTube-VOS 2018 videos	$80.2$	$84.5$	$81.1$	$0.295$	$1.62$

Table 4: Comparison of data efficiency, performance, and inference overhead. Mask scores use the complete pipelines, including SAM/SAM2 where applicable; PMD and GeoMotion scores include SAM2. Trainable Params count optimized parameters only. Added FLOPs exclude SAM/SAM2 and, for PMD, frozen-backbone inference; end-to-end latency and measurement details are reported in Appendix E .

Panel of Figure 4 — Figure 4: Low-rank structure of the probe-induced static–dynamic discriminative representation on $\pi^{3}$ X-PMD, illustrated for DAVIS-all [ 27 ] . (a) Principal Component Analysis (PCA) illustrates clear separation between static and dynamic tokens. (b) The singular spectrum decays steeply in the leading components, especially in earlier layers, revealing strong rank concentration. (c) Performance saturates at a very low effective rank, indicating that the probe-induced motion cues are compact.

Panel of Figure 5 — Figure 5: Scene-diversity starvation analysis. (a) Low-rank probing ( $k=8$ ) matches or exceeds full-rank probing under reduced training data, indicating a compact static–dynamic discriminative structure. (b) Downstream mask quality degrades gradually rather than collapsing.

Probe / labels	Params	Pair accuracy (%)	Gain (pp)	Balanced accuracy (%)
$k=4$ , correct	0.147M	95.70	+11.23	91.60
$k=8$ , correct	0.295M	95.72	+11.25	91.67
$k=8$ , shuffled	0.295M	84.48	+0.01	50.34
Always-same baseline	0	84.47	0.00	50.00

Table 5: Selectivity of low-rank quadratic probes on frozen

\pi^{3}

X tokens. Gain is the percentage-point improvement in pair accuracy over the always-same-state baseline.

Panel of Figure 6 — Figure 6: Semantic leakage in $\pi^{3}$ X-PMD. The two static background flamingos receive elevated dynamic scores alongside the moving foreground instance, illustrating imperfect separation of motion and semantic cues.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Rule	DAVIS-all	SegTrackv2	FBMS-59	Mean	Std.
Fixed 0.30	64.5	69.4	66.6	66.8	1.98
Fixed 0.40	68.1	73.3	71.1	70.8	2.15
Fixed 0.50	69.9	73.3	73.9	72.4	1.75
Fixed 0.60	69.7	69.4	74.2	71.1	2.20
Fixed 0.70	66.5	61.6	71.2	66.4	3.93
Otsu	69.1	72.2	70.8	70.7	1.28

Appendix

Table 6: Threshold robustness of coarse

\pi^{3}

X-PMD without SAM2. Mean and Std. summarize J-M across the three benchmarks. The last two rows instead report across-rule standard deviations for each benchmark and for the three-benchmark mean.

Split	# Seq.	# GT Frames	Static Pixels (%)	Static Upper-left $14{\times}14$ Patch (%)
Full official split
Train	2,985	89,750	82.75	97.03
Valid	492	16,563	83.08	96.61
Our subset
Train	160	4,933	87.03	97.73
Valid	80	2,893	84.79	96.58

Appendix

Table 7: Fixed-reference staticness diagnostic on YouTube-VOS 2018 [ 41 ] . The last column reports the percentage of frames in which the upper-left

14\times 14

patch corresponding to the fixed reference token is static under the projected-mask labeling rule.

Panel of Figure 7 — Figure 7: Stress test of the fixed anchor token $a=0$ convention on a DAVIS-all [ 27 ] evaluation example for the two non-padded backbones in the main analysis. The upper-left reference patch is marked in green in the input frame. For each backbone, we compare a standard case where the fixed $a=0$ patch is static with a rare violation where the same patch is fully dynamic according to the ground-truth mask. For both DA3-PMD and DUSt3R-PMD, the violation moderately increases the background response, but it does not invert the prediction: the moving foreground remains separable and is recovered by both the coarse and refined masks.

Dataset	# Valid Tokens	Static Tokens (%)	Dynamic Tokens (%)
DAVIS-all [ 27 ]	392,512	92.0	8.0
SegTrackv2 [ 17 ]	68,224	95.3	4.7
FBMS-59 [ 14 ]	46,080	88.4	11.6

Appendix

Table 8: Representative token-level class imbalance on the evaluation benchmarks. Static tokens dominate across all test sets. Ratios are computed from the

\pi^{3}

X [ 36 ] tokenization of the probed features; normalized-gain baselines are computed separately for each backbone using its own tokenization.

Rank $k$	$P_{\mathrm{PMD}}(k)$	$P_{\mathrm{PMD}}(k)/P_{\mathrm{full}}$	Accuracy (%) $\uparrow$	NG $\times 10^{2}$ $\uparrow$
4	147,474	0.78%	96.10	60.60
8	294,930	1.56%	96.12	60.77
16	589,842	3.12%	96.10	60.55
32	1,179,666	6.25%	96.11	60.63
64	2,359,314	12.50%	96.08	60.38
128	4,718,610	25.00%	96.08	60.38

Appendix

Table 9: Validation-based rank selection for PMD. Parameter counts are reported across candidate rank hyperparameters

k

, compared with the implemented full-rank quadratic probe at

d=1024

and

L_{\mathrm{agg}}=18

aggregated blocks. For

\pi^{3}

X-PMD, we report the best token-level Accuracy and Normalized Gain on the shared YouTube-VOS validation subset (80 sequences) over all inter-frame blocks. This validation subset, not any target evaluation benchmark, is used to select the default rank

k=8

Rank $k$	$P_{\mathrm{PMD}}(k)$	$P_{\mathrm{PMD}}(k)/P_{\mathrm{full}}$	Accuracy (%) $\uparrow$	NG $\times 10^{2}$ $\uparrow$
4	147,474	0.78%	95.54	66.48
8	294,930	1.56%	95.57	66.70
16	589,842	3.12%	95.54	66.49
32	1,179,666	6.25%	95.60	66.95
64	2,359,314	12.50%	95.48	66.01
128	4,718,610	25.00%	95.36	65.09

Appendix

Table 10: Target-benchmark diagnostic rank sweep on DAVIS-all [ 27 ] . Parameter counts are reported across candidate rank hyperparameters

k

, compared with the implemented full-rank quadratic probe at

d=1024

and

L_{\mathrm{agg}}=18

aggregated blocks. For

\pi^{3}

X-PMD, we report the best token-level Accuracy and Normalized Gain over all inter-frame blocks. This analysis is used only to diagnose whether the compact-rank behavior transfers to DAVIS-all, not for selecting

k

PMD rank $k$	$P_{\mathrm{GeoMotion}}/P_{\mathrm{PMD}}(k)$	$L_{\mathrm{agg}}kd/(Ld_{\mathrm{dec}}^{2})\,\downarrow$
4	$5.45\text{\times}{10}^{2}$	$1.41\text{\times}{10}^{-2}$
8	$2.72\text{\times}{10}^{2}$	$2.81\text{\times}{10}^{-2}$
16	$1.36\text{\times}{10}^{2}$	$5.63\text{\times}{10}^{-2}$
32	$6.81\text{\times}{10}^{1}$	$1.13\text{\times}{10}^{-1}$
64	$3.40\text{\times}{10}^{1}$	$2.25\text{\times}{10}^{-1}$
128	$1.70\text{\times}{10}^{1}$	$4.50\text{\times}{10}^{-1}$

Appendix

Table 11: Comparison between PMD and the trainable GeoMotion [ 8 ] motion head. The second column reports the concrete parameter ratio. The third column reports a simple degree-of-freedom proxy

L_{\mathrm{agg}}kd/(Ld_{\mathrm{dec}}^{2})

, ignoring constant factors, with

L_{\mathrm{agg}}=18

d=d_{\mathrm{dec}}=1024

, and

L=5

; it is not a formal sample-complexity bound.

Method	Backbone	Trainable Module	Trainable Params $\downarrow$	Added FLOPs / Frame $\downarrow$
Large-scale supervised
SegAnyMo [ 10 ]	DINOv2 + Depth Anything v2 + TAPIR	Trajectory-based motion segmentation network	$3.44$ M	> $2.08$ G
Flow-SAM [ 39 ]	SAM + RAFT	FlowP-SAM decoder + FlowI-SAM decoder	$9.83$ M	> $3.63\text{\times}{10}^{4}$ G
GeoMotion [ 8 ]	$\pi^{3}$ + RAFT	Confidence decoder + confidence head + motion-aware decoder	$80.3$ M	> $3.09\text{\times}{10}^{2}$ G
Ours (PMD, $k=8$ )
PMD	$\pi^{3}$	Low-rank quadratic probe	$0.295$ M	$1.62$ G

Appendix

Table 12: Inference efficiency comparison.

Pipeline	Included inference scope	Latency (ms/frame)	Peak VRAM (GiB)
$\pi^{3}$ X backbone	Preprocessing and frozen-backbone forward	480.6	4.47
$\pi^{3}$ X+coarse PMD	Above, plus probe, aggregation, upsampling, thresholding	484.8	4.47
$\pi^{3}$ X+PMD+SAM2	Above, plus sequence refinement	952.8	4.47
GeoMotion coarse [ 8 ]	Released $\pi^{3}$ + RAFT + motion-decoder pipeline	371.3	8.83
GeoMotion+SAM2	Above, plus released sequence refinement	526.7	9.80
Flow-SAM [ 39 ]	Released RAFT + FlowP/FlowI + SAM pipeline	3,104.5	12.33

Appendix

Table 13: End-to-end inference measurements. SAM2 is included only in the corresponding refined pipelines; native SAM/SAM2 components are included for Flow-SAM and SegAnyMo.

Figure 8: Additional layer-wise motion decodability on DAVIS-all [ 27 ] : VGGT [ 34 ] ,

\pi^{3}

[ 36 ] , and MapAnything [ 13 ] .

Model	Refine	DAVIS17 (J-M / J-R) $\uparrow$	SegTrackv2 (J-M / J-R) $\uparrow$	FBMS-59 (J-M / J-R) $\uparrow$	Avg. $\Delta$ J-R $\downarrow$	Std. $\Delta$ J-R $\downarrow$
$\pi^{3}$ -PMD	w/o SAM2	72.7 / 89.9	72.9 / 92.9	72.6 / 83.4	2.59	1.45
$\pi^{3}$ -PMD	w/ SAM2	80.2 / 91.1	84.5 / 97.5	81.1 / 85.4	2.59	1.45
$\pi^{3}$ X-PMD	w/o SAM2	73.7 / 91.0	72.9 / 94.0	73.5 / 88.8	2.37	2.11
$\pi^{3}$ X-PMD	w/ SAM2	81.9 / 92.3	86.4 / 99.3	83.2 / 89.3	2.37	2.11
DA3-PMD	w/o SAM2	69.6 / 85.8	66.3 / 80.3	69.8 / 81.4	6.36	2.96
DA3-PMD	w/ SAM2	77.7 / 88.2	75.7 / 89.8	82.3 / 88.6	6.36	2.96

Appendix

Table 14: Additional PMD benchmarking of different feed-forward 3D foundation model backbones. Each benchmark cell reports J-Mean / J-Recall.

\Delta

J-R denotes the J-Recall gain from SAM2 [ 29 ] refinement. Avg.

\Delta

J-R and Std.

\Delta

J-R are computed over DAVIS2017 [ 27 ] , SegTrackv2 [ 17 ] , and FBMS-59 [ 14 ] ; smaller gains indicate less change in the fraction of frames exceeding 0.5 Jaccard, not necessarily smaller changes in mask quality.

Backbone	Architecture	Source-val. NG mean $\pm$ std.		Target J-M ( $\uparrow$ )
Multi-view backbones (probe group: inter-frame)
$\pi^{3}$ X [ 36 ]	DINOv2-L + 36 alternating frame/global blocks	72.46 $\pm$	2.06	73.35
$\pi^{3}$ [ 36 ]	DINOv2-L + 36 alternating frame/global blocks	76.76 $\pm$	0.78	72.77
VGGT [ 34 ]	DINOv2-L + 24 paired frame/global stages	73.60 $\pm$	4.18	72.31
DA3 [ 21 ]	40-block ViT-G; local/global alternation from block 13	61.87 $\pm$	6.60	68.58
MapAnything [ 13 ]	DINOv2-G + separate 16-block UniCeption stack	67.20 $\pm$	2.10	66.86

Appendix

Table 15: Source-validation pairwise decodability and target coarse-mask performance. Source NG is

\mathrm{NG}_{\mathrm{pair}}\times 10^{2}

(mean

\pm

standard deviation across blocks in each selected group); target J-M is averaged over DAVIS17, SegTrackv2, and FBMS-59.

Broad design regime	Backbones	Mean source NG $\times 10^{2}$	Mean target J-M
Separate ViT-L encoder + alternating frame/global multi-view stack	$\pi^{3}$ X, $\pi^{3}$ , VGGT	74.27	72.81
ViT-G-scale local/global multi-view stack + dense geometry heads	DA3, MapAnything	64.54	67.72
Pairwise asymmetric encoder–decoder with reciprocal cross-attention	DUSt3R	33.73	61.59

Appendix

Table 16: Descriptive grouping by broad architectural design regime. Source-validation NG and target J-M are averaged over the backbones in each row.

Comparison	Spearman $\rho$	Pearson $r$
All six backbones	0.77	0.93
Five multi-view backbones	0.60	0.80
Three design-regime means	1.00	0.97

Appendix

Table 17: Descriptive associations between source-validation pairwise NG and mean target coarse-mask J-M.

Method	Flow	w/o SAM2		w/ SAM2		$\Delta$ J-R $\downarrow$
Method	Flow	J-M $\uparrow$	J-R $\uparrow$	J-M $\uparrow$	J-R $\uparrow$	$\Delta$ J-R $\downarrow$
Easi3R dust3r [ 5 ]	$\times$	46.9	50.5	60.1	65.3	14.8
Easi3R monst3r [ 5 ]	$\times$	54.8	66.2	67.9	76.1	9.9
MonST3R [ 46 ]	$\checkmark$	38.1	36.1	56.4	59.6	23.5
DAS3R [ 40 ]	$\times$	44.5	44.0	57.4	61.3	17.3
VGGT4D [ 9 ]	$\times$	56.5	65.6	–	–	–

Appendix

Table 18: Feed-forward geometry-based motion segmentation comparison on DAVIS2017 [ 27 ] . We report J-Mean (J-M), J-Recall (J-R), and the J-Recall gain from SAM2 refinement.

\Delta

J-R measures changes in the fraction of frames above 0.5 Jaccard; J-M separately captures changes in average mask overlap. Note that PMD introduces substantially fewer additional parameters and lower FLOP overhead than GeoMotion; see Table 12 .

Figure 9: Additional scene-diversity starvation analysis on DAVIS-all [ 27 ] . We report token-level Normalized Gain for the default low-rank quadratic probe (

k=8

) across

\pi^{3}

X [ 36 ] , DA3 [ 21 ] , and DUSt3R [ 35 ] while reducing the number of YouTube-VOS training videos. Solid lines denote the mean over available random subset seeds, and shaded bands denote one standard deviation when multiple seeds are available. Performance degrades gradually as training diversity is reduced, while the uncertainty bands remain moderate overall and widen mainly in the most data-starved regimes.

Method	DAVIS16 (J-M $\uparrow$ )	DAVIS17 (J-M $\uparrow$ )	DAVIS-all (J-M $\uparrow$ )	SegTrackv2 (J-M $\uparrow$ )	FBMS-59 (J-M $\uparrow$ )
Unsupervised / Self-supervised
CIS [ 43 ]	71.5	–	–	62.0	63.5
OCLR [ 37 ]	80.9	–	–	72.3	69.8
DyStaB [ 44 ]	82.8	–	–	72.7	68.6
RCF [ 20 ]	80.9	–	–	76.7	69.9
OCMOS [ 38 ]	81.1	–	–	76.6	81.9

Appendix

Table 19: Extended motion segmentation benchmarking. All non-PMD scores are quoted from the cited papers; only PMD rows are evaluated with our evaluator. These complete-pipeline results have differing supervision and inference priors, separated in Table 2 . Trained on only 160 YouTube-VOS 2018 [ 41 ] videos, PMD transfers across DAVIS [ 26 , 27 ] , SegTrackv2 [ 17 ] , and FBMS-59 [ 14 ] ; its strongest multi-view pretrained variants approach supervised and self-supervised methods. Gray entries denote in-domain training, fine-tuning, or test-time optimization on the target benchmark. Methods using SAM/SAM2 [ 15 , 29 ] , either for external refinement or as a native pipeline component, are marked with † .

Panel of Figure 10 — Figure 10: Scene-diversity starvation and layer-wise semantic leakage on $\pi^{3}$ X [ 36 ] . Visualization of PMD dynamic score maps across network depth under varying amounts of training data. While deeper representations entangle motion with semantics, reducing the training subset to only 10 videos exacerbates this leakage across all layers. The starved probe incorrectly highlights static foreground objects, whereas the 160-video model maintains a focus on actual motion, particularly in its shallow and intermediate layers.

Panel of Figure 11 — Figure 11: Limitation: Fine-grained Motion vs. Binary Detection. PMD highlights the breakdancer with the most prominent motion and assigns moderately elevated scores to nearby people with subtler in-place motion relative to the static background. However, the pipeline ultimately produces a binary static–dynamic mask rather than distinguishing different motion types or motion magnitudes.

Asset	Use in this work	License / terms
SAM2 [ 29 ]	Optional mask refinement	SAM2 code and model weights are released under Apache-2.0; the SAM2 evaluation code is released under BSD-3-Clause. We use SAM2 only as an optional refinement module and follow the official license terms.
$\pi^{3}$ / $\pi^{3}$ X [ 36 ]	Frozen backbone	Code is released under BSD-3-Clause; $\pi^{3}$ and $\pi^{3}$ X model weights are released under CC BY-NC 4.0 and are restricted to non-commercial research and educational use.
Depth Anything 3 [ 21 ]	Frozen backbone	Code is released under Apache-2.0. The DA3-GIANT-1.1 model weights used in this work are released under CC BY-NC 4.0 and are restricted to non-commercial use.
DUSt3R [ 35 ]	Frozen backbone	Code and released materials are distributed under CC BY-NC-SA 4.0, with non-commercial use only. We use DUSt3R only for research evaluation.
VGGT [ 34 ]	Frozen backbone	The VGGT repository is distributed under the custom VGGT License. According to the official repository, only the newer VGGT-1B-Commercial checkpoint is licensed for commercial usage, whereas the original VGGT-1B checkpoint remains non-commercial. Our experiments use the original VGGT-1B checkpoint only for non-commercial research evaluation and follow the corresponding license terms.
MapAnything [ 13 ]	Frozen backbone	Code is released under Apache-2.0. The official pretrained model variants have separate licenses: facebook/map-anything is CC BY-NC 4.0, while facebook/map-anything-apache is Apache-2.0. We follow the license of the specific checkpoint used in each experiment.

Appendix

Table 20: Existing pretrained models and tools used in this work. We obtain all models and tools from their official sources and follow their respective licenses and terms of use.

Asset	Use in this work	License / terms
YouTube-VOS 2018 [ 41 ]	Probe training and validation source	Annotations are licensed under CC BY 4.0; data are released for non-commercial research purposes only. We use the dataset only for research and obtain it from the official source.
DAVIS 2016/2017 [ 26 , 27 ]	Evaluation benchmark	DAVIS 2017 challenge annotations are licensed under CC BY 4.0; the official DAVIS 2017 evaluation code is released under BSD-3-Clause. We use the dataset only for benchmark evaluation and cite the official DAVIS papers.
SegTrackv2 [ 17 ]	Evaluation benchmark	The official dataset page does not explicitly state a standard license. We use SegTrackv2 only for non-commercial research evaluation, obtain it from the official project page, and cite the original dataset paper.
FBMS-59 [ 14 ]	Evaluation benchmark	Custom research-only terms: the dataset is provided only for research purposes and without warranty; commercial use is prohibited. We use FBMS-59 only for research evaluation and cite the corresponding dataset paper.

Appendix

Table 21: Existing datasets and benchmarks used in this work. We obtain all datasets from their official sources and follow their respective licenses and terms of use.

Motion Segmentation

See all 17 papers

Dyna3: VLM-Guided Training-Free 4D Reconstruction via Depth Foundation Models
Oct 1, 2026Xinhao Xiang, Weiyang Li, Zhijie Zheng +2Multi-View Depth Estimation 4D Reconstruction
IFM Lab, University of California, Davis, CA, USA · Genies Inc., CA, USA
Less Context, Better Geometry: Masked Geometric Encoder for Robust 3D Foundation Models
Oct 5, 2026Zhimin Shao, Xijun Liu, Zhaoliang Zhang +43D Foundation Models Token Merging
Department of Electrical and Computer Engineering Johns Hopkins University · Department of Data Science University of Virginia
VI3: Grounding Pretrained 3D Foundation Models with Inertial Cues
Sep 3, 2026Ernesto Lozano, Alberto Jaenal, Javier Civera3D Foundation Models
I3A, Universidad de Zaragoza, Spain

Oct 1, 2026J/K move · Enter open · S save

Dyna3: VLM-Guided Training-Free 4D Reconstruction via Depth Foundation Models

Xinhao Xiang, Weiyang Li, Zhijie Zheng, Abhijeet Rastogi+1

IFM Lab, University of California, Davis, CA, USA · Genies Inc., CA, USA

Multi-View Depth Estimation 4D Reconstruction Dynamic Scene Reconstruction 4D Scene Understanding

Open paper AskPDF arXiv

Recent depth foundation models like Depth Anything 3 (DA3) achieve remarkable multi-view depth estimation but assume static 3D scenes, limiting their applicability to real-world dynamic environments. Existing training-free 4D methods like Easi3R and VGGT4D rely on correspondence-trained backbones whose attention encodes cross-frame matching, a property absent in depth-only models like DA3. We present Dyna3, a training-free framework that extends DA3 for 4D dynamic scene reconstruction without any fine-tuning. Our key insight is that DA3's cross-view features, though trained only for depth consistency, implicitly encode motion-discriminative signals when combined with best-match feature search across frames. Its static surfaces find consistent matches globally, while dynamic objects cannot. We further adopt vision-language models (VLM) to automatically generate scene-specific semantic prompts for SAM 3, enabling precise instance-level segmentation that distinguishes which objects move from what objects exist. For reconstruction, we decouple the scene into a cross-frame aligned static background and per-frame dynamic point clouds. Experiments on four datasets demonstrate that Dyna3 surpasses correspondence-trained methods with +5.5pp J-Mean over state-of-the-art VGGT4D on dynamic object segmentation, while achieving up to 13x faster pose estimation and 3x faster 4D reconstruction with 4 to 8x lower memory. Dyna3 could therefore enable much denser temporal sampling that prior methods cannot support.

Abstract

Motion Segmentation

Explore similar work

Dyna3: VLM-Guided Training-Free 4D Reconstruction via Depth Foundation Models

Less Context, Better Geometry: Masked Geometric Encoder for Robust 3D Foundation Models

VI3: Grounding Pretrained 3D Foundation Models with Inertial Cues