M2P: Improving Visual Foundation Models with Mask-to-Point Weakly-Supervised Learning for Dense Point Tracking
Authors: Qiangqiang Wu, Tianyu Yang, Bo Fang, Jia Wan, Matias Di Martino, Guillermo Sapiro, Antoni B. Chan
Organizations: Department of Computer Science, City University of Hong Kong, China · Department of Electrical and Computer Engineering, Princeton University, Princeton, NJ 08544 USA · Meituan, Shenzhen, China · School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen 518066, China · Department of Electrical and Computer Engineering, Duke University, Durham, NC 27708 USA
Tracking Any Point (TAP) has emerged as a fundamental tool for video understanding. Current approaches adapt Vision Foundation Models (VFMs) like DINOv2 via offline finetuning or test-time optimization. However, these VFMs rely on static image pre-training, which is inherently sub-optimal for capturing dense temporal correspondence in videos. To address this, we propose Mask-to-Point (M2P) learning, which leverages rich video object segmentation (VOS) mask annotations to improve VFMs for dense point tracking. Our M2P introduces three new mask-based constraints for weakly-supervised representation learning. First, we propose a local structure consistency loss, which leverages Procrustes analysis to model the cohesive motion of points lying within a local structure, achieving more reliable point-to-point matching learning. Second, we propose a mask label consistency (MLC) loss, which enforces that sampled foreground points strictly match foreground regions across frames. The proposed MLC loss can be regarded as a regularization, which stabilizes training and prevents convergence to trivial solutions. Finally, mask boundary constrain is applied to explicitly supervise boundary points. We show that our weaklysupervised M2P models significantly outperform baseline VFMs with efficient training by using only 3.6K VOS training videos. Notably, M2P achieves 12.8% and 14.6% performance gains over DINOv2-B/14 and DINOv3-B/16 on the TAP-Vid-DAVIS benchmark, respectively. Moreover, the proposed M2P models are used as pre-trained backbones for both test-time optimized and offline fine-tuned TAP tasks, demonstrating its potential to serve as general pre-trained models for point tracking. Code will be made publicly available upon acceptance.
Figures & tables
Fig. 1: Our proposed Mask-to-Point (M2P) learning improves visual foundation models (e.g., DINOs) on dense point tracking, while being trained on only 3.6K VOS videos.
Fig. 2: Overall pipeline of the proposed Mask-to-Point (M2P) weakly-supervised learning, which enhances existing visual foundation models (VFMs) for dense point tracking by leveraging video object segmentation (VOS) datasets with mask annotations. Our M2P introduces three new mask-based constraints for representation learning, including local structure consistency , mask label consistency and mask boundary constrain .
Fig. 3: Illustration of our proposed local structure consistency loss LLSC . The top- Ke strong matches are found between query points pg and predicted correspondences, {p~g}i=1Ke . These strongly matched points are used to estimate a simalarity transformation, which is then applied to the other points to obtain remaining pseudo-labels p~ig , i>Ke , for supervision.
Fig. 4: Illustration of the confidence score Sig in our mask label consistency constraint. Zoom in for clearer visualization.
Fig. 5: An overall pipeline of the proposed M2P-Tracker for test-time optimization TAP. The proposed M2P-Tracker uses our M2Pv3-S/16 backbone for feature extraction, providing strong temporal prior for more effective online adaptation.
Config
Value
optimizer
AdamW [ 55 ]
base learning rate
2.0e-5
weight decay
1.0e-7
batch size
16
training iteration
150,000
lora rank
64
TABLE I: The pre-training setting for our M2P learning.
Config
Value
optimizer
AdamW [ 55 ]
base learning rate
1.5e-4
weight decay
1.0e-4
batch size
4
training iteration
100,000
warm iteration
300
TABLE II: The training setup for offline TAP fine-tuning on the synthetic Kubric [ 11 ] dataset..
Model
Setup
Size
#L.P.
Data
TAP-VID-DAVIS
TAP-VID-Kinetics
RGB-Stacking
MAE [ 57 ]
ZS
32 × 32
-
S
23.5
27.6
43.2
DeiT [ 58 ]
ZS
32 × 32
-
S
24.0
22.4
23.3
CLIP [ 59 ]
ZS
32 × 32
-
S
25.4
25.0
33.8
DINO [ 60 ]
ZS
32 × 32
-
S
34.5
34.4
39.3
DINOv2- S /14 [ 14 ]
ZS
32 × 32
-
S
37.1
33.4
36.9
DINOv2- B /14-Reg [ 14 ]
ZS
32 × 32
-
S
37.4
33.6
35.9
TABLE III: Comparison on TAP-VID-DAVIS [ 17 ] , TAP-VID-Kinetics [ 56 ] and RGB-Stacking [ 19 ] under the queried-first mode . Models are commonly evaluated at a final feature map resolution of 32×32 or 30×45 , following the original evaluation in image-based and video-based models. ZS, WS and FS denote zero-shot, weakly-supervised and fully-supervised learning settings, respectively. D [ 17 ] , Y [ 15 ] , and SA-V [ 16 ] denote the training video splits of corresponding VOS datasets with mask annotations, while S and V refers to static-image and video training data, respectively. #L.P. represents the number of learnable parameters. Our M2Pv2 and M2Pv3 are built upon DINOv2 and DINOv3 with specified LoRA ranks (R). The best result for each backbone is shown in bold.
Backbone
TAP-VID-DAVIS-First
TAP-VID-Kinetics-First
TAP-VID-DAVIS-Strided
δ0
δ1
δ2
δ3
δ4
δavgx
δ0
δ1
δ2
δ3
δ4
δavgx
δ0
δ1
δ2
δ3
δ4
δavgx
ResNet-18 [ 11 , 68 ]
9.0
27.3
54.9
73.7
84.1
49.8
8.6
28.8
56.5
74.2
83.3
50.3
9.7
31.1
60.6
78.3
87.0
53.3
TSM-ResNet-18 [ 19 , 69 ]
7.3
23.1
46.7
66.6
79.2
44.6
7.8
28.1
55.2
71.4
80.2
48.6
8.2
26.8
53.6
73.5
83.9
49.2
CoTracker [ 8 ]
27.8
46.4
58.6
64.1
68.5
53.1
24.1
41.9
55.1
62.2
67.0
50.1
33.7
52.5
63.4
68.3
72.0
58.0
DINOv2 (ViT-S) [ 14 ]
6.0
19.9
47.5
73.7
84.5
46.3
4.2
13.6
36.3
65.9
79.2
39.8
7.3
22.7
52.9
80.0
89.5
50.4
DINOv2 (ViT-B) [ 14 ]
8.9
24.7
53.8
77.0
87.1
50.3
5.1
16.0
40.0
67.9
80.4
41.9
10.0
28.6
59.5
82.9
87.4
54.4
TABLE IV: Backbone comparison on the TAP-Vid datasets [ 19 ] . For fair comparison, all backbones are evaluated at a final feature map resolution of 64×64 following [ 13 ] , without applying any iterative refinement . For each column, the best results are shown in bold.
Backbone
TAP-VID-Kinetics-Strided
δ0
δ1
δ2
δ3
δ4
δavgx
ResNet-18 [ 11 , 68 ]
10.5
35.3
65.7
81.3
88.6
56.3
TSM-ResNet-18 [ 19 , 69 ]
9.15
33.2
64.6
79.2
86.5
54.5
CoTracker [ 8 ]
31.0
31.0
50.5
70.6
74.9
58.2
DINOv2 (ViT-S) [ 14 ]
4.9
15.8
41.9
73.8
86.0
44.5
DINOv2 (ViT-B) [ 14 ]
5.9
18.6
45.9
75.7
86.9
46.6
TABLE V: Backbone comparison on the TAP-Vid-Kinetics dataset [ 19 ] with the strided query mode.
Fig. 6: Training on Kubric [ 11 ] and validation on TAP-VID-DAVIS with the strided query mode. Our M2P-Chrono significantly outperforms both the original Chrono [ 13 ] and the Chrono variant with DINOv3-S/16 during the entire training, demonstrating that M2P provides a stronger temporal matching prior, which leads to more stable and efficient fine-tuning on synthetic data.
Fig. 7: Qualitative visualization of query-to-frame correlation maps obtained from DINOv2-B/14 [ 14 ] , DINOv3-B/16 [ 48 ] and our proposed M2Pv3-B/16, where red stars denote query/GT points, and yellow stars denote predicted points. Our M2Pv3 generates sharper and more discriminative correlation maps, while DINOv3 and DINOv2 show noisy responses over distractor regions, which often result in drifting tracks. Zoom in for clearer visualization.
Method
Mode
#L.P.
Hours
DAVIS-480
BADJA
δavgx
OA
AJ
δseg
RAFT [ 70 ]
-
-
-
66.7
-
-
45.0
DINOv2 [ 14 ]
-
-
-
66.7
-
-
62.8
TAPNet [ 19 ]
S
-
-
66.4
79.0
46.0
45.4
PIPs++ [ 20 ]
S
-
-
73.6
-
-
59.0
TAPIR [ 11 ]
S
-
-
77.3
89.5
65.7
68.7
TABLE VI: Online optimization-based tracker comparison on TAP-Vid-DAVIS-480 [ 19 ] and BADJA [ 72 ] . Our test-time online optimized M2P-Tracker outperforms DINO-Tracker by using 15× fewer learnable parameters. ‘#L.P.’ denotes learnable parameters. Modes ‘S’ and ‘TT’ indicate supervised and test-time optimization, respectively. ‘Hours’ denotes the optimization time per video for each test-time model optimized on an RTX4090 GPU.
Fig. 8: Occlusion accuracy (OA) on TAP-Vid-DAVIS-480 [ 7 , 17 ] under varying occlusion levels. We partition the TAP-Vid-DAVIS-480 test set into three subsets based on each video’s average occlusion rate, computed from the ground-truth visibility masks.
Fig. 9: Qualitative visualization of the average query-to-frame correlation maps and point tracking results produced by DINOv2-B/14, DINOv3-B/16, and our M2Pv3-B/16. The average correlation maps are computed by averaging multiple query-to-frame correlation maps. Ground-truth points are shown as open circles, and their corresponding predicted points (i.e., filled circles) are connected with dashed lines to indicate the prediction errors. Please zoom in for clearer details.
VFM
LSC
MLC
MBC
δavgx
DINOv2-B/14
38.0
DINOv2-B/14
✓
49.0 (+ 11.0% )
DINOv2-B/14
✓
✓
50.0 (+ 12.0% )
DINOv2-B/14
✓
✓
50.3 (+ 12.3% )
DINOv2-B/14
✓
✓
✓
50.8 (+ 12.8% )
DINOv3-S/16
38.0
TABLE VII: Component analysis of our M2P learning on TAP-VID-DAVIS with the first query mode.
Fig. 10: Qualitative visualization of the dense point tracking results obtained by DINO-Tracker [ 7 ] and our M2P-tracker on TAP-VID-DAVIS 480. We use red circles to highlight the regions with inaccurate tracked points.
Variant
Module
Rank
#L.P.
δavgx
DINOv2-B/14
-
-
-
38.0
DINOv2-B/14
QKV
8
0.3M
48.3
DINOv2-B/14
QKV
16
0.6M
49.0
DINOv2-B/14
QKV
32
1.2M
49.4
DINOv2-B/14
QKV
64
2.4M
50.5
DINOv2-B/14
QKV
128
4.7M
50.1
TABLE VIII: Ablation of LoRA training on TAP-VID-DAVIS with the first query mode.
Variant
δavgx
DINOv2-B/14
38.0
w/ Pseudo Point Learning
45.6
w/ M2P Learning
50.8
M2Pv2-B/14
50.8
w/ More Data (D + Y + MOOSE [ 73 ] )
51.5
M2Pv3-S/16
50.5
TABLE IX: Component analysis of our M2P learning on TAP-VID-DAVIS with the first query mode.
Fig. 11: (a) Impact of MLC on M2Pv2-B/16-R@8 (QKV); (b) Ablation on the point number K within each group.
Point tracking follows selected or dense image points through video, enabling motion analysis, 3D reconstruction, and precise spatiotemporal understanding.
Despite achieving strong results on standard benchmarks, current point tracking methods rely on feature backbones that are rarely designed with the temporal coherence needed for robust real-world performance. While recent works incorporate powerful visual foundation model (VFM) features into tracking pipelines, no prior work has systematically analyzed which VFM provides the most robust representations for point tracking. We present the first such analysis, evaluating diverse VFMs in a zero-shot setting on both standard and robustness benchmarks for point tracking. Our study reveals that video diffusion transformers (DiTs) consistently yield the most temporally coherent and discriminative features, even surpassing ResNet backbones explicitly supervised on tracking data. We hypothesize this advantage stem from large-scale video pretraining, full 3D spatio-temporal attention, and a diffusion training objective. Motivated by this finding, we propose DiTracker, which integrates video DiT features into existing tracking frameworks through query-key matching cost computation, cost-level fusion with a lightweight ResNet branch, and LoRA adaptation. Under the same tracking head, DiTracker is trained solely on synthetic data with far fewer iterations, yet outperforms CoTracker3 trained with additional real-world videos, with the largest gains under challenging and corrupted scenarios. It further generalizes across tracking heads and scales with backbone size, confirming that generative video pretraining provides real-world priors that reduce the dependence on large-scale real-data supervision.
Dynamic point cloud pretraining is still dominated by masked reconstruction objectives. However, these objectives inherit two key limitations. Existing methods inject ground-truth tube centers as decoder positional embeddings, causing spatio-temporal positional leakage. Moreover, they supervise inter-frame motion with deterministic proxy targets that systematically discard distributional structure by collapsing multimodal trajectory uncertainty into conditional means. To address these limitations, we propose Diffusion Masked Pretraining (DiMP), a unified self-supervised framework for dynamic point clouds. DiMP introduces diffusion modeling into both positional inference and motion learning. It first applies forward diffusion noise only to masked tube centers, then predicts clean centers from visible spatio-temporal context. This removes positional leakage while preserving visible coordinates as clean temporal anchors. DiMP also reformulates point-wise inter-frame displacement supervision as a DDPM noise-prediction objective conditioned on decoded representations. This design drives the encoder to target the full conditional distribution of plausible motions under a variational surrogate, rather than collapsing to a single deterministic estimate. Extensive experiments demonstrate that DiMP consistently improves downstream accuracy over the backbone alone, with absolute gains of 11.21% on offline action segmentation and 13.65% under causally constrained online inference.Codes are available at https://github.com/InitalZ/DiMP.git.
Zhuoyue Zhang, Yiding Sun, Haozhe Cheng +4
School of Software Engineering, Xi’an Jiaotong University, Xi’an, China · School of Artificial Intelligence and Robotics, Hunan University, Changsha, China · Department of Computer Science and Software Engineering, University of Western Australia, Perth, Australia
Existing point tracking models face a fundamental tradeoff: they can either track a sparse set of query points over long horizons, or track all points across only short clips. We introduce TrackEverything, a 3D point tracker that breaks this trade-off by representing videos as persistent 3D scene tracks in world coordinates. Grounded in the insight that videos are 2D projections of an underlying 3D world, TrackEverything decouples model complexity from video duration, allowing it to scale with unique physical scene geometry instead. Our approach introduces three key innovations. First, we employ a voxelization-based de-duplication mechanism at sliding-window boundaries to merge co-located tracks, preventing repeated observations of the same surface from redundantly accumulating. Second, we decompose tracking into an endpoint refiner that predicts each point's destination and static-versus-dynamic classification, followed by a lightweight trajectory refiner that decodes dense trajectories exclusively for dynamic points. Third, we propose 3D WAFT, replacing memory-prohibitive 4D correlation volumes with efficient feature sampling in the scene cloud. To the best of our knowledge, TrackEverything is the first 3D tracker capable of tracking all visible points across videos exceeding 1000 frames within 40 GB of GPU memory. On TAPVid-3D, TrackEverything outperforms all open-source all-frame dense 3D trackers by more than 20% APD on short clips, while remaining competitive with state-of-the-art sparse trackers on long sequences, despite tracking far more points.