Models for long-term point tracking are typically trained on large synthetic datasets. The performance of these models degrades in real-world videos due to different characteristics and the absence of dense ground-truth annotations. Self-training on unlabeled videos has been explored as a practical solution, but the quality of pseudo-labels strongly depends on the reliability of teacher models, which vary across frames and scenes. In this paper, we address the problem of real-world fine-tuning and introduce verifier, a meta-model that learns to assess the reliability of tracker predictions and guide pseudo-label generation. Given candidate trajectories from multiple pretrained trackers, the verifier evaluates them per frame and selects the most trustworthy predictions, resulting in high-quality pseudo-label trajectories. When applied for fine-tuning, verifier-guided pseudo-labeling substantially improves the quality of supervision and enables data-efficient adaptation to unlabeled videos. Extensive experiments on four real-world benchmarks demonstrate that our approach achieves state-of-the-art results while requiring less data than prior self-training methods. Project page: https://kuis-ai.github.io/track_on_r
Figures & tables
Figure 1: Verifier-guided real-world adaptation. (Left) Given a query point in a real-world video, multiple off-the-shelf trackers produce alternative trajectory hypotheses. Verifier evaluates these per-frame predictions and selects the most reliable ones, forming a refined pseudo-label trajectory. (Right) Unlike naïve self-training, which randomly selects a teacher model for pseudo-label generation, the verifier adaptively combines predictions from multiple teachers, providing cleaner supervision for the student tracker during real-world fine-tuning.
Figure 2: Teacher inconsistency and oracle performance. (a) Across 4 real-world datasets, six off-the-shelf teacher models (shown on the legend) are compared against an oracle that, at each frame, selects the most accurate teacher prediction. Individual teachers (colored circles) cluster below the oracle (diamonds), while the black horizontal line marks the performance of random teacher selection. The large gap between the oracle and both individual models and random selection highlights the substantial headroom available for adaptive, per-frame selection. (b) Example from TAP-Vid Kinetics [ 11 ] : Teacher predictions whose pixel errors fluctuate across time. The upper plot shows per-frame pixel error curves, with occluded frames shaded in gray. Colored lines correspond to the same trackers as in (a), illustrating that accuracy varies across time. The lower row shows uniformly sampled frames with teacher predictions and the ground-truth point (white star).
Figure 3: Verifier overview. Given query points at frame t0 and their candidate predictions (teacher outputs during inference or randomly augmented trajectories during training), we extract local features for both the query and each candidate, producing query features fq (replicated across time) and candidate features ft (a vector for each candidate, per frame). The query features are then decoded by the Candidate Transformer ( right ), which consists of restricted cross-attention, where each frame-level query attends only to its corresponding candidates, followed by self-attention and feed-forward layers. The transformer outputs per-frame reliability distributions over candidates, capturing spatial and temporal consistency based on feature similarity.
Figure 4: Verifier as inference time ensemble. Comparison of the verifier ensemble against individual teacher models and the random-teacher baseline on real-world datasets. All teacher results are reproduced using their official checkpoints. The verifier consistently achieves the best performance across datasets, demonstrating its ability to exploit the complementary strengths of different models.
Model
EgoPoints
RoboTAP
Kinetics
DAVIS
δavgx
OA
AJ
δavgx
OA
AJ
δavgx
OA
AJ
δavgx
OA
Synthetic Pretraining
TAPIR [ 12 ]
50.2
79.9
59.6
73.4
87.0
49.6
64.2
85.0
56.2
70.0
86.5
LocoTrack [ 8 ]
59.6
88.6
62.3
76.2
87.1
52.9
66.8
85.3
63.0
75.3
87.2
TAPNext-B [ 48 ]
31.8
73.8
59.5
72.8
88.0
53.3
67.9
87.0
62.4
76.6
90.5
CoTracker3 (Window) [ 21 ]
—
—
60.8
73.7
87.1
54.1
66.6
87.1
64.5
76.7
89.7
Table 1: Quantitative results on real-world datasets. Comparison with prior work on EgoPoints, RoboTAP, TAP-Vid Kinetics, and TAP-Vid DAVIS in terms of AJ, δavgx , and OA (higher is better). Models are grouped into synthetic pretraining and real-world fine-tuning . The latter includes methods fine-tuned on additional real-world videos using semi-supervised strategies. We report officially published results when available; if no reported values exist, we evaluate the authors’ official checkpoints. Missing checkpoints are shown as “—”. † AllTracker leverages additional optical-flow datasets (synthetic and real-world with ground-truth), without pseudo-label fine-tuning.
Teacher Models
DAVIS
RoboTAP
A
B
C
D
E
Rand.
Ver.
Rand.
Ver.
(i)
✓
✓
✗
✗
✗
79.5
80.6
77.4
81.8
(ii)
✓
✓
✓
✗
✗
79.4
80.6
77.3
82.4
(iii)
✓
✓
✓
✓
✗
77.5
80.8
77.4
82.8
(iv)
✓
✓
✓
✓
✓
77.7
81.1
78.0
83.1
Table 2: Effect of teacher composition on verifier performance. We report δavgx for the random-teacher baseline (Rand.) and the verifier-based selection (Ver.) across different teacher subsets.
Data
EgoPoints
RoboTAP
Kinetics
DAVIS
δavgx
OA
δavgx
OA
δavgx
OA
δavgx
OA
Real
66.9
90.2
82.7
93.8
70.9
90.4
80.3
92.4
Mix
65.6
90.7
82.3
93.9
70.8
90.3
80.7
92.5
Mix + Schedule
67.3
90.2
82.6
94.0
71.0
90.5
80.3
92.5
Table 3: Synthetic vs. real data during fine-tuning. We compare three configurations: Real (only Dreal ), Mix ( Dreal+Dsyn ), and Mix + Schedule, where the loss weight of real-world samples is gradually increased during training.
Figure 5: Localized Feature Extraction. Given frame-wise features of the query frame t0 and target frame t , denoted by Ft0 and Ft , we first bilinearly sample the reference feature qsample at the query location. A deformable attention module ϕdef then aggregates localized context around each candidate location ( Ct ), producing descriptors ht . We concatenate displacement embeddings η(⋅) with an identity embedding (query vs. candidate) and project via ϕproj to obtain the final query and candidate features consumed by the candidate transformer.
Model
DR
PointOdyssey
δavgx↑
δavgx↑
MTE ↓
Survival ↑
BootsTAPNext
46.2
9.9
88.2
12.8
CoTracker3
72.3
44.5
20.7
56.3
Track-On2
74.5
45.1
22.0
57.7
Track-On-R (Ours)
75.1
53.4
13.4
63.1
Table 4: Quantitative results on synthetic benchmarks. We compare the synthetic baseline Track-On2 with our real-world fine-tuned variant (Track-On-R) on Dynamic Replica (DR) and PointOdyssey.
Ensemble
EgoPoints
RoboTAP
Kinetics
DAVIS
Geometric median
60.1
81.7
70.3
80.6
Agreement-based
61.8
81.8
70.4
80.5
Kalman constant-vel.
52.1
76.5
65.8
72.8
Minimum acceleration
54.0
78.3
67.2
77.8
Verifier
64.8
82.8
71.2
80.8
Table 5: Non-learning ensemble baselines vs. verifier. Comparison of fixed ensemble heuristics and the learned verifier on four benchmarks measured by δavgx .
Dreal
Size
EgoPoints
RoboTAP
Kinetics
DAVIS
δavgx
OA
δavgx
OA
δavgx
OA
δavgx
OA
TAO
2.9K
66.1
91.1
82.6
93.6
70.8
90.2
80.3
92.3
Mix
4.9K
66.9
90.2
82.7
93.8
70.9
90.4
80.3
92.4
Table 6: Effect of the Dreal dataset composition. We compare fine-tuning using TAO only (2921 videos) versus the full collection of TAO, OVIS, and VSPW (4864 videos).
Figure 6: Verifier selection behavior across videos. Each row corresponds to a different video from TAP-Vid Kinetics. Frames are uniformly sampled among visible ones, and a 50×50 crop centered at the ground-truth point is shown. Colored dots indicate predictions from teacher trackers, the star marks the ground-truth location, and the legend lists the verifier reliability scores, with the selected candidate highlighted in bold. The verifier adaptively switches between trackers across frames, assigning higher scores to spatially accurate predictions while suppressing unreliable ones.
Point tracking follows selected or dense image points through video, enabling motion analysis, 3D reconstruction, and precise spatiotemporal understanding.
We introduce S3T (Self-Supervised Self-Distillation over Time), which, to the best of our knowledge, is the first fully self-contained framework for continuous video state tracking. Our method treats temporal sampling density as privileged information, based on the hypothesis that a denser view of the same clip recovers the running state more accurately. This view serves as the teacher, while a sparse-view student with the same weights learns to match its next-token distribution. The model generates its own target, so training requires no labels, separate teacher, or reward signal, and adds no inference cost. On LLaVA-OneVision-2-8B, S3T improves VSTAT accuracy by +1.74 as a single model, +2.38 with souping, and +2.70 with additional vision-encoder adaptation, while prior self-evolving methods leave state tracking largely unchanged. The capability learned from unlabeled synthetic clips transfers to real videos, improving performance by +7.95 on VSTAT-YouTube state-tracking questions and +4.50 on MVBench Action Count.
Shravan Venkatraman, Wenshuai Zhao, Mohammad Hassan Vali +1
Mohamed bin Zayed University of Artificial Intelligence, UAE · Aalto University, Finland · ELLIS Institute Finland & Department of Computer Science, Aalto University, Finland
Point-supervised temporal action localization (PTAL) aims to recognize and localize action instances in untrimmed videos using only point-level annotations. Existing pseudo-label-based methods have made substantial progress in PTAL. However, confidence-based initialization can overlook well-localized proposals. Meanwhile, existing methods either keep pseudo-labels fixed during detector training or update them without fully exploiting alternative boundary evidence from the original proposal pool. These limitations constrain supervision quality, thereby hindering detection performance. To learn high-quality pseudo-labels, we propose a framework that integrates prior-guided initialization with consensus-guided refinement during detector training. Our framework comprises two modules: Prior-Guided Initialization (PGI) and Consensus-Guided Refinement (CGR). PGI combines proposal confidence with point-derived priors to establish instance-specific reliability for initial pseudo-label selection and candidate retention. CGR continually uses the retained original candidates to construct boundary targets, combining their consensus with evolving detector predictions to refine pseudo-labels throughout training. Experiments on four benchmarks demonstrate that our method achieves competitive detection performance compared with existing methods. In particular, our method achieves 68.5% average mAP over tIoU thresholds 0.1--0.7 on THUMOS14, surpassing PseR by 3.7 percentage points. Code is available at https://github.com/zhenyingfang/LHQP
Vision-based aerial tracking is critical in GPS-denied environments. Reliable perception for tracking depends on large-scale labeled data, yet most photorealistic datasets rely on heavy manual annotation and are time-consuming to produce. We present FalconTrack, a unified perception-and-tracking framework that (i) leverages a photorealistic editable simulator for automated label generation and (ii) combines multi-head perception with physics-aware tracking for zero-shot sim-to-real transfer. FalconTrack provides an automated labeling pipeline in a Gaussian Splatting simulator that isolates target Gaussians from short object videos and composites them with randomized backgrounds to generate RGB, mask, class, and 6-DoF pose labels, producing about 10k labeled images in under 20 minutes. Using this dataset, we train a multi-head perception module with staged learning and reprojection consistency, and fuse its outputs with class-conditioned dynamics priors in an EKF for tracking. Our perception model outperforms two baselines and reaches 96-100% class accuracy in zero-shot sim-to-real transfer on three geometrically diverse objects and two environments, while maintaining consistent performance in unseen simulated and real scenes. In real hardware closed-loop visual tracking, the onboard system runs at about 25 Hz and achieves 100% success in sim-to-real F1-tenth and gate tracking in five trajectories across two environments, while a mask-centered vision baseline drops to 60% success on F1-tenth during fast out-of-view scenarios.
Yan Miao, Karteek Gandiboyina, Noah Giles +4
Department of Electrical and Computer Engineering, University of Illinois at Urbana Champaign · Toyota Motor North America R&D