Real-World Point Tracking with Verifier-Guided Pseudo-Labeling

Görkay Aydemir; Fatma Güney; Weidi Xie

cs.CVMar 12, 2026

Real-World Point Tracking with Verifier-Guided Pseudo-Labeling

Authors: Görkay Aydemir, Fatma Güney, Weidi Xie

Organizations: Department of Computer Engineering, Koc¸ University · KUIS AI Center · School of Artificial Intelligence, Shanghai Jiao Tong University

Abstract

Models for long-term point tracking are typically trained on large synthetic datasets. The performance of these models degrades in real-world videos due to different characteristics and the absence of dense ground-truth annotations. Self-training on unlabeled videos has been explored as a practical solution, but the quality of pseudo-labels strongly depends on the reliability of teacher models, which vary across frames and scenes. In this paper, we address the problem of real-world fine-tuning and introduce verifier, a meta-model that learns to assess the reliability of tracker predictions and guide pseudo-label generation. Given candidate trajectories from multiple pretrained trackers, the verifier evaluates them per frame and selects the most trustworthy predictions, resulting in high-quality pseudo-label trajectories. When applied for fine-tuning, verifier-guided pseudo-labeling substantially improves the quality of supervision and enables data-efficient adaptation to unlabeled videos. Extensive experiments on four real-world benchmarks demonstrate that our approach achieves state-of-the-art results while requiring less data than prior self-training methods. Project page: https://kuis-ai.github.io/track_on_r

Figures & tables

Figure 1: Verifier-guided real-world adaptation. (Left) Given a query point in a real-world video, multiple off-the-shelf trackers produce alternative trajectory hypotheses. Verifier evaluates these per-frame predictions and selects the most reliable ones, forming a refined pseudo-label trajectory. (Right) Unlike naïve self-training, which randomly selects a teacher model for pseudo-label generation, the verifier adaptively combines predictions from multiple teachers, providing cleaner supervision for the student tracker during real-world fine-tuning.

Panel of Figure 2 — Figure 2: Teacher inconsistency and oracle performance. (a) Across 4 real-world datasets, six off-the-shelf teacher models (shown on the legend) are compared against an oracle that, at each frame, selects the most accurate teacher prediction. Individual teachers (colored circles) cluster below the oracle (diamonds), while the black horizontal line marks the performance of random teacher selection. The large gap between the oracle and both individual models and random selection highlights the substantial headroom available for adaptive, per-frame selection. (b) Example from TAP-Vid Kinetics [ 11 ] : Teacher predictions whose pixel errors fluctuate across time. The upper plot shows per-frame pixel error curves, with occluded frames shaded in gray. Colored lines correspond to the same trackers as in (a), illustrating that accuracy varies across time. The lower row shows uniformly sampled frames with teacher predictions and the ground-truth point (white star).

Figure 3: Verifier overview. Given query points at frame $t_{0}$ and their candidate predictions (teacher outputs during inference or randomly augmented trajectories during training), we extract local features for both the query and each candidate, producing query features $\mathbf{f}^{q}$ (replicated across time) and candidate features $\mathbf{f}_{t}$ (a vector for each candidate, per frame). The query features are then decoded by the Candidate Transformer ( right ), which consists of restricted cross-attention, where each frame-level query attends only to its corresponding candidates, followed by self-attention and feed-forward layers. The transformer outputs per-frame reliability distributions over candidates, capturing spatial and temporal consistency based on feature similarity.

Figure 4: Verifier as inference time ensemble. Comparison of the verifier ensemble against individual teacher models and the random-teacher baseline on real-world datasets. All teacher results are reproduced using their official checkpoints. The verifier consistently achieves the best performance across datasets, demonstrating its ability to exploit the complementary strengths of different models.

Model	EgoPoints		RoboTAP			Kinetics			DAVIS
Model	$\delta_{\text{avg}}^{x}$	OA	AJ	$\delta_{\text{avg}}^{x}$	OA	AJ	$\delta_{\text{avg}}^{x}$	OA	AJ	$\delta_{\text{avg}}^{x}$	OA
Synthetic Pretraining
TAPIR [ 12 ]	50.2	79.9	59.6	73.4	87.0	49.6	64.2	85.0	56.2	70.0	86.5
LocoTrack [ 8 ]	59.6	88.6	62.3	76.2	87.1	52.9	66.8	85.3	63.0	75.3	87.2
TAPNext-B [ 48 ]	31.8	73.8	59.5	72.8	88.0	53.3	67.9	87.0	62.4	76.6	90.5
CoTracker3 (Window) [ 21 ]	—	—	60.8	73.7	87.1	54.1	66.6	87.1	64.5	76.7	89.7

Table 1: Quantitative results on real-world datasets. Comparison with prior work on EgoPoints, RoboTAP, TAP-Vid Kinetics, and TAP-Vid DAVIS in terms of AJ,

\delta_{\text{avg}}^{x}

, and OA (higher is better). Models are grouped into synthetic pretraining and real-world fine-tuning . The latter includes methods fine-tuned on additional real-world videos using semi-supervised strategies. We report officially published results when available; if no reported values exist, we evaluate the authors’ official checkpoints. Missing checkpoints are shown as “—”. † AllTracker leverages additional optical-flow datasets (synthetic and real-world with ground-truth), without pseudo-label fine-tuning.

	Teacher Models					DAVIS		RoboTAP
	A	B	C	D	E	Rand.	Ver.	Rand.	Ver.
(i)	✓	✓	✗	✗	✗	79.5	80.6	77.4	81.8
(ii)	✓	✓	✓	✗	✗	79.4	80.6	77.3	82.4
(iii)	✓	✓	✓	✓	✗	77.5	80.8	77.4	82.8
(iv)	✓	✓	✓	✓	✓	77.7	81.1	78.0	83.1

Table 2: Effect of teacher composition on verifier performance. We report

\delta_{\text{avg}}^{x}

for the random-teacher baseline (Rand.) and the verifier-based selection (Ver.) across different teacher subsets.

Data	EgoPoints		RoboTAP		Kinetics		DAVIS
Data	$\delta_{\text{avg}}^{x}$	OA	$\delta_{\text{avg}}^{x}$	OA	$\delta_{\text{avg}}^{x}$	OA	$\delta_{\text{avg}}^{x}$	OA
Real	66.9	90.2	82.7	93.8	70.9	90.4	80.3	92.4
Mix	65.6	90.7	82.3	93.9	70.8	90.3	80.7	92.5
Mix + Schedule	67.3	90.2	82.6	94.0	71.0	90.5	80.3	92.5

Table 3: Synthetic vs. real data during fine-tuning. We compare three configurations: Real (only

\mathcal{D}_{\text{real}}

), Mix (

\mathcal{D}_{\text{real}}+\mathcal{D}_{\text{syn}}

), and Mix + Schedule, where the loss weight of real-world samples is gradually increased during training.

Figure 5: Localized Feature Extraction. Given frame-wise features of the query frame $t_{0}$ and target frame $t$ , denoted by $\mathbf{F}_{t_{0}}$ and $\mathbf{F}_{t}$ , we first bilinearly sample the reference feature $\mathbf{q}_{\text{sample}}$ at the query location. A deformable attention module $\phi_{\text{def}}$ then aggregates localized context around each candidate location ( $\mathbf{C}_{t}$ ), producing descriptors $\mathbf{h}_{t}$ . We concatenate displacement embeddings $\eta(\cdot)$ with an identity embedding (query vs. candidate) and project via $\phi_{\text{proj}}$ to obtain the final query and candidate features consumed by the candidate transformer.

Model	DR	PointOdyssey
Model	$\delta_{\text{avg}}^{x}$ $\uparrow$	$\delta_{\text{avg}}^{x}$ $\uparrow$	MTE $\downarrow$	Survival $\uparrow$
BootsTAPNext	46.2	9.9	88.2	12.8
CoTracker3	72.3	44.5	20.7	56.3
Track-On2	74.5	45.1	22.0	57.7
Track-On-R (Ours)	75.1	53.4	13.4	63.1

Table 4: Quantitative results on synthetic benchmarks. We compare the synthetic baseline Track-On2 with our real-world fine-tuned variant (Track-On-R) on Dynamic Replica (DR) and PointOdyssey.

Ensemble	EgoPoints	RoboTAP	Kinetics	DAVIS
Geometric median	60.1	81.7	70.3	80.6
Agreement-based	61.8	81.8	70.4	80.5
Kalman constant-vel.	52.1	76.5	65.8	72.8
Minimum acceleration	54.0	78.3	67.2	77.8
Verifier	64.8	82.8	71.2	80.8

Table 5: Non-learning ensemble baselines vs. verifier. Comparison of fixed ensemble heuristics and the learned verifier on four benchmarks measured by

\delta_{\text{avg}}^{x}

$\mathcal{D}_{\text{real}}$	Size	EgoPoints		RoboTAP		Kinetics		DAVIS
$\mathcal{D}_{\text{real}}$	Size	$\delta_{\text{avg}}^{x}$	OA	$\delta_{\text{avg}}^{x}$	OA	$\delta_{\text{avg}}^{x}$	OA	$\delta_{\text{avg}}^{x}$	OA
TAO	2.9K	66.1	91.1	82.6	93.6	70.8	90.2	80.3	92.3
Mix	4.9K	66.9	90.2	82.7	93.8	70.9	90.4	80.3	92.4

Table 6: Effect of the

\mathcal{D}_{\text{real}}

dataset composition. We compare fine-tuning using TAO only (2921 videos) versus the full collection of TAO, OVIS, and VSPW (4864 videos).

Panel of Figure 6 — Figure 6: Verifier selection behavior across videos. Each row corresponds to a different video from TAP-Vid Kinetics. Frames are uniformly sampled among visible ones, and a $50\times 50$ crop centered at the ground-truth point is shown. Colored dots indicate predictions from teacher trackers, the star marks the ground-truth location, and the legend lists the verifier reliability scores, with the selected candidate highlighted in bold. The verifier adaptively switches between trackers across frames, assigning higher scores to spatially accurate predictions while suppressing unreliable ones.

Point Tracking

Point tracking follows selected or dense image points through video, enabling motion analysis, 3D reconstruction, and precise spatiotemporal understanding.

See all 35 papers

Temporal Self-Distillation: Learning Visual State Tracking in Videos Without Supervision
Sep 3, 2026Shravan Venkatraman, Wenshuai Zhao, Mohammad Hassan Vali +1Self-Supervised Learning State Tracking
Mohamed bin Zayed University of Artificial Intelligence, UAE · Aalto University, Finland · ELLIS Institute Finland & Department of Computer Science, Aalto University, Finland
Learning High-Quality Pseudo-Labels for Point-Supervised Temporal Action Localization
Oct 26, 2026Zhenying FangTemporal Action Localization Temporal Video Understanding
Independent Researcher
FalconTrack: Photorealistic Auto-Labeled Perception and Physics-Aware Vision-Based Aerial Tracking
Jun 29, 2026Yan Miao, Karteek Gandiboyina, Noah Giles +4Visual Object Tracking Synthetic Data Generation
Department of Electrical and Computer Engineering, University of Illinois at Urbana Champaign · Toyota Motor North America R&D

Sep 3, 2026J/K move · Enter open · S save

Temporal Self-Distillation: Learning Visual State Tracking in Videos Without Supervision

Shravan Venkatraman, Wenshuai Zhao, Mohammad Hassan Vali, Arno Solin

Mohamed bin Zayed University of Artificial Intelligence, UAE · Aalto University, Finland · ELLIS Institute Finland & Department of Computer Science, Aalto University, Finland

Self-Supervised Learning State Tracking Self-Distillation Temporal Video Understanding

Open paper AskPDF arXiv

We introduce S

^3

T (Self-Supervised Self-Distillation over Time), which, to the best of our knowledge, is the first fully self-contained framework for continuous video state tracking. Our method treats temporal sampling density as privileged information, based on the hypothesis that a denser view of the same clip recovers the running state more accurately. This view serves as the teacher, while a sparse-view student with the same weights learns to match its next-token distribution. The model generates its own target, so training requires no labels, separate teacher, or reward signal, and adds no inference cost. On LLaVA-OneVision-2-8B, S

^3

T improves VSTAT accuracy by

+1.74

as a single model,

+2.38

with souping, and

+2.70

with additional vision-encoder adaptation, while prior self-evolving methods leave state tracking largely unchanged. The capability learned from unlabeled synthetic clips transfers to real videos, improving performance by

+7.95

on VSTAT-YouTube state-tracking questions and

+4.50

on MVBench Action Count.

Abstract

Point Tracking

Explore similar work

Temporal Self-Distillation: Learning Visual State Tracking in Videos Without Supervision

Learning High-Quality Pseudo-Labels for Point-Supervised Temporal Action Localization

FalconTrack: Photorealistic Auto-Labeled Perception and Physics-Aware Vision-Based Aerial Tracking