GOT-JEPA: Generic Object Tracking with Model Adaptation and Occlusion Handling using Joint-Embedding Predictive Architecture
Authors: Shih-Fang Chen, Jun-Cheng Chen, I-Hong Jhuo, Yen-Yu Lin
Organizations: Department of Computer Science, National Yang Ming Chiao Tung University · Research Center for Information Technology Innovation, Academia Sinica · Microsoft AI
The human visual system tracks objects by integrating current observations with previously observed information, adapting to target and scene changes, and reasoning about occlusion at fine granularity. In contrast, recent generic object trackers are often optimized for training targets, which limits robustness and generalization in unseen scenarios, and their occlusion reasoning remains coarse, lacking detailed modeling of occlusion patterns. To address these limitations in generalization and occlusion perception, we propose GOT-JEPA, a model-predictive pretraining framework that extends JEPA from predicting image features to predicting tracking models. Given identical historical information, a teacher predictor generates pseudo-tracking models from a clean current frame, and a student predictor learns to predict the same pseudo-tracking models from a corrupted version of the current frame. This design provides stable pseudo supervision and explicitly trains the predictor to produce reliable tracking models under occlusions, distractors, and other adverse observations, improving generalization to dynamic environments. Building on GOT-JEPA, we further propose OccuSolver to enhance occlusion perception for object tracking. OccuSolver adapts a point-centric point tracker for object-aware visibility estimation and detailed occlusion-pattern capture. Conditioned on object priors iteratively generated by the tracker, OccuSolver incrementally refines visibility states, strengthens occlusion handling, and produces higher-quality reference labels that progressively improve subsequent model predictions. Extensive evaluations on seven benchmarks show that our method effectively enhances tracker generalization and robustness.
Figures & tables
Fig. 1: (a) GOT-JEPA extends the JEPA architecture to build a robust model predictor for model adaptation. The t-Predictor generates diverse tracking models for pseudo-labeling, while the s-Predictor predicts them using corrupted current frames. Both predictors use identical historical frames and tracking results as few-shot examples to aid target identification, while variations in the current frame drive robustness in tracking-model prediction. (b) OccuSolver enhances occlusion-handling capabilities. T1’ shows initial point sampling. T2’ shows that OccuSolver filters redundant points as invisible states, retaining essential points as visible states. This information is then utilized to enhance occlusion perception in the proposed GOT tracker.
Fig. 2: Overview of the proposed framework. (a) We pre-train a robust model predictor using a JEPA-based approach. Conditioned on identical past information, a student predictor (s-Predictor) learns from a corrupted current frame to predict the tracking models generated by a teacher (t-Predictor) with the uncorrupted input. This process compels the student to learn representations that are robust to frame variations. Details are provided in Sec. III-B . (b) The pre-trained student predictor is then integrated into the tracking head with classification and regression decoders and fine-tuned for precise object localization. Details are provided in Sec. III-A and Sec. III-B . (c) To address occlusions, OccuSolver adapts a point tracker to be object-aware using priors from the object tracker. The resulting point visibility states are then integrated with visual features via an Ensemble Network, enabling the final model to better handle occluded targets and generate more accurate tracking models over time. Refer to Sec. III-C for details.
Training-Test Class Overlap
Low or No Overlap
Full Overlap
Dataset
AVisT
NfS
OTB
GOT*
LaSOT
TrackingNet
Tracker
Year
SUC
SUC
SUC
AO
NPr
Pr
SUC
NPr
SUC
GOT-JEPA (Ours)
-
63.7
70.8
73.2
79.6
85.3
83.2
75.4
90.6
86.4
UniSOT [ 105 ]
2026
57.8
67.6
-
-
-
78.3
71.3
-
84.1
SAMURAI [ 106 ]
2026
-
69.2
71.5
81.7
82.7
80.2
74.2
-
85.3
PiVOT [ 5 ]
2025
62.2
68.2
71.2
76.9
84.7
82.1
73.4
90.0
85.3
TABLE I: Comparison of our method with SOTA trackers on various datasets. All trackers utilise the same training data for consistency. “*” indicates adherence to specific GOT-10k (GOT) guidelines [ 104 ] . For existing methods, we report their highest-resolution variant, prioritising ViT-L backbones when available, otherwise their best-performing variant.
Metrics
GOT-JEPA
MixFormerL
OSTrackSTB
TransT_M
SwinTrack
tomp
Robustness
0.898
0.859
0.869
0.849
0.803
0.818
AUC
0.728
0.708
0.680
0.639
0.626
0.628
TABLE II: Comparisons among different trackers on VOT-STb2022. SOTA results are reported from the VOT2022 paper [ 119 ] .
Fig. 3: This figure presents the attribute analysis of OTB-100, AVisT, and LaSOT from left to right, with the average scores at the bottom.
Dataset
Tracker
Background Clutter
Deformation
Occlusion
Out-of-View
Target Visibility
OTB
Ours
71.4
69.5
70.3
72.5
–
ToMP-L
68.2
67.1
67.3
69.4
–
PiVOT
68.4
68.6
67.8
66.1
–
ROMTrack
67.4
68.0
68.9
69.5
–
AVisT
Ours
67.2
49.0
61.3
–
67.8
ToMP-L
65.5
45.7
57.4
–
65.2
TABLE III: Attribute-wise results on OTB, AVisT, and LaSOT using large-resolution variants for several important attributes.
Dataset
NfS
AVisT
LaSOT
Tracker / Metric
OP50
Ours
89.60
73.67
86.46
ToMP-L
85.69
72.55
84.78
LoRAT-L
85.56
-
85.11
UVLTrack-L
85.58
65.11
83.14
GRM-L
83.53
63.83
82.75
TABLE IV: Comparisons among different trackers on multiple datasets using OP50 as a metric.
Fig. 4: Comparison of methods using NPr, Pr, and SUC plots on the NfS dataset, from left to right.
Fig. 5: Comparison of methods using NPr, Pr, and SUC plots on the AVisT dataset, from left to right.
Fig. 6: Comparison of methods using NPr, Pr, and SUC plots on the LaSOT dataset, from left to right.
JEPA-Pretrain
OccuSolver
AVisT
LaSOT
OTB
Inv. Loss
Cov. Loss
(1)
-
-
-
59.2
70.7
69.2
(2)
-
-
✓
59.6
71.1
69.5
(3)
✓
-
-
60.4
72.3
70.2
(4)
✓
✓
-
60.8
72.6
70.3
(5)
✓
✓
✓
61.7
73.4
70.6
TABLE V: Ablation studies on tracker components were evaluated using SUC on three datasets under GOT-JEPA-252.
Regularization Method
α
β
AVisT
LaSOT
Inv
1
0
60.4
72.3
Inv + Cov
1
1
60.1
72.1
Inv + Cov
10
1
60.6
72.5
Inv + Cov
25
1
60.8
72.6
Inv + Cov
50
1
60.7
72.6
TABLE VI: Effect of invariance and covariance loss weights in GOT-JEPA pre-training (L-252, without OccuSolver).
Tracker
JEPA Pretrain
Occu Solver
Obstruction Effects
Target Effects
Imaging Effects
Weather Conditions
Camouflage
ToMP-L
-
-
56.74
42.84
54.06
62.39
65.00
GOT-JEPA
✓
-
58.75
46.66
54.94
66.08
62.42
GOT-JEPA
✓
✓
61.86
50.59
57.44
65.83
62.45
Attribute Descriptions
Occlusion
Distractor Deformation
Low-light Archival
Target- Visibility
Background- Clutter
TABLE VII: Ablation studies on the impact of proposed components with varying attributes on the AVisT [ 115 ] dataset. The bottom row highlights more specific descriptions of the attributes.
JEPA-Pretrain
Copy-Paste
Masking
AVisT
LaSOT
(1)
59.2
70.7
(2)
✓
59.5
71.2
(3)
✓
✓
60.8
72.6
(4)
✓
✓
60.0
71.3
TABLE VIII: Evaluation of corruption types and their impact on pre-training strategies under GOT-JEPA-252.
Tracker
ProjNet
AVisT
LaSOT
OTB-100
(1)
ToMP-L
-
59.2
70.7
69.2
(2)
ToMP-L
✓
58.9
70.5
69.2
(3)
GOT-JEPA
-
59.6
71.4
69.3
(4)
GOT-JEPA
✓
60.8
72.6
70.3
TABLE IX: Evaluating the impact of ProjNet on the tracker. GOT-JEPA is evaluated at a resolution of 252 without OccuSolver.
Fig. 7: An analysis of the validation curve to study how tracker pre-training, with or without JEPA, affects the learning curve.
Model Predictor
Reg/Cls Decoders
OccuSolver
Total
Backbone
Predictor
Point Tracker
Adapters
Latency (ms)
23.82
4.39
2.30
9.43
1.39
41.34
Latency (%)
57.65
10.6
5.6
22.8
3.4
100
Trainable Parameter (M)
0
17.4
2.5
0
7.7
27.6
MACs (G)
125.5
13.9
1.8
182.8
1.1
325.1
TABLE X: The analysis quantifies computational costs of each component in GOT-JEPA-L-378.
#Points
AVisT
LaSOT
OTB-100
64
61.2
72.9
70.3
128
61.7
73.3
70.6
256
61.8
73.3
70.4
TABLE XI: Comparison of different query points for OccuSolver AUC as a metric across three datasets.
First Frame Constraint
AVisT
LaSOT
OTB-100
-
63.55
75.03
73.24
✓
63.69
75.36
73.24
TABLE XII: An ablation study on whether preventing point sampling from the occluded frame can benefit tracking.
Tracker
w/ Point Tracker
w/ LS-FineTune
w/ Obj. Prior
NPr
SUC
OP50
GOT-JEPA
✓
81.2
63.0
73.2
✓
✓
81.3
63.3
73.3
✓
✓
✓
81.9
63.7
73.7
TABLE XIII: Comparison between using the original Point Tracker for OccuSolver and our refinement strategy on the AVisT.
Fig. 8: Visual comparisons of tracking results from raw annotations, GOT-JEPA, and the baseline tracker ToMP-L across diverse video sequences under adverse scenarios are presented. Rows 1 to 3 illustrate object tracking under occlusion, including partial and full occlusion, and the tracker’s recovery after occlusion removal. Row 4 shows occlusion under distractor cases. Row 5 shows tracking with small targets. Row 6 presents tracking results for deformation cases. Best viewed when zoomed in; more detailed visual comparisons among trackers are provided in the video appendix.
Fig. 9: An ablation study investigates the frame gap between consecutive sampled frames for OccuSolver during inference, using the SUC as the metric.
Fig. 10: Comparison of relative learning rates between ProjNet and other components during the JEPA training stage.
Fig. 11: Col 1: Initial frame and initial point sampling. Col 2: Points that are more object-aware and can handle partial occlusion. Col 3: Cases of full occlusion. The original input image is shown in the yellow box.
Visual object tracking follows objects across video frames, maintaining their locations and identities despite motion, occlusion, and changing appearance.
Shenzhen International Graduate School, Tsinghua University, Shenzhen 518055, China · Department of Automation, Tsinghua University, Beijing 100084, China