GOT-JEPA: Generic Object Tracking with Model Adaptation and Occlusion Handling using Joint-Embedding Predictive Architecture

Shih-Fang Chen; Jun-Cheng Chen; I-Hong Jhuo; Yen-Yu Lin

cs.CVFeb 16, 2026

GOT-JEPA: Generic Object Tracking with Model Adaptation and Occlusion Handling using Joint-Embedding Predictive Architecture

Authors: Shih-Fang Chen, Jun-Cheng Chen, I-Hong Jhuo, Yen-Yu Lin

Organizations: Department of Computer Science, National Yang Ming Chiao Tung University · Research Center for Information Technology Innovation, Academia Sinica · Microsoft AI

Abstract

The human visual system tracks objects by integrating current observations with previously observed information, adapting to target and scene changes, and reasoning about occlusion at fine granularity. In contrast, recent generic object trackers are often optimized for training targets, which limits robustness and generalization in unseen scenarios, and their occlusion reasoning remains coarse, lacking detailed modeling of occlusion patterns. To address these limitations in generalization and occlusion perception, we propose GOT-JEPA, a model-predictive pretraining framework that extends JEPA from predicting image features to predicting tracking models. Given identical historical information, a teacher predictor generates pseudo-tracking models from a clean current frame, and a student predictor learns to predict the same pseudo-tracking models from a corrupted version of the current frame. This design provides stable pseudo supervision and explicitly trains the predictor to produce reliable tracking models under occlusions, distractors, and other adverse observations, improving generalization to dynamic environments. Building on GOT-JEPA, we further propose OccuSolver to enhance occlusion perception for object tracking. OccuSolver adapts a point-centric point tracker for object-aware visibility estimation and detailed occlusion-pattern capture. Conditioned on object priors iteratively generated by the tracker, OccuSolver incrementally refines visibility states, strengthens occlusion handling, and produces higher-quality reference labels that progressively improve subsequent model predictions. Extensive evaluations on seven benchmarks show that our method effectively enhances tracker generalization and robustness.

Figures & tables

Fig. 1: (a) GOT-JEPA extends the JEPA architecture to build a robust model predictor for model adaptation. The t-Predictor generates diverse tracking models for pseudo-labeling, while the s-Predictor predicts them using corrupted current frames. Both predictors use identical historical frames and tracking results as few-shot examples to aid target identification, while variations in the current frame drive robustness in tracking-model prediction. (b) OccuSolver enhances occlusion-handling capabilities. T1’ shows initial point sampling. T2’ shows that OccuSolver filters redundant points as invisible states, retaining essential points as visible states. This information is then utilized to enhance occlusion perception in the proposed GOT tracker.

Fig. 2: Overview of the proposed framework. (a) We pre-train a robust model predictor using a JEPA-based approach. Conditioned on identical past information, a student predictor (s-Predictor) learns from a corrupted current frame to predict the tracking models generated by a teacher (t-Predictor) with the uncorrupted input. This process compels the student to learn representations that are robust to frame variations. Details are provided in Sec. III-B . (b) The pre-trained student predictor is then integrated into the tracking head with classification and regression decoders and fine-tuned for precise object localization. Details are provided in Sec. III-A and Sec. III-B . (c) To address occlusions, OccuSolver adapts a point tracker to be object-aware using priors from the object tracker. The resulting point visibility states are then integrated with visual features via an Ensemble Network, enabling the final model to better handle occluded targets and generate more accurate tracking models over time. Refer to Sec. III-C for details.

Training-Test Class Overlap		Low or No Overlap				Full Overlap
Dataset		AVisT	NfS	OTB	GOT*	LaSOT			TrackingNet
Tracker	Year	SUC	SUC	SUC	AO	NPr	Pr	SUC	NPr	SUC
GOT-JEPA (Ours)	-	63.7	70.8	73.2	79.6	85.3	83.2	75.4	90.6	86.4
UniSOT [ 105 ]	2026	57.8	67.6	-	-	-	78.3	71.3	-	84.1
SAMURAI [ 106 ]	2026	-	69.2	71.5	81.7	82.7	80.2	74.2	-	85.3
PiVOT [ 5 ]	2025	62.2	68.2	71.2	76.9	84.7	82.1	73.4	90.0	85.3

TABLE I: Comparison of our method with SOTA trackers on various datasets. All trackers utilise the same training data for consistency. “*” indicates adherence to specific GOT-10k (GOT) guidelines [ 104 ] . For existing methods, we report their highest-resolution variant, prioritising ViT-L backbones when available, otherwise their best-performing variant.

Metrics	GOT-JEPA	MixFormerL	OSTrackSTB	TransT_M	SwinTrack	tomp
Robustness	0.898	0.859	0.869	0.849	0.803	0.818
AUC	0.728	0.708	0.680	0.639	0.626	0.628

TABLE II: Comparisons among different trackers on VOT-STb2022. SOTA results are reported from the VOT2022 paper [ 119 ] .

Fig. 3: This figure presents the attribute analysis of OTB-100, AVisT, and LaSOT from left to right, with the average scores at the bottom.

Dataset	Tracker	Background Clutter	Deformation	Occlusion	Out-of-View	Target Visibility
OTB	Ours	71.4	69.5	70.3	72.5	–
	ToMP-L	68.2	67.1	67.3	69.4	–
	PiVOT	68.4	68.6	67.8	66.1	–
	ROMTrack	67.4	68.0	68.9	69.5	–
AVisT	Ours	67.2	49.0	61.3	–	67.8
	ToMP-L	65.5	45.7	57.4	–	65.2

TABLE III: Attribute-wise results on OTB, AVisT, and LaSOT using large-resolution variants for several important attributes.

Dataset	NfS	AVisT	LaSOT
Tracker / Metric	OP50
Ours	89.60	73.67	86.46
ToMP-L	85.69	72.55	84.78
LoRAT-L	85.56	-	85.11
UVLTrack-L	85.58	65.11	83.14
GRM-L	83.53	63.83	82.75

TABLE IV: Comparisons among different trackers on multiple datasets using OP50 as a metric.

Fig. 4: Comparison of methods using NPr, Pr, and SUC plots on the NfS dataset, from left to right.

Fig. 5: Comparison of methods using NPr, Pr, and SUC plots on the AVisT dataset, from left to right.

Fig. 6: Comparison of methods using NPr, Pr, and SUC plots on the LaSOT dataset, from left to right.

	JEPA-Pretrain		OccuSolver	AVisT	LaSOT	OTB
	Inv. Loss	Cov. Loss	OccuSolver	AVisT	LaSOT	OTB
(1)	-	-	-	59.2	70.7	69.2
(2)	-	-	✓	59.6	71.1	69.5
(3)	✓	-	-	60.4	72.3	70.2
(4)	✓	✓	-	60.8	72.6	70.3
(5)	✓	✓	✓	61.7	73.4	70.6

TABLE V: Ablation studies on tracker components were evaluated using SUC on three datasets under GOT-JEPA-252.

Regularization Method	$\alpha$	$\beta$	AVisT	LaSOT
Inv	1	0	60.4	72.3
Inv + Cov	1	1	60.1	72.1
Inv + Cov	10	1	60.6	72.5
Inv + Cov	25	1	60.8	72.6
Inv + Cov	50	1	60.7	72.6

TABLE VI: Effect of invariance and covariance loss weights in GOT-JEPA pre-training (L-252, without OccuSolver).

Tracker	JEPA Pretrain	Occu Solver	Obstruction Effects	Target Effects	Imaging Effects	Weather Conditions	Camouflage
ToMP-L	-	-	56.74	42.84	54.06	62.39	65.00
GOT-JEPA	✓	-	58.75	46.66	54.94	66.08	62.42
GOT-JEPA	✓	✓	61.86	50.59	57.44	65.83	62.45
Attribute Descriptions			Occlusion	Distractor Deformation	Low-light Archival	Target- Visibility	Background- Clutter

TABLE VII: Ablation studies on the impact of proposed components with varying attributes on the AVisT [ 115 ] dataset. The bottom row highlights more specific descriptions of the attributes.

	JEPA-Pretrain	Copy-Paste	Masking	AVisT	LaSOT
(1)				59.2	70.7
(2)		✓		59.5	71.2
(3)	✓	✓		60.8	72.6
(4)	✓		✓	60.0	71.3

TABLE VIII: Evaluation of corruption types and their impact on pre-training strategies under GOT-JEPA-252.

Tracker		ProjNet	AVisT	LaSOT	OTB-100
(1)	ToMP-L	-	59.2	70.7	69.2
(2)	ToMP-L	✓	58.9	70.5	69.2
(3)	GOT-JEPA	-	59.6	71.4	69.3
(4)	GOT-JEPA	✓	60.8	72.6	70.3

TABLE IX: Evaluating the impact of ProjNet on the tracker. GOT-JEPA is evaluated at a resolution of 252 without OccuSolver.

Fig. 7: An analysis of the validation curve to study how tracker pre-training, with or without JEPA, affects the learning curve.

	Model Predictor		Reg/Cls Decoders	OccuSolver		Total
	Backbone	Predictor	Reg/Cls Decoders	Point Tracker	Adapters	Total
Latency (ms)	23.82	4.39	2.30	9.43	1.39	41.34
Latency (%)	57.65	10.6	5.6	22.8	3.4	100
Trainable Parameter (M)	0	17.4	2.5	0	7.7	27.6
MACs (G)	125.5	13.9	1.8	182.8	1.1	325.1

TABLE X: The analysis quantifies computational costs of each component in GOT-JEPA-L-378.

#Points	AVisT	LaSOT	OTB-100
64	61.2	72.9	70.3
128	61.7	73.3	70.6
256	61.8	73.3	70.4

TABLE XI: Comparison of different query points for OccuSolver AUC as a metric across three datasets.

First Frame Constraint	AVisT	LaSOT	OTB-100
-	63.55	75.03	73.24
✓	63.69	75.36	73.24

TABLE XII: An ablation study on whether preventing point sampling from the occluded frame can benefit tracking.

Tracker	w/ Point Tracker	w/ LS-FineTune	w/ Obj. Prior	NPr	SUC	OP50
GOT-JEPA	✓			81.2	63.0	73.2
	✓	✓		81.3	63.3	73.3
	✓	✓	✓	81.9	63.7	73.7

TABLE XIII: Comparison between using the original Point Tracker for OccuSolver and our refinement strategy on the AVisT.

Fig. 8: Visual comparisons of tracking results from raw annotations, GOT-JEPA, and the baseline tracker ToMP-L across diverse video sequences under adverse scenarios are presented. Rows 1 to 3 illustrate object tracking under occlusion, including partial and full occlusion, and the tracker’s recovery after occlusion removal. Row 4 shows occlusion under distractor cases. Row 5 shows tracking with small targets. Row 6 presents tracking results for deformation cases. Best viewed when zoomed in; more detailed visual comparisons among trackers are provided in the video appendix.

Fig. 9: An ablation study investigates the frame gap between consecutive sampled frames for OccuSolver during inference, using the SUC as the metric.

Fig. 10: Comparison of relative learning rates between ProjNet and other components during the JEPA training stage.

Fig. 11: Col 1: Initial frame and initial point sampling. Col 2: Points that are more object-aware and can handle partial occlusion. Col 3: Cases of full occlusion. The original input image is shown in the yellow box.

Visual Object Tracking

Visual object tracking follows objects across video frames, maintaining their locations and identities despite motion, occlusion, and changing appearance.

See all 175 papers

Jul 1, 2026

Rethinking Generic Object Tracking Toward Human-Level Perceptual Intelligence

At the heart of human visual perception lies the ability to maintain a continuous and coherent understanding of the external world. By integrating observations with accumulated experience, the human visual system can continuously adapt to variations in both the target and its surrounding environment, while preserving robust visual continuity as scene dynamics evolve. Human vision can therefore integrate prior knowledge, spatial geometry, and semantic context to understand complex scenes and their changes. As a core problem in computer vision, visual object tracking aims to bring machine perception closer to human visual perception. These capabilities are central to the task of Generic Object Tracking (GOT). In this task, a visual tracker is initialized only with the bounding box of an arbitrarily specified target in the first frame, and must continuously localize the target in subsequent dynamic visual streams. However, future events, observations, and real-world variations are inherently unpredictable; therefore, the model's generalization and online adaptation capabilities remain bottlenecks. Tracking reliability can deteriorate when the target undergoes severe deformation, is affected by complex distractors, encounters significant environmental changes, or belongs to a category unseen during training. This dissertation aims to narrow the gap between machine visual tracking systems and human visual perception by proposing a series of methods that systematically enhance the target discrimination, robust adaptation, and geometric reasoning capabilities of tracking models.

Shih-Fang Chen

National Yang Ming Chiao Tung University Institute of Computer Science and Engineering

Visual Object Tracking

Ask PDF

Jul 2, 2026

Mask-supervised Object-centric Representation Learning with LeJEPA

Self-supervised image encoders deliver strong features for downstream tasks but need many images for training. A natural remedy to counter this is to make each image count for more. A scene contains many objects, and given masks from human annotators or an off-the-shelf segmentation model, pre-training can focus on aligning per-object rather than image-wide representations, extracting more signal from every image. Existing mask-supervised methods do this through reconstruction or contrastive losses that leverage negative objects. We instead use two separate projection spaces for the alignment. In a semantic space, per-object representations from different views are aligned. To avoid collapse, instead of using negative objects, which requires category definitions, we extend the negative-free LeJEPA objective and show that its distributional anti-collapse regularizer ports naturally from whole images to the variable-sized set of objects in a scene. In an instance space, a contrastive loss separates per-object representations from their context and co-occurring instances, including those of the same category. To separate object representations from their context, we copy objects and paste them into other contexts, where each pasted copy serves as an additional view of the original object. Trained on COCO with ground-truth masks, our method outperforms image-level and mask-guided baselines on tracking (DAVIS), classification (ImageNet-1k) and re-identification (NAVI), matches the best of them on semantic segmentation (ADE20k) and keeps its lead over image-level LeJEPA and a supervision-matched alternative on COCO fractions down to 256 images.

Jakob Geusen, Ender Konukoglu

Biomedical Image Computing Group, ETH Zurich

Contrastive Learning Representation Learning Joint-Embedding Predictive Architecture Object-Centric Representation Learning Self-Supervised Visual Representation Learning

Ask PDF

May 21, 2026

Segment Anything with Motion, Geometry, and Semantic Adaptation for Complex Nonlinear Visual Object Tracking

Traditional visual object tracking (VOT) methods typically rely on task-specific supervised training, limiting their generalization to unseen objects and challenging scenarios with distractors, occlusion, and nonlinear motion. Recent vision foundation models, exemplified by SAM 2, learn strong video understanding priors from large-scale pretraining and offer a promising foundation for building more robust and generalizable trackers. However, directly applying SAM 2 to VOT remains suboptimal, as it does not explicitly model target motion dynamics or enforce geometric and semantic consistency across frames, both of which are essential for reliable tracking. To address this issue, we propose SAMOSA, a new tracking framework that adapts SAM 2 to complex VOT scenarios by explicitly leveraging motion, geometry, and semantic cues. Specifically, we introduce a lightweight nonlinear motion predictor to model target dynamics and guide mask selection as well as memory filtering. We further exploit semantic cues to detect target shifts and recover from tracking failures, while geometric cues are incorporated as structural constraints to improve tracking stability. In this way, SAMOSA bridges the gap between the implicit video understanding prior of SAM 2 and explicit tracking-oriented modeling. Extensive experiments show that SAMOSA consistently outperforms state-of-the-art SAM 2--based approaches on general benchmarks, demonstrates stronger generalization than supervised VOT methods, and achieves substantial gains on anti-UAV datasets, which typify complex nonlinear motion scenarios. Our code is available at https://github.com/DurYi/SAMOSA.

Deyi Zhu, Yuji Wang, Yong Liu +4

Shenzhen International Graduate School, Tsinghua University, Shenzhen 518055, China · Department of Automation, Tsinghua University, Beijing 100084, China

Visual Object Tracking Vision Foundation Model Adaptation

Ask PDF

Abstract

Visual Object Tracking

Explore similar work

Rethinking Generic Object Tracking Toward Human-Level Perceptual Intelligence

Mask-supervised Object-centric Representation Learning with LeJEPA

Segment Anything with Motion, Geometry, and Semantic Adaptation for Complex Nonlinear Visual Object Tracking