Drift-Resilient Temporal Priors for Visual Tracking

Yuqing Huang; Liting Lin; Weijun Zhuang; Zhenyu He; Xin Li

cs.CVApr 3, 2026

Drift-Resilient Temporal Priors for Visual Tracking

Authors: Yuqing Huang, Liting Lin, Weijun Zhuang, Zhenyu He, Xin Li

Organizations: Harbin Institute of Technology, Shenzhen · Pengcheng Laboratory · Lero, the Research Ireland Centre for Software, University of Limerick · Pazhou Lab (Huangpu)

Abstract

Temporal information is crucial for visual tracking, but existing multi-frame trackers are vulnerable to model drift caused by naively aggregating noisy historical predictions. In this paper, we introduce DTPTrack, a lightweight and generalizable module designed to be seamlessly integrated into existing trackers to suppress drift. Our framework consists of two core components: (1) a Temporal Reliability Calibrator (TRC) mechanism that learns to assign a per-frame reliability score to historical states, filtering out noise while anchoring on the ground-truth template; and (2) a Temporal Guidance Synthesizer (TGS) module that synthesizes this calibrated history into a compact set of dynamic temporal priors to provide predictive guidance. To demonstrate its versatility, we integrate DTPTrack into three diverse tracking architectures--OSTrack, ODTrack, and LoRAT-and show consistent, significant performance gains across all baselines. Our best-performing model, built upon an extended LoRATv2 backbone, sets a new state-of-the-art on several benchmarks, achieving a 77.5% Success rate on LaSOT and an 80.3% AO on GOT-10k.

Figures & tables

Figure 1: Comparison of temporal modeling strategies in visual tracking. (a) Autoregressive trackers propagate historical predictions through a sequence model, making them vulnerable to cumulative errors. (b) Dynamic memory trackers update an internal memory over time, but noisy predictions can contaminate the memory state. (c) Online spatial–temporal trackers process short video clips jointly but treat all historical frames as equally reliable. (d) DTPTrack uses Temporal Reliability Calibrator(TRC) to reliability-weight historical frames and Temporal Guidance Synthesizer (TGS) to produce temporal prior tokens that guide the tracking backbone and suppress drift.

Figure 2: Architectural Overview of the DTPTrack Module within our Extended LoRATv2 Backbone. Our module operates in two stages before the main Transformer blocks. First, the Temporal Reliability Calibrator (TRC) block summarizes the feature embeddings of the template ( $z_{0}$ ) and three historical reference frames ( $z_{1},z_{2},z_{3}$ ) and computes a reliability weight for each. The confidence for the initial ground-truth frame ( $z_{0}$ ) is fixed to 1.0. Second, the Temporal Guidance Synthesizer (TGS) block converts these calibrated summaries into a set of dynamic prior tokens. These tokens are then prepended to the original sequence of visual features to guide the host tracker’s backbone through its Frame-Wise Causal Attention layers.

Method	Source	LaSOT			VastTrack		GOT-10k*			TrackingNet
Method	Source	AUC	P ${}_{\textrm{Norm}}$	P	AUC	P	AO	SR 0.5	SR 0.75	AUC	P ${}_{\textrm{Norm}}$
DTPTrack-B 224	Ours	74.3	83.6	80.7	40.7	40.2	76.5	87.6	74.9	85.6	90.1
DTPTrack-L 378	Ours	77.5	86.5	84.7	47.2	49.7	80.3	89.9	81.9	86.9	90.8
LoRATv2-B 224 [ 30 ]	NIPS25	72.0	81.3	77.9	39.1	38.7	74.7	84.7	72.7	84.1	-
LoRATv2-L 378 [ 30 ]	NIPS25	76.1	85.1	83.1	44.2	46.7	78.2	86.8	79.1	85.7	-
SPMTrack-L [ 4 ]	CVPR25	76.8	85.9	84.0	-	-	80.0	89.4	79.9	86.9	91.0

Table 1: State-of-the-art comparison on LaSOT, VastTrack, GOT-10k and TrackingNet. ‘*’ denotes for trackers trained only with GOT-10k train split. The best three results are highlighted in red , blue and bold , respectively.

Method	UAV123	OTB2015	TNL2K
DTPTrack-L 378	72.3	74.7	63.7
DTPTrack-B 224	71.9	73.2	60.3
SPMTrack-B	71.7	72.7	62.0
HIPTrack [ 3 ]	70.5	71.0	-
ARTrackV2-B 256 [ 1 ]	69.9	-	59.2
ODTrack-B 384 [ 53 ]	-	72.3	60.9

Table 2: Comparison with state-of-the-art methods on additional benchmarks in the AUC score.

Method	#	MACs (G)	Params (M)	FPS
OSTrack-256 [ 52 ]	2	21.5	92.1	106.2
OSTrack-384 [ 52 ]	2	48.4	92.1	62.0
ODTrack-B [ 53 ]	5	73.1	92.1	40.5
ODTrack-L [ 53 ]	5	223.3	311.3	17.2
LoRAT-B 378 [ 31 ]	2	96.6	88.5	189.0
LoRAT-L 378 [ 31 ]	2	325.0	308.0	97.5

Table 3: Efficiency comparison with state-of-the-art trackers. The number of frames processed per inference step is included to contextualize the computational cost. All measurements were conducted on a single NVIDIA A100 GPU. # indicates number of frames incorporated.

Model	MACs(G)	Params(M)	FPS	LaSOT		VastTrack
Model	MACs(G)	Params(M)	FPS	AUC	P ${}_{\textrm{Norm}}$	AUC	P
OSTrack-256	29.05	92.12	118.5	69.1	78.7	33.6	31.5
+ DTPTrack	29.57	92.66	104.2	70.1 (+1.0)	79.5 (+0.8)	35.4 (+1.8)	33.0 (+1.5)
ODTrack-B	73.10	92.12	40.5	73.2	83.2	-	-
+ DTPTrack	73.41	93.40	32.0	73.7 (+0.5)	84.0 (+0.8)	38.2	37.5
LoRAT-B224	33.24	98.71	176.0	71.1	80.9	38.7	37.8

Table 4: Generalizability analysis of DTPTrack. Our module consistently improves performance across three diverse host trackers with minimal computational overhead.

Variant	Modification	LaSOT		VastTrack
Variant	Modification	AUC	P ${}_{\textrm{Norm}}$	AUC	P
Analysis of Temporal Reliability Calibrator (TRC)
(a)	Fixed Threshold	72.0	77.7	38.2	37.0
(b)	Fully Gated ( $z_{0}$ )	73.2	79.3	40.1	39.4
Analysis of Temporal Guidance Synthesizer (TGS)
(c)	w/o Base Token	72.7	78.4	39.0	38.2

Table 5: Ablation studies on the core components of DTPTrack. All experiments are based on the ViT-Base backbone. The final row compares our full model against its direct multi-frame baseline.

Variant	Modification	LaSOT		VastTrack
Variant	Modification	AUC	P ${}_{\textrm{Norm}}$	AUC	P
Analysis of Dynamic Prior Token Generation
(a)	Momentum-based	73.8	80.3	40.1	39.5
(b)	Optical Flow-based	73.2	79.3	39.4	38.7
(c)	Ours	74.3	83.6	40.7	40.2

Table 6: Ablation study on the dynamic prior token generation method.

Variant	Modification	LaSOT		VastTrack
Variant	Modification	AUC	P ${}_{\textrm{Norm}}$	AUC	P
Analysis of Temporal Context Length
(a)	2 Frames	72.0	81.3	39.1	38.7
(b)	3 Frames	73.2	82.3	39.7	39.2
(c)	4 Frames	73.7	83.1	40.3	39.8
(d)	5 Frames (Ours)	74.3	83.6	40.7	40.2

Table 7: Ablation study on the number of historical frames. We evaluate the model’s performance by varying the temporal context length.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Item	Value
template area factor	2
search region area factor	4 (B-224) / 5 (L-378)
scale jitter	0.25
translation jitter	3
horizontal flip	0.5
color jitter	0.4

Appendix

Table 8: Hyper-parameters used in DTPTrack.

Method	SV	ARC	ROT	POC	FOC	DEF	BC	LR
Baseline	71.8	71.8	73.0	70.8	65.6	74.7	66.8	66.8
+ DTPTrack	74.1	72.8	73.5	71.5	66.7	75.0	68.2	68.4

Appendix

Table 9: Attribute-wise AUC (%) comparison on the LaSOT.

Method	Qual.	Acc.	Rob.	DRE $\downarrow$
Baseline	0.668	0.742	0.809	0.100
+ DTPTrack	0.680	0.760	0.840	0.080

Appendix

Table 10: Comparison of tracking stability and DRE on the DiDi dataset.

Benchmark	Method	Backbone	EAO $\uparrow$
VOT-STB2022	MixFormer-L	ViT-L	0.582
VOT-STB2022	DTPTrack-B 224	ViT-B	0.610
VOTS2024	LoRAT-g	ViT-L	0.536
VOTS2024	DTPTrack-L 378	ViT-L	0.630

Appendix

Table 11: EAO performance comparison on VOT-STB2022 and VOT2024 benchmarks.

Figure 3: Visualization of the gating mechanism. The model assigns lower scores (indicated by darker colors or lower magnitude) to frames with artifacts or occlusions.

Figure 4: Qualitative comparison with state-of-the-art trackers on challenging scenarios.

Visual Object Tracking

Visual object tracking follows objects across video frames, maintaining their locations and identities despite motion, occlusion, and changing appearance.

See all 171 papers

Continuous Token-Level Spatio-Temporal Context Modeling for Visual Object Tracking
Sep 7, 2026Ding Xia, Meiqin Liu, Jing Zhou +1Spatial Context Modeling Visual Object Tracking
State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Xi’an Jiaotong University, Xi’an 710049, China. · College of Electrical Engineering, Zhejiang University, Hangzhou 310027, China. · School of Electronics and Information Engineering, Xi’an Jiaotong University, Xi’an 710049, China.
SENTRY: SAM2-Enhanced Neighbor-Aware and Temporally Reasoned Memory for Visual Tracking
Jun 23, 2026Mohamad Alansari, Yonathan Michael, Hasan AlMarzouqi +3Memory-Augmented Neural Networks Visual Object Tracking
Khalifa University, Abu Dhabi, UAE · University of Western Australia, Australia
RELO: Reinforcement Learning to Localize for Visual Object Tracking
May 8, 2026Xin Chen, Chuanyu Sun, Jiao Xu +4Reinforcement Learning Visual Object Tracking
City University of Hong Kong · Dalian University of Technology · Hunyuan Team, Tencent

Sep 7, 2026J/K move · Enter open · S save

Continuous Token-Level Spatio-Temporal Context Modeling for Visual Object Tracking

Ding Xia, Meiqin Liu, Jing Zhou, Jian Lan

State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Xi’an Jiaotong University, Xi’an 710049, China. · College of Electrical Engineering, Zhejiang University, Hangzhou 310027, China. · School of Electronics and Information Engineering, Xi’an Jiaotong University, Xi’an 710049, China.

Spatial Context Modeling Visual Object Tracking Spatiotemporal Reasoning Temporal Attention

Open paper AskPDF arXiv

Spatio-temporal context has become increasingly crucial for visual tracking. However, most existing approaches extract spatio-temporal cues via discrete sampling strategies, which inherently deviate from the continuity of spatio-temporal context, thereby deteriorating tracking performance. To address this challenge, we propose TLCTrack, a novel tracking framework that models token-level spatio-temporal context through continuously updated salient tokens, enabling more accurate target representation. Specifically, TLCTrack incorporates three components: Masked Unidirectional Attention (MUA), Spatial Salient Token Collection (SSTC), and Temporal Salient Token Bank (TSTB) modules. By explicitly integrating spatio-temporal context, MUA extracts discriminative targetaware spatial features in the search region. To avoid the negative impact of background on feature learning, SSTC progressively suppresses background interference, thereby enhancing target spatial representation. Finally, TSTB captures high-quality spatio-temporal information through continuous salient token updates. Extensive experiments on five benchmarks demonstrate that our method achieves superior performance over state-of-the-art trackers. Code and models are available at https://github.com/xiading123/TLCTrack.

Abstract

Visual Object Tracking

Explore similar work

Continuous Token-Level Spatio-Temporal Context Modeling for Visual Object Tracking

SENTRY: SAM2-Enhanced Neighbor-Aware and Temporally Reasoned Memory for Visual Tracking

RELO: Reinforcement Learning to Localize for Visual Object Tracking