Uncertainty-Guided Inference-Time Depth Adaptation for Transformer-Based Visual Tracking

Patrick Poggi; Divake Kumar; Theja Tulabandhula; Amit Ranjan Trivedi

cs.CVFeb 18, 2026

Uncertainty-Guided Inference-Time Depth Adaptation for Transformer-Based Visual Tracking

Authors: Patrick Poggi, Divake Kumar, Theja Tulabandhula, Amit Ranjan Trivedi

Organizations: ∗University of Illinois at Chicago

Abstract

Transformer-based single-object trackers achieve state-of-the-art accuracy but rely on fixed-depth inference, executing the full encoder--decoder stack for every frame regardless of visual complexity, thereby incurring unnecessary computational cost in long video sequences dominated by temporally coherent frames. We propose UncL-STARK, an architecture-preserving approach that enables dynamic, uncertainty-aware depth adaptation in transformer-based trackers without modifying the underlying network or adding auxiliary heads. The model is fine-tuned to retain predictive robustness at multiple intermediate depths using random-depth training with knowledge distillation, thus enabling safe inference-time truncation. At runtime, we derive a lightweight uncertainty estimate directly from the model's corner localization heatmaps and use it in a feedback-driven policy that selects the encoder and decoder depth for the next frame based on the prediction confidence by exploiting temporal coherence in video. Extensive experiments on GOT-10k and LaSOT demonstrate up to 12% GFLOPs reduction, 8.9% latency reduction, and 10.8% energy savings while maintaining tracking accuracy within 0.2% of the full-depth baseline across both short-term and long-term sequences.

Figures & tables

Fig. 1: Overview of the UncL-STARK framework. Uncertainty derived from corner localization heatmaps at frame $t$ drives inference-time depth adaptation by selecting the encoder–decoder depth for frame $t+1$ . This feedback mechanism exploits temporal coherence in video to reduce computation while preserving tracking accuracy.

Fig. 2: Architecture-preserving depth truncation in UncL-STARK. The decoder attends to the output of a selected encoder layer, and predictions are produced from a selected decoder layer, enabling inference at arbitrary depths without modifying the prediction head.

Fig. 3: Mean IoU versus encoder–decoder depth before and after random-depth (RD) training with knowledge distillation (KD) on GOT-10k (val) and LaSOT (test).

Combination	GOT-10k (val)	LaSOT (test)	Average	$\Delta$ Avg (%)
(1,1)	0.7670	0.6102	0.6886	–
(2,2)	0.7930	0.6459	0.7195	+4.48
(3,3)	0.8012	0.6545	0.7279	+1.17
(4,4)	0.8031	0.6587	0.7309	+0.41
(5,5)	0.8084	0.6581	0.7333	+0.33

TABLE I: Mean IoU performance for different encoder–decoder depth combinations. Results are reported separately for each dataset and averaged across datasets. The last column reports the percentage increase in average performance with respect to the previous depth configuration.

Dataset	AO / AUC	$\boldsymbol{\Delta}$	GFLOPs	Lat./En.
LaSOT (test)	65.69 / 64.60	$-$ 0.20 / $-$ 0.17	12.0	8.9 / 10.8
GOT-10k (val)	0.8198 / 80.42	$-$ 0.19 / $-$ 0.17	11.9	7.1 / 4.3
GOT-10k (test)	0.672 / –	$-$ 1.04 / –	11.4	7.8 / 6.1

TABLE II: Adaptive UncL-STARK performance relative to fixed-depth STARK (5,5). Accuracy deltas are relative to baseline; resource values denote percentage reduction.

Fig. 4: Profiled GFLOPs and average latency across depth configurations, both scaling approximately linearly with the number of executed transformer layers.

Fig. 5: Correlation–calibration trade-off of heatmap-derived confidence proxies. Each point is plotted by Pearson correlation with IoU (x-axis) and Expected Calibration Error (ECE, y-axis). Higher absolute correlation and lower ECE indicate better alignment with tracking accuracy. The dashed line denotes the Pareto-optimal frontier; the selected top- $k$ mass estimator lies on this frontier, achieving a favorable balance between correlation and calibration.

Panel of Figure 8 — Fig. 8: Qualitative comparison between the fixed-depth baseline and the proposed adaptive tracker on the same video sequence.

Fig. 9: Attribute-based analysis of tracking performance on GOT-10k (val). The attributes represent, in clockwise order, the degrees to which the object is visible (from ‘Low’ to ‘Full’ visibility). The red line represents the adaptive tracking framework’s AO, whereas the blue one that of the baseline.

Dataset	Method	Acc.	Drop (%)	FLOPs	Latency	Energy	Adapt. vs. Fixed (3,3)
LaSOT (test)	Baseline (5,5)	64.71 (AUC)
	Fixed (3,3)	64.38 (AUC)	0.50	12.7%	9.4%	11.5%	✓
	Adaptive Tracker	64.60 (AUC)	0.17	12.0%	8.9%	10.8%	✓
GOT-10k (val)	Baseline (5,5)	0.8213 (AO)
	Fixed (3,3)	0.8162 (AO)	0.62	12.6%	7.3%	4.5%	✓
	Adaptive Tracker	0.8198 (AO)	0.19	11.9%	7.1%	4.3%	✓

TABLE III: Comparison of baseline (5,5), fixed-depth (3,3), and adaptive tracker variants across LaSOT and GOT-10k. Accuracy is reported using dataset-standard metrics (AUC for LaSOT, AO for GOT-10k). Percentage variations and efficiency gains are computed w.r.t. the fixed-depth baseline. Latency refers to measured end-to-end latency; energy is GPU-only (NVML).

Dataset	Metric	Random	Threshold	$\Delta$ (%)
GOT-10k (val)	Mean IoU (seq.)	0.8015	0.8038	+0.29
	AO (frames)	0.8171	0.8198	+0.33
	Mean Loss	1.8841	1.8819	+0.11
	GFLOPs (mean)	21.98	21.61	-1.7
	Latency (ms)	23.09	22.91	-1.2
	Energy (J)	234.46	237.61	+3.3

TABLE IV: Comparison between random depth selection and confidence-threshold-based adaptive depth selection on GOT-10k (val) and LaSOT (test). The threshold-based policy consistently achieves equal or better accuracy while reducing computational cost, latency, or energy consumption, indicating that the adaptive mechanism successfully exploits confidence information to guide depth selection.

Fig. 10: Accuracy–efficiency trade-off of alternative heatmap-derived confidence proxies. Each point represents a different proxy, plotted by the mean IoU achieved by the adaptive tracker (x-axis) and the corresponding GFLOPs savings obtained when the proxy is used to drive depth selection (y-axis). Although some proxies estimate confidence and others uncertainty (with inverted semantics), all serve as scalar indicators of prediction reliability. The dashed line is the Pareto-frontier.

Visual Object Tracking

Visual object tracking follows objects across video frames, maintaining their locations and identities despite motion, occlusion, and changing appearance.

See all 175 papers

Difference Feature Map Distillation: Transferring Inter-Sample Relational Knowledge Towards Efficient Transformer-Based Tracking
Oct 5, 2026Zhicheng Ding, Xinyu Chu, Qing TianVisual Object Tracking Autonomous Driving Perception
Department of Computer Science, Bowling Green State University, OH 43402, USA · Department of Computer Science, University of Alabama at Birmingham, AL 35294, USA
LoopTrack: A Simple Baseline for Parameter-Efficient Transformer Tracking
Sep 27, 2026Liang Peng, Chenxiao Li, Libo Zhang +2Visual Object Tracking Recurrent Transformers
School of Computer Science, National Engineering Research Center for Multimedia Software, Institute of Artificial Intelligence, Hubei Key Laboratory of Multimedia and Network Communication Engineering, Wuhan University · University of North Texas · Institute of Software, Chinese Academy of Sciences
FeedbackTrack: Visual-Cortex-Inspired Cross-Frame Feedback for Transformer Tracking
Aug 10, 2026Yueyang Cang, Xiaoteng Zhang, Zhiyuan Ning +2Visual Object Tracking
Tsinghua University Beijing, China

Oct 5, 2026J/K move · Enter open · S save

Difference Feature Map Distillation: Transferring Inter-Sample Relational Knowledge Towards Efficient Transformer-Based Tracking

Zhicheng Ding, Xinyu Chu, Qing Tian

Department of Computer Science, Bowling Green State University, OH 43402, USA · Department of Computer Science, University of Alabama at Birmingham, AL 35294, USA

Visual Object Tracking Autonomous Driving Perception Relational Knowledge Distillation Feature-Based Knowledge Distillation

Open paper AskPDF arXiv

In autonomous driving perception, visual object tracking systems must satisfy stringent latency and power constraints while remaining robust in complex and dynamic environments. Although transformer-based trackers achieve state-of-the-art accuracy, their substantial computational and memory overheads hinder deployment on real-time, resource-constrained platforms. To move toward this goal, we propose Difference Feature Map Knowledge Distillation (DFM-KD), a novel relational distillation framework tailored for transformer-based visual object tracking. Unlike conventional feature distillation methods that minimize point-wise discrepancies (e.g., mean squared error) between teacher and student feature representations, DFM-KD transfers knowledge through inter-sample feature differences, explicitly aligning the relational structure of the feature space. By distilling how the teacher models appearance variation and consistency across samples, rather than enforcing similarity in absolute activations, DFM-KD enables the student to better capture the structural dynamics of visual changes within a batch. As a result, the distilled model exhibits enhanced feature robustness and improved tracking performance. Extensive experiments demonstrate that DFM-KD consistently outperforms conventional feature-level distillation methods in both tracking precision and success rates.

Abstract

Visual Object Tracking

Explore similar work

Difference Feature Map Distillation: Transferring Inter-Sample Relational Knowledge Towards Efficient Transformer-Based Tracking

LoopTrack: A Simple Baseline for Parameter-Efficient Transformer Tracking

FeedbackTrack: Visual-Cortex-Inspired Cross-Frame Feedback for Transformer Tracking