Architecture and evaluation protocol for transformer-based visual object tracking in UAV applications

Augustin Borne; Pierre Notin; Christophe Hennequin; Sebastien Changey; Stephane Bazeille; Christophe Cudel; Franz Quint

cs.CVMar 4, 2026

Architecture and evaluation protocol for transformer-based visual object tracking in UAV applications

Authors: Augustin Borne, Pierre Notin, Christophe Hennequin, Sebastien Changey, Stephane Bazeille, Christophe Cudel, Franz Quint

Organizations: ISL, Hochschule Karlsruhe -- Technik und Wirtschaft Karlsruhe University of Applied Sciences, IRIMAS · French-German Research Institute of Saint-Louis, Saint-Louis, France · Université de Haute-Alsace, Mulhouse, France · Karlsruhe University of Applied Sciences, Karlsruhe, Germany ·

Abstract

Object tracking from Unmanned Aerial Vehicles (UAVs) is challenged by platform dynamics, camera motion, and limited onboard resources. Existing visual trackers either lack robustness in complex scenarios or are too computationally demanding for real-time embedded use. We propose an Modular Asynchronous Tracking Architecture (MATA) that combines a transformer-based tracker with an Extended Kalman Filter, integrating ego-motion compensation from sparse optical flow and an object trajectory model. We further introduce a hardware-independent, embedded oriented evaluation protocol and a new metric called Normalized time to Failure (NT2F) to quantify how long a tracker can sustain a tracking sequence without external help. Experiments on UAV benchmarks, including an augmented UAV123 dataset with synthetic occlusions, show consistent improvements in Success and NT2F metrics across multiple tracking processing frequency. A ROS 2 implementation on a Nvidia Jetson AGX Orin confirms that the evaluation protocol more closely matches real-time performance on embedded systems.

Figures & tables

Figure 1: Overview of the Modular Asynchronous Tracking Architecture: Example with tracker running at 10 Hz, estimation filter at 30 Hz and ego motion at 30 Hz.

Panel of Figure 2 — Figure 2: Probability mass function with detected peak and AUC region.

Panel of Figure 3 — Figure 3: The proposed EOP evalution protocol

Panel of Figure 4 — Figure 4: Example of shape applied to sequences on UAV123 dataset

Protocol	VOT	MATA	UAV123		UAV123+occ		VTUAV
Protocol	VOT	MATA	SR	NT2F	SR	NT2F	SR	NT2F
LTP	OSTrack	✗	82.8	79.5	75.9	72.7	74.2	59.3
	OSTrack	✓	83.6	79.5	78.5	74.1	74.8	61.1
	MixFormerV2	✗	84.3	79.1	78.2	68.8	75.6	55.2
	MixFormerV2	✓	83.2	79.7	77.2	72.0	73.9	59.0
EOP @ 30 Hz	OSTrack	✗	80.1	76.9	73.3	70.1	74.9	59.4

Table 1: Performance of OSTrack and MixFormerV2, with and without MATA, on UAV123, UAV123+occ, and VTUAV under LTP and EOP protocols. EOP is evaluated at 30 Hz and 10 Hz, the latter matching the runtime observed on NVIDIA Jetson Orin.

VOT	MATA	Occlusion		Distractor		High dynamic		Out of view
VOT	MATA	SR	NT2F	SR	NT2F	SR	NT2F	SR	NT2F
OSTrack	✗	50.2	43.4	67.8	60.6	32.0	16.1	41.2	39.8
OSTrack	✓	59.9	49.9	74.6	65.0	38.3	24.0	47.1	35.9
MixFormerV2	✗	52.9	37.0	70.3	50.6	36.2	14.9	46.5	32.8
MixFormerV2	✓	59.1	50.8	72.4	61.1	35.1	25.1	46.4	36.7

Table 2: Metrics per challenges on UAV123+occ with EOP evaluation with tracker at 10Hz

Configuration	Ego motion	EKF	Score	Details	PR	SR	MSR	NT2F
1	✗	✗	✗	MixFormerV2 (Baseline)	79.3	60.8	51.8	44.5
2	✓	✓	✓	MATA	78.1	64.7	53.5	60.3
3	✗	✗	✓	section 2.1	78.8	60.8	51.5	46.4
4	✗	✓	✓	no camera compensation	76.4	61.6	51.5	48.1
5	✓	✓	✓	constant acceleration model	77.9	64.5	53.3	57.0
6	✓	✓	✓	constant jerk model	75.4	62.4	51.6	54.9

Table 3: Impact of individual components on tracking performance on UAV123+occ, with tracking at 10 Hz using EOP.

VOT	MATA	Baseline			Absolute difference w.r.t. Baseline
		Embedded ( ROS 2 )			EOP			LTP
		SR	NT2F	Freq.	$\Delta$ SR	$\Delta$ NT2F	Freq.	$\Delta$ SR	$\Delta$ NT2F	Freq.
OSTrack	✗	58.6	51.0	10.5	3.7	4	10.5	17.3	21.7	30
OSTrack	✓	58.4	51.3	9.2	0.5	1.6	9.2	20.1	22.8	30
MixFormerV2	✗	63.8	51.0	11.1	5.7	8.5	11.1	14.4	17.8	30
MixFormerV2	✓	59.7	49.8	9.7	1.0	3.8	9.7	17.5	22.2	30

Table 4: Embedded performance on UAV123+occ (Jetson AGX Orin). EOP and LTP report absolute differences (

\Delta

) in SR and NT2F with respect to the Embedded baseline. For example, the absolute difference in SR between EOP and the baseline is computed as

\Delta SR=\left|SR_{\mathrm{EOP}}-SR_{\mathrm{baseline}}\right|

. Freq. denotes the processing frequency in Hz.

Visual Object Tracking

Visual object tracking follows objects across video frames, maintaining their locations and identities despite motion, occlusion, and changing appearance.

See all 175 papers

Dual-branch Distilled Transformer for Efficient Asymmetric UAV Tracking
May 27, 2026Hongtao Yang, Bineng Zhong, Qihua Liang +4Visual Object Tracking Efficient ViTs
Key Lab of Education Blockchain and Intelligent Technology, Ministry of Education, Guangxi Normal University, Guilin, 541004, China · Guangxi Key Lab of Multi-Source Information Mining and Security, Guangxi Normal University, Guilin, 541004, China · Nanjing University of Science and Technology +1
JitTrack: Onboard Multi-Object Tracking Against Viewpoint Jitter for Agile UAVs
Aug 11, 2026Yachun Shan, Feitian ZhangAerial Robotics Vision-Based Robot Control
The Department of Advanced Manufacturing and Robotics, Peking University, Beijing, China
ASUMOT: Motion-Consistency-Based Asynchronous UAV Detection and Tracking with Event Cameras
Jul 13, 2026Baofeng Jia, Xiaoyu Chen, Jingyuan Zhang +4Event-Based Vision Event-Based Object Detection
State key Laboratory of Extreme Environment Optoelectronic Dynamic Testing Technology and Instrument, Nanjing University of Science and Technology, Nanjing 210094, China · Jiangsu Key Laboratory of Visual Sensing and Intelligent Perception, Nanjing University of Science and Technology, Nanjing 210094, China

May 27, 2026J/K move · Enter open · S save

Dual-branch Distilled Transformer for Efficient Asymmetric UAV Tracking

Hongtao Yang, Bineng Zhong, Qihua Liang, Yaozong Zheng+3

Key Lab of Education Blockchain and Intelligent Technology, Ministry of Education, Guangxi Normal University, Guilin, 541004, China · Guangxi Key Lab of Multi-Source Information Mining and Security, Guangxi Normal University, Guilin, 541004, China · Nanjing University of Science and Technology+1

Visual Object Tracking Efficient ViTs Knowledge Distillation

Open paper AskPDF arXiv

Given the real-time demands of UAV tracking, many methods simplify the backbone to reduce computation, but this often weakens feature representation and degrades performance in complex scenarios. To alleviate this issue, we propose EATrack, an efficient and asymmetric UAV tracking framework centered around a teacher-guided dual-branch distillation strategy that enhances the feature expressiveness of the lightweight student model. Specifically, EATrack investigates two complementary perspectives of knowledge transfer: spatially focused feature-level distillation that compensates for weakened representations by guiding the student to learn strong target representations, and prediction-level distillation that enhances spatial localization by learning the teacher's capability for accurate target localization. Furthermore, to enhance robustness against appearance variations, we introduce a fine-grained target-aware distillation strategy that selectively transfers the teacher's target modeling capacity to the student. A temporal adaptation module is incorporated at inference to enhance robustness over time. Experiments on five UAV benchmarks demonstrate that EATrack achieves a favorable balance between accuracy and speed. Code: https://github.com/GXNU-ZhongLab/EATrack

Abstract

Visual Object Tracking

Explore similar work

Dual-branch Distilled Transformer for Efficient Asymmetric UAV Tracking

JitTrack: Onboard Multi-Object Tracking Against Viewpoint Jitter for Agile UAVs

ASUMOT: Motion-Consistency-Based Asynchronous UAV Detection and Tracking with Event Cameras