Test-Time Selective Adaptation for Cross-Modal UAV Object Tracking with Token Shift
Organizations: Anhui University, Hefei, China. · Anhui University of Science and Technology, Huainan, China. · SDAIA-KFUPM Joint Research Centre for AI, Information and Computer Science Department, King Fahd University of Petroleum and Minerals, Dhahran 31261, Saudi Arabia
Abstract
Cross-modal unmanned aerial vehicle (UAV) object tracking faces modality-induced appearance discrepancies and abrupt target-position shifts that can arise from modality switching, UAV motion, and inter-sensor offsets. Although existing methods model modality and spatial variations, they primarily rely on offline representation learning and offer limited localization-oriented adaptation during testing. To address these challenges, we propose Test-Time Selective Adaptation with Token Shift (TSAT), which derives spatial and temporal adaptation signals from unlabeled test sequences for online optimization. Specifically, we design a controllable token shift auxiliary task, which imposes explicit spatial correspondence between predefined token translations and resultant localization offsets, generating stable annotation-free spatial adaptation signals. Furthermore, we introduce a temporal backward prediction auxiliary task to exploit frame-wise temporal dependencies, effectively mitigating cumulative localization errors in continuous tracking. To suppress interference from unreliable tracking samples, we develop a reliability-aware update gating strategy to screen valid adaptation signals and stabilize online parameter updates. Experiments on CM-UOT demonstrate that TSAT achieves state-of-the-art performance against existing test-time adaptation methods, exhibiting superior robustness under frequent modality switching and severe spatial displacement. Experiments across tracking frameworks and datasets further validate the effectiveness and generalization capability of TSAT.
Figures & tables
| Method | Pub. Info. | PR | NPR | SR |
|---|---|---|---|---|
| TransT [ 28 ] | CVPR’21 | 39.5 | 33.6 | 29.2 |
| OSTrack [ 29 ] | ECCV’22 | 37.0 | 32.7 | 28.0 |
| HiT [ 30 ] | ICCV’23 | 40.0 | 34.3 | 30.1 |
| SeqTrack [ 31 ] | CVPR’23 | 43.3 | 39.2 | 33.5 |
| GRM [ 32 ] | CVPR’23 | 38.5 | 33.5 | 28.8 |
| EVPTrack [ 33 ] | AAAI’24 | 46.5 | 41.7 | 35.7 |
| Method | Pub. Info. | PR | NPR | SR | ||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Before | After | All | Before | After | All | Before | After | All | ||
| AdaBN [ 38 ] | PR’18 | 90.99 | 43.12 | 50.42 | 86.48 | 37.32 | 44.69 | 71.50 | 32.16 | 38.05 |
| TENT [ 19 ] | ICLR’21 | 90.97 | 42.46 | 49.79 | 86.71 | 36.54 | 43.98 | 71.67 | 31.65 | 37.57 |
| CoTTA [ 20 ] | CVPR’22 | 90.97 | 40.81 | 48.55 | 86.62 | 35.32 | 43.08 | 71.66 | 30.57 | 36.77 |
| ETA/EATA [ 21 ] | ICML’22 | 90.90 | 42.30 | 49.55 | 86.60 | 36.56 | 43.89 | 71.61 | 31.57 | 37.42 |
| MEMO [ 39 ] | NeurIPS’22 | 91.18 | 43.89 | 51.19 | 86.68 | 37.86 | 45.24 | 71.71 | 32.72 | 38.61 |
| Method | 20–40 | 40–60 | 60–80 | 80–100 | 100 | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| PR | NPR | SR | PR | NPR | SR | PR | NPR | SR | PR | NPR | SR | PR | NPR | SR | |
| AdaBN | 49.90 | 44.40 | 37.77 | 46.56 | 41.45 | 35.08 | 48.62 | 41.88 | 34.24 | 35.41 | 33.82 | 26.21 | 33.84 | 30.74 | 25.96 |
| TENT | 48.20 | 42.51 | 36.39 | 46.06 | 40.40 | 34.96 | 47.29 | 42.00 | 33.88 | 46.57 | 42.81 | 33.70 | 29.83 | 27.47 | 23.17 |
| CoTTA | 50.81 | 44.89 | 38.30 | 43.49 | 39.23 | 33.30 | 46.95 | 42.15 | 33.82 | 47.30 | 44.86 | 35.44 | 27.24 | 24.44 | 21.06 |
| ETA/EATA | 48.86 | 43.64 | 36.94 | 45.32 | 40.05 | 34.23 | 46.83 | 40.99 | 33.09 | 43.09 | 41.15 | 31.88 | 30.22 | 27.85 | 23.38 |
| MEMO | 49.73 | 44.13 | 37.47 | 48.91 | 43.43 | 36.78 | 49.74 | 42.42 | 34.66 | 41.38 | 39.87 | 31.40 | 33.39 | 30.78 | 26.16 |
| No. | RUG | PR | NPR | SR | ||
|---|---|---|---|---|---|---|
| 1 | 50.60 | 44.80 | 38.00 | |||
| 2 | ✓ | 51.80 | 45.96 | 39.13 | ||
| 3 | ✓ | 51.37 | 45.77 | 38.92 | ||
| 4 | ✓ | ✓ | 52.10 | 46.23 | 39.37 | |
| 5 | ✓ | ✓ | 52.78 | 46.92 | 39.91 | |
| 6 | ✓ | ✓ | 51.39 | 45.76 | 38.94 |
| Frames since switch | Without TBP | With TBP | Error reduction |
|---|---|---|---|
| 0–9 | 94.75 | 91.89 | 2.86 |
| 10–29 | 98.81 | 95.76 | 3.05 |
| 30–59 | 106.49 | 102.58 | 3.91 |
| 60–119 | 118.55 | 113.30 | 5.25 |
| 147.72 | 144.20 | 3.52 |
| PR (%) | NPR (%) | SR (%) | |
|---|---|---|---|
| 1 | 51.92 | 46.14 | 39.26 |
| 2 | 51.76 | 46.01 | 39.14 |
| 3 | 53.19 | 47.28 | 40.23 |
| 4 | 52.33 | 46.39 | 39.55 |
| 5 | 51.62 | 45.86 | 39.04 |
| Threshold | PR (%) | NPR (%) | SR (%) |
|---|---|---|---|
| 0.16 | 52.00 | 46.17 | 39.28 |
| 0.20 | 52.25 | 46.46 | 39.55 |
| 0.24 | 53.19 | 47.28 | 40.23 |
| 0.28 | 52.53 | 46.69 | 39.70 |
| 0.36 | 52.21 | 46.56 | 39.52 |
| Interval | PR (%) | NPR (%) | SR (%) |
|---|---|---|---|
| 5 | 51.09 | 45.41 | 38.69 |
| 8 | 51.45 | 45.87 | 38.90 |
| 10 | 53.19 | 47.28 | 40.23 |
| 15 | 52.06 | 46.40 | 39.46 |
| 20 | 51.34 | 45.67 | 38.85 |
| Learning rate | PR (%) | NPR (%) | SR (%) |
|---|---|---|---|
| 50.56 | 45.12 | 38.36 | |
| 52.06 | 46.23 | 39.32 | |
| 53.19 | 47.28 | 40.23 | |
| 52.50 | 46.40 | 39.56 | |
| 51.72 | 46.09 | 39.14 |
| PR | NPR | SR |
| 53.19 | 47.28 | 40.23 |
| 52.97 | 47.09 | 40.12 |
| 52.54 | 46.70 | 39.77 |
| 52.66 | 46.79 | 39.81 |
| Method | FPS | FLOPs (G) | Params (M) | PR | NPR | SR |
|---|---|---|---|---|---|---|
| Baseline | 106.41 | 51.1 | 90.1 | 50.60 | 44.80 | 38.00 |
| TSAT | 72.87 | 72.2 | 90.1 | 53.19 | 47.28 | 40.23 |
| Method | PR | NPR | SR | |||
|---|---|---|---|---|---|---|
| OSTrack ∗ | 45.45 | 40.00 | 34.09 | |||
| Ours | 47.67 | (+2.22) | 42.02 | (+2.02) | 35.75 | (+1.66) |
| Method | PR | NPR | SR | |||
|---|---|---|---|---|---|---|
| OSTrack | 74.30 | 76.80 | 63.30 | |||
| SARLA | 76.08 | 78.74 | 64.85 | |||
| TSAT | 76.87 | (+0.79) | 79.46 | (+0.72) | 65.45 | (+0.60) |
| Yun Xiao received the M.S. and Ph.D. degrees from the School of Computer Science and Technology, Anhui University, Hefei, China, in 2011 and 2019, respectively. She is currently an Associate Professor with the School of Artificial Intelligence, Anhui University. Her research interests include computer vision and deep learning. |
| Hailong Ye is currently pursuing a master’s degree at Anhui University, Hefei, China. His research interests include computer vision, cross-modal object tracking, and test-time adaptation. |
| Zhicheng Zhao is currently an Associate Professor with the School of Artificial Intelligence, Anhui University, Hefei, China. His research interests include computer vision, multimodal learning, and remote sensing image analysis. |
| Lei Liu received the B.S. and Ph.D. degrees from Anhui University, Hefei, China, in 2019 and 2024, respectively. He is currently with the School of Artificial Intelligence, Anhui University of Science and Technology, Huainan, China. His research interests include computer vision and deep learning. |
| Amir Hussain received the B.Eng. (Hons.) and Ph.D. degrees in electronic and electrical engineering from the University of Strathclyde, Glasgow, U.K., in 1992 and 1997, respectively. He is currently a Chair Professor with the Department of Information and Computer Science, King Fahd University of Petroleum and Minerals, Dhahran, Saudi Arabia. His research interests include artificial intelligence, cognitive data science, and their applications in healthcare and industrial systems. |
Visual Object Tracking
Visual object tracking follows objects across video frames, maintaining their locations and identities despite motion, occlusion, and changing appearance.
See all 182 papers