Dual-level Adaptation for Multi-Object Tracking: Building Test-Time Calibration from Experience and Intuition

Wen Guo; Pengfei Zhao; Zongmeng Wang; Yufan Hu; Junyu Gao

cs.CVMar 23, 2026

Dual-level Adaptation for Multi-Object Tracking: Building Test-Time Calibration from Experience and Intuition

Authors: Wen Guo, Pengfei Zhao, Zongmeng Wang, Yufan Hu, Junyu Gao

Organizations: Shandong Technology and Business University · Inner Mongolia University · University of Science and Technology Beijing · Institute of Automation, Chinese Academy of Sciences

Abstract

Multiple Object Tracking (MOT) has long been a fundamental task in computer vision, with broad applications in various real-world scenarios. However, due to distribution shifts in appearance, motion pattern, and catagory between the training and testing data, model performance degrades considerably during online inference in MOT. Test-Time Adaptation (TTA) has emerged as a promising paradigm to alleviate such distribution shifts. However, existing TTA methods often fail to deliver satisfactory results in MOT, as they primarily focus solely on frame-level adaptation while neglecting temporal consistency and identity association across frames and videos. Inspired by human decision-making process, this paper propose a Test-time Calibration from Experience and Intuition (TCEI) framework. In this framework, the Intuitive system utilizes transient memory to recall recently observed objects for rapid predictions, while the Experiential system leverages the accumulated experience from prior test videos to reassess and calibrate these intuitive predictions. Furthermore, both confident and uncertain objects during online testing are exploited as historical priors and reflective cases, respectively, enabling the model to adapt to the testing environment and alleviate performance degradation. Extensive experiments demonstrate that the proposed TCEI framework consistently achieves superior performance across multiple benchmark datasets and significantly enhances the model's adaptability under distribution shifts. The code will be released at https://github.com/1941Zpf/TCEI.

Figures & tables

Figure 1: Illustration of the proposed TCEI framework. (a) The upper part illustrates MOT under distribution shift. Due to significant discrepancies between the training and testing domains, the baseline model produces incorrect ID predictions. (b) The lower part presents our TCEI framework. The Intuitive System exploits transient memory from recently observed objects to provide rapid test-time guidance, and the Experiential System utilizes accumulated historical experience to calibrate these intuitive predictions.

Figure 2: Overview of the proposed Test-time Calibration from Experience and Intuition (TCEI) framework. The Intuitive system performs rapid inference using transient memory, while the Experiential system refines predictions with historical test experience. Confident and uncertain objects are stored in caches to provide temporal priors and reflective cues. “Exp. Embeds” denotes the experience embeddings, while “Query” represents the query embeddings of the Transformer decoder. The experience embeddings evolve along with the query embeddings to capture object-specific characteristics.

Methods	HOTA	DetA	AssA	IDF1
CNN based:
FairMOT [ 56 ]	39.7	66.7	23.8	40.8
CenterTrack [ 58 ]	41.8	78.1	22.6	35.7
TreDeS [ 48 ]	43.3	74.5	25.4	41.2
ByteTrack [ 55 ]	47.7	71.0	32.1	53.9
GTR [ 59 ]	48.0	72.5	31.9	50.3

Table 1: Performance comparison with state-of-the-art methods on DanceTrack. The best performance is marked in bold. The experimental results are obtained from reproductions based on the official code and from the official reports. The tables are organized in ascending order of the HOTA metric.

Methods	HOTA	DetA	AssA	IDF1
CNN based:
FairMOT [ 56 ]	49.3	70.2	34.7	53.5
GTR [ 59 ]	54.5	64.8	45.9	55.8
QDTrack [ 6 ]	60.4	77.5	47.2	62.3
ByteTrack [ 55 ]	62.1	76.5	50.5	69.1
CenterTrack [ 58 ]	62.7	82.1	48.0	60.0

Table 2: Performance comparison with state-of-the-art methods on the SportsMOT test set. The results are reported using the official implementations, without any additional training data.

Datasets	Method	HOTA	DetA	AssA	IDF1	FPS
DanceTrack	No Adap.	69.5	80.4	60.2	74.6	14
	Tent	69.4	80.3	60.1	74.2	7
	TCEI	70.6	80.2	62.3	75.6	12
SportsMOT	No Adap.	72.6	83.5	63.2	77.1	12
	Tent	72.8	83.5	63.5	77.2	7
	TCEI	73.0	83.5	64.0	77.5	9

Table 3: Comparison between Tent and our proposed TCEI framework on the DanceTrack and SportsMOT test sets. FPS indicates inference speed.

Intuitive	Experiential	HOTA	DetA	AssA	IDF1
–	–	69.5	80.4	60.2	74.6
✓	–	70.5	80.2	62.1	75.5
–	✓	70.4	80.3	61.9	75.6
✓	✓	70.6	80.2	62.3	75.6

Table 4: Ablation study on the Intuitive and Experiential systems conducted on the DanceTrack. The combination of both components yields the best performance, confirming their complementary contributions to association accuracy.

CO	UO	HOTA	DetA	AssA	IDF1
–	–	69.5	80.4	60.2	74.6
✓	–	69.6	80.3	60.4	74.6
–	✓	70.2	80.2	61.5	75.3
✓	✓	70.5	80.2	62.1	75.6

Table 5: Ablation study on the effects of confident (CO) and uncertain (UO) historical objects on the DanceTrack. “CO” and “UO” respectively denote the use of confident objects as temporal priors and uncertain objects as reflective cases. Combining both leads to the highest performance, confirming their complementary contributions to association improvement.

Figure 3: Analysis of the maximum capacity of the confident and uncertain objects on the DanceTrack dataset.

Methods	HOTA	DetA	AssA	IDF1
No Adap.	69.5	80.4	60.2	74.6
Average	69.7	80.4	60.7	74.4
Entropy	69.9	80.4	61.0	74.9
Ours	70.6	80.2	62.3	75.6

Table 6: Comparison of different calibration strategies for the Experiential system on the DanceTrack. “Average” denotes the naive averaging of Intuitive and Experiential predictions, and “Entropy” selects the prediction with lower entropy. Our selective calibration strategy, which corrects only uncertain components of the Intuitive system, achieves the best overall performance.

Multi-Object Tracking

Multi-object tracking (MOT) links detections across video frames to maintain object identities through motion, occlusion, and changing viewpoints.

See all 193 papers

Test-Time Adaptation via Dual Distillation for Videos Under Severe Distribution Shifts
Jul 27, 2026André Sacilotti, Samuel Felipe dos Santos, Jurandy AlmeidaSource-Free Domain Adaptation Test-Time Adaptation
University of São Paulo São Carlos, São Paulo, Brazil · Federal University of São Carlos Sorocaba, São Paulo, Brazil
Continual Test-Time Adaptation in Computer Vision: Methods, Benchmarks, and Future Directions
Jul 9, 2026Sarthak Kumar Maharana, Shambhavi Mishra, Yunbei Zhang +6Source-Free Domain Adaptation Distribution Shift Robustness
The University of Texas at Dallas, USA · LIVIA ETS Montreal, ILLS International Laboratory on Learning Systems (ILLS), McGill - ÉTS - MILA - CNRS - Université Paris-Saclay - CentraleSupélec · Tulane University, USA +2
FACTOR: Counterfactual Training-Free Test-Time Adaptation for Open-Vocabulary Object Detection
May 5, 2026Kaixiang Zhao, Mao Ye, Lihua Zhou +4Open-Vocabulary Object Detection Distribution Shift Robustness
School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, China · Institute of Machine Intelligence (IMI), University of Shanghai for Science and Technology, Shanghai, China · Surrey Institute for People-Centred Artificial Intelligence, CVSSP, University of Surrey, Guildford, UK.

Jul 27, 2026J/K move · Enter open · S save

Test-Time Adaptation via Dual Distillation for Videos Under Severe Distribution Shifts

André Sacilotti, Samuel Felipe dos Santos, Jurandy Almeida

University of São Paulo São Carlos, São Paulo, Brazil · Federal University of São Carlos Sorocaba, São Paulo, Brazil

Source-Free Domain Adaptation Test-Time Adaptation Video Action Recognition

Open paper AskPDF arXiv

Deep learning models have achieved state-of-the-art performance in several computer vision tasks. However, they experience severe performance degradation when applied to real-world scenarios due to unanticipated distribution shifts. Test-Time Adaptation (TTA) attempts to solve this problem by using unlabeled data from the target domain to dynamically adapt to the test distribution at inference time, without access to the source data. However, TTA remains a challenging problem when adapting to continuous, temporally correlated data, such as videos, and in scenarios where the target domain contains severe domain shifts. For this reason, few works in the literature explore TTA for videos under such extreme conditions. To overcome these limitations, we propose Test-time Adaptation via Dual Distillation (TADD), an online adaptation framework that relies on a lightweight projection adapter to bridge the domain gap. The adapter module is pre-trained on the source domain and then adapted to the target using our proposed complementary losses: (i) zero-shot distillation, which encourages alignment with the domain-agnostic features from a pre-trained vision-language model (VLM); and (ii) target distillation, which retains the source domain discriminative knowledge encoded in the pre-trained adapter. Built upon a frozen CLIP backbone, our method introduces this lightweight projection adapter as the sole updatable component during inference. We conducted extensive evaluations on three well-known video action recognition benchmarks: UCF-HMDB, Daily-DA, and Sports-DA. Our experiments in the closed-set scenario demonstrate that our method consistently outperforms state-of-the-art TTA baselines. Notably, our TTA approach improves upon previous methods by up to +3.81% on UCF-HMDB, +2.63% on Daily-DA, and +3.03% on Sports-DA.

Abstract

Multi-Object Tracking

Explore similar work

Test-Time Adaptation via Dual Distillation for Videos Under Severe Distribution Shifts

Continual Test-Time Adaptation in Computer Vision: Methods, Benchmarks, and Future Directions

FACTOR: Counterfactual Training-Free Test-Time Adaptation for Open-Vocabulary Object Detection