Calibrating Track-Query Updates at Test Time for Multi-Object Tracking

Riku Inoue; Shogo Sato; Yu Mitsuzumi; Kazuhiko Murasaki; Tomoyasu Shimada; Takashi Sano

cs.CVOct 28, 2026

Calibrating Track-Query Updates at Test Time for Multi-Object Tracking

Authors: Riku Inoue, Shogo Sato, Yu Mitsuzumi, Kazuhiko Murasaki, Tomoyasu Shimada, Takashi Sano

Organizations: NTT, Inc.

Abstract

Multi-Object Tracking (MOT) requires localizing multiple objects and maintaining their identities over time. Recent query-propagation trackers carry target-specific representations across frames and update them with current-frame image features, yet identity association remains difficult when visually similar targets overlap or cross. During such interactions, current-frame query updates may contain components directed toward nearby active tracks, which may affect identity association in later frames. To investigate this issue, we analyze track-query updates preceding identity switches (IDSWs), where the same target is assigned a different track ID. Our analysis shows that (i) propagated track-query representations change more abruptly before IDSWs, (ii) current-frame updates have larger positive projections along directions toward other active tracks, and (iii) bounding-box intersection over union (IoU) retrieves the counterpart track involved in a subsequent IDSW with high recall when that track remains active before the switch. Motivated by these findings, we propose Test-time Track-Query Calibration (TQC), a training-free and gradient-free method for query-propagation trackers. TQC calibrates the tracker output before propagation using a bounded correction estimated from update components directed toward nearby active tracks. It requires only track-query representations and bounding boxes available during inference, without retraining or architectural modification. Across nine DanceTrack and SportsMOT test settings, TQC improves HOTA and AssA in all settings and IDF1 in eight, with an average HOTA gain of +0.38 and limited inference overhead. Controlled variants on six validation settings further show that TQC's correction direction yields more consistent HOTA improvements than alternative directions with matched correction norms.

Figures & tables

Figure 1: Standard query propagation and Test-time Track-Query Calibration (TQC). Left: A current-frame query update may contain a component directed toward a nearby active-track representation. TQC calibrates the tracker output before propagation using a bounded correction estimated from update components directed toward nearby active tracks. Right: In this example, standard query propagation leads to an identity switch at frame $t+1$ , whereas TQC preserves the identities.

Figure 2: Distributions of query-update statistics before IDSWs. (a) Norm of the frame-to-frame change in propagated query representations. (b) Average of the five largest normalized positive projections over all other active tracks. (c) Average normalized positive projection over IoU top-5 nearby active tracks.

Figure 3: Candidate retrieval recall for IDSW counterpart tracks. For each evaluated pre-IDSW state in which the counterpart track is active, other active tracks are ranked by bounding-box IoU, query cosine similarity, or the normalized positive projection in Eq. 4 .

Figure 4: Overview of TQC. The upper flow shows the standard query-propagation process, in which track-query representations are updated with current-frame image features and propagated to the next frame. TQC is inserted after the tracker forward pass and calibrates the tracker output representations before subsequent propagation. It retrieves nearby active tracks using bounding-box IoU, estimates update components directed toward these tracks, and subtracts a bounded correction.

Deformable:
Method	HOTA	DetA	AssA	MOTA	IDF1
MOTR	54.21	73.49	40.19	79.68	51.49
+ TQC	54.92	73.79	41.08	80.07	52.30
MeMOTR	63.24	76.95	52.13	85.45	65.43
+ TQC	64.15	77.55	53.24	86.12	66.30
CO-MOT	69.96	82.07	59.78	90.99	72.25

Table 1: Test results on DanceTrack. † DualMOT is evaluated using the publicly released checkpoint without confidence-guided update suppression. Reproduction details are provided in the supplementary material.

Deformable:
Method	HOTA	DetA	AssA	MOTA	IDF1
MeMOTR	68.78	82.01	57.77	90.21	69.89
+ TQC	69.19	82.15	58.36	90.40	70.46
DAB:
MeMOTR	70.00	83.04	59.09	91.46	71.38
+ TQC	70.14	83.14	59.26	91.64	71.53

Table 2: Test results on SportsMOT.

Dataset	$\Delta$ HOTA	$\Delta$ AssA	$\Delta$ IDF1	Pos.
Largest baseline $\mathrm{DetA}-\mathrm{AssA}$
DanceTrack	+1.53	+2.03	+1.88	6/6
SportsMOT	+1.07	+1.72	+1.27	3/3
Overall	+1.37	+1.93	+1.68	9/9
Lowest baseline AssA
DanceTrack	+1.40	+1.71	+1.74	6/6

Table 3: Performance on association-hard validation sequences. For each tracker and dataset setting, we independently select ten sequences under each of two baseline criteria. Values are mean TQC improvements across tracker settings, computed from sequence-averaged metrics. Pos. denotes the number of settings with positive

\Delta

HOTA.

Variant	$\Delta$ HOTA	$\Delta$ AssA	$\Delta$ IDF1	Pos.
Full TQC	+0.71	+1.06	+0.88	6/6
Correction direction
Reversed correction	-0.91	-1.28	-1.36	0/6
Random directions	-0.04	-0.05	-0.07	3/6
Orthogonal directions	-0.06	-0.09	+0.02	2/6
Non-overlapping-track directions	+0.10	+0.15	+0.01	4/6

Table 4: Controlled variants on six validation settings. Values are average changes from the corresponding baselines. Pos. denotes the number of settings with positive

\Delta

HOTA. Direction variants preserve the TQC correction norm, and random directions are averaged over eight seeds. Per-setting results are provided in the supplementary material.

	Tracker output after TQC	Track query used for propagation
Projection decreased	$98.38\%$	$97.21\%$
Median projection retention	$0.317$	$0.235$
Median update-norm retention	$0.980$	$0.792$

Table 5: Effect of TQC on query updates. Projection and update-norm retention are state-wise ratios between the TQC branch and the paired branch with the current-frame correction disabled.

Figure 5: Qualitative examples of TQC. Examples from DanceTrack and SportsMOT are shown using MeMOTR-DAB. Values denote the mean normalized positive projection over up to five nearby active tracks with positive bounding-box IoU.

Tracker	Base FPS $\uparrow$	+TQC FPS $\uparrow$	Slowdown $\downarrow$	Test $\Delta$ HOTA $\uparrow$
MOTR	9.44	9.35	1.02%	+0.71
MeMOTR-Deformable	8.26	8.17	1.13%	+0.91
CO-MOT	9.33	9.26	0.83%	+0.20
MeMOTR-DAB	7.29	7.22	0.95%	+0.41
SambaMOTR-DAB	7.29	7.23	0.91%	+0.32
DualMOT	6.20	6.16	0.61%	+0.20

Table 6: Inference efficiency and HOTA improvements on DanceTrack. Inference speed is measured on the validation set using a DGX Spark, while

\Delta

HOTA denotes the test-set improvement over the corresponding baseline. Slowdown is computed from unrounded per-frame latencies.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Tracker	$\beta$	$\rho$
DanceTrack	MOTR [ 37 ]	1.50	0.15
	MeMOTR-Deformable [ 10 ]	1.25	0.05
	CO-MOT [ 33 ]	1.00	0.10
	MeMOTR-DAB [ 10 ]	1.75	0.15
	SambaMOTR-DAB [ 25 ]	0.75	0.50
	DualMOT † [ 15 ]	2.00	0.25

Appendix

Table 7: Hyperparameters used for test-set evaluation. The parameters are selected using HOTA on the corresponding validation set and fixed for test-set evaluation. † DualMOT uses the publicly released checkpoint without confidence-guided update suppression.

Figure 6: Distributions of query-update statistics before IDSWs with SambaMOTR-DAB. (a) Norm of the frame-to-frame change in propagated track-query representations. (b) Average of the five largest normalized positive projections over all other active tracks. (c) Average normalized positive projection over IoU top-5 nearby active tracks.

DanceTrack
	All other active tracks		IoU top-5 candidates
Tracker	AUROC	$95\%$ CI	AUROC	$95\%$ CI
MeMOTR-Deformable	0.680	[0.625, 0.735]	0.657	[0.615, 0.705]
MeMOTR-DAB	0.699	[0.661, 0.736]	0.682	[0.647, 0.717]
SambaMOTR-DAB	0.691	[0.649, 0.734]	0.666	[0.635, 0.697]
SportsMOT

Appendix

Table 8: AUROC of normalized positive projections after covariance whitening. Results are reported on six validation settings. Brackets indicate

95\%

confidence intervals obtained from

1{,}000

sequence-level bootstrap resamples.

Candidate set	Track states	IDSW events
IoU top-5	$865/873$ ( $99.08\%$ )	$405/408$ ( $99.26\%$ )

Appendix

Table 9: Coverage of the future IDSW counterpart by the candidate set used by TQC.

Measure	Result
Nonzero counterpart correction term	$767/865$ ( $88.67\%$ )
Counterpart projection decreased	$752/767$ ( $98.04\%$ )
Median update-norm retention	$0.971$
Median counterpart-projection retention	$0.378$

Appendix

Table 10: Effect of TQC on the update component directed toward the IDSW counterpart.

Representation	Projection decreased	Median projection retention [IQR]	Median update-norm retention [IQR]
Tracker output after TQC	$98.38\%$	$0.317$ $[0.109,0.467]$	$0.980$ $[0.970,0.990]$
Track query used for propagation	$97.21\%$	$0.235$ $[0.002,0.503]$	$0.792$ $[0.623,0.920]$

Appendix

Table 11: Effect of TQC on candidate-directed projections immediately after TQC and after subsequent query processing. Each percentage is computed over states for which the corresponding paired comparison is defined. Retention values are state-wise ratios reported as medians and interquartile ranges.

Figure 7: Candidate retrieval recall for future IDSW counterparts with SambaMOTR-DAB. For each pre-IDSW state in which the counterpart track is already active, IoU-based retrieval considers only tracks with positive IoU, as in TQC. Query cosine similarity and the normalized positive projection rank all other active tracks.

	$\Delta$ HOTA						Average
Variant	MeMOTR DAB DT	MeMOTR Def DT	SambaMOTR DAB DT	MeMOTR DAB SP	MeMOTR Def SP	SambaMOTR DAB SP	$\Delta$ HOTA	$\Delta$ AssA	$\Delta$ IDF1
Full TQC	+0.85	+0.96	+0.62	+0.50	+1.22	+0.09	+0.71	+1.06	+0.88
Correction direction
Reversed correction	-2.88	-0.64	-0.52	-0.41	-0.81	-0.20	-0.91	-1.28	-1.36
Random directions	-0.48	+0.27	-0.10	+0.13	+0.11	-0.15	-0.04	-0.05	-0.07
Orthogonal directions	-0.35	-0.31	+0.77	+0.06	-0.22	-0.29	-0.06	-0.09	+0.02

Appendix

Table 12: Controlled variants on six validation settings. Values are changes from the corresponding baseline. Direction variants preserve the TQC correction norm, and random directions are averaged over eight seeds. Norm-preserving correction restores the original update norm, update damping reduces the magnitude of the current-frame query update by the magnitude of the TQC correction, and uniform damping scales continuing-track updates by

\gamma

Tracker	Base	TQC	$\Delta$
Deformable:
MOTR	2,383	2,330	-53
MeMOTR	1,477	1,378	-99
CO-MOT	1,254	1,184	-70
DAB:
MeMOTR	1,115	1,106	-9

Appendix

Table 13: Identity-switch counts on the DanceTrack test set. Lower values are better.

\Delta

denotes TQC

-

Base.

Tracker	Base	TQC	$\Delta$
Deformable:
MeMOTR	3,359	3,238	-121
DAB:
MeMOTR	3,393	3,292	-101
SambaMOTR	3,129	2,835	-294

Appendix

Table 14: Identity-switch counts on the SportsMOT test set. Lower values are better.

\Delta

denotes TQC

-

Base.

Tracker	Metric	Base	TQC	$\Delta$
Deformable:
MOTR	HOTA	51.79	52.77	+0.98
	AssA	33.79	34.75	+0.96
	IDF1	47.28	48.51	+1.23
MeMOTR	HOTA	47.12	48.68	+1.56
	AssA	32.37	34.54	+2.17

Appendix

Table 15: Performance on association-hard DanceTrack validation sequences. For each tracker, we select the ten sequences with the largest values of baseline

\mathrm{DetA}-\mathrm{AssA}

. All values are averages of sequence-level metrics. Differences are computed from unrounded values.

Deformable:
Tracker	Metric	Base	TQC	$\Delta$
MeMOTR	HOTA	67.00	68.63	+1.63
	AssA	52.39	55.04	+2.65
	IDF1	67.15	68.77	+1.62
DAB:
MeMOTR	HOTA	68.18	69.27	+1.09

Appendix

Table 16: Performance on association-hard SportsMOT validation sequences. For each tracker, we select the ten sequences with the largest values of baseline

\mathrm{DetA}-\mathrm{AssA}

. All values are averages of sequence-level metrics. Differences are computed from unrounded values.

Tracker	Metric	Base	TQC	$\Delta$
Deformable:
MOTR	HOTA	39.94	40.16	+0.22
	DetA	64.29	65.51	+1.22
	AssA	25.36	25.15	-0.21
	MOTA	66.83	68.49	+1.67
	IDF1	37.26	37.56	+0.30

Appendix

Table 17: DanceTrack results using an alternative definition of association-hard sequences. For each tracker, we select the ten validation sequences with the lowest baseline AssA. All values are averages of sequence-level metrics. Differences are computed from unrounded values.

Deformable:
Tracker	Metric	Base	TQC	$\Delta$
MeMOTR	HOTA	65.66	68.12	+2.46
	DetA	84.29	84.56	+0.27
	AssA	51.33	55.28	+3.96
	MOTA	93.10	93.52	+0.42
	IDF1	66.29	69.16	+2.87

Appendix

Table 18: SportsMOT results using an alternative definition of association-hard sequences. For each tracker, we select the ten validation sequences with the lowest baseline AssA. All values are averages of sequence-level metrics. Differences are computed from unrounded values.

Deformable:
Tracker	Param.	$\beta$	$\rho$	HOTA	AssA	IDF1
MOTR	$\beta$	1.25	0.15	51.85	38.88	49.80
	$\beta$	1.50	0.15	51.87	38.82	49.80
	$\beta$	1.75	0.15	51.84	38.77	49.87
	$\rho$	1.50	0.10	51.20	38.11	48.94
	$\rho$	1.50	0.15	51.87	38.82	49.80

Appendix

Table 19: Hyperparameter sensitivity on the DanceTrack validation set. For each tracker, we vary one parameter at a time around the selected value while keeping the other parameter fixed. Best values within each one-parameter sweep are shown in bold.

Deformable:
Tracker	Param.	$\beta$	$\rho$	HOTA	AssA	IDF1
MeMOTR	$\beta$	1.25	0.15	75.48	66.66	78.26
	$\beta$	1.50	0.15	75.65	66.92	78.48
	$\beta$	1.75	0.15	75.64	66.93	78.46
	$\rho$	1.50	0.10	75.48	66.62	78.22
	$\rho$	1.50	0.15	75.65	66.92	78.48

Appendix

Table 20: Hyperparameter sensitivity on the SportsMOT validation set. For each tracker, we vary one parameter at a time around the selected value while keeping the other parameter fixed. Best values within each one-parameter sweep are shown in bold.

Tracker	Transfer	$(\beta,\rho)$	Transferred $\Delta$ HOTA	Target-specific $\Delta$ HOTA
MeMOTR-Deformable	DT $\rightarrow$ SP	(1.25, 0.05)	+0.10	+0.41
MeMOTR-DAB	DT $\rightarrow$ SP	(1.75, 0.15)	+0.10	+0.14
SambaMOTR-DAB	DT $\rightarrow$ SP	(0.75, 0.50)	+0.19	+0.08
MeMOTR-Deformable	SP $\rightarrow$ DT	(1.50, 0.15)	+0.88	+0.91
MeMOTR-DAB	SP $\rightarrow$ DT	(0.75, 0.10)	-0.87	+0.41
SambaMOTR-DAB	SP $\rightarrow$ DT	(0.50, 0.75)	+0.48	+0.32

Appendix

Table 21: Cross-dataset transfer of TQC hyperparameters on the test sets. DT and SP denote DanceTrack and SportsMOT, respectively. Transferred parameters are selected on the source validation set, whereas target-specific parameters are selected on the target validation set.

Dataset	Method	HOTA	DetA	AssA	MOTA	IDF1
MOT17	MeMOTR-DAB	58.8	59.6	58.4	72.9	71.5
	+ TQC	58.8	59.7	58.3	73.0	71.7
MOT20	MeMOTR-DAB	36.8	33.7	40.4	40.6	46.0
	+ TQC	37.5	35.1	40.2	42.6	47.4

Appendix

Table 22: Test results on MOT17 and MOT20 with MeMOTR-DAB. Both evaluations use the official MOT17 checkpoint without additional training on MOT20. TQC uses

(\beta,\rho)=(0.75,0.25)

, selected on MOT20 train and fixed for both test sets.

Method	mTETA	mHOTA	mLocA	mAssocA	mAssA
MeMOTR-DAB	52.85	40.59	39.63	56.05	52.74
+ TQC	52.90	40.64	39.62	56.28	52.78

Appendix

Table 23: Results on BDD100K validation. All metrics are averaged over eight classes. mAssocA and mAssA are association metrics from TETA and HOTA, respectively.

Deformable:
Tracker	#Q	Params	Base (GFLOPs)	TQC (MFLOPs)
MOTR	2	44.14M	233.28	0.0114
	23	44.14M	233.57	1.1305
MeMOTR	2	42.88M	233.28	0.0114
	23	42.88M	233.50	1.1275
CO-MOT	6	41.94M	232.04	0.0836

Appendix

Table 24: Computational complexity of TQC. Base denotes the estimated computation of the tracker forward pass, and TQC denotes the additional calibration cost. We follow a common profiling convention that counts each multiply-accumulate as one operation. Here, #Q denotes the number of active track-query slots, and Params denotes the total number of model parameters. TQC introduces no additional learnable parameters.

Multi-Object Tracking

Multi-object tracking (MOT) links detections across video frames to maintain object identities through motion, occlusion, and changing viewpoints.

See all 194 papers

Aug 18, 2026

Probing Association Instability with Track-State Perturbations for Clip-Level Active Learning in Query-Propagation Multi-Object Tracking

Training query-propagation end-to-end multi-object tracking (MOT) models requires dense bounding-box and identity annotations across video sequences, making dataset construction expensive. Clip-level active learning reduces this cost by selecting video clips for annotation, but prior acquisition criteria based on output-level temporal uncertainty may miss clips whose informativeness comes from association instability in propagated track states. We propose QPID (Query-Propagation Instability and Diversity), a clip acquisition method for query-propagation MOT that targets association instability in propagated track states. QPID estimates this instability by applying two-sided perturbations to internal track states and measuring prediction differences from a clean reference branch. The key idea is that, in stable clips, each propagated track should continue to follow the same target under small perturbations, whereas in ambiguous clips, small changes in the track state can alter which target the track follows, leading to changes in localization or confidence. QPID measures these perturbation-induced prediction differences with two metrics: Localization Drift and Entropy-Weighted Confidence Discrepancy. These metrics are aggregated into a clip-level association-instability score. To avoid redundant uncertainty-only selection, QPID selects a representative annotation batch from high-instability clips using Uncertainty-Weighted Visual Coverage with track-level visual prototypes. Experiments on DanceTrack and SportsMOT with MeMOTR and SambaMOTR show that QPID achieves strong performance compared with active learning baselines under the same annotation budget.

Riku Inoue, Shogo Sato, Kazuhiko Murasaki +3

Human Informatics Laboratories NTT, Inc. Kanagawa, Japan

Multi-Object Tracking Active Learning Uncertainty Quantification

Ask PDF

Apr 1, 2026

Out of Sight, Out of Track: Adversarial Attacks on Propagation-based Multi-Object Trackers via Query State Manipulation

Recent Tracking-by-Query-Propagation (TBP) methods have advanced Multi-Object Tracking (MOT) by enabling end-to-end (E2E) pipelines with long-range temporal modeling. However, this reliance on query propagation introduces unexplored architectural vulnerabilities to adversarial attacks. We present FADE, a novel attack framework designed to exploit these specific vulnerabilities. FADE employs two attack strategies targeting core TBP mechanisms: (i) Temporal Query Flooding: Generates spurious temporally consistent track queries to exhaust the tracker's limited query budget, forcing it to terminate valid tracks. (ii) Temporal Memory Corruption: Directly attacks the query updater's memory by severing temporal links via state de-correlation and erasing the learned feature identity of matched tracks. Furthermore, we introduce a differentiable pipeline to optimize these attacks for physical-world realizability by leveraging simulations of advanced perception sensor spoofing. Experiments on MOT17 and MOT20 benchmarks demonstrate that FADE is highly effective against state-of-the-art TBP trackers, causing significant identity switches and track terminations.

Halima Bouzidi, Haoyu Liu, Yonatan Gizachew Achamyeleh +2

University of California, Irvine, CA, USA

Adversarial Attacks Multi-Object Tracking Physical Adversarial Attacks

Ask PDF

Mar 14, 2026

QTrack: Query-Driven Reasoning for Multi-modal MOT

Category-level multi-object trackers estimate trajectories for all detected instances, whereas some applications require propagating only targets specified by a user. We study offline, reference-grounded video tracking: given an ordered video clip, a natural-language query, and the reference-frame boxes of the specified targets, a model predicts their subsequent trajectories. We introduce QTrack, a vision-language policy trained with structured trajectory rewards and Temporal Perception-Aware Policy Optimization (TAPO). TAPO contrasts the likelihood of a generated response under the observed clip and a reference-frame repetition, encouraging the output to depend on visual evidence beyond the reference frame. We also introduce RMOT26, a controlled diagnostic benchmark constructed from existing tracking datasets. Under the reported protocol, QTrack obtains a motion- consistency score of 0.30, MOTP of 0.75, center-location error of 44.61 pixels, and normalized distance error of 0.39; its motion-consistency score ties the strongest comparison method. Component ablations suggest complementary effects from the motion reward and temporal contrast, while an exploratory four- clip, 50-frame evaluation examines transfer beyond the primary short-clip setting. The results support further study of temporal-evidence learning in trajectory- generating vision-language models. Code and data are available at https://github.com/gaash-lab/QTrack

Tajamul Ashraf, Tavaheed Tariq, Sonia Yadav +4

King Abdullah University of Science and Technology (KAUST), Saudi Arabia · Gaash Research Lab, National Institute of Technology Srinagar, India · The University of Queensland, Australia

Referring MOT Multimodal Visual Object Tracking Multi-Object Tracking Vision-Language Models Multimodal Reasoning

Ask PDF

Abstract

Multi-Object Tracking

Explore similar work

Probing Association Instability with Track-State Perturbations for Clip-Level Active Learning in Query-Propagation Multi-Object Tracking

Out of Sight, Out of Track: Adversarial Attacks on Propagation-based Multi-Object Trackers via Query State Manipulation

QTrack: Query-Driven Reasoning for Multi-modal MOT