Out of Sight, Out of Track: Adversarial Attacks on Propagation-based Multi-Object Trackers via Query State Manipulation

Halima Bouzidi; Haoyu Liu; Yonatan Gizachew Achamyeleh; Praneetsai Vasu Iddamsetty; Mohammad Abdullah Al Faruque

cs.CVApr 1, 2026

Out of Sight, Out of Track: Adversarial Attacks on Propagation-based Multi-Object Trackers via Query State Manipulation

Authors: Halima Bouzidi, Haoyu Liu, Yonatan Gizachew Achamyeleh, Praneetsai Vasu Iddamsetty, Mohammad Abdullah Al Faruque

Organizations: University of California, Irvine, CA, USA

Abstract

Recent Tracking-by-Query-Propagation (TBP) methods have advanced Multi-Object Tracking (MOT) by enabling end-to-end (E2E) pipelines with long-range temporal modeling. However, this reliance on query propagation introduces unexplored architectural vulnerabilities to adversarial attacks. We present FADE, a novel attack framework designed to exploit these specific vulnerabilities. FADE employs two attack strategies targeting core TBP mechanisms: (i) Temporal Query Flooding: Generates spurious temporally consistent track queries to exhaust the tracker's limited query budget, forcing it to terminate valid tracks. (ii) Temporal Memory Corruption: Directly attacks the query updater's memory by severing temporal links via state de-correlation and erasing the learned feature identity of matched tracks. Furthermore, we introduce a differentiable pipeline to optimize these attacks for physical-world realizability by leveraging simulations of advanced perception sensor spoofing. Experiments on MOT17 and MOT20 benchmarks demonstrate that FADE is highly effective against state-of-the-art TBP trackers, causing significant identity switches and track terminations.

Figures & tables

Figure 1: Overview of the Proposed FADE Attack Strategies.

Figure 2: Overview of the FADE Attack Pipeline. (a) A PGD differentiable optimization loop crafts digital or physical perturbations $\omega$ . (b) The TBP tracker ( $f_{MOT}$ ) processes the perturbation, and its outputs/hidden-states ( $\hat{Y}^{t}$ and $\mathcal{H}^{t}$ ) are used to calculate (bottom) $\mathcal{L}_{\text{FADE}}\in\{\mathcal{L}_{\text{TQF}},\mathcal{L}_{\text{TMC}}\}$ . (c) The visualized results show $\mathcal{L}_{\text{TQF}}$ causing query starvation via spurious track injection and $\mathcal{L}_{\text{TMC}}$ causing re-association failure via temporal memory corruption. The loss gradient is fed back to the optimization loop (a), closing the FADE pipeline.

Figure 3: Visualization of the TQF attack effectiveness against MeMOTR [ 7 ] on a video sequence from the MOT17 benchmark.

MOT17 Dataset – Average scene density $\sim$ 21 objects/frame.
Tracker	Attacker	HOTA $\downarrow$	DetA $\downarrow$	AssA $\downarrow$	IDF1 $\downarrow$	IDR $\downarrow$	IDP $\downarrow$	IDSW $\uparrow$
MOTR [ 36 ]	Clean	58.63	49.90	70.38	69.35	68.48	71.40	7.23
	Daedalus [ 31 ]	45.85	51.26	42.19	50.48	46.37	55.88	8.74
	Hijacking [ 11 ]	46.08	51.24	42.61	50.77	46.78	55.97	8.89
	F&F [ 40 ]	46.59	51.10	43.71	51.55	47.44	56.92	8.71
	$\text{FADE}_{\text{TMC}}$	45.89	51.36	42.18	50.45	46.35	55.79	8.77

Table 1: Digital Attack Performance on MOT17.

MOT20 Dataset – High Density: Avg. $\sim$ 150 objects/frame.
Tracker	Attacker	HOTA $\downarrow$	DetA $\downarrow$	AssA $\downarrow$	IDF1 $\downarrow$	IDR $\downarrow$	IDP $\downarrow$	IDSW $\uparrow$
MOTR [ 36 ]	Clean	55.06	40.57	75.89	64.10	57.65	74.80	5.14
	Daedalus [ 31 ]	42.07	41.54	43.63	49.30	45.22	54.37	8.78
	Hijacking [ 11 ]	43.62	38.04	51.64	51.46	45.43	60.22	9.02
	F&F [ 40 ]	41.36	31.84	55.04	47.94	39.30	64.88	7.72
	$\text{FADE}_{\text{TMC}}$	40.38	39.70	42.24	47.90	38.96	64.03	6.61

Table 2: Digital Attack Performance on MOT20.

MOT17 Dataset – Average scene density $\sim$ 21 objects/frame.
Tracker	Attack_Vector	HOTA $\downarrow$	DetA $\downarrow$	AssA $\downarrow$	IDF1 $\downarrow$	IDR $\downarrow$	IDP $\downarrow$	IDSW $\uparrow$
MOTR [ 36 ]	Clean	58.63	49.90	70.38	69.35	68.48	71.40	7.23
	TMC_AAI	45.09	47.37	44.21	51.57	46.44	58.43	8.64
	TMC_EAI	42.81	47.87	39.37	47.76	42.77	54.55	9.22
MOTRv2 [ 39 ]	Clean	59.96	49.15	74.71	71.99	60.89	89.68	1.75
	TMC_AAI	53.66	44.12	66.57	66.61	53.83	88.96	2.00

Table 3: Simulated Physical Attack Performance on MOT17.

MOT20 Dataset – High Density: Avg. $\sim$ 150 objects/frame.
Tracker	Attack_Vector	HOTA $\downarrow$	DetA $\downarrow$	AssA $\downarrow$	IDF1 $\downarrow$	IDR $\downarrow$	IDP $\downarrow$	IDSW $\uparrow$
MOTR [ 36 ]	Clean	55.06	40.57	75.89	64.10	57.65	74.80	5.14
	TMC_AAI	36.17	34.06	39.72	43.08	31.83	69.95	5.64
	TMC_EAI	41.78	35.99	51.65	48.65	37.36	71.02	5.70
MOTRv2 [ 39 ]	Clean	59.56	44.74	80.71	69.55	54.90	96.81	0.73
	TMC_AAI	54.44	41.32	72.97	66.04	50.52	97.12	0.74

Table 4: Simulated Physical Attack Performance on MOT20.

Figure 4: Qualitative comparison of the TMC attack with AAI (upper) and EAI (bottom) attack vectors on the MOT17 benchmark.

Adv. Loss	CO-MOT [ 18 ]			MeMOTR [ 7 ]			Samba [ 25 ]
Adv. Loss	HOTA	IDF1	IDSW	HOTA	IDF1	IDSW	HOTA	IDF1	IDSW
Clean	58.16	69.87	1.83	67.35	80.83	0.81	62.91	73.67	1.02
$\mathcal{L}_{\text{Flood}}$	40.25	48.51	14.45	41.04	50.99	4.47	51.50	61.85	1.91
w/o $\mathcal{L}_{\text{Flood}}$	46.54	56.51	8.89	54.27	66.13	2.40	51.86	61.72	1.58
w/o $\mathcal{L}_{\text{Cost}}$	36.44	43.99	9.77	42.53	52.67	4.10	53.14	64.02	1.98
w/o $\mathcal{L}_{\text{Siphon}}$	39.00	46.44	15.28	41.88	51.94	4.44	44.96	51.87	2.28

Table 5: Ablation Study on FADE Adversarial Losses.

Attack Vector	Metrics	Number of PGD Iterations
Attack Vector	Metrics	T=0	T=5	T=10	T=50	T=100	T=150
$\text{FADE}_{\text{TMC}}$ AAI	HOTA	52.03	49.51	49.62	43.02	39.85	39.81
	IDF1	60.35	57.34	58.02	48.89	44.95	44.86
	IDSW	4.39	6.05	5.47	6.26	6.85	7.05
$\text{FADE}_{\text{TMC}}$ EAI	HOTA	52.03	43.42	42.39	42.38	41.06	41.92
	IDF1	60.35	49.23	48.01	47.62	46.40	47.47

Table 6: Simulated Physical Attack Optimization Convergence.

Figure 5: Physical Adversarial Attacks Scenarios and Models.

Figure 6: Illustration of the adversarial frame $F^{adv}_{t}$ from the $\mathcal{P}_{AAI}$ transformation. The top row when fixing $x,y,\phi$ at minimum values while varying hyperparameter $\mathcal{D}$ from 5 (low) to 30 (high). The top row when fixing $x,y,\phi$ at maximum values while varying hyperparameter $\mathcal{D}$ from 5 (low) to 30 (high).

Figure 7: Natural Gaussian blur motion Vs. AAI-induced blur.

Figure 8: Illustration of the adversarial frame $F^{adv}_{t}$ from the $\mathcal{P}_{EAI}$ transformation. The top row when applying random color stripes while varying $\mathcal{N}$ from 3 (low) to 20 (high).

Attack	Dataset	MOTR	MOTRv2	MeMOTR	Samba	CO-MOT
Clean	MOT17 \| MOT20	58.63 \| 55.40	59.96 \| 56.70	67.35 \| 68.20	62.91 \| 64.50	58.16 \| 61.20
Untargeted PGD	MOT17 \| MOT20	49.90 \| 40.10	57.41 \| 57.77	59.85 \| 60.74	58.38 \| 57.61	55.85 \| 60.95
BBA_Blind	MOT17 \| MOT20	48.51 \| 41.19	56.07 \| 59.36	59.10 \| 58.47	56.79 \| 54.27	53.02 \| 57.56
BBA_Blur	MOT17 \| MOT20	46.92 \| 35.85	57.13 \| 58.13	62.54 \| 64.67	56.52 \| 51.63	53.55 \| 58.97
$\text{FADE}_{\text{TMC}}$	MOT17 \| MOT20	45.90 \| 42.63	39.29 \| 29.64	41.41 \| 57.67	45.53 \| 46.85	37.26 \| 33.37
$\text{FADE}_{\text{TQF}}$	MOT17 \| MOT20	45.89 \| 40.38	46.76 \| 56.68	41.56 \| 37.70	48.04 \| 54.91	41.73 \| 49.28

Table 7: FADE vs. Untargeted PGD and recent MOT attack.

Attack	Dataset	MOTR	MOTRv2	MeMOTR	Samba	CO-MOT
TQF w/ GT	MOT17 \| MOT20	45.90 \| 42.63	39.29 \| 29.64	41.41 \| 57.67	45.53 \| 46.85	37.26 \| 33.37
TQF w/ Pred	MOT17 \| MOT20	45.91 \| 42.46	39.17 \| 29.69	42.26 \| 58.25	44.82 \| 46.31	37.11 \| 33.30
TQF w/o $\mathcal{L}_{\text{Cost}}$	MOT17 \| MOT20	45.84 \| 42.42	38.48 \| 29.67	42.53 \| 59.38	53.14 \| 55.62	36.44 \| 33.58

Table 8: Ablation on Cost Mimicry in the TQF Loss (HOTA)

MOT17 Dataset – Average scene density $\sim$ 21 objects/frame.
Tracker	Attack_Vector	HOTA $\downarrow$	DetA $\downarrow$	AssA $\downarrow$	IDF1 $\downarrow$	IDR $\downarrow$	IDP $\downarrow$	IDSW $\uparrow$
MOTR	Clean	58.63	49.90	70.38	69.35	68.48	71.40	7.23
	TQF_AAI	52.66	45.19	62.80	64.66	61.06	70.08	7.70
	TQF_EAI	42.74	48.15	39.04	47.68	42.49	54.77	8.87
MOTRv2	Clean	59.96	49.15	74.71	71.99	60.89	89.68	1.75
	TQF_AAI	52.50	43.08	65.32	65.06	52.48	87.28	2.64

Table 9: Simulated Physical FADE-TQF Attacks on MOT17.

MOT20 Dataset – High Density: Avg. $\sim$ 150 objects/frame.
Tracker	Attack_Vector	HOTA $\downarrow$	DetA $\downarrow$	AssA $\downarrow$	IDF1 $\downarrow$	IDR $\downarrow$	IDP $\downarrow$	IDSW $\uparrow$
MOTR	Clean	55.06	40.57	75.89	64.10	57.65	74.80	5.14
	TQF_AAI	50.47	37.94	68.31	61.04	52.36	75.36	5.45
	TQF_EAI	41.97	36.05	52.00	48.81	38.48	68.60	6.60
MOTRv2	Clean	59.56	44.74	80.71	69.55	54.90	96.81	0.73
	TQF_AAI	53.75	40.34	72.93	64.76	49.18	97.02	0.80

Table 10: Simulated Physical FADE-TQF Attacks on MOT20.

Figure 9: Analysis of Long-term Memory Integrity in TBP Trackers. The figure depicts the state of the MeMOTR [ 7 ] ’s memory bank before (PGD Step 0) and after (PGD Step 149) the FADE attack, showing how FADE undermines the tracking identity memory mechanism.

Surrogate	Target $\rightarrow$	MOTR	MOTRv2	MeMOTR	Samba	CO-MOT
Surrogate	Clean HOTA	MOTR	MOTRv2	MeMOTR	Samba	CO-MOT
None (Baseline)		58.63	59.96	67.35	62.91	58.16
MOTR	TMC \| TQF	42.74 \| 42.72	51.89 \| 52.04	60.31 \| 60.31	56.12 \| 55.53	48.97 \| 49.51
MOTRv2	TMC \| TQF	42.76 \| 42.79	37.08 \| 27.98	60.65 \| 60.11	55.94 \| 56.56	48.09 \| 44.78
MeMOTR	TMC \| TQF	47.35 \| 46.26	54.11 \| 53.76	46.01 \| 29.10	52.10 \| 52.43	52.16 \| 52.44
Samba	TMC \| TQF	47.60 \| 47.72	54.24 \| 53.88	55.20 \| 56.45	43.22 \| 36.30	52.68 \| 52.13

Table 11: Black-box Transferability Check (HOTA) on MOT17.

MOT17 Dataset – Average scene density $\sim$ 21 objects/frame.
Tracker	Attack_Vector	Defense	HOTA	DetA	AssA	IDF1	IDR	IDP	IDSW
MOTR	Clean	None	58.63	49.90	70.38	69.35	68.48	71.40	7.23
	$\text{FADE}_{\text{TMC}}$	CJ	46.00	47.45	45.88	51.67	46.68	58.57	9.67
	$\text{FADE}_{\text{TMC}}$	GN	46.38	47.90	46.22	51.87	46.63	59.05	9.51
	$\text{FADE}_{\text{TMC}}$	SS	46.02	47.88	45.54	51.48	46.13	58.82	9.56
	$\text{FADE}_{\text{TMC}}$	None	45.89	51.36	42.18	50.45	46.35	55.79	8.77

Table 12: FADE-TMC Performances under Defenses on MOT17.

MOT17 Dataset – Average scene density $\sim$ 21 objects/frame.
Tracker	Attack_Vector	Defense	HOTA	DetA	AssA	IDF1	IDR	IDP	IDSW
MOTR	Clean	None	58.63	49.90	70.38	69.35	68.48	71.40	7.23
	$\text{FADE}_{\text{TQF}}$	CJ	47.00	47.47	47.87	53.09	47.88	60.24	9.17
	$\text{FADE}_{\text{TQF}}$	GN	47.03	47.92	47.51	53.19	48.23	60.04	9.69
	$\text{FADE}_{\text{TQF}}$	SS	47.18	47.26	48.43	53.21	48.15	60.08	9.47
	$\text{FADE}_{\text{TQF}}$	None	45.90	51.40	42.17	50.51	46.40	55.86	8.73

Table 13: FADE-TQF Performances under Defenses on MOT17.

Adversarial Attacks

Adversarial attacks exploit crafted inputs, poisoned data, or system interactions to manipulate AI predictions, behavior, privacy, or reliability.

See all 444 papers

Probing Association Instability with Track-State Perturbations for Clip-Level Active Learning in Query-Propagation Multi-Object Tracking
Aug 18, 2026Riku Inoue, Shogo Sato, Kazuhiko Murasaki +3Multi-Object Tracking Active Learning
Human Informatics Laboratories NTT, Inc. Kanagawa, Japan
QTrack: Query-Driven Reasoning for Multi-modal MOT
Mar 14, 2026Tajamul Ashraf, Tavaheed Tariq, Sonia Yadav +4Referring MOT Multimodal Visual Object Tracking
King Abdullah University of Science and Technology (KAUST), Saudi Arabia · Gaash Research Lab, National Institute of Technology Srinagar, India · The University of Queensland, Australia
Tracking-by-detection in Multi-object Tracking: Survey and Experiments
Sep 8, 2026Yujin Yang, Kyujin Shim, Kangwook Ko +1Data Association Multi-Object Tracking
School of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, 34141, South Korea.

Aug 18, 2026J/K move · Enter open · S save

Probing Association Instability with Track-State Perturbations for Clip-Level Active Learning in Query-Propagation Multi-Object Tracking

Riku Inoue, Shogo Sato, Kazuhiko Murasaki, Tomoyasu Shimada+2

Human Informatics Laboratories NTT, Inc. Kanagawa, Japan

Multi-Object Tracking Active Learning Uncertainty Quantification

Open paper AskPDF arXiv

Training query-propagation end-to-end multi-object tracking (MOT) models requires dense bounding-box and identity annotations across video sequences, making dataset construction expensive. Clip-level active learning reduces this cost by selecting video clips for annotation, but prior acquisition criteria based on output-level temporal uncertainty may miss clips whose informativeness comes from association instability in propagated track states. We propose QPID (Query-Propagation Instability and Diversity), a clip acquisition method for query-propagation MOT that targets association instability in propagated track states. QPID estimates this instability by applying two-sided perturbations to internal track states and measuring prediction differences from a clean reference branch. The key idea is that, in stable clips, each propagated track should continue to follow the same target under small perturbations, whereas in ambiguous clips, small changes in the track state can alter which target the track follows, leading to changes in localization or confidence. QPID measures these perturbation-induced prediction differences with two metrics: Localization Drift and Entropy-Weighted Confidence Discrepancy. These metrics are aggregated into a clip-level association-instability score. To avoid redundant uncertainty-only selection, QPID selects a representative annotation batch from high-instability clips using Uncertainty-Weighted Visual Coverage with track-level visual prototypes. Experiments on DanceTrack and SportsMOT with MeMOTR and SambaMOTR show that QPID achieves strong performance compared with active learning baselines under the same annotation budget.

Attack	Dataset	MOTR	MOTRv2	MeMOTR	Samba	CO-MOT
Clean	MOT17 \| MOT20	58.63 \| 55.40	59.96 \| 56.70	67.35 \| 68.20	62.91 \| 64.50	58.16 \| 61.20
Untargeted PGD	MOT17 \| MOT20	49.90 \| 40.10	57.41 \| 57.77	59.85 \| 60.74	58.38 \| 57.61	55.85 \| 60.95
BBA_Blind	MOT17 \| MOT20	48.51 \| 41.19	56.07 \| 59.36	59.10 \| 58.47	56.79 \| 54.27	53.02 \| 57.56
BBA_Blur	MOT17 \| MOT20	46.92 \| 35.85	57.13 \| 58.13	62.54 \| 64.67	56.52 \| 51.63	53.55 \| 58.97
$\text{FADE}_{\text{TMC}}$	MOT17 \| MOT20	45.90 \| 42.63	39.29 \| 29.64	41.41 \| 57.67	45.53 \| 46.85	37.26 \| 33.37
$\text{FADE}_{\text{TQF}}$	MOT17 \| MOT20	45.89 \| 40.38	46.76 \| 56.68	41.56 \| 37.70	48.04 \| 54.91	41.73 \| 49.28

Attack	Dataset	MOTR	MOTRv2	MeMOTR	Samba	CO-MOT
TQF w/ GT	MOT17 \| MOT20	45.90 \| 42.63	39.29 \| 29.64	41.41 \| 57.67	45.53 \| 46.85	37.26 \| 33.37
TQF w/ Pred	MOT17 \| MOT20	45.91 \| 42.46	39.17 \| 29.69	42.26 \| 58.25	44.82 \| 46.31	37.11 \| 33.30
TQF w/o $\mathcal{L}_{\text{Cost}}$	MOT17 \| MOT20	45.84 \| 42.42	38.48 \| 29.67	42.53 \| 59.38	53.14 \| 55.62	36.44 \| 33.58

Abstract

Adversarial Attacks

Explore similar work

Probing Association Instability with Track-State Perturbations for Clip-Level Active Learning in Query-Propagation Multi-Object Tracking

QTrack: Query-Driven Reasoning for Multi-modal MOT

Tracking-by-detection in Multi-object Tracking: Survey and Experiments