TAQ-MOTR: Tracking-Aware Detection Query Formation for End-to-End Multiple Object Tracking
Authors: Hanyang Liu, Hong Zhang, Ding Yuan, Yang Han, Yifan Yang
Organizations: School of Astronautics, Beihang University, 37 Xueyuan Road, Haidian District, Beijing 100191, China · Key Laboratory of Spacecraft Design Optimization and Dynamic Simulation Technology, Ministry of Education · State Key Laboratory of High-Efficiency Reusable Aerospace Transportation Technology · School of Software, Beihang University, 37 Xueyuan Road, Haidian District, Beijing 100191, China
Tracking-by-propagation (TBP) methods perform end-to-end multi-object tracking by propagating track queries across frames, jointly handling detection and identity association without heuristic post-processing while naturally modeling temporal information. However, as detection and association are tightly coupled, detection queries in TBP are expected to discover currently untracked objects and provide complementary current-frame evidence for challenging tracks whose queries and identities are retained. Existing methods largely inherit detection-query formation from single-frame detection and construct such queries before the decoder with little awareness of trajectory states. Consequently, currently untracked objects may be missed, while challenging tracked objects with reduced visibility under occlusion may lack complementary evidence, causing association failures or duplicate responses. To address these issues, we propose TAQ-MOTR, a tracking-aware detection query formation framework that employs a pre-decoder Hybrid Query Generator (HQG) to fuse current-frame image features, track memory, and learnable memory. By conditioning detection queries on trajectory states, TAQ-MOTR adapts them to \rev{the discovery of currently untracked objects and tracked-object complementation We further introduce a training-only auxiliary detection branch whose one-to-one label assignment is decoupled from that of the main tracking branch, providing additional detection supervision to strengthen the shared HQG and decoder for the discovery of currently untracked objects. TAQ-MOTR improves performance on DanceTrack, SportsMOT, and BFT, achieving 72.0 HOTA, 63.1 AssA, and 76.6 IDF1 on the DanceTrack test set. Code is available at https://github.com/Magicmiya/TAQ-MOTR
Figures & tables
Figure 1: Detection query formation before decoding. (a) MOTR: learnable queries; (b) MOTRv2: detector-guided queries; (c) TAQ-MOTR: tracking-aware queries. The comparison highlights current-frame information, trajectory-state priors, and learnable embeddings.
Figure 2: Overall architecture of TAQ-MOTR. TAQ-MOTR forms tracking-aware detection queries before decoding. In the main tracking branch, HQG uses image-based detection proposals as queries and reads track memory and learnable memory as keys/values. The generated detection queries are concatenated with updated track queries and jointly decoded. During training, an auxiliary detection branch shares the detection proposals and HQG layers but removes track memory, producing a separate detection-query group supervised by all current-frame objects. Attention masks isolate this auxiliary group from the main query groups. The right panel shows an HQG layer.
Figure 3: Prior carriers used in HQG. Learnable memory provides learnable detection priors that are independent of specific trajectories. Track memory encodes existing trajectory states from propagated track queries, including content, position, and disappearance-time information. Detection proposals provide current-frame object evidence and coarse spatial priors.
Figure 4: Positive label assignments during training on DanceTrack. All ratios use the total GT count per epoch, including clip-initial frames. The auxiliary branch is disabled on each clip’s first frame, so Auxiliary Detection/GT remains below 100% while enabled. Dashed lines mark clip-length changes; shading marks branch shutdown from epoch 24. The auxiliary-only ratio is then zero and omitted. Epoch numbering starts at 0.
Method
HOTA ↑
DetA ↑
AssA ↑
MOTA ↑
IDF1 ↑
Tracking-by-Detection-like
Hybrid-SORT-ReID † [ 34 ] (AAAI 2024)
65.7
–
–
91.8
67.4
DiffMOT † [ 22 ] (CVPR 2024)
62.3
82.5
47.2
92.8
63.0
PlugTrack † [ 14 ] (AAAI 2026)
63.3
82.5
48.4
92.4
64.1
MOTIP [ 7 ] (CVPR 2025)
69.6
80.4
60.4
90.6
74.7
OA-SORT † [ 15 ] (CVPR 2026)
63.1
–
48.5
91.7
64.2
Table 1: Comparison with state-of-the-art methods on the DanceTrack test set. Best and second-best results among Tracking-by-Propagation (TBP) methods are marked in bold and underlined , respectively. Tracking-by-Detection-like methods are listed for reference only. ‡ denotes additional data for tracking-model training, and † denotes additional models. Train+val results are excluded from ranking.
Method
HOTA ↑
DetA ↑
AssA ↑
MOTA ↑
IDF1 ↑
Tracking-by-Detection-like
DiffMOT † [ 22 ] (CVPR 2024)
72.1
86.0
60.5
94.5
72.8
MOTIP [ 7 ] (CVPR 2025)
72.6
83.5
63.2
92.4
77.1
ETTrack † [ 9 ] (AI 2025)
72.2
86.9
60.1
94.9
72.5
FDTA † [ 28 ] (CVPR 2026)
74.2
84.1
65.5
93.0
78.5
Tracking-by-Propagation (TBP)
Table 2: Comparison with state-of-the-art methods on the SportsMOT test set. Best and second-best results among Tracking-by-Propagation (TBP) methods are marked in bold and underlined , respectively. Tracking-by-Detection-like methods are listed for reference only. ‡ denotes additional data for tracking-model training, and † denotes additional models. Our mixed variant uses additional CrowdHuman data and is excluded from ranking.
Method
HOTA ↑
DetA ↑
AssA ↑
MOTA ↑
IDF1 ↑
Tracking-by-Detection-like
SORT † [ 2 ] (ICIP 2016)
61.2
60.6
62.3
75.5
77.2
ByteTrack † [ 38 ] (ECCV 2022)
62.5
61.2
64.1
77.2
82.3
OC-SORT † [ 3 ] (CVPR 2023)
66.8
65.4
68.7
77.1
79.3
MOTIP [ 7 ] (CVPR 2025)
70.5
69.6
71.8
77.1
82.1
FDTA † [ 28 ] (CVPR 2026)
72.2
70.1
74.5
78.2
84.2
Table 3: Comparison with state-of-the-art methods on the BFT test set. Best and second-best results among Tracking-by-Propagation (TBP) methods are marked in bold and underlined , respectively. Tracking-by-Detection-like methods are listed for reference only. ‡ denotes additional data for tracking-model training, and † denotes additional models.
Method
Parameters (M) ↓
FPS ↑
MOTR [ 36 ]
43.9
21.0
MOTRv2 [ 39 ] (with YOLOX)
140.6
16.1
CO-MOT [ 33 ]
41.7
25.1
MeMOTR [ 8 ]
42.9
22.5
TAQ-MOTR
48.9
41.1
Table 4: Efficiency comparison with open-source TBP methods on DanceTrack. All methods follow the same measurement protocol. MOTRv2 includes YOLOX in both its parameter count and inference time.
Figure 5: Module-level efficiency analysis of TAQ-MOTR. Rounded parameter counts and reported per-frame runtimes are shown for the backbone, hybrid encoder, decoder, HQG, and query updater.
DP
TM
LM
HOTA ↑
DetA ↑
AssA ↑
IDF1 ↑
DetRe ↑
IDSW ↓
Baselines
Original
63.9
74.6
55.0
67.1
–
1237
Unified basic model
62.9
76.0
52.3
66.1
85.4
1530
HQG variants
✓
–
–
58.1
71.9
47.1
61.9
–
–
✓
✓
–
63.3
74.1
54.2
68.0
83.7
996
Table 5: Ablation of HQG priors on the full DanceTrack validation set. DP, TM, and LM denote detection proposals, track memory, and learnable memory, respectively. Original reports the published MeMOTR results [ 8 ] . Unified basic model adopts the same backbone, encoder, and pretrained weights as TAQ-MOTR while retaining the rest of MeMOTR’s tracking design. The best result for each metric is shown in bold, and unreported values are indicated by a dash.
Level
HOTA ↑
DetA ↑
AssA ↑
IDF1 ↑
Stride 8
59.5
66.7
53.2
68.2
Stride 16
61.2
68.6
54.7
68.9
Stride 16+32
64.8
74.7
56.4
70.0
Stride 32 (Ours)
65.3
76.5
56.0
70.3
Table 6: Ablation of feature levels for constructing image-based detection proposals on the DanceTrack validation set.
Retain lost tracks
HOTA ↑
DetA ↑
AssA ↑
IDF1 ↑
0 frames (Ours)
65.3
76.5
56.0
70.3
≤ 1 frame
61.7
78.3
48.8
63.1
≤ 3 frames
63.9
77.9
52.5
66.3
≤ 5 frames
63.7
77.5
52.5
66.4
Table 7: Ablation of retaining recently lost tracks in HQG track memory.
Tokens (shared set)
HOTA ↑
DetA ↑
AssA ↑
IDF1 ↑
0 (None)
63.3
74.1
54.2
68.0
4 (Learnable)
64.2
74.6
55.3
70.2
32 (Fixed random)
64.6
74.7
56.0
69.6
32 (Learnable, Ours)
65.3
76.5
56.0
70.3
64 (Learnable)
65.7
76.2
56.6
70.8
Table 8: Effect of memory type and shared-set token count in HQG on the DanceTrack validation set. All HQG layers use the same memory set. Fixed random memory uses frozen, randomly initialized embeddings.
Layers
HOTA ↑
DetA ↑
AssA ↑
IDF1 ↑
1
63.7
76.8
52.9
68.4
2
64.1
76.1
54.3
69.0
3 (Ours)
65.3
76.5
56.0
70.3
6
63.7
75.2
54.1
68.2
Table 9: Ablation of the number of HQG layers.
Auxiliary schedule
HOTA ↑
DetA ↑
AssA ↑
IDF1 ↑
None (0, 0)
64.2
74.8
55.2
69.3
Staged (8, 24)
64.6
75.3
55.7
69.0
Staged (14, 24) (Ours)
65.3
76.5
56.0
70.3
Staged (21, 24)
64.8
76.7
55.6
69.1
Staged (24, 24)
65.0
77.0
54.9
68.6
Staged (14, 21)
64.4
75.2
55.3
70.1
Table 10: Auxiliary supervision schedules on the DanceTrack validation set. Parenthesized epochs indicate when intermediate-layer auxiliary decoder losses are removed and when the entire auxiliary branch is disabled, respectively. None denotes training without auxiliary supervision.
Configuration
HOTA ↑
DetA ↑
AssA ↑
IDF1 ↑
w/o HQG + A
62.7
76.1
51.9
66.1
w/o HQG + B
60.0
75.7
47.7
63.0
w/o HQG + C
62.9
76.0
52.3
66.1
w/ HQG + A
65.1
76.3
55.6
69.7
w/ HQG + B
57.0
74.9
43.5
60.9
w/ HQG + C
65.3
76.5
56.0
70.3
Table 11: Effect of target-related detection-to-track interaction on DanceTrack. A: unrelated masking; B: related masking; C: normal self-attention. The C rows are reference results for the corresponding models, evaluated with different inference settings from A and B. All metrics are percentages. Signed changes denote absolute differences on the percentage scale.
Figure 6: Qualitative comparison of detection-query formation strategies on DanceTrack. Green boxes denote ground truth. Yellow, red, and blue boxes denote tracking outputs from three query-formation variants: image-based detection queries without HQG, learned queries, and HQG-generated queries (TAQ-MOTR), respectively. The status tables report matched IDs and tracking states: “Lost” denotes no matched result, “IDSW” an identity switch, and “track” correct tracking. In the illustrated frames, TAQ-MOTR preserves the queried identities through occlusion, crossing, and overlap; this comparison provides qualitative evidence of association behavior.
Figure 7: Query-focus visualization of detection queries on DanceTrack. The heatmaps aggregate cross-attention between detection queries and multi-scale image features in the first decoder layer; each panel is independently normalized, so heatmap colors indicate relative spatial focus rather than absolute attention magnitudes. Green boxes mark established tracking outputs and red boxes missing or unstable outputs; these colors do not indicate track-query retention. Panels (a)–(b) show video initialization, and (c)–(e) illustrate changes in query focus in a crowded scene.
Figure 8: IoU distributions of learned and HQG-initialized detection queries on DanceTrack. The horizontal axis shows each query box’s maximum IoU with currently untracked (left) or tracked (right) GT; the logarithmic vertical axis shows query counts.
Multi-Object Tracking (MOT) requires localizing multiple objects and maintaining their identities over time. Recent query-propagation trackers carry target-specific representations across frames and update them with current-frame image features, yet identity association remains difficult when visually similar targets overlap or cross. During such interactions, current-frame query updates may contain components directed toward nearby active tracks, which may affect identity association in later frames. To investigate this issue, we analyze track-query updates preceding identity switches (IDSWs), where the same target is assigned a different track ID. Our analysis shows that (i) propagated track-query representations change more abruptly before IDSWs, (ii) current-frame updates have larger positive projections along directions toward other active tracks, and (iii) bounding-box intersection over union (IoU) retrieves the counterpart track involved in a subsequent IDSW with high recall when that track remains active before the switch. Motivated by these findings, we propose Test-time Track-Query Calibration (TQC), a training-free and gradient-free method for query-propagation trackers. TQC calibrates the tracker output before propagation using a bounded correction estimated from update components directed toward nearby active tracks. It requires only track-query representations and bounding boxes available during inference, without retraining or architectural modification. Across nine DanceTrack and SportsMOT test settings, TQC improves HOTA and AssA in all settings and IDF1 in eight, with an average HOTA gain of +0.38 and limited inference overhead. Controlled variants on six validation settings further show that TQC's correction direction yields more consistent HOTA improvements than alternative directions with matched correction norms.
Recent Tracking-by-Query-Propagation (TBP) methods have advanced Multi-Object Tracking (MOT) by enabling end-to-end (E2E) pipelines with long-range temporal modeling. However, this reliance on query propagation introduces unexplored architectural vulnerabilities to adversarial attacks. We present FADE, a novel attack framework designed to exploit these specific vulnerabilities. FADE employs two attack strategies targeting core TBP mechanisms: (i) Temporal Query Flooding: Generates spurious temporally consistent track queries to exhaust the tracker's limited query budget, forcing it to terminate valid tracks. (ii) Temporal Memory Corruption: Directly attacks the query updater's memory by severing temporal links via state de-correlation and erasing the learned feature identity of matched tracks. Furthermore, we introduce a differentiable pipeline to optimize these attacks for physical-world realizability by leveraging simulations of advanced perception sensor spoofing. Experiments on MOT17 and MOT20 benchmarks demonstrate that FADE is highly effective against state-of-the-art TBP trackers, causing significant identity switches and track terminations.
This paper presents a comprehensive experimental evaluation and detailed analysis of state-of-the-art multi-object tracking algorithms, with an emphasis on quantifying the individual contributions of detection and association components to overall tracking performance. Unlike existing surveys that primarily offer theoretical categorizations or taxonomies of tracking methods, our work adopts a rigorous experimental perspective grounded in publicly available implementations, providing practical guidance for researchers and practitioners in method selection and system design. We introduce a unified pipeline diagram that consolidates the core components across the two main branches of visual multi-object tracking: tracking-by-detection and end-to-end deep learning paradigms, and systematically analyze the object detection, feature extraction, and data association modules. Through extensive empirical studies on standard benchmarks, including MOT16, MOT17, MOT20, SportsMOT, DanceTrack, and CrowdTrack datasets, we reveal critical insights: (1) detection quality dominates association strategy performance, with detector improvements yielding more than 10% gains compared to less than 5% from refined association strategies; (2) modern deep learning detectors paired with specialized re-identification models significantly outperform joint detection and embedding approaches; and (3) transformer-based end-to-end methods exhibit greater robustness to detection quality variations but at a substantial computational cost. Our findings from extensive experiments provide key insights into component-level effects in MOT, particularly the dominant influence of detection quality relative to association, while offering practical insights for designing and optimizing MOT systems under varying performance and robustness requirements. Code and experimental setups are available at github.com/linh-gist/VisualMOT.
Linh Van Ma, Juhua Hu, Wei Cheng +2
School of Electrical Engineering and Computer Science at GIST, Gwangju, Korea · School of Engineering and Technology, University of Washington Tacoma, United States