TAQ-MOTR: Tracking-Aware Detection Query Formation for End-to-End Multiple Object Tracking
Authors: Hanyang Liu, Hong Zhang, Ding Yuan, Yang Han, Yifan Yang
Organizations: School of Astronautics, Beihang University, 37 Xueyuan Road, Haidian District, Beijing 100191, China · Key Laboratory of Spacecraft Design Optimization and Dynamic Simulation Technology, Ministry of Education · State Key Laboratory of High-Efficiency Reusable Aerospace Transportation Technology · School of Software, Beihang University, 37 Xueyuan Road, Haidian District, Beijing 100191, China
Tracking-by-propagation (TBP) methods perform end-to-end multi-object tracking by propagating track queries across frames, jointly handling detection and identity association without heuristic post-processing while naturally modeling temporal information. However, as detection and association are tightly coupled, detection queries in TBP are expected to discover currently untracked objects and provide complementary current-frame evidence for challenging tracks whose queries and identities are retained. Existing methods largely inherit detection-query formation from single-frame detection and construct such queries before the decoder with little awareness of trajectory states. Consequently, currently untracked objects may be missed, while challenging tracked objects with reduced visibility under occlusion may lack complementary evidence, causing association failures or duplicate responses. To address these issues, we propose TAQ-MOTR, a tracking-aware detection query formation framework that employs a pre-decoder Hybrid Query Generator (HQG) to fuse current-frame image features, track memory, and learnable memory. By conditioning detection queries on trajectory states, TAQ-MOTR adapts them to \rev{the discovery of currently untracked objects and tracked-object complementation We further introduce a training-only auxiliary detection branch whose one-to-one label assignment is decoupled from that of the main tracking branch, providing additional detection supervision to strengthen the shared HQG and decoder for the discovery of currently untracked objects. TAQ-MOTR improves performance on DanceTrack, SportsMOT, and BFT, achieving 72.0 HOTA, 63.1 AssA, and 76.6 IDF1 on the DanceTrack test set. Code is available at https://github.com/Magicmiya/TAQ-MOTR
Figures & tables
Figure 1: Detection query formation before decoding. (a) MOTR: learnable queries; (b) MOTRv2: detector-guided queries; (c) TAQ-MOTR: tracking-aware queries. The comparison highlights current-frame information, trajectory-state priors, and learnable embeddings.
Figure 2: Overall architecture of TAQ-MOTR. TAQ-MOTR forms tracking-aware detection queries before decoding. In the main tracking branch, HQG uses image-based detection proposals as queries and reads track memory and learnable memory as keys/values. The generated detection queries are concatenated with updated track queries and jointly decoded. During training, an auxiliary detection branch shares the detection proposals and HQG layers but removes track memory, producing a separate detection-query group supervised by all current-frame objects. Attention masks isolate this auxiliary group from the main query groups. The right panel shows an HQG layer.
Figure 3: Prior carriers used in HQG. Learnable memory provides learnable detection priors that are independent of specific trajectories. Track memory encodes existing trajectory states from propagated track queries, including content, position, and disappearance-time information. Detection proposals provide current-frame object evidence and coarse spatial priors.
Figure 4: Positive label assignments during training on DanceTrack. All ratios use the total GT count per epoch, including clip-initial frames. The auxiliary branch is disabled on each clip’s first frame, so Auxiliary Detection/GT remains below 100% while enabled. Dashed lines mark clip-length changes; shading marks branch shutdown from epoch 24. The auxiliary-only ratio is then zero and omitted. Epoch numbering starts at 0.
Method
HOTA ↑
DetA ↑
AssA ↑
MOTA ↑
IDF1 ↑
Tracking-by-Detection-like
Hybrid-SORT-ReID † [ 34 ] (AAAI 2024)
65.7
–
–
91.8
67.4
DiffMOT † [ 22 ] (CVPR 2024)
62.3
82.5
47.2
92.8
63.0
PlugTrack † [ 14 ] (AAAI 2026)
63.3
82.5
48.4
92.4
64.1
MOTIP [ 7 ] (CVPR 2025)
69.6
80.4
60.4
90.6
74.7
OA-SORT † [ 15 ] (CVPR 2026)
63.1
–
48.5
91.7
64.2
Table 1: Comparison with state-of-the-art methods on the DanceTrack test set. Best and second-best results among Tracking-by-Propagation (TBP) methods are marked in bold and underlined , respectively. Tracking-by-Detection-like methods are listed for reference only. ‡ denotes additional data for tracking-model training, and † denotes additional models. Train+val results are excluded from ranking.
Method
HOTA ↑
DetA ↑
AssA ↑
MOTA ↑
IDF1 ↑
Tracking-by-Detection-like
DiffMOT † [ 22 ] (CVPR 2024)
72.1
86.0
60.5
94.5
72.8
MOTIP [ 7 ] (CVPR 2025)
72.6
83.5
63.2
92.4
77.1
ETTrack † [ 9 ] (AI 2025)
72.2
86.9
60.1
94.9
72.5
FDTA † [ 28 ] (CVPR 2026)
74.2
84.1
65.5
93.0
78.5
Tracking-by-Propagation (TBP)
Table 2: Comparison with state-of-the-art methods on the SportsMOT test set. Best and second-best results among Tracking-by-Propagation (TBP) methods are marked in bold and underlined , respectively. Tracking-by-Detection-like methods are listed for reference only. ‡ denotes additional data for tracking-model training, and † denotes additional models. Our mixed variant uses additional CrowdHuman data and is excluded from ranking.
Method
HOTA ↑
DetA ↑
AssA ↑
MOTA ↑
IDF1 ↑
Tracking-by-Detection-like
SORT † [ 2 ] (ICIP 2016)
61.2
60.6
62.3
75.5
77.2
ByteTrack † [ 38 ] (ECCV 2022)
62.5
61.2
64.1
77.2
82.3
OC-SORT † [ 3 ] (CVPR 2023)
66.8
65.4
68.7
77.1
79.3
MOTIP [ 7 ] (CVPR 2025)
70.5
69.6
71.8
77.1
82.1
FDTA † [ 28 ] (CVPR 2026)
72.2
70.1
74.5
78.2
84.2
Table 3: Comparison with state-of-the-art methods on the BFT test set. Best and second-best results among Tracking-by-Propagation (TBP) methods are marked in bold and underlined , respectively. Tracking-by-Detection-like methods are listed for reference only. ‡ denotes additional data for tracking-model training, and † denotes additional models.
Method
Parameters (M) ↓
FPS ↑
MOTR [ 36 ]
43.9
21.0
MOTRv2 [ 39 ] (with YOLOX)
140.6
16.1
CO-MOT [ 33 ]
41.7
25.1
MeMOTR [ 8 ]
42.9
22.5
TAQ-MOTR
48.9
41.1
Table 4: Efficiency comparison with open-source TBP methods on DanceTrack. All methods follow the same measurement protocol. MOTRv2 includes YOLOX in both its parameter count and inference time.
Figure 5: Module-level efficiency analysis of TAQ-MOTR. Rounded parameter counts and reported per-frame runtimes are shown for the backbone, hybrid encoder, decoder, HQG, and query updater.
DP
TM
LM
HOTA ↑
DetA ↑
AssA ↑
IDF1 ↑
DetRe ↑
IDSW ↓
Baselines
Original
63.9
74.6
55.0
67.1
–
1237
Unified basic model
62.9
76.0
52.3
66.1
85.4
1530
HQG variants
✓
–
–
58.1
71.9
47.1
61.9
–
–
✓
✓
–
63.3
74.1
54.2
68.0
83.7
996
Table 5: Ablation of HQG priors on the full DanceTrack validation set. DP, TM, and LM denote detection proposals, track memory, and learnable memory, respectively. Original reports the published MeMOTR results [ 8 ] . Unified basic model adopts the same backbone, encoder, and pretrained weights as TAQ-MOTR while retaining the rest of MeMOTR’s tracking design. The best result for each metric is shown in bold, and unreported values are indicated by a dash.
Level
HOTA ↑
DetA ↑
AssA ↑
IDF1 ↑
Stride 8
59.5
66.7
53.2
68.2
Stride 16
61.2
68.6
54.7
68.9
Stride 16+32
64.8
74.7
56.4
70.0
Stride 32 (Ours)
65.3
76.5
56.0
70.3
Table 6: Ablation of feature levels for constructing image-based detection proposals on the DanceTrack validation set.
Retain lost tracks
HOTA ↑
DetA ↑
AssA ↑
IDF1 ↑
0 frames (Ours)
65.3
76.5
56.0
70.3
≤ 1 frame
61.7
78.3
48.8
63.1
≤ 3 frames
63.9
77.9
52.5
66.3
≤ 5 frames
63.7
77.5
52.5
66.4
Table 7: Ablation of retaining recently lost tracks in HQG track memory.
Tokens (shared set)
HOTA ↑
DetA ↑
AssA ↑
IDF1 ↑
0 (None)
63.3
74.1
54.2
68.0
4 (Learnable)
64.2
74.6
55.3
70.2
32 (Fixed random)
64.6
74.7
56.0
69.6
32 (Learnable, Ours)
65.3
76.5
56.0
70.3
64 (Learnable)
65.7
76.2
56.6
70.8
Table 8: Effect of memory type and shared-set token count in HQG on the DanceTrack validation set. All HQG layers use the same memory set. Fixed random memory uses frozen, randomly initialized embeddings.
Layers
HOTA ↑
DetA ↑
AssA ↑
IDF1 ↑
1
63.7
76.8
52.9
68.4
2
64.1
76.1
54.3
69.0
3 (Ours)
65.3
76.5
56.0
70.3
6
63.7
75.2
54.1
68.2
Table 9: Ablation of the number of HQG layers.
Auxiliary schedule
HOTA ↑
DetA ↑
AssA ↑
IDF1 ↑
None (0, 0)
64.2
74.8
55.2
69.3
Staged (8, 24)
64.6
75.3
55.7
69.0
Staged (14, 24) (Ours)
65.3
76.5
56.0
70.3
Staged (21, 24)
64.8
76.7
55.6
69.1
Staged (24, 24)
65.0
77.0
54.9
68.6
Staged (14, 21)
64.4
75.2
55.3
70.1
Table 10: Auxiliary supervision schedules on the DanceTrack validation set. Parenthesized epochs indicate when intermediate-layer auxiliary decoder losses are removed and when the entire auxiliary branch is disabled, respectively. None denotes training without auxiliary supervision.
Configuration
HOTA ↑
DetA ↑
AssA ↑
IDF1 ↑
w/o HQG + A
62.7
76.1
51.9
66.1
w/o HQG + B
60.0
75.7
47.7
63.0
w/o HQG + C
62.9
76.0
52.3
66.1
w/ HQG + A
65.1
76.3
55.6
69.7
w/ HQG + B
57.0
74.9
43.5
60.9
w/ HQG + C
65.3
76.5
56.0
70.3
Table 11: Effect of target-related detection-to-track interaction on DanceTrack. A: unrelated masking; B: related masking; C: normal self-attention. The C rows are reference results for the corresponding models, evaluated with different inference settings from A and B. All metrics are percentages. Signed changes denote absolute differences on the percentage scale.
Figure 6: Qualitative comparison of detection-query formation strategies on DanceTrack. Green boxes denote ground truth. Yellow, red, and blue boxes denote tracking outputs from three query-formation variants: image-based detection queries without HQG, learned queries, and HQG-generated queries (TAQ-MOTR), respectively. The status tables report matched IDs and tracking states: “Lost” denotes no matched result, “IDSW” an identity switch, and “track” correct tracking. In the illustrated frames, TAQ-MOTR preserves the queried identities through occlusion, crossing, and overlap; this comparison provides qualitative evidence of association behavior.
Figure 7: Query-focus visualization of detection queries on DanceTrack. The heatmaps aggregate cross-attention between detection queries and multi-scale image features in the first decoder layer; each panel is independently normalized, so heatmap colors indicate relative spatial focus rather than absolute attention magnitudes. Green boxes mark established tracking outputs and red boxes missing or unstable outputs; these colors do not indicate track-query retention. Panels (a)–(b) show video initialization, and (c)–(e) illustrate changes in query focus in a crowded scene.
Figure 8: IoU distributions of learned and HQG-initialized detection queries on DanceTrack. The horizontal axis shows each query box’s maximum IoU with currently untracked (left) or tracked (right) GT; the logarithmic vertical axis shows query counts.
School of Electrical Engineering and Computer Science at GIST, Gwangju, Korea · School of Engineering and Technology, University of Washington Tacoma, United States