MOTIP2: Spatial Priors for End-to-End Multi-Object Tracking

Benoît Roussel; Damien Bouet; Liming Chen; Pierre Perrault

cs.CVOct 7, 2026

MOTIP2: Spatial Priors for End-to-End Multi-Object Tracking

Authors: Benoît Roussel, Damien Bouet, Liming Chen, Pierre Perrault

Organizations: École Centrale de Lyon LIRIS, UMR CNRS 5205 Lyon, France · IDEMIA Courbevoie, France

Abstract

End-to-end multi-object trackers have narrowed the gap with classical tracking-by-detection on association-difficult benchmarks. Yet they still make spatially implausible errors no classical tracker would, such as assigning one identity to objects on opposite sides of the frame. A model could learn to avoid them, but tracking annotations are scarce, so we encode spatial priors explicitly instead, while keeping inference fully end-to-end with no post-hoc association. We propose three spatial priors, at the data, loss, and representation stages. Spatial ID Switches bias trajectory permutations toward spatially overlapping objects, reducing the mismatch between training and inference confusions. Spatial ID Loss scales each identity's penalty by its box distance, so a distant switch costs more than a nearby one. Spatial Anchor gives each track token its frame position, an explicit spatial cue for attention. We instantiate the three priors in MOTIP2, a tracker adapted from MOTIP and built on the real-time DEIM detection transformer. Trained without extra data, its main model, MOTIP2-L, sets a new state of the art: 73.4 HOTA on DanceTrack, 76.0 on SportsMOT, and 71.1 IDF1 on PersonPath22. MOTIP2 is a family of models spanning the speed-accuracy trade-off: a lighter model, MOTIP2-S, matches the original MOTIP at over 3x the speed, and MOTIP2-X reaches 74.8 HOTA on DanceTrack.

Figures & tables

Panel of Figure 1 — Figure 1: Accuracy vs. latency. HOTA on DanceTrack-val vs. per-frame PyTorch FP32 latency (ms) on a single V100. At matched accuracy, MOTIP2-S is $3.2\times$ faster than MOTIP. Marker shape denotes input resolution: square 640 $\times$ 640, gray triangle 800 $\times$ 1333; MOTIP (black) and MOTIP2 (circles) run at 800 $\times$ 1440. † Estimated from RTX 4090 measurements [ Liao et al.(2026)Liao, Yang, Wu, Yu, Li, and Zhang ] scaled by MOTIP’s V100/4090 ratio.

Figure 2: Overview of MOTIP2. DEIM produces detection embeddings independently per frame. Three explicit spatial priors then inject geometric structure at distinct stages: (i) Spatial ID Switches perturb the training-time historical context by GIoU-biased identity permutations that mirror the near-track confusions seen at inference; (ii) Spatial ID Loss supervises the ID decoder with a geometric penalty; (iii) Spatial Anchor carries each detection’s positional encoding into the ID decoder, so every token holds an explicit cue to its image position.

Figure 3: Spatial ID Switches. A DanceTrack [ Sun et al.(2022)Sun, Cao, Jiang, Yuan, Bai, Kitani, and Luo ] frame with four annotated objects, of which only $2$ and $3$ overlap, and the identity permutation the augmentation draws for this frame. We perturb the pairwise GIoU affinity with i.i.d. Gaussian noise $\varepsilon_{ij}\sim\mathcal{N}(0,\,\sigma^{2})$ and let the Hungarian algorithm return the assignment maximizing it; cell $(i,j)$ means object $i$ takes identity $j$ , light where the identity is kept and dark where it is swapped. Since only spatially close pairs carry positive affinity, the perturbation swaps $2\!\leftrightarrow\!3$ while the well-separated objects keep their identity.

Tracking-by-detection
Method	HOTA	DetA	AssA	MOTA	IDF1
FairMOT [ Zhang et al.(2021b)Zhang, Wang, Wang, Zeng, and Liu ]	39.7	66.7	23.8	82.2	40.8
CenterTrack [ Zhou et al.(2020)Zhou, Koltun, and Krähenbühl ]	41.8	78.1	22.6	86.8	35.7
ByteTrack [ Zhang et al.(2022)Zhang, Sun, Jiang, Yu, Weng, Yuan, Luo, Liu, and Wang ]	47.7	71.0	32.1	89.6	53.9
OC-SORT [ Cao et al.(2023)Cao, Pang, Weng, Khirodkar, and Kitani ]	55.1	80.3	38.3	92.0	54.6
StrongSORT [ Du et al.(2023)Du, Zhao, Song, Zhao, Su, Gong, and Meng ]	55.6	80.7	38.6	91.1	55.2

Table 1: State-of-the-art comparison on the DanceTrack [ Sun et al.(2022)Sun, Cao, Jiang, Yuan, Bai, Kitani, and Luo ] test set. The best end-to-end result is in bold .

\dagger

denotes extra training data (MOTIP2: DanceTrack train+val). MOTIP2-L/X (w/o spatial priors) disable all three priors under the same training setup, keeping every other component, which isolates their contribution from the DEIM detector.

Tracking-by-detection
Method	HOTA	DetA	AssA	MOTA	IDF1
FairMOT [ Zhang et al.(2021b)Zhang, Wang, Wang, Zeng, and Liu ]	49.3	70.2	34.7	86.4	53.5
GTR [ Zhou et al.(2022)Zhou, Yin, Koltun, and Krähenbühl ]	54.5	64.8	45.9	67.9	55.8
QDTrack [ Pang et al.(2021)Pang, Qiu, Li, Chen, Li, Darrell, and Yu ]	60.4	77.5	47.2	90.1	62.3
CenterTrack [ Zhou et al.(2020)Zhou, Koltun, and Krähenbühl ]	62.7	82.1	48.0	90.8	60.0
ByteTrack [ Zhang et al.(2022)Zhang, Sun, Jiang, Yu, Weng, Yuan, Luo, Liu, and Wang ]	62.8	77.1	51.2	94.1	69.8

Table 2: State-of-the-art comparison on the SportsMOT [ Cui et al.(2023)Cui, Zeng, Zhao, Yang, Wu, and Wang ] test set. The best end-to-end result is in bold .

Method	IDF1	MOTA	IDSW $\downarrow$	FP $\downarrow$	FN $\downarrow$
Tracking-by-detection
CenterTrack [ Zhou et al.(2020)Zhou, Koltun, and Krähenbühl ]	46.4	59.3	10 319	24K	72K
FairMOT [ Zhang et al.(2021b)Zhang, Wang, Wang, Zeng, and Liu ]	61.1	61.8	5 095	15K	80K
ByteTrack [ Zhang et al.(2022)Zhang, Sun, Jiang, Yu, Weng, Yuan, Luo, Liu, and Wang ]	66.8	75.4	5 931	17K	40K
OC-SORT [ Cao et al.(2023)Cao, Pang, Weng, Khirodkar, and Kitani ]	68.2	74.0	4 931	17K	41K
Path Consistency [ Lu et al.(2024)Lu, Shuai, Chen, Xu, and Modolo ]	69.4	75.8	4 663	15K	38K

Table 3: State-of-the-art comparison on the PersonPath22 [ Shuai et al.(2022)Shuai, Bergamo, Büchler, Berneshawi, Boden, and Tighe ] test set. The best end-to-end result is in bold . Numbers for comparison methods are compiled from [ Shuai et al.(2022)Shuai, Bergamo, Büchler, Berneshawi, Boden, and Tighe ] and [ Lu et al.(2024)Lu, Shuai, Chen, Xu, and Modolo ] ; these works do not report HOTA-family metrics. For completeness, MOTIP2-L additionally achieves HOTA 62.8, DetA 66.1, and AssA 60.4.

	Accuracy			V100 PT FP32			T4 TRT FP16
Method	HOTA	DetA	AssA	Det.	ID Dec	Total	Det.	ID Dec	Total
MOTIP [ Gao et al.(2025)Gao, Qi, and Wang ]	64.0	75.4	54.7	72.7	14.2	86.9	–
MOTIP2-X	70.1	80.7	61.1	71.2	9.9	81.1	38.8	1.3	40.1
MOTIP2-L	68.0	77.3	60.0	45.0	9.8	54.8	21.9	1.3	23.2
MOTIP2-M	67.3	76.4	59.7	32.5	6.8	39.3	15.8	0.9	16.7
MOTIP2-S	64.5	74.0	56.5	21.8	5.3	27.1	9.4	0.7	10.1

Table 4: Per-component latency breakdown. HOTA on DanceTrack-val and per-component latency (ms). Det. = detector (Deformable DETR for MOTIP, DEIM for MOTIP2); PT = PyTorch, TRT = TensorRT. MOTIP has no standard TensorRT path. Input resolution:

800\times 1440

. ID Decoder profiled with

T{=}30

frames, 50 trajectories, and 50 detections.

	PersonPath22				DanceTrack
	HOTA	DetA	AssA	IDF1	HOTA	DetA	AssA	IDF1
Full model	61.6	63.8	60.3	70.4	64.5	74.0	56.5	68.8
uniform IDSW	56.1	58.6	54.4	64.0	63.7	73.0	55.8	68.1
w/o IDSW	59.5	64.2	55.8	66.8	60.1	73.3	49.7	63.0
w/o Spatial ID Loss	61.4	64.2	59.4	70.1	63.5	73.3	55.5	67.3
w/o Spatial Anchor	61.6	64.3	59.9	70.4	64.0	73.3	56.1	68.6

Table 5: Ablation of the spatial priors. MOTIP2-S, each variant trained and evaluated per benchmark (PersonPath22+CrowdHuman; DanceTrack train split). uniform IDSW downgrades our Spatial ID Switches to a non-spatial permutation; w/o IDSW removes the augmentation entirely. † With all priors off, the data-stage prior is removed on PersonPath22 but set to uniform on DanceTrack, where uniform is near-neutral rather than harmful (cf. rows above). The full model (bold row label) is the reference each removal is compared against.

Figure 4: Qualitative tracking comparison on PersonPath22 (uid_vid_00063). Rows (top to bottom): our GIoU-based sampling ( Ours ), no augmentation ( w/o IDSW ), and MOTIP’s uniform permutation ( uniform IDSW ). Columns are ordered in time, left to right. Two pedestrians cross and occlude; only ours recovers the correct identities afterward.

Method	$\alpha$	HOTA	DetA	AssA	MOTA	IDF1
Rel. PE	–	61.3	64.1	59.3	73.4	70.2
RoPE	1.0	60.4	63.7	58.0	72.7	68.7
	1.5	61.0	63.7	59.1	72.8	69.6
	2.0	61.6	63.8	60.3	72.9	70.4
	2.5	61.6	63.8	60.3	72.9	70.4

Table 6: Effect of positional encoding type and time rescaling factor

\alpha

. Results on PersonPath22 test. We compare RoPE with the learned relative positional encoding from MOTIP [ Gao et al.(2025)Gao, Qi, and Wang ] (Rel. PE). The best result for each metric is shown in bold .

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Tracking-by-detection
Method	HOTA	DetA	AssA	MOTA	IDF1
CenterTrack [ Zhou et al.(2020)Zhou, Koltun, and Krähenbühl ]	52.2	53.8	51.0	67.8	64.7
FairMOT [ Zhang et al.(2021b)Zhang, Wang, Wang, Zeng, and Liu ]	59.3	60.9	58.0	73.7	72.3
DeepSORT [ Wojke et al.(2017)Wojke, Bewley, and Paulus ]	61.2	63.1	59.7	78.0	74.5
SORT [ Bewley et al.(2016)Bewley, Ge, Ott, Ramos, and Upcroft ]	63.0	64.2	62.2	80.1	78.2
ByteTrack [ Zhang et al.(2022)Zhang, Sun, Jiang, Yu, Weng, Yuan, Luo, Liu, and Wang ]	63.1	64.5	62.0	80.3	77.3

Appendix

Table 7: State-of-the-art comparison on the MOT17 [ Milan et al.(2016)Milan, Leal-Taixé, Reid, Roth, and Schindler ] test set. The best end-to-end result is in bold . MOTIP2-L/X are trained on MOT17 + CrowdHuman, matching MOTIP’s recipe.

Panel of Figure 5 — Figure 5: ID-switch augmentation variants. PersonPath22 (test) HOTA over joint-training epochs (MOTIP2-S): GIoU-based sampling ( Ours ), no augmentation ( w/o IDSW ), and MOTIP’s uniform permutation ( uniform IDSW ). The uniform permutation diverges and ends below the no-augmentation baseline, while ours trains stably to the highest HOTA.

	Training			Inference
Sampling	Swap%	LR%	$\overline{\mathrm{GIoU}}$	HOTA	LR-sw
DanceTrack
None	–	–	–	60.1	120
Uniform	37%	85%	$-0.40$	63.7	260
Ours	10%	26%	$0.20$	64.5	134
PersonPath22

Appendix

Table 8: Spatial ID-switch augmentation behavior. Training , over one epoch: the realized swap rate (Swap%), the fraction of swapped pairs that are long-range (LR%: GIoU

\leq 0

, non-overlapping), and their mean overlap (

\overline{\mathrm{GIoU}}

). Inference : the number of long-range ID switches produced by each variant (LR-sw).

Figure 6: A long-distance identity switch our spatial priors prevent. DanceTrack dancetrack0090 , consecutive frames $710$ and $711$ . Top: with the spatial priors disabled, identity 2 jumps $\approx\!975$ px across the frame (yellow arrow), merging two distant dancers into a single track ( Sec. 1 ). Bottom: our full model keeps both identities stable.

$N_{\text{init}}$	HOTA	DetA	AssA	MOTA	IDF1	IDSW $\downarrow$	HOTA	DetA	AssA	MOTA	IDF1	IDSW $\downarrow$
	PersonPath22						DanceTrack
1	61.5	63.7	60.0	72.5	70.3	5 612	64.3	73.7	56.4	81.9	67.7	3 287
2	61.6	63.8	60.3	72.9	70.4	5 069	64.5	74.0	56.5	82.9	68.8	2 489
3	61.6	63.8	60.3	73.1	70.4	4 911	64.5	74.2	56.3	83.2	68.3	2 356

Appendix

Table 9: Effect of tracklet confirmation

N_{\text{init}}

. MOTIP2-S on PersonPath22 test and DanceTrack val.

N_{\text{init}}

is the number of consecutive frames a new tracklet must be matched before being confirmed. Best per metric and benchmark in bold .

End-to-end
Method	HOTA	Latency (ms)	Resolution
FastTrackTr-R18-Dec3 [ Liao et al.(2026)Liao, Yang, Wu, Yu, Li, and Zhang ]	47.4	17.1 †	640 $\times$ 640
FastTrackTr-Dec3 [ Liao et al.(2026)Liao, Yang, Wu, Yu, Li, and Zhang ]	51.2	27.2 †	640 $\times$ 640
PuTR [ Liu et al.(2025a)Liu, Li, Wang, and Xu ]	54.5	39.9 †	800 $\times$ 1333
FastTrackTr (640) [ Liao et al.(2026)Liao, Yang, Wu, Yu, Li, and Zhang ]	54.8	31.5 †	640 $\times$ 640
FastTrackTr (1333) [ Liao et al.(2026)Liao, Yang, Wu, Yu, Li, and Zhang ]	56.9	53.6 †	800 $\times$ 1333

Appendix

Table 10: Latency comparison with end-to-end baselines. HOTA on DanceTrack-val and per-frame latency (ms) in PyTorch on a single V100. † Estimated from RTX 4090 measurements [ Liao et al.(2026)Liao, Yang, Wu, Yu, Li, and Zhang ] scaled by MOTIP’s V100/4090 latency ratio.

Explore similar work

Clip-level Uncertainty and Temporal-aware Active Learning for End-to-End Multi-Object Tracking
May 11, 2026Riku Inoue, Shogo Sato, Kazuhiko Murasaki +3Multi-Object Tracking Pool-Based Active Learning
NTT, Inc., Kanagawa, Japan
VLA-ReID: Video-Level Association for Re-Identification in Multi-Object Tracking with Highly Similar Objects
Jul 19, 2026Yanrong Qin, Xiaoyan Cao, Yao YaoMulti-Object Tracking Re-Identification
Glasgow College, University of Electronic Science and Technology of China, Chengdu, China · Key Laboratory for Urban Habitat Environmental Science and Technology, School of Environment and Energy, Peking University Shenzhen Graduate School, Shenzhen, China · School of Management Science and Real Estate, Chongqing University, Chongqing, China
SpikingMOT: A Spike-Driven Multi-Object Tracker
Jul 22, 2026Yiding Sun, Xiangyang Yang, Dongxu Zhang +7Multi-Object Tracking Time-To-First-Spike
1Xi’an Jiaotong University · 2Peking University · 3Guilin University of Technology

May 11, 2026, cs.CVJ/K move · Enter open · S save

Clip-level Uncertainty and Temporal-aware Active Learning for End-to-End Multi-Object Tracking

Riku Inoue, Shogo Sato, Kazuhiko Murasaki, Tomoyasu Shimada+2

NTT, Inc., Kanagawa, Japan

Multi-Object Tracking Pool-Based Active Learning Active Learning Large Inter-Frame Displacements Uncertainty

Open paper AskPDF arXiv

Multi-Object Tracking (MOT) in dynamic environments relies on robust temporal reasoning to maintain consistent object identities over time. Transformer-based end-to-end MOT models achieve strong performance by explicitly modeling temporal dependencies, yet training them requires extensive bounding-box and identity annotations. Given the high labeling cost and strong redundancy in videos, Active Learning (AL) is an effective approach to improve annotation efficiency. However, existing AL methods for MOT primarily operate at the frame level, which is structurally misaligned with modern end-to-end trackers whose inference and training rely on multi-frame clips. To bridge this gap, we formulate clip-level active learning and propose Clip-level Uncertainty and Temporal-aware Active Learning (CUTAL). In contrast to frame-based approaches, CUTAL scores each clip using uncertainty metrics derived from multi-frame predictions to capture inter-frame correspondence ambiguities, while enforcing temporal diversity to select an informative and non-redundant subset. Experiments show that CUTAL achieves stronger overall performance than baselines at the same label budgets across MeMOTR and SambaMOTR. Notably, CUTAL achieves performance comparable to full supervision for MeMOTR on both datasets using only 50% of the labeled training data.

Abstract

Explore similar work

Clip-level Uncertainty and Temporal-aware Active Learning for End-to-End Multi-Object Tracking

VLA-ReID: Video-Level Association for Re-Identification in Multi-Object Tracking with Highly Similar Objects

SpikingMOT: A Spike-Driven Multi-Object Tracker