Beyond the Survey: A Systematic Empirical Study of Detection and Association in Visual MOT

Linh Van Ma; Juhua Hu; Wei Cheng; Unse Fatima; Moongu Jeon

cs.CVSep 14, 2026

Beyond the Survey: A Systematic Empirical Study of Detection and Association in Visual MOT

Authors: Linh Van Ma, Juhua Hu, Wei Cheng, Unse Fatima, Moongu Jeon

Organizations: School of Electrical Engineering and Computer Science at GIST, Gwangju, Korea · School of Engineering and Technology, University of Washington Tacoma, United States

Abstract

This paper presents a comprehensive experimental evaluation and detailed analysis of state-of-the-art multi-object tracking algorithms, with an emphasis on quantifying the individual contributions of detection and association components to overall tracking performance. Unlike existing surveys that primarily offer theoretical categorizations or taxonomies of tracking methods, our work adopts a rigorous experimental perspective grounded in publicly available implementations, providing practical guidance for researchers and practitioners in method selection and system design. We introduce a unified pipeline diagram that consolidates the core components across the two main branches of visual multi-object tracking: tracking-by-detection and end-to-end deep learning paradigms, and systematically analyze the object detection, feature extraction, and data association modules. Through extensive empirical studies on standard benchmarks, including MOT16, MOT17, MOT20, SportsMOT, DanceTrack, and CrowdTrack datasets, we reveal critical insights: (1) detection quality dominates association strategy performance, with detector improvements yielding more than 10% gains compared to less than 5% from refined association strategies; (2) modern deep learning detectors paired with specialized re-identification models significantly outperform joint detection and embedding approaches; and (3) transformer-based end-to-end methods exhibit greater robustness to detection quality variations but at a substantial computational cost. Our findings from extensive experiments provide key insights into component-level effects in MOT, particularly the dominant influence of detection quality relative to association, while offering practical insights for designing and optimizing MOT systems under varying performance and robustness requirements. Code and experimental setups are available at github.com/linh-gist/VisualMOT.

Figures & tables

Fig. 1: Typical pipeline of visual MOT algorithms. In this structure, objects are first localized within a frame and subsequently linked to those identified in other frames. While this framework is conventionally categorized as tracking-by-detection, recent end-to-end approaches can also be interpreted through these functional operations.

Focus	Year	Ref.	Contributions	Limitation
Generic texttt	2021	[ 10 ]	Comprehensive review of MOT formulation and methods.	Rely on qualitative analysis or aggregated published results, without controlled matched-condition experiments isolating individual components.
	2025	[ 8 ]	Groups TBD methods into five categories, compares E2E versus TBD, and assesses tracker generality across domains.
	2025	[ 12 ]	Classifies TBD and E2E methods; provides an open-source evaluation and visualization toolkit.
Data Assoc.	2020	[ 11 ]	Unifies probabilistic, end-to-end, and affinity-learning association under the assignment formulation.	Focus exclusively on the association step; the detector is treated as a fixed input, so its dominant effect on tracking is not examined.
Data Assoc.	2022	[ 7 ]	Reviews similarity functions and filters for traffic MOT association, comparing stability, accuracy, robustness, and speed.
Component text	2020	[ 13 ]	Mostly review detection algorithms and aggregate results from published papers to perform a numerical comparison.	Single-component scope; no controlled cross-component comparison of detection and association.

TABLE I: Comparison between existing MOT surveys and our empirical study. Existing works rely on qualitative summaries or single-component analysis, whereas our work provides a controlled, cross-component empirical benchmark.

Fig. 2: Overview of end-to-end approaches for visual multi-object tracking. While object detection has been extensively studied, most deep learning-based MOT methods mainly concentrate on improving data association, enabling both modules to be trained in an end-to-end manner.

Fig. 3: The tracking algorithms used in our experiments span from 2016 to 2025 and are categorized based on their data association strategies. Methods using analytical data association, typically solved via linear assignment, have long dominated MOT research and account for most published approaches. In contrast, learned feature-based association methods remain relatively limited. Recently, inspired by DETR architectures, E2E trackers have emerged as a new paradigm that jointly learns detection and association. Reference numbers appear in square brackets, and publication years are shown in parentheses.

Fig. 4: The detection performance of seven multi-object detectors, measured by MODA and F1 score with respect to the confidence threshold, increases from 0.2, peaks at around 0.5-0.6 for most detectors and at 0.7 for the YOLOX detector, and then gradually decreases as the threshold continues to increase.

Fig. 5: Effect of detection confidence threshold on tracking performance for different trackers in DanceTrack dataset [ 45 ] . Careful selection of this detection confidence threshold can improve tracking performance by more than 5% in terms of HOTA.

Fig. 6: Performance evaluation of tracking-by-detection frameworks under synthetic detection noise on the CrowdTrack test set. Evaluation metrics (HOTA, MOTA, IDF1, and TEM) are plotted against clutter rates (

\lambda_{c}\in\{0.5,2,5,10\}

), where

\lambda_{c}=0

denotes the clean, unperturbed original ground-truth baseline (the original ground truth without any modifications). To ensure a controlled evaluation, all frameworks share identical input detections under a uniform perturbation protocol (

P_{D}=0.8

via a Bernoulli process). For appearance-based association, appearance embeddings are consistently extracted using an SBS50 ReID model pre-trained on the CrowdTrack training sequences.

Fig. 7: Qualitative re-identification similarity of different detectors/networks on MOT16-04 sequence (top two rows) of MOT16 dataset, and MOT20-05 (the bottom row) at frame index 100 with detection confidence score higher than 0.3. The lighter/whiter the similarity visualization is, the better the discrimination of networks. Diagonal values represent the comparison of the identity feature.

Fig. 8: Quantitative re-identification similarity of different detectors/networks on MOT16 and MOT20 validation sequences. Each bar represents its mean value and standard deviation across frames in the same sequence. Lower mean values illustrate high network performance in distinguishing between detected objects. The experiment was done with detections that had a confidence score higher than 0.3.

Fig. 9: We evaluate three tracking algorithms StrongSORT [ 32 ] (first row), FairMOT (second row), and MOTDT (third row) using three MOT detectors that output both bounding boxes and appearance features, TraDeS [ 61 ] , JDE [ 59 ] , and FairMOT [ 29 ] , by replacing their original re-identification features with those extracted from a stronger SBS50 network while keeping all other tracking components unchanged.

Fig. 10: Tracking performance comparison (HOTA-sorted) of different trackers with different input detectors on the MOT17 validation dataset. For each detector, the average HOTA score is computed across all trackers using that detector, and detectors are arranged left to right, top to bottom in increasing order of this average. Within each detector’s panel, trackers are arranged left to right in increasing order of their individual HOTA scores. In both cases, the best-performing result is toward the right or bottom. SORT, AdaptByteTrack, and OCSORT do not use re-identification feature. The rest of the trackers are using bounding boxes and re-id feature as input. YOLOv11 detection was obtained with YOLOv11-X pretrained weight, and re-identification feature is extracted using SBS50 following DeepOCSORT.

Fig. 11: Tracking performance comparison (HOTA-Sorted) of different trackers and different input detectors in MOT20 validation dataset. The best-performing result is toward the right. SORT, AdaptByteTrack, and OCSORT do not use the re-identification feature. The rest of the trackers are using a bounding box and re-id feature as input. YOLOv11 detection was obtained with YOLOv11-X pretrained weights on the COCO dataset, and re-identification feature is extracted using SBS50 following DeepOCSORT.

Fig. 12: Tracking performance comparison across different trackers and input detectors on the MOT17 validation set. Results are presented as heatmaps, where darker colors indicate lower performance. The first row shows results using the original detections from our selected detectors, while the second row presents results after post-processing the detections to reduce localization errors, along with an oracle detection setting that uses MOT17 ground-truth bounding boxes paired with SBS50 features. In each subplot, rows and columns are sorted by average score in descending order (from top to bottom and left to right).

Component	Transformation	Explanation
$x_{c}$ ; $\dot{x}_{c}$	$a_{11}x_{c}+a_{12}y_{c}+(a_{13})$	rotate & scale (plus translate)
$y_{c}$ ; $\dot{y}_{c}$	$a_{21}x_{c}+a_{22}y_{c}+(a_{23})$	rotate & scale (plus translate)
$t$ ; $\dot{t}$	$a_{11}t+a_{12}l+(a_{13})$	rotate & scale (plus translate)
$l$ ; $\dot{l}$	$a_{21}t+a_{22}l+(a_{23})$	rotate & scale (plus translate)
$s$ ; $\dot{s}$	$s\sigma^{2}$	area scales by $\det(M)=\sigma^{2}$
$a$ ; $\dot{a}$	$a$	aspect ratio invariant under isotropic scaling

TABLE II: Transformation rules for each object-state component under camera motion compensation. Velocity components (such as

\dot{x_{c}},\dot{w}

) follow the same rule as their parent term (the underlying term from which a velocity is derived).

Form	Expression of $\bm{\mathbf{P}}_{k\|k}$
Standard	$(\bm{\mathbf{I}}-\bm{\mathbf{K}}_{k}\bm{\mathbf{H}}_{k}){{\bm{\mathbf{P}}}}^{\prime}_{k\|k-1}$
Symmetric	${{\bm{\mathbf{P}}}}^{\prime}_{k\|k-1}-\bm{\mathbf{K}}_{k}\bm{\mathbf{S}}_{k}\bm{\mathbf{K}}_{k}^{T}$
Joseph form	$(\bm{\mathbf{I}}-\bm{\mathbf{K}}_{k}\bm{\mathbf{H}}_{k}){{\bm{\mathbf{P}}}}^{\prime}_{k\|k-1}(\bm{\mathbf{I}}-\bm{\mathbf{K}}_{k}\bm{\mathbf{H}}_{k})^{T}+\bm{\mathbf{K}}_{k}\bm{\mathbf{R}}_{k}\bm{\mathbf{K}}_{k}^{T}$

TABLE III: There are three common forms and expressions used to implement the covariance state update in a Kalman filter. The standard form is simple but not symmetric. The symmetric form is numerically stable and symmetric, as implemented by ByteTrack [ 24 ] . The Joseph form is also numerically stable and symmetric, and is implemented by OCSORT [ 25 , 30 ] .

Sequence	CMC	IDF1 ( $\uparrow$ )	MOTA ( $\uparrow$ )	HOTA ( $\uparrow$ )	IDs ( $\downarrow$ )	FPS ( $\uparrow$ )
MOT16-02	$\times$	60.3	76.1	54.3	187	2918.7
MOT16-02	✓	58.4	76.4	52.9	179	17.5
MOT16-04	$\times$	95.6	97.9	88.8	10	2283.3
MOT16-04	✓	95.6	97.9	88.8	10	17.3
MOT16-09	$\times$	74.5	86.5	64.7	36	8804.5
MOT16-09	✓	74.5	86.5	64.7	36	17.3

TABLE IV: Performance comparison of AdaptByteTrack [ 75 ] with (✓) and without (

\times

) Camera Motion Compensation (CMC) [ 81 ] on MOT16 validation set using YOLOX detections. Best per-sequence results are in bold .

Seq.	Tracker	CMC	HOTA ( $\uparrow$ )	AssA ( $\uparrow$ )	IDF1 ( $\uparrow$ )	MOTA ( $\uparrow$ )	FPS ( $\uparrow$ )
track0023	ByteTrack	$\times$	41.0	23.7	41.4	72.0	517.5
	ByteTrack	$\checkmark$	64.1	51.9	69.3	89.7	33.1
	VisualRFS	$\times$	45.6	34.0	48.6	58.6	5.1
	VisualRFS	$\checkmark$	61.6	50.2	65.6	79.9	5.9
	OCSORT	$\times$	50.0	25.6	42.0	80.5	223.3
	OCSORT	$\checkmark$	74.7	56.2	67.8	97.1	27.9

TABLE V: Performance comparison of tracking methods with (✓) and without (

\times

) Camera Motion Compensation (CMC). Best results for each tracker per sequence are in bold . Evaluations utilize ground-truth detections from the CrowdTrack test set, with an SBS50 model trained on the CrowdTrack training set to extract re-identification features. For compactness, only sequences featuring significant camera motion are listed. Best per-sequence results are highlighted in bold .

Fig. 13: Effect of the matching threshold across five selected tracking algorithms on the DanceTrack validation dataset. The horizontal axis indicates the threshold used for each method’s specific cost matrix, while the vertical axis shows performance metrics in percentages. Fine-tuning this threshold is critical, as a well-calibrated value yields substantial performance gains.

Fig. 14: Effect of additional rounds of data association on tracking performance of MOT17 dataset. Improvements are generally small (below 5% in HOTA, IDF1, and MOTA), except for OCSORT with TraDeS and YOLOv11 detectors, where larger gains occur due to the weakness of its first association round.

Fig. 15: Impact of supplemental data association rounds on DanceTrack performance using the YOLOX detector. Observed improvements across HOTA, IDF1, and MOTA are marginal, consistently remaining below 5%.

Det.	Tracker	Setting	HOTA $\uparrow$	IDF1 $\uparrow$	MOTA $\uparrow$	TEM $\uparrow$
YOLOX	ByteTrack	One DA	73.1 (5.9)	81.7 (7.1)	83.5 (12.3)	0.7 (0.2)
	ByteTrack	Origin	75.9 (0.4)	84.5 (0.7)	90.0 (0.7)	0.9 (0.0)
	DeepOCSORT	One DA	66.8 (1.2)	71.8 (2.2)	84.0 (3.2)	0.7 (0.0)
	DeepOCSORT	Origin	66.8 (1.3)	71.6 (2.4)	85.7 (0.8)	0.8 (0.0)
	FairMOT	One DA	74.8 (1.1)	82.5 (1.4)	86.5 (3.0)	0.8 (0.0)
	FairMOT	Origin	76.0 (0.6)	84.2 (1.1)	88.6 (1.0)	0.8 (0.0)

TABLE VI: Performance comparison across various detectors, trackers, and data association settings on the MOT17 dataset over a confidence score sweep (0.2-0.8). One DA denotes a single round of linear DA, whereas Origin refers to the original tracker implementation utilizing two DA rounds. Values represent the mean tracking score followed by 1 standard deviation across the confidence sweep (in parentheses). Upward arrows

\uparrow

indicate that higher values are better.

Fig. 16: Experiments of SUSHI [ 33 ] and LG-MOT [ 35 ] with different detectors on MOT17 dataset. We use the default pretrained weights of YOLOv11 on the COCO dataset and do not retrain it on MOT17. This highlights the importance of detection accuracy versus tracking performance. The confidence score is set to 0.5 and fixed when changing detectors.

Fig. 17: Performance of end-to-end tracking methods across confidence thresholds (0.1–0.9) is evaluated using HOTA, MOTA, and IDF1 on the validation sets of the DanceTrack and SportsMOT datasets. Across all subplots, the vertical axis represents percentage scores, while the horizontal axis denotes the confidence threshold. Transformer-based methods demonstrate greater robustness to detection threshold variations compared to traditional tracking-by-detection approaches. Specifically, MOTR and CO-MOT remain remarkably stable as the confidence score changes; this suggests a favorable architecture, as these systems are less susceptible to fluctuations in detection quality.

D.	Split	Method	Detector	HOTA $\uparrow$	IDF1 $\uparrow$	MOTA $\uparrow$
SportsMOT	Test	SambaMOTR	DAB-D-DETR	72.1	74.8	91.9
	Test	SambaMOTR	D-DETR	70.5	73.3	90.4
	Val	MOTIP	DAB-D-DETR	78.8	83.3	94.4
		MOTIP	D-DETR	76.5	82.2	94.1
		MeMOTR	DAB-D-DETR	76.4	79.4	94.3
		MeMOTR	D-DETR	74.4	77.0	93.4

TABLE VII: Comparison of tracking performance using different trackers and detectors on the SportsMOT and DanceTrack datasets. The two detectors listed in the table, DAB-Deformable-DETR and Deformable-DETR, both use ResNet-50 as the backbone. DAB-Deformable-DETR, trained with different loss coefficients, achieves a box AP of 48.7, whereas Deformable-DETR with iterative bounding box refinement obtains a box AP of 46.2 on the COCO dataset.

Dataset	Method	Detector	HOTA $\uparrow$	IDF1 $\uparrow$	MOTA $\uparrow$	FPS $\uparrow$
DanceTrack	CO-MOT	D-DETR	64.6 (0.9)	66.9 (0.6)	84.7 (2.9)	7.1 (0.1)
	MOTR	DETR	51.3 (0.4)	49.2 (0.3)	74.7 (1.7)	6.2 (0.0)
	MOTIP	DAB-D-DETR	61.7 (3.2)	65.3 (3.7)	81.2 (2.8)	5.1 (0.1)
	SambaMOTR	DAB-D-DETR	53.8 (16.5)	56.3 (19.1)	71.6 (23.0)	3.2 (1.5)
	MOTIP	D-DETR	60.3 (3.3)	64.4 (4.1)	82.5 (3.4)	2.7 (0.1)
SportsMOT	MOTIP	D-DETR	76.0 (1.5)	81.4 (1.6)	93.7 (1.0)	6.1 (0.0)

TABLE VIII: Inference speed and tracking performance comparison across varying confidence thresholds. We compare end-to-end (E2E) tracking methods on the DanceTrack and SportsMOT datasets. Values report the mean and one standard deviation (in parentheses) averaged across confidence thresholds from 0.1 to 0.9. Within each dataset, methods are sorted in descending order of FPS. Bold numbers indicate the best performance for each metric.

Group	Method	Detector	Det $\uparrow$	ReID $\uparrow$	Track $\uparrow$	Total $\uparrow$	HOTA $\uparrow$
E2E	CO-MOT	D-DETR	8.06	-	60.32	7.11	65.42
	MOTIP	D-DETR	6.86	-	4.40	2.68	64.10
	MeMOTR	DAB-D-DETR	9.11	-	6.12	3.66	63.93
	SambaMOTR	DAB-D-DETR	9.81	-	4.84	3.24	62.65
TBD	ByteTrack	YOLOX $\|$ SBS50	10.91	35.31	92.72	7.65	46.50
	HybridSORT		10.91	35.31	60.87	7.35	59.91

TABLE IX: Performance comparison between tracking-by-detection and end-to-end multi-object tracking methods on the DanceTrack validation set. Speed is reported in frames per second (FPS

\uparrow

) for Det

\uparrow

, ReID

\uparrow

, Track

\uparrow

and Total

\uparrow

execution time for the whole pipeline. Tracking accuracy is measured by Higher Order Tracking Accuracy (HOTA

\uparrow

). Upward arrows indicate that higher values indicate better performance.

Multi-Object Tracking

Multi-object tracking (MOT) links detections across video frames to maintain object identities through motion, occlusion, and changing viewpoints.

See all 193 papers

Tracking-by-detection in Multi-object Tracking: Survey and Experiments
Sep 8, 2026Yujin Yang, Kyujin Shim, Kangwook Ko +1Data Association Multi-Object Tracking
School of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, 34141, South Korea.
COVTrack++: Learning Open-Vocabulary Multi-Object Tracking from Continuous Videos via a Synergistic Paradigm
Mar 25, 2026Zekun Qian, Wei Feng, Ruize Han +1Visual Object Tracking Multi-Object Tracking
School of Computer Science and Technology, Tianjin University, Tianjin, China · Department of Computer Science, City University of Hong Kong, Hong Kong SAR, China · Faculty of Computer Science and Artificial Intelligence, Shenzhen University of Advanced Technology
MOTIP2: Spatial Priors for End-to-End Multi-Object Tracking
Oct 7, 2026Benoît Roussel, Damien Bouet, Liming Chen +1Multi-Object Tracking Data Association
École Centrale de Lyon LIRIS, UMR CNRS 5205 Lyon, France · IDEMIA Courbevoie, France

Sep 8, 2026J/K move · Enter open · S save

Tracking-by-detection in Multi-object Tracking: Survey and Experiments

Yujin Yang, Kyujin Shim, Kangwook Ko, Changick Kim

School of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, 34141, South Korea.

Data Association Multi-Object Tracking

Open paper AskPDF arXiv

Multi-object tracking (MOT) is an essential computer vision task that simultaneously tracks multiple objects in video sequences, with various applications in surveillance, autonomous navigation, and human-computer interaction. The tracking-by-detection (TBD) paradigm, which combines object detection with temporal association, has emerged as a leading approach, driven by innovative algorithms. Despite recent progress, fair evaluation of TBD-based methods remains a challenge. Many studies introduce modules such as similarity metrics, data association strategies, or motion models, but they are often evaluated under inconsistent protocols, with different baseline trackers, hyperparameters, and datasets. Such inconsistencies obscure the genuine contribution of each module and hinder objective comparison. This survey systematically reviews TBD-based MOT techniques, including similarity measurements, data association, camera motion compensation, and interpolation strategies. Starting from a minimal baseline tracker, we fairly evaluate the contributions of each method across diverse datasets and accumulate well-balanced methods. Our findings establish a strong baseline tracker and provide a foundation for the principled design of robust and versatile MOT systems suitable for real-world deployment.

Abstract

Multi-Object Tracking

Explore similar work

Tracking-by-detection in Multi-object Tracking: Survey and Experiments

COVTrack++: Learning Open-Vocabulary Multi-Object Tracking from Continuous Videos via a Synergistic Paradigm

MOTIP2: Spatial Priors for End-to-End Multi-Object Tracking