UGO: Unified Architecture for General Multi-Object Tracking by Segmentation

Jer Pelhan; Alan Lukezic; Matej Kristan

cs.CVSep 29, 2026

UGO: Unified Architecture for General Multi-Object Tracking by Segmentation

Authors: Jer Pelhan, Alan Lukezic, Matej Kristan

Organizations: Faculty of Computer and Information Science, University of Ljubljana

Abstract

General multi-object tracking (GMOT) tracks all instances of a user-specified category from a single first-frame exemplar. Prior work relies on bounding boxes and surrogate training, and struggles with non-rigid objects, crowded scenes, and distractors. We introduce UGO, a unified GMOT tracker that pairs a pretrained exemplar-conditioned detection head with an instance-propagation head in a common architecture. A novel training-free, energy-minimization consolidation method converts overlapping proposals into exclusive pixel-wise masks and detections, resolving over-segmentation, duplicates, and conflicts. A hierarchical memory spanning global and instance levels improves recall and per-instance segmentation accuracy using a new memory management protocol. UGO sets a new state-of-the-art on GMOT benchmarks and video object counting, and is competitive with specialist MOT methods, establishing a strong paradigm for unified, open-category multi-object tracking.

Figures & tables

Figure 1: First row: Tracker ID=1 oversegments an occluding fish tracked by ID=2, while a single detector fires on both. UGO resolves the oversegmentations and redundant detection by a novel consolidation module. Second row: The new paradigm enables robust general multi-object tracking by segmentation in cluttered dense scenes through occlusions.

Figure 2: UGO unifies detection and frame-to-frame association in a single network. Detector and instance tracker heads produce compatible per-pixel outputs, which enables optimization-based, training-free consolidation and robust tracking.

Figure 3: Oversegmented object at $T=1$ and tracking in subsequent frames with and without consolidation-driven DRM update. Individual colors indicate an ID, with dashed pattern indicating overlapping IDs. Consolidation-based memory update leads to distractor resolution, while ignoring it leads to tracker failure.

	Detector	Tracker	IDF1 $\uparrow$	MT $\uparrow$	ML $\downarrow$	FP $\downarrow$	FN $\downarrow$	F1 $\uparrow$	IDSw $\downarrow$	MOTA $\uparrow$	HOTA $\uparrow$
Open-Vocabulary	OVTrack [ 21 ]	DeepSORT [ 38 ]	21.2	165	1367	49984	160378	47.7	1470	20.2	-
		ByteTrack [ 44 ]	20.6	164	1345	51356	156329	49.1	1669	19.9	-
		BoT-SORT [ 1 ]	20.3	167	1328	45721	163378	47.1	3278	20.0	-
		TbQ [ 23 ] -SwT	18.7	186	1304	50784	154893	49.7	6381	21.3	-
	GLIP-T [ 20 ]	DeepSORT [ 38 ]	41.6	401	877	46610	141330	55.0	2892	25.5	-
		ByteTrack [ 44 ]	45.1	447	746	52591	131759	57.5	2706	27.0	-

Table 1: State-of-the-art comparison on the GMOT-40 [ 3 ] benchmark.

Figure 4: Incorrect masks spreading over several objects are corrected by the proposed detector-tracker consolidation method, enabling accurate memory updates and preventing drifts.

Method	HOTA $\uparrow$	MOTA $\uparrow$	IDF1 $\uparrow$	IDP $\uparrow$	IDR $\uparrow$	MT $\uparrow$	PT	ML $\downarrow$	F1 $\uparrow$	IDSw $\downarrow$	FM $\downarrow$
ByteTrack [ 44 ]	40.1	38.5	51.2	64.9	42.3	310	465	329	0.631	1309	3513
IOUTrack [ 6 ]	41.6	55.7	45.7	51.9	40.7	388	454	262	0.762	4639	5259
SORT [ 5 ]	42.8	55.6	49.2	58.5	42.4	333	470	301	0.749	2530	3730
OMC [ 22 ]	43.0	53.4	50.3	61.8	42.4	324	478	302	0.735	4938	7162
Tracktor++ [ 4 ]	44.2	55.2	51.0	58.5	45.1	364	472	268	0.751	1976	4149
TransTrack [ 35 ]	45.4	48.3	53.4	63.4	46.1	327	416	361	0.705	1978	6459

Table 2: State-of-the-art comparison on AnimalTrack [ 43 ] .

Method	HOTA	DetA	AssA
MPNTrack17	46.6 (7%)	46.2 (-2%)	47.3 (18%)
eTC17	45.1 (11%)	44.1 (3%)	46.4 (19%)
Tracktor++v2	45.1 (11%)	45.3 (0%)	45.0 (24%)
ByteTrack [ 44 ]	63.1 (-21%)	64.5 (-30%)	62.0 (-10%)
OC-Sort [ 8 ]	63.2 (-21%)	63.2 (-28%)	63.4 (-12%)
OC-Sort ∗ [ 8 ]	54.6 (-8%)	47.8 (-5%)	57.6 (-3%)

Table 3: Tracking performance on the MOT17. The colors denote UGO performing better / worse compared to the individual method, ∗ denotes public detection setup.

Method	HOTA	DetA	AssA
MPNTrack17	46.6 (7%)	46.2 (-2%)	47.3 (18%)
eTC17	45.1 (11%)	44.1 (3%)	46.4 (19%)
Tracktor++v2	45.1 (11%)	45.3 (0%)	45.0 (24%)
ByteTrack [ 44 ]	63.1 (-21%)	64.5 (-30%)	62.0 (-10%)
OC-Sort [ 8 ]	63.2 (-21%)	63.2 (-28%)	63.4 (-12%)
OC-Sort ∗ [ 8 ]	54.6 (-8%)	47.8 (-5%)	57.6 (-3%)

Table 3: Tracking performance on the MOT17. The colors denote UGO performing better / worse compared to the individual method, ∗ denotes public detection setup.

	Penguins		Crystals
Method	MAE↓	RMSE↓	MAE↓	RMSE↓
GDINO ${}_{\text{MASA}}$ * [ 24 ]	9.0	11.5	72.3	87.9
CountGDB ${}_{\text{ByteTrack}}$ * [ 2 ]	4.3	5.5	71.6	88.1
CountVid [ 2 ] *	4.0	5.3	69.1	86.0
CountGDB ${}_{\text{ByteTrack}}$ [ 2 ]	4.0	4.2	31.1	52.8
GeCo ${}_{\text{SAM2.1}}$ [ 31 ]	11.3	14.5	46.1	82.8

Table 4: Video object counting results on Science-Count [ 2 ] . Methods marked with * accept text prompts instead of exemplars.

Method	HOTA	DetA	AssA	MOTA	FN	FP	IDSw	MT	PT	ML	FM	IDF1
UGO ${}_{\overline{\text{CONS}}}$	50.98	49.11	54.36	43.12	61835	59794	24173	1084	671	189	8414	61.47
UGO HM	52.35	39.42	70.55	5.88	34457	204395	2402	1460	328	156	4553	60.76
UGO ${}_{\overline{\Theta_{1}}}$	52.39	55.56	50.48	57.11	39900	64272	5773	1360	421	163	4478	56.86
UGO D4S	61.00	53.80	70.49	57.33	51892	55349	2132	1252	430	262	4409	72.81
UGO ${}_{\overline{\Theta_{2}}}$	62.76	55.55	72.11	59.16	40059	63251	1375	1351	419	174	4633	75.66
UGO ${}_{\overline{\text{DRM}}}$	63.19	56.23	72.20	60.30	41057	59463	1244	1331	434	179	4554	76.10

Table 5: UGO ablation on GMOT-40 [ 3 ] .

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: HAM overview: stable tracks provide robust updates for detector category-level memory, while instance-level memories are updated robustly by the result of consolidation analysis.

Method	HOTA $\uparrow$	MOTA $\uparrow$	IDF1 $\uparrow$	IDP $\uparrow$	IDR $\uparrow$	MT $\uparrow$	PT	ML $\downarrow$	FP $\downarrow$	FN $\downarrow$	Pr $\uparrow$	Re $\uparrow$	F1 $\uparrow$	IDSw $\downarrow$	FM $\downarrow$
JDE	26.8	27.3	31.0	51.0	22.0	106	414	584	17887	155623	0.830	0.360	0.502	3187	5031
FairMOT	30.6	29.0	38.8	62.8	28.0	143	462	499	17653	152624	0.837	0.372	0.515	2335	5447
Trackformer	31.0	20.4	36.5	40.9	32.8	230	491	383	70404	118724	0.639	0.512	0.568	4355	3725
TADAM	32.5	36.5	37.2	44.4	32.0	258	495	351	41728	110048	0.761	0.547	0.637	2538	4469
DeepSORT	32.8	41.4	35.2	49.7	27.2	213	452	439	14131	124747	0.893	0.487	0.630	3503	4527
ByteTrack	40.1	38.5	51.2	64.9	42.3	310	465	329	31591	116587	0.800	0.521	0.631	1309	3513

Appendix

Table 6: State-of-the-art comparison on AnimalTrack [ 43 ] .

Figure 6: Two sequences from AnimalTrack where UGO produces the most false detections. Red bounding boxes are overlaid to highlight the relevant regions. Ground-truth bounding boxes are shown in green – note that some instances are missing in the ground truth annotations.

Parameter	Default	Role
assoc_iou_thresh	$0.1$	Loose detection-to-track IoU threshold for deciding whether a detection is new.
trk_assoc_iou_thresh	$0.5$	Stricter IoU threshold for deciding whether a tracker masklet is matched.
new_det_thresh	$0.7$	Minimum detector score required to spawn a new object.
HIGH_CONF_THRESH	$0.8$	Threshold for high-confidence detector reconditioning.
iou_thresh_recondition	$0.8$	Default IoU threshold for detector-to-tracker memory refresh.
suppress_overlapping_based

Appendix

Table 7: Threshold summary for SAM3 [ 9 ] .

Method	HOTA	DetA	AssA	MOTA	FN	FP	IDSw	MT	PT	ML	FM	IDF1
$\theta_{0}=0.3$	63.71	56.60	72.91	61.12	40195	58347	1130	1348	426	170	4494	76.85
$\theta_{0}=0.4$	63.75	56.62	72.96	61.15	40000	58416	1165	1345	431	168	4500	76.92
$\theta_{0}=0.6$	63.82	56.65	73.09	61.20	39769	58605	1092	1353	422	169	4463	77.03
$\theta_{0}=0.7$	63.77	56.64	72.98	61.20	39695	58666	1107	1344	433	167	4487	77.00
$\theta_{0}=0.8$	63.62	56.57	72.72	60.98	39996	58942	1082	1347	431	166	4492	76.78
$\tau_{\text{md}}=0.45$	63.45	56.33	72.70	60.37	38370	61941	1269	1359	419	166	4575	76.49

Appendix

Table 8: UGO ablation on GMOT-40 [ 3 ] .

Method	Quality	Accuracy	Robustness
SAM2.1	0.649	0.720	0.887
DAM4SAM	0.694 1	0.727 1	0.944 1
UGO	0.685 2	0.724 2	0.932 2

Appendix

Table 9: Single-target tracking performance on the DiDi dataset [ 36 ] measured using the following standard performance measures: tracking quality, accuracy and robustness.

Figure 7: Qualitative examples of tracking with UGO. Each instance is represented by a colored mask and line, denoting its trajectory.

Explore similar work

COVTrack++: Learning Open-Vocabulary Multi-Object Tracking from Continuous Videos via a Synergistic Paradigm
Mar 25, 2026Zekun Qian, Wei Feng, Ruize Han +1Multi-Object Tracking Open-Vocabulary Object Detection
School of Computer Science and Technology, Tianjin University, Tianjin, China · Department of Computer Science, City University of Hong Kong, Hong Kong SAR, China · Faculty of Computer Science and Artificial Intelligence, Shenzhen University of Advanced Technology
Selective Mask Propagation for Multi-Object Tracking
Jun 11, 2026Alexander HolmbergMulti-Object Tracking Video Object Segmentation
KTH Royal Institute of Technology
Unified Multimodal Visual Tracking with Dual Mixture-of-Experts
May 5, 2026Lingyi Hong, Jinglun Li, Xinyu Zhou +6Multi-Object Tracking Multimodal Fusion
Shanghai Key Lab of Intelligent Information Processing, College of Computer Science and Artificial Intelligence, Fudan University

Mar 25, 2026, cs.CVJ/K move · Enter open · S save

COVTrack++: Learning Open-Vocabulary Multi-Object Tracking from Continuous Videos via a Synergistic Paradigm

Zekun Qian, Wei Feng, Ruize Han, Junhui Hou

School of Computer Science and Technology, Tianjin University, Tianjin, China · Department of Computer Science, City University of Hong Kong, Hong Kong SAR, China · Faculty of Computer Science and Artificial Intelligence, Shenzhen University of Advanced Technology

Multi-Object Tracking Open-Vocabulary Object Detection Intermediate States

Open paper AskPDF arXiv

Multi-Object Tracking (MOT) has traditionally focused on a few specific categories, restricting its applicability to real-world scenarios involving diverse objects. Open-Vocabulary Multi-Object Tracking (OVMOT) addresses this by enabling tracking of arbitrary categories, including novel objects unseen during training. However, current progress is constrained by two challenges: the lack of continuously annotated video data for training, and the lack of a customized OVMOT framework to synergistically handle detection and association. We address the data bottleneck by constructing C-TAO, the first continuously annotated training set for OVMOT, which increases annotation density by 26x over the original TAO and captures smooth motion dynamics and intermediate object states. For the framework bottleneck, we propose COVTrack++, a synergistic framework that achieves a bidirectional reciprocal mechanism between detection and association through three modules: (1) Multi-Cue Adaptive Fusion (MCF) dynamically balances appearance, motion, and semantic cues for association feature learning; (2) Multi-Granularity Hierarchical Aggregation (MGA) exploits hierarchical spatial relationships in dense detections, where visible child nodes (e.g., object parts) assist occluded parent objects (e.g., whole body) for association feature enhancement; (3) Temporal Confidence Propagation (TCP) recovers flickering detections through high-confidence tracked objects boosting low-confidence candidates across frames, stabilizing trajectories. Extensive experiments on TAO demonstrate state-of-the-art performance, with novel TETA reaching 35.4% and 30.5% on validation and test sets, improving novel AssocA by 4.8% and novel LocA by 5.8% over previous methods, and show strong zero-shot generalization on BDD100K.

Abstract

Explore similar work

COVTrack++: Learning Open-Vocabulary Multi-Object Tracking from Continuous Videos via a Synergistic Paradigm

Selective Mask Propagation for Multi-Object Tracking

Unified Multimodal Visual Tracking with Dual Mixture-of-Experts