DepTR-MOT: Unveiling the Potential of Depth-Informed Trajectory Refinement for Multi-Object Tracking

Buyin Deng; Lingxin Huang; Kai Luo; Fei Teng; Kailun Yang

cs.CVSep 22, 2025

DepTR-MOT: Unveiling the Potential of Depth-Informed Trajectory Refinement for Multi-Object Tracking

Authors: Buyin Deng, Lingxin Huang, Kai Luo, Fei Teng, Kailun Yang

Abstract

Visual Multi-Object Tracking (MOT) is a crucial component of robotic perception, yet existing Tracking-By-Detection (TBD) methods often rely on 2D cues, such as bounding boxes and motion modeling, which struggle under occlusions and close-proximity interactions. Trackers relying on these 2D cues are particularly unreliable in robotic environments, where dense targets and frequent occlusions are common. While depth information has the potential to alleviate these issues, most existing MOT datasets lack depth annotations, leading to its underexploited role in the domain. To unveil the potential of depth-informed trajectory refinement, we introduce DepTR-MOT, a category-agnostic DETR-based detector enhanced with instance-level depth information. Specifically, we propose two key innovations: (i) foundation model-based instance-level soft depth label supervision, which refines depth prediction, and (ii) the distillation of dense depth maps to maintain global depth consistency. These strategies enable DepTR-MOT to embed instance-level depth awareness into detector and to output instance-level depth during inference. All designs strictly follow the prevailing Tracking-By-Detection (TBD) paradigm, requiring no additional training or inference stages. By incorporating depth cues, our method enhances the robustness of the TBD paradigm and effectively resolves occlusion and close-proximity challenges. Experiments on the QuadTrack, DanceTrack, SportsMOT, and VisDrone19 datasets demonstrate the effectiveness of our approach, achieving HOTA scores of 27.59, 44.939, 61.2, and 42.5, respectively. These consistent gains across aerial, sports, and surveillance platforms highlight the advantages of our method in handling occlusion and close-proximity challenges in robotic tracking. The source code will be made publicly available at https://github.com/warriordby/DepTR-MOT.

Figures & tables

Fig. 1: Overview of the proposed DepTR: by cultivating intrinsic depth perception via foundation-model guided supervision, DepTR-MOT jointly regresses instance-specific absolute depth and bounding box coordinates during the detection stage, seamlessly incorporating depth cues into data association to enhance trajectory continuity.

Fig. 2: Comparison of tracking paradigms: (A) conventional model outputs only bounding boxes and confidence scores; (B) current depth-informed MOT methods rely on external modules for additional depth or appearance features, leading to bulky inference; (C) our proposed model directly outputs instance-level depth in the detection head, achieving lightweight and depth-aware tracking with improved robustness under occlusion.

Fig. 3: Pipeline of DepTR-MOT. A prompt-based knowledge-guided foundation module leverages frozen SAM and Depth models to generate instance-level depth labels from 2D box annotations, providing label supervision and depth feature distillation to align the decoding-depth feature map during training. During inference, DepTR directly predicts 3D spatial locations using stacked object and depth-awareness blocks.

Panel of Table 4 — TABLE I: Comparison of different data loading strategies on the DanceTrack dataset. Here, $L$ denotes the length of the data loading window, while $\rho$ represents the sliding interval between two consecutive windows.

Fig. 4: Illustration of the pattern that bounding-box-based depth indexing is susceptible to background interference (left) and neighboring individuals (right), leading to biased or inconsistent depth values. Incorporating SAM2 mitigates these issues by applying instance masks, which remove environmental depth values and yield more accurate depth distributions.

Panel of Table 6 — TABLE II: Results on QuadTrack, DanceTrack, SportsMOT, and VisDrone19. DepTR-MOT delivers robust performance improvements across diverse scenarios for major TBD trackers, with particularly substantial gains in IDF1 and AssA.

Fig. 5: Real-world application of DepTR-MOT: Deployed on a quadruped robot equipped with a panoramic camera, evaluating pedestrian tracking performance in sidewalk scenarios.

$\lambda$	$\gamma$	HOTA $\uparrow$	IDF1 $\uparrow$	MOTA $\uparrow$	DetA $\uparrow$	AssA $\uparrow$
1.0	0	23.666	22.201	-2.6575	30.044	19.342
0.1	0.9	13.384	12.456	-65.001	17.523	11.555
0.2	0.8	17.492	17.012	-47.486	20.841	16.050
0.3	0.7	20.623	20.655	-36.736	22.973	20.095
0.4	0.6	20.747	20.663	-32.180	23.896	19.461
0.5	0.5	22.553	22.835	-29.161	24.630	22.514

TABLE III: Performance of DepTR-MOT on QuaTrack under different

\lambda

and

\gamma

weight configurations with ByteTrack [ 40 ] .

Exp.	$W_{d}$	$L_{reg}$	$L_{align}$	HOTA $\uparrow$	IDF1 $\uparrow$	MOTA $\uparrow$	DetA $\uparrow$	AssA $\uparrow$
(1)	–	–	–	42.343	83.352	26.640	67.691	45.922
(2)	–	✓	✓	43.336	82.204	27.955	67.617	44.907
(3)	✓	–	–	44.669	86.126	28.859	69.515	47.176
(4)	✓	–	✓	43.722	85.705	27.779	69.201	46.231
(5)	✓	✓	–	43.086	85.804	26.875	69.464	44.770
(6)	✓	✓	✓	44.939	86.271	29.133	69.687	47.956

TABLE IV: Ablation study on the key design components of DepTR on the DanceTrack dataset.

Multi-Object Tracking

Multi-object tracking (MOT) links detections across video frames to maintain object identities through motion, occlusion, and changing viewpoints.

See all 189 papers

DRMOT: A Dataset and Framework for RGBD Referring Multi-Object Tracking
Feb 4, 2026Sijia Chen, Shaofeng Liang, Lijuan Ma +7Referring MOT Multi-Object Tracking
Huazhong University of Science and Technology · South-Central Minzu University
SAMOFT: Robust Multi-Object Tracking via Region and Flow
May 10, 2026Yanchao Wang, Dawei Zhang, Chengzhuan Yang +5Query Re-Identification Multi-Object Tracking
School of Computer Science and Technology, Zhejiang Normal University, Jinhua 321004, China · School of Automation and Intelligent Sensing, the Institute of Image Processing and Pattern Recognition, and the Institute of Medical Robotics, Shanghai Jiao Tong University, Shanghai 200240, China · Institute for Sustainable Industries and Liveable Cities, College of Engineering and Science, Victoria University, Melbourne, VIC 8001, Australia +1
Occlusion-Aware SORT: Observing Occlusion for Robust Multi-Object Tracking
Mar 6, 2026Chunjiang Li, Jianbo Ma, Li Shen +2Multi-Object Tracking Occlusion Reasoning
College of Computer Science, Sichuan University, Chengdu, China · Institute of Optics and Electronics, CAS, Chengdu, China

Feb 4, 2026J/K move · Enter open · S save

DRMOT: A Dataset and Framework for RGBD Referring Multi-Object Tracking

Sijia Chen, Shaofeng Liang, Lijuan Ma, Yanqiu Yu+6

Huazhong University of Science and Technology · South-Central Minzu University

Referring MOT Multi-Object Tracking Multimodal Large Language Models

Open paper AskPDF arXiv

Referring Multi-Object Tracking (RMOT) aims to track specific targets based on language descriptions, playing a vital role in interactive AI systems such as robotics and autonomous driving. However, existing RMOT methods rely exclusively on 2D RGB observations. Since similar 2D projections can correspond to different 3D configurations, the available observations may not uniquely determine the referred targets. This creates a fundamental mismatch between the task requirements and the available input. To advance research on the DRMOT task, we construct DRSet, a dataset tailored for the DRMOT task. DRSet comprises 2,076 referring expressions over 186 RGB-D source sequences. It features synchronized RGB images and depth maps paired with language descriptions, including explicit depth-dependent descriptions, to systematically evaluate spatial-semantic detection and tracking capabilities. Furthermore, we propose DRTrack, an MLLM-guided depth-referring tracking framework. DRTrack leverages a Geometric-Aware GRPO fine-tuned Multimodal Large Language Model (MLLM) to achieve precise spatial-semantic detection from joint L+RGB+D inputs, while incorporating explicit depth cues to enforce robust trajectory association. Extensive experiments on the DRSet dataset demonstrate the effectiveness of our framework. Our dataset and code will be available at https://github.com/chen-si-jia/DRMOT.

Abstract

Multi-Object Tracking

Explore similar work

DRMOT: A Dataset and Framework for RGBD Referring Multi-Object Tracking

SAMOFT: Robust Multi-Object Tracking via Region and Flow

Occlusion-Aware SORT: Observing Occlusion for Robust Multi-Object Tracking