cs.CVSep 22, 2025

DepTR-MOT: Unveiling the Potential of Depth-Informed Trajectory Refinement for Multi-Object Tracking

Authors: Buyin Deng, Lingxin Huang, Kai Luo, Fei Teng, Kailun Yang

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Visual Multi-Object Tracking (MOT) is a crucial component of robotic perception, yet existing Tracking-By-Detection (TBD) methods often rely on 2D cues, such as bounding boxes and motion modeling, which struggle under occlusions and close-proximity interactions. Trackers relying on these 2D cues are particularly unreliable in robotic environments, where dense targets and frequent occlusions are common. While depth information has the potential to alleviate these issues, most existing MOT datasets lack depth annotations, leading to its underexploited role in the domain. To unveil the potential of depth-informed trajectory refinement, we introduce DepTR-MOT, a category-agnostic DETR-based detector enhanced with instance-level depth information. Specifically, we propose two key innovations: (i) foundation model-based instance-level soft depth label supervision, which refines depth prediction, and (ii) the distillation of dense depth maps to maintain global depth consistency. These strategies enable DepTR-MOT to embed instance-level depth awareness into detector and to output instance-level depth during inference. All designs strictly follow the prevailing Tracking-By-Detection (TBD) paradigm, requiring no additional training or inference stages. By incorporating depth cues, our method enhances the robustness of the TBD paradigm and effectively resolves occlusion and close-proximity challenges. Experiments on the QuadTrack, DanceTrack, SportsMOT, and VisDrone19 datasets demonstrate the effectiveness of our approach, achieving HOTA scores of 27.59, 44.939, 61.2, and 42.5, respectively. These consistent gains across aerial, sports, and surveillance platforms highlight the advantages of our method in handling occlusion and close-proximity challenges in robotic tracking. The source code will be made publicly available at https://github.com/warriordby/DepTR-MOT.

Figures & tables

Fig. 1: Overview of the proposed DepTR: by cultivating intrinsic depth perception via foundation-model guided supervision, DepTR-MOT jointly regresses instance-specific absolute depth and bounding box coordinates during the detection stage, seamlessly incorporating depth cues into data association to enhance trajectory continuity.
Fig. 2: Comparison of tracking paradigms: (A) conventional model outputs only bounding boxes and confidence scores; (B) current depth-informed MOT methods rely on external modules for additional depth or appearance features, leading to bulky inference; (C) our proposed model directly outputs instance-level depth in the detection head, achieving lightweight and depth-aware tracking with improved robustness under occlusion.
Fig. 3: Pipeline of DepTR-MOT. A prompt-based knowledge-guided foundation module leverages frozen SAM and Depth models to generate instance-level depth labels from 2D box annotations, providing label supervision and depth feature distillation to align the decoding-depth feature map during training. During inference, DepTR directly predicts 3D spatial locations using stacked object and depth-awareness blocks.
TABLE I: Comparison of different data loading strategies on the DanceTrack dataset. Here, LL denotes the length of the data loading window, while ρ\rho represents the sliding interval between two consecutive windows.
Fig. 4: Illustration of the pattern that bounding-box-based depth indexing is susceptible to background interference (left) and neighboring individuals (right), leading to biased or inconsistent depth values. Incorporating SAM2 mitigates these issues by applying instance masks, which remove environmental depth values and yield more accurate depth distributions.
TABLE II: Results on QuadTrack, DanceTrack, SportsMOT, and VisDrone19. DepTR-MOT delivers robust performance improvements across diverse scenarios for major TBD trackers, with particularly substantial gains in IDF1 and AssA.
Fig. 5: Real-world application of DepTR-MOT: Deployed on a quadruped robot equipped with a panoramic camera, evaluating pedestrian tracking performance in sidewalk scenarios.
λ\lambdaγ\gammaHOTA ↑\uparrowIDF1 ↑\uparrowMOTA ↑\uparrowDetA ↑\uparrowAssA ↑\uparrow
1.0023.66622.201-2.657530.04419.342
0.10.913.38412.456-65.00117.52311.555
0.20.817.49217.012-47.48620.84116.050
0.30.720.62320.655-36.73622.97320.095
0.40.620.74720.663-32.18023.89619.461
0.50.522.55322.835-29.16124.63022.514
TABLE III: Performance of DepTR-MOT on QuaTrack under different λ\lambda and γ\gamma weight configurations with ByteTrack [ 40 ] .
Exp.WdW_{d}LregL_{reg}LalignL_{align}HOTA ↑\uparrowIDF1 ↑\uparrowMOTA ↑\uparrowDetA ↑\uparrowAssA ↑\uparrow
(1)–––42.34383.35226.64067.69145.922
(2)–✓✓43.33682.20427.95567.61744.907
(3)✓––44.66986.12628.85969.51547.176
(4)✓–✓43.72285.70527.77969.20146.231
(5)✓✓–43.08685.80426.87569.46444.770
(6)✓✓✓44.93986.27129.13369.68747.956
TABLE IV: Ablation study on the key design components of DepTR on the DanceTrack dataset.
More in this topic

Multi-Object Tracking

Multi-object tracking (MOT) links detections across video frames to maintain object identities through motion, occlusion, and changing viewpoints.

See all 189 papers

Explore similar work

CardsList
Feb 4, 2026

DRMOT: A Dataset and Framework for RGBD Referring Multi-Object Tracking

Referring Multi-Object Tracking (RMOT) aims to track specific targets based on language descriptions, playing a vital role in interactive AI systems such as robotics and autonomous driving. However, existing RMOT methods rely exclusively on 2D RGB observations. Since similar 2D projections can correspond to different 3D configurations, the available observations may not uniquely determine the referred targets. This creates a fundamental mismatch between the task requirements and the available input. To advance research on the DRMOT task, we construct DRSet, a dataset tailored for the DRMOT task. DRSet comprises 2,076 referring expressions over 186 RGB-D source sequences. It features synchronized RGB images and depth maps paired with language descriptions, including explicit depth-dependent descriptions, to systematically evaluate spatial-semantic detection and tracking capabilities. Furthermore, we propose DRTrack, an MLLM-guided depth-referring tracking framework. DRTrack leverages a Geometric-Aware GRPO fine-tuned Multimodal Large Language Model (MLLM) to achieve precise spatial-semantic detection from joint L+RGB+D inputs, while incorporating explicit depth cues to enforce robust trajectory association. Extensive experiments on the DRSet dataset demonstrate the effectiveness of our framework. Our dataset and code will be available at https://github.com/chen-si-jia/DRMOT.

Sijia Chen, Shaofeng Liang, Lijuan Ma +7

Huazhong University of Science and Technology · South-Central Minzu University

Referring MOTMulti-Object TrackingMultimodal Large Language Models
AskPDF
May 10, 2026

SAMOFT: Robust Multi-Object Tracking via Region and Flow

Multi-object tracking (MOT) is a fundamental task in computer vision that requires continuously tracking multiple targets while maintaining consistent identities across frames. However, most existing approaches primarily rely on instance-level object features for trajectory association, which often leads to degraded performance under challenging conditions such as object deformation, nonlinear motion, and occlusion. In this work, we propose SAMOFT, a robust tracker that leverages pixel-level cues to improve robustness under complex motion scenarios. Specifically, we introduce a Pixel Motion Matching (PMM) module that integrates the Segment Anything Model (SAM) with dense optical flow to refine Kalman filter-based motion prediction using instantaneous foreground pixel motion. To further enhance robustness under unreliable detections, we design a Centroid Distance Matching (CDM) module that performs flexible mask-based centroid matching for low-confidence or partially occluded observations. Moreover, a Distribution-Based Correction (DBC) module models long-tailed motion patterns in a training-free manner using historical optical flow statistics and dynamically corrects trajectory states online. We also incorporate a Cluster-Aware ReID (CA-ReID) strategy to improve the stability and discriminative power of trajectory appearance features. Extensive experiments on the DanceTrack and MOTChallenge benchmarks demonstrate that SAMOFT consistently improves baseline trackers and achieves competitive performance compared with recent state-of-the-art methods, validating the effectiveness of leveraging pixel-level cues for robust multi-object tracking.

Yanchao Wang, Dawei Zhang, Chengzhuan Yang +5

School of Computer Science and Technology, Zhejiang Normal University, Jinhua 321004, China · School of Automation and Intelligent Sensing, the Institute of Image Processing and Pattern Recognition, and the Institute of Medical Robotics, Shanghai Jiao Tong University, Shanghai 200240, China · Institute for Sustainable Industries and Liveable Cities, College of Engineering and Science, Victoria University, Melbourne, VIC 8001, Australia +1

Query Re-IdentificationMulti-Object Tracking
AskPDF
Mar 6, 2026

Occlusion-Aware SORT: Observing Occlusion for Robust Multi-Object Tracking

Multi-object tracking (MOT) involves analyzing object trajectories and counting the number of objects in video sequences. However, 2D MOT faces challenges due to positional cost confusion arising from partial occlusion. To address this issue, we present the novel Occlusion-Aware SORT (OA-SORT) framework, a plug-and-play and training-free framework that includes the Occlusion-Aware Module (OAM), the Occlusion-Aware Offset (OAO), and the Bias-Aware Momentum (BAM). Specifically, OAM analyzes the occlusion status of objects, where a Gaussian Map (GM) is introduced to reduce background influence. In contrast, OAO and BAM leverage the OAM-described occlusion status to mitigate cost confusion and suppress estimation instability. Comprehensive evaluations on the DanceTrack, SportsMOT, and MOT17 datasets demonstrate the importance of occlusion handling in MOT. On the DanceTrack test set, OA-SORT achieves 63.1% and 64.2% in HOTA and IDF1, respectively. Furthermore, integrating the Occlusion-Aware framework into the four additional trackers improves HOTA and IDF1 by an average of 2.08% and 3.05%, demonstrating the reusability of the occlusion awareness.

Chunjiang Li, Jianbo Ma, Li Shen +2

College of Computer Science, Sichuan University, Chengdu, China · Institute of Optics and Electronics, CAS, Chengdu, China

Multi-Object TrackingOcclusion Reasoning
AskPDF