DRMOT: A Dataset and Framework for RGBD Referring Multi-Object Tracking

Sijia Chen; Shaofeng Liang; Lijuan Ma; Yanqiu Yu; Nan Li; Wenshuo Chen; Runwei Guan; Dongming Wu

cs.CVFeb 4, 2026

DRMOT: A Dataset and Framework for RGBD Referring Multi-Object Tracking

Authors: Sijia Chen, Shaofeng Liang, Lijuan Ma, Yanqiu Yu, Nan Li, Wenshuo Chen, Runwei Guan, Dongming Wu, +2 more

Organizations: Huazhong University of Science and Technology · South-Central Minzu University

Abstract

Referring Multi-Object Tracking (RMOT) aims to track specific targets based on language descriptions, playing a vital role in interactive AI systems such as robotics and autonomous driving. However, existing RMOT methods rely exclusively on 2D RGB observations. Since similar 2D projections can correspond to different 3D configurations, the available observations may not uniquely determine the referred targets. This creates a fundamental mismatch between the task requirements and the available input. To advance research on the DRMOT task, we construct DRSet, a dataset tailored for the DRMOT task. DRSet comprises 2,076 referring expressions over 186 RGB-D source sequences. It features synchronized RGB images and depth maps paired with language descriptions, including explicit depth-dependent descriptions, to systematically evaluate spatial-semantic detection and tracking capabilities. Furthermore, we propose DRTrack, an MLLM-guided depth-referring tracking framework. DRTrack leverages a Geometric-Aware GRPO fine-tuned Multimodal Large Language Model (MLLM) to achieve precise spatial-semantic detection from joint L+RGB+D inputs, while incorporating explicit depth cues to enforce robust trajectory association. Extensive experiments on the DRSet dataset demonstrate the effectiveness of our framework. Our dataset and code will be available at https://github.com/chen-si-jia/DRMOT.

Figures & tables

Fig. 1: Comparison between RMOT and DRMOT tasks. (a) The RMOT model relying solely on language (L) description and RGB images is unable to correctly detect the referred target under depth-dependent spatial descriptions. (b) DRMOT leverages depth cues to resolve spatial ambiguity, thereby achieving accurate target detection and maintaining temporal identity consistency.

Fig. 2: Annotation Process. The annotation process includes four steps: (1) Attribute Table Creation: we build an attribute table that categorizes object descriptions into static attributes and dynamic behaviors; (2) Object Selection: we review the entire video and select representative targets based on their attributes and behaviors; (3) Language Description Annotation: we draw bounding boxes frame by frame and compose language descriptions according to the predefined attributes; and (4) Annotation Verification: we perform a cross-review by dual-annotator to ensure valid bounding box coordinates, consistent object IDs, and accurate language descriptions. Finally, all verified annotations are packed to generate the DRSet dataset.

Tracking Dataset	Language	RGB	Depth	Sequences	Expressions	Avg.Words
Refer-KITTI [ 6 ]	✓	✓	✗	18	818	49
Refer-Dance [ 26 ]	✓	✓	✗	65	1.9K	25
LaMOT [ 30 ]	✓	✓	✗	62	145	9
CDTB [ 33 ]	✗	✓	✓	80	–	–
DepthTrack [ 34 ]	✗	✓	✓	200	–	–
ARKitTrack [ 7 ]	✗	✓	✓	300	–	–

TABLE I: Comparison of existing tracking datasets and the expanded DRSet dataset.

Subject Type	Attribute Dimension	Example Attribute Values
Human	Clothing	Clothing color, clothing type, accessories
	Hand-held objects	Backpack, basketball, umbrella, etc.
	Gender / Age	Male, female, child
	Actions	Waving, walking, eating, standing, etc.
	Scene interaction	Leaving, approaching
	Spatial position	Right side, front, back

TABLE II: Examples of attribute dimensions and values for different subject types.

Fig. 3: Overview of the DRSet dataset statistics. (A) Distribution of English words per referring expression. (B) Expressions per RGB-D source sequence. (C) Distribution of annotated temporal spans. (D) Distribution of target identities per expression. (E) Joint distribution of target density and temporal span, with depth and spatial expressions highlighted. (F) Environment composition within major semantic categories. (G) Proportions of key challenge attributes. (H) Dominant language concepts in the referring expressions.

Fig. 4: Overall pipeline of DRTrack. DRTrack consists of two stages. In Stage 1, the Depth-Promoted Language-Guided Detection stage leverages an MLLM to jointly process language descriptions, RGB images, and depth maps, and is fine-tuned via Geometric-Aware GRPO with format and IoU rewards to produce accurate depth-aware 2D bounding boxes of referred objects. In Stage 2, the Depth-Enhanced Association stage integrates depth information to compute the RGBD joint similarity as the association cost, thereby yielding ID-consistent trajectories of the referred targets.

Modality	Method	Venue	HOTA↑	DetA↑	AssA↑	DetRe↑	DetPr↑	AssRe↑	AssPr↑	LocA↑
L-RGB	TransRMOT [ 6 ]	CVPR 2023	1.16	0.24	5.79	0.26	2.57	5.90	90.28	75.00
L-RGB	TempRMOT [ 23 ]	ArXiv 2024	2.75	0.72	10.58	0.82	5.28	11.94	63.42	75.07
L-RGB	YOLOX [ 41 ] +ByteTrack [ 12 ] + iKUN [ 26 ]	CVPR 2024	7.89	2.87	21.78	4.70	6.78	25.48	62.27	77.42
L-RGB	YOLOX [ 41 ] +OC-SORT [ 13 ] + iKUN [ 26 ]	CVPR 2024	8.36	3.00	23.49	4.66	7.63	27.30	65.99	78.32
L-RGB	CRTracker [ 25 ]	AAAI 2025	8.23	2.67	27.25	3.85	7.79	29.39	66.16	72.63
L-RGB	DKGTrack [ 42 ]	ICCV 2025	2.99	0.42	21.71	0.44	12.71	22.28	75.55	75.46

TABLE III: Performance of different methods on the test set of the DRSet dataset. ↑ indicates that higher score is better. The best results are marked in bold .

MLLMs		Metrics
Name	Size	HOTA↑	DetA↑	AssA↑	DetRe↑	DetPr↑	AssRe↑	AssPr↑	LocA↑	FPS↑	AES↑
DeepSeek-VL [ 38 ]	7B	9.76	9.79	10.53	19.49	14.32	12.43	39.65	63.46	5.63	8.51
InternVL3.5 [ 46 ]	8B	5.61	6.49	5.29	9.20	15.82	5.78	49.86	66.28	2.34	4.51
LLaVA-NEXT [ 37 ]	8B	14.37	14.24	16.26	27.05	20.67	18.92	50.11	69.75	4.71	10.87
Qwen2.5-VL [ 8 ]	3B	11.45	8.26	20.14	11.69	20.08	22.22	56.48	71.51	5.83	9.67
Qwen2.5-VL [ 8 ]	7B	17.70	18.76	18.06	41.71	23.23	21.13	50.02	72.76	3.75	12.01

TABLE IV: Comparison of different Multimodal Large Language Models (MLLMs) on the test set of the DRSet dataset under the L+RGB input setting. ↑ indicates that higher score is better. The best results are marked in bold .

Settings			Metrics
Input Data	Association	GRPO	HOTA↑	DetA↑	AssA↑	DetRe↑	DetPr↑	AssRe↑	AssPr↑	LocA↑
L+RGB	SORT	✗	11.45	8.26	20.14	11.69	20.08	22.22	56.48	71.51
L+RGB+D	SORT	✗	11.23	7.82	20.26	10.88	19.92	22.46	56.88	71.68
L+RGB	Ours	✗	13.29	8.26	24.97	10.35	25.95	29.91	51.62	71.96
L+RGB+D	Ours	✗	12.75	7.70	24.66	9.58	25.40	29.80	51.90	72.12
L+RGB	SORT	✓	15.62	14.84	17.84	45.33	16.91	21.08	47.92	72.40

TABLE V: Component ablation of DRTrack with Qwen2.5-VL-3B-Instruct [ 8 ] on the test set of the DRSet dataset. “Ours” denotes the proposed depth-enhanced association. ↑ indicates that higher score is better. The best results are marked in bold .

Reward		Metrics
$R_{\text{format}}$	$R_{\text{IoU}}$	HOTA↑	DetA↑	AssA↑	DetRe↑	DetPr↑	AssRe↑	AssPr↑	LocA↑
✓	✗	27.28	22.01	34.78	34.23	32.84	48.24	44.95	71.79
✗	✓	27.10	21.94	34.46	35.53	31.56	47.25	45.13	71.73
✓	✓	28.25	22.87	35.90	34.53	34.50	49.88	45.68	71.81

TABLE VI: Ablation of the reward functions in Geometric-Aware GRPO. All variants use Qwen2.5-VL-3B-Instruct [ 8 ] , L+RGB+D inputs, and the proposed association on the test set of the DRSet dataset. ↑ indicates that higher score is better. The best results are marked in bold .

Multimodal Large Language Models (MLLMs)			Input Data	Fusion Weight	Metrics
Name	Size	RL Fine-tuning	Input Data	$\alpha$	HOTA↑	DetA↑	AssA↑	DetRe↑	DetPr↑	AssRe↑	AssPr↑	LocA↑
Qwen2.5-VL-3B-Instruct	3B	✓ (GRPO)	L+RGB+D	0.0	28.18	22.80	35.85	34.47	34.39	49.95	45.31	71.79
Qwen2.5-VL-3B-Instruct	3B	✓ (GRPO)	L+RGB+D	0.1	28.24	22.86	35.89	34.53	34.47	49.88	45.67	71.81
Qwen2.5-VL-3B-Instruct	3B	✓ (GRPO)	L+RGB+D	0.2	28.23	22.87	35.87	34.54	34.49	49.84	45.65	71.81
Qwen2.5-VL-3B-Instruct	3B	✓ (GRPO)	L+RGB+D	0.4	28.21	22.82	35.89	34.53	34.38	49.88	45.68	71.81
Qwen2.5-VL-3B-Instruct	3B	✓ (GRPO)	L+RGB+D	0.5	28.10	22.67	35.84	34.56	34.03	49.76	45.72	71.81

TABLE VII: Analysis of fusion weights

\alpha

in Equation ( 17 ). ↑ indicates that higher score is better. The best results are marked in bold .

$\sigma$	Metrics
$\sigma$	HOTA↑	DetA↑	AssA↑	DetRe↑	DetPr↑	AssRe↑	AssPr↑	LocA↑
1	20.54	19.31	23.13	35.73	26.29	29.34	46.99	71.66
5	26.95	22.00	34.05	34.84	32.29	46.18	46.41	71.79
10	27.98	22.54	35.75	34.60	33.69	49.51	45.83	71.80
15	28.17	22.78	35.84	34.54	34.29	49.79	45.66	71.82
30	28.24	22.87	35.90	34.53	34.50	49.88	45.68	71.81

TABLE VIII: Sensitivity to the depth scaling factor

\sigma

on the DRSet test split. ↑ indicates that higher score is better. The best results are marked in bold .

Depth Setting	Metrics
Depth Setting	HOTA↑	DetA↑	AssA↑	AssRe↑
Gaussian 20 mm	28.23	22.88	35.83	49.83
Gaussian 50 mm	28.24	22.88	35.84	49.81
Gaussian 100 mm	28.22	22.87	35.80	49.80
Multiplicative 5%	28.23	22.87	35.81	49.80
Multiplicative 10%	28.23	22.88	35.85	49.87

TABLE IX: Robustness to depth noise on the DRSet test split. “Ours” denotes the original depth input without perturbation. ↑ indicates that higher score is better. The best results are marked in bold .

Interval	Metrics
(frames)	HOTA↑	DetA↑	AssA↑	DetRe↑	AssRe↑	FPS↑	AES↑
1	38.36	32.49	45.89	48.07	60.61	0.49	12.88
5	33.76	27.80	41.89	40.96	56.21	2.42	17.46
10	28.25	22.87	35.90	34.53	49.88	4.82	18.16
20	21.76	16.95	29.05	26.69	42.12	9.15	17.52
30	19.14	14.38	26.80	23.13	39.21	12.94	17.36

TABLE X: Sensitivity to the MLLM inference interval on the DRSet test split. ↑ indicates that higher score is better. The best results are marked in bold .

Fig. 5: Qualitative results of DRTrack on the DRSet dataset. For each example, the top row shows RGB frames with predicted bounding boxes and identities, while the bottom row shows the corresponding depth maps. The examples illustrate how depth cues help resolve depth-dependent spatial descriptions, such as “closer to” and “behind”, while maintaining temporally consistent identities.

Explore similar work

ORMOT: A Dataset and Framework for Omnidirectional Referring Multi-Object Tracking
Mar 5, 2026Zihan Zhou, Sijia Chen, Yanqiu Yu +2Visual Object Tracking Multi-Object Tracking
National Key Laboratory of Science and Technology on Multi-spectral Information Processing, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan 430074, China
RT-RMOT: A Dataset and Framework for RGB-Thermal Referring Multi-Object Tracking
Feb 25, 2026Yanqiu Yu, Zhifan Jin, Sijia Chen +4Visual Object Tracking Multi-Object Tracking
Huazhong University of Science and Technology, Wuhan, Hubei, China · South-Central Minzu University, Wuhan, Hubei, China
STORM: End-to-End Referring Multi-Object Tracking in Videos
Apr 12, 2026Zijia Lu, Jingru Yi, Jue Wang +4Multi-Object Tracking Multimodal Grounding
Amazon

Mar 5, 2026J/K move · Enter open · S save

ORMOT: A Dataset and Framework for Omnidirectional Referring Multi-Object Tracking

Zihan Zhou, Sijia Chen, Yanqiu Yu, En Yu+1

National Key Laboratory of Science and Technology on Multi-spectral Information Processing, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan 430074, China

Visual Object Tracking Multi-Object Tracking

Open paper AskPDF arXiv

Omnidirectional cameras provide 360° spatial coverage, making them increasingly valuable in applications such as autonomous driving and video surveillance. While Multi-Object Tracking (MOT) and its language-guided extension, Referring Multi-Object Tracking (RMOT), have achieved notable progress, existing methods rely on conventional cameras with limited fields of view, causing critical contextual cues to be lost when targets move outside the frame. This fundamentally limits the model's ability to interpret long-horizon language descriptions involving sequential actions, spatial relations, and group behaviors. In this work, we propose Omnidirectional Referring Multi-Object Tracking (ORMOT), a novel task extending RMOT to omnidirectional imagery, where 360° coverage ensures complete scene context for accurate language-guided tracking. To advance this task, we construct ORSet, a dataset comprising 27 omnidirectional scenes, 848 language descriptions, and 3,401 annotated objects. Furthermore, we propose ORTrack, an LVLM-driven framework that enables zero-shot language-guided detection and robust cross-frame association in complex 360° environments. Experiments on ORSet demonstrate that ORTrack achieves state-of-the-art performance, providing a strong baseline for future research. The dataset and code will be open-sourced at https://github.com/chen-si-jia/ORMOT.

Abstract

Explore similar work

ORMOT: A Dataset and Framework for Omnidirectional Referring Multi-Object Tracking

RT-RMOT: A Dataset and Framework for RGB-Thermal Referring Multi-Object Tracking

STORM: End-to-End Referring Multi-Object Tracking in Videos