RT-RMOT: A Dataset and Framework for RGB-Thermal Referring Multi-Object Tracking

Yanqiu Yu; Zhifan Jin; Sijia Chen; Tongfei Chu; En Yu; Liman Liu; Wenbing Tao

cs.CVFeb 25, 2026

RT-RMOT: A Dataset and Framework for RGB-Thermal Referring Multi-Object Tracking

Authors: Yanqiu Yu, Zhifan Jin, Sijia Chen, Tongfei Chu, En Yu, Liman Liu, Wenbing Tao

Organizations: Huazhong University of Science and Technology, Wuhan, Hubei, China · South-Central Minzu University, Wuhan, Hubei, China

Abstract

Referring Multi-Object Tracking has attracted increasing attention due to its human-friendly interactive characteristics, yet it exhibits limitations in low-visibility conditions, such as nighttime, smoke, and other challenging scenarios. To overcome this limitation, we propose a new RGB-Thermal RMOT task, named RT-RMOT, which aims to fuse RGB appearance features with the illumination robustness of the thermal modality to enable all-day referring multi-object tracking. To promote research on RT-RMOT, we construct the first Referring Multi-Object Tracking dataset under RGB-Thermal modality, named RefRT. It contains 388 language descriptions, 1,250 tracked targets, and 166,147 Language-RGB-Thermal (L-RGB-T) triplets. Furthermore, we propose RTrack, a framework built upon a multimodal large language model (MLLM) that integrates RGB, thermal, and textual features. Since the initial framework still leaves room for improvement, we introduce a Group Sequence Policy Optimization (GSPO) strategy to further exploit the model's potential. To alleviate training instability during RL fine-tuning, we introduce a Clipped Advantage Scaling (CAS) strategy to suppress gradient explosion. In addition, we design Structured Output Reward and Comprehensive Detection Reward to balance exploration and exploitation, thereby improving the completeness and accuracy of target perception. Extensive experiments on the RefRT dataset demonstrate the effectiveness of the proposed RTrack framework.

Figures & tables

Figure 1: The difference between RMOT and RT-RMOT. RMOT model takes RGB images as input, but pedestrian positions cannot be reliably obtained from RGB alone, often leading to tracking failure. RT-RMOT model leverages thermal (T) images to obtain pedestrian contours and locations, and combines them with crosswalk regions provided by RGB images, enabling precise identification and tracking of people described by language.

Figure 2: Data annotation process. The annotation process consists of three steps: (1) Target Pre-selection and Annotation; (2) GPT-Assisted Generation of Attribute Descriptions; (3) Human Verification and Refinement. First, visually similar targets are selected, and bounding boxes are annotated across the entire sequence. Then, annotated key frames and task instructions are provided to GPT to analyze the target’s category, scene, appearance, motion, and spatial position. Finally, GPT-generated attributes are integrated into initial descriptions, which is refined through multiple annotators’ review to produce the final language descriptions.

Panel of Figure 3 — Figure 3: Visualization results on the RefRT dataset. (a) The word cloud of the RefRT dataset contains a rich set of keywords. (b) The RefRT dataset covers diverse scenes, targets and attributes. (c) After normalizing the three-dimensional data of the language descriptions in the RefRT dataset, the broad distribution characteristics of the dataset are revealed.

Figure 4: Overall pipeline of RTrack. The RTrack framework consists of three modules: (1) Large-model perception module: performs inference detection of the target in the current frame based on the language description. (2) Trajectory Prediction Module: predicts the target box in the current frame based on the historical trajectory. (3) Identity Association Module: matches the trajectory box and the detection box to generate the target ID. Among them, the GSPO algorithm includes three aspects: (a) CAS strategy: constrains relative rewards to avoid gradient explosion. (b) Structured output reward: requires the model to format the output and limits the output length. (c) Comprehensive detection reward: completes the output encouragement and accurate output based on the IoU value.

Modality	Method	Venue	HOTA↑	DetA↑	AssA↑	DetRe↑	DetPr↑	AssRe↑	AssPr↑	LocA↑
RGB	TransRMOT [ 20 ]	CVPR 2023	8.69	2.57	29.96	3.01	14.46	30.73	85.49	79.63
	TempRMOT [ 26 ]	ArXiv 2024	8.19	1.86	36.23	2.04	16.68	39.28	75.39	77.48
	CRTracker [ 3 ]	AAAI2025	9.30	2.37	37.01	3.81	5.83	40.10	67.48	73.25
	YOLOX+ByteTrack+ iKUN [ 5 ]	CVPR 2024	2.32	0.29	19.86	0.29	12.71	21.18	61.45	69.70
	Qwen2.5-VL-3B [ 2 ]	ArXiv 2025	2.09	0.93	5.28	0.97	17.14	5.40	87.46	76.69
RGBT	DeformCAT [ 7 ] +SORT+iKUN	IEEE TMM	2.03	0.41	11.25	0.77	0.87	12.07	47.65	62.61

Table 1: Performance of many methods on the RefRT test set. ↑ indicates that higher scores are better. The best results are marked in bold .

Modality	Method	Size	RL Fine-tuning	HOTA↑	DetA↑	AssA↑	DetRe↑	DetPr↑	AssRe↑	AssPr↑	LocA↑
RGB	Qwen2.5-VL [ 2 ]	3B	✗	2.09	0.93	5.28	0.97	17.14	5.40	87.46	76.69
RGB	RTrack	3B	✓	12.49	11.05	15.25	16.14	23.95	16.18	79.39	73.90
RGBT	Qwen2.5-VL [ 2 ]	3B	✗	4.98	2.59	10.19	3.05	14.29	10.65	83.40	75.52
RGBT	RTrack	3B	✓	15.53	12.39	20.79	20.15	22.78	22.02	81.99	75.53

Table 2: Results of more experiments on the RefRT test set.↑ indicates that higher scores are better. The best results are marked in bold .

Module	Method	Size	RL Fine-tuning	HOTA↑	DetA↑	AssA↑	DetRe↑	DetPr↑	AssRe↑	AssPr↑	LocA↑
A. MLLMs	LLaVA [ 13 ]	7B	✗	1.13	0.29	5.32	8.84	0.28	7.32	19.47	57.69
	LLaVA-NeXT [ 12 ]	8B	✗	2.99	0.84	13.35	5.70	0.96	17.84	38.32	60.17
	InternVL2 [ 4 ]	8B	✗	0.82	0.18	4.77	10.77	0.18	5.71	31.96	60.10
	Qwen2.5-VL [ 2 ]	3B	✗	4.98	2.59	10.19	3.05	14.29	10.65	83.40	75.52
B. Training Strategy	w/o CAS	3B	✓	5.27	3.93	7.37	4.32	28.11	7.43	90.32	78.63
	w/o Struct. Reward	3B	✓	11.24	9.47	14.25	16.81	17.10	14.79	86.08	77.20

Table 3: Ablation study results of RTrack on the RefRT test set. Module A represent the untrained results, while Module B reflects the results after training. ↑ indicates higher is better. Best results are in bold .

Figure 5: Qualitative zero-shot results of RTrack on the test set of RT-RMOT task.

Dataset	Source	Videos	Scene	Desc.	Instance	Instance/Desc.	Bbox	Multi-Object	RGBT	Real-world
RGBT234	IJCV2019	234	Urban, campus	234	–	–	234	×	✓	✓
GTOT	CVPR2016	50	Urban, campus	–	–	–	15.8k	×	✓	✓
RGBT-Tiny	arXiv 2024	115	Port, urban roads	–	–	–	1.2M	×	✓	✓
Refer-KITTI	CVPR2023	18	Urban, road mixed	818	8.7k	10.7	0.36M	✓	×	✓
Refer-KITTI-V2	arXiv 2024	21	Urban, highways, etc.	9.8k	65.4k	607	3.06M	✓	×	✓
Refer-Dance	CVPR2024	65	Mainly indoor	1.9k	650	0.34	0.55M	✓	×	✓

Table 4: Comparison with Existing Datasets. The Desc. indicates the number of language descriptions, and Instance/Desc. represents the average number of target instances per description. Check marks indicate whether a dataset supports Multi-Object Tracking, includes RGBT modalities data, and contains real-world scenarios. The comparison results indicate that the RefRT dataset fulfills the combined requirements of RGBT modalities inputs, diverse object categories, and comprehensive semantic annotations.

Prompt Design	HOTA↑	DetA↑	AssA↑	DetRe↑	DetPr↑	AssRe↑	AssPr↑	LocA↑
Short Prompt	13.92	11.99	17.42	17.92	24.75	18.30	82.87	75.64
Description-First Prompt	13.00	11.10	16.25	20.21	18.75	17.05	81.68	75.65
No Coordinate Example Prompt	16.29	12.26	22.90	17.95	25.85	24.49	81.25	75.35
Base Prompt (Ours)	15.53	12.39	20.79	20.15	22.78	22.02	81.99	75.53

Table 5: Ablation study on different prompt designs in RTrack.

Batch Size	Epochs	Learning Rate	LoRA- $r$	LoRA- $\alpha$	LoRA Dropout	Generation Num	Max Len	Beta	Epsilon	$Scale_{\max}$	$L_{\min}$	$L_{\max}$	$L_{low}$	$L_{high}$	$\alpha$	$\beta$	$\gamma$	$\lambda$
1	1	$1\times 10^{-5}$	8	16	0.05	4	512	0.001	$1\times 10^{-3}$	3.0	80	600	140	200	0.5	1	0.5	2

Table 6: Hyperparameter configuration used in the RL fine-tuning process.

Figure 6: More qualitative results of our RTrack framework on the test set of the RefRT dataset . It is worth noting that our framework can perform long-term tracking, rather than tracking only in the initial frame.

Figure 7: Ground-Truth Results in the RefRT dataset provide diverse, RGBT aligned real-world data.

Explore similar work

RAGTrack: Language-aware RGBT Tracking with Retrieval-Augmented Generation
Mar 4, 2026Hao Li, Yuhao Wang, Wenning Hao +3Visual Object Tracking Multimodal RAG
College of Command and Control Engineering, Army Engineering University of PLA · School of Future Technology, Dalian University of Technology · School of Information and Communication Engineering, Dalian University of Technology
DRMOT: A Dataset and Framework for RGBD Referring Multi-Object Tracking
Feb 4, 2026Sijia Chen, Shaofeng Liang, Lijuan Ma +7Multi-Object Tracking Multimodal Large Language Models
Huazhong University of Science and Technology · South-Central Minzu University
STORM: End-to-End Referring Multi-Object Tracking in Videos
Apr 12, 2026Zijia Lu, Jingru Yi, Jue Wang +4Multi-Object Tracking Multimodal Grounding
Amazon

Mar 4, 2026J/K move · Enter open · S save

RAGTrack: Language-aware RGBT Tracking with Retrieval-Augmented Generation

Hao Li, Yuhao Wang, Wenning Hao, Pingping Zhang+2

College of Command and Control Engineering, Army Engineering University of PLA · School of Future Technology, Dalian University of Technology · School of Information and Communication Engineering, Dalian University of Technology

Visual Object Tracking Multimodal RAG

Open paper AskPDF arXiv

RGB-Thermal (RGBT) tracking aims to achieve robust object localization across diverse environmental conditions by fusing visible and thermal infrared modalities. However, existing RGBT trackers rely solely on initial-frame visual information for target modeling, failing to adapt to appearance variations due to the absence of language guidance. Furthermore, current methods suffer from redundant search regions and heterogeneous modality gaps, causing background distraction. To address these issues, we first introduce textual descriptions into RGBT tracking benchmarks. This is accomplished through a pipeline that leverages Multi-modal Large Language Models (MLLMs) to automatically produce texual annotations. Afterwards, we propose RAGTrack, a novel Retrieval-Augmented Generation framework for robust RGBT tracking. To this end, we introduce a Multi-modal Transformer Encoder (MTE) for unified visual-language modeling. Then, we design an Adaptive Token Fusion (ATF) to select target-relevant tokens and perform channel exchanges based on cross-modal correlations, mitigating search redundancies and modality gaps. Finally, we propose a Context-aware Reasoning Module (CRM) to maintain a dynamic knowledge base and employ a Retrieval-Augmented Generation (RAG) to enable temporal linguistic reasoning for robust target modeling. Extensive experiments on four RGBT benchmarks demonstrate that our framework achieves state-of-the-art performance across various challenging scenarios. The source code is available https://github.com/IdolLab/RAGTrack.

Abstract

Explore similar work

RAGTrack: Language-aware RGBT Tracking with Retrieval-Augmented Generation

DRMOT: A Dataset and Framework for RGBD Referring Multi-Object Tracking

STORM: End-to-End Referring Multi-Object Tracking in Videos