RAGTrack: Language-aware RGBT Tracking with Retrieval-Augmented Generation

Hao Li; Yuhao Wang; Wenning Hao; Pingping Zhang; Dong Wang; Huchuan Lu

cs.CVMar 4, 2026

RAGTrack: Language-aware RGBT Tracking with Retrieval-Augmented Generation

Authors: Hao Li, Yuhao Wang, Wenning Hao, Pingping Zhang, Dong Wang, Huchuan Lu

Organizations: College of Command and Control Engineering, Army Engineering University of PLA · School of Future Technology, Dalian University of Technology · School of Information and Communication Engineering, Dalian University of Technology

Abstract

RGB-Thermal (RGBT) tracking aims to achieve robust object localization across diverse environmental conditions by fusing visible and thermal infrared modalities. However, existing RGBT trackers rely solely on initial-frame visual information for target modeling, failing to adapt to appearance variations due to the absence of language guidance. Furthermore, current methods suffer from redundant search regions and heterogeneous modality gaps, causing background distraction. To address these issues, we first introduce textual descriptions into RGBT tracking benchmarks. This is accomplished through a pipeline that leverages Multi-modal Large Language Models (MLLMs) to automatically produce texual annotations. Afterwards, we propose RAGTrack, a novel Retrieval-Augmented Generation framework for robust RGBT tracking. To this end, we introduce a Multi-modal Transformer Encoder (MTE) for unified visual-language modeling. Then, we design an Adaptive Token Fusion (ATF) to select target-relevant tokens and perform channel exchanges based on cross-modal correlations, mitigating search redundancies and modality gaps. Finally, we propose a Context-aware Reasoning Module (CRM) to maintain a dynamic knowledge base and employ a Retrieval-Augmented Generation (RAG) to enable temporal linguistic reasoning for robust target modeling. Extensive experiments on four RGBT benchmarks demonstrate that our framework achieves state-of-the-art performance across various challenging scenarios. The source code is available https://github.com/IdolLab/RAGTrack.

Figures & tables

Figure 1: Comparison with different RGBT tracking paradigms. (a) Existing RGBT trackers suffer from inadequate appearance modeling, search redundancy, and modality gap. (b) Our RAGTrack introduces linguistic reasoning, dynamic token selection, and adaptive channel exchange.

Figure 2: Overall framework. Our method begins by tokenizing input texts and images with reasoning tokens. MTE then performs unified visual-language modeling, while ATF utilizes text-guided attention to dynamically select target-relevant tokens and enables adaptive channel exchange. Subsequently, CRM retrieves relevant contexts from a dynamic knowledge base for context-aware reasoning. Finally, the prediction head outputs tracking results, which are used by MLLMs to generate updated textual descriptions for following frames.

Method	Publication	Resolution	GTOT		RGBT210		RGBT234		LasHeR
Method	Publication	Resolution	MPR $\uparrow$	MSR $\uparrow$	PR $\uparrow$	SR $\uparrow$	MPR $\uparrow$	MSR $\uparrow$	PR $\uparrow$	NPR $\uparrow$	SR $\uparrow$
CMPP [ 63 ]	CVPR 2020	256 $\times$ 256	92.6	73.8	$-$	$-$	82.3	57.5	$-$	$-$	$-$
CAT [ 26 ]	ECCV 2020	256 $\times$ 256	88.9	71.7	79.2	53.3	80.4	56.1	45.0	39.5	31.4
APFNet [ 70 ]	AAAI 2022	256 $\times$ 256	90.5	73.7	$-$	$-$	82.7	57.9	50.0	43.9	36.2
CMD [ 75 ]	CVPR 2023	256 $\times$ 256	89.2	73.4	$-$	$-$	82.4	58.4	59.0	54.6	46.4
ViPT [ 85 ]	CVPR 2023	256 $\times$ 256	$-$	$-$	$-$	$-$	83.5	61.7	65.1	$-$	52.5

Table 1: Performance comparison on four RGBT tracking benchmarks. The best results are in bold.

Figure 4: Attribute-based evaluations on the LasHeR dataset.


Method	MPR $\uparrow$	MSR $\uparrow$
Baseline	87.9	64.5
w/ CRM*	89.1	65.0
w/ MTE+CRM*	91.1	66.7
w/ MTE+CRM	91.8	67.4
w/ MTE+CRM+ATF	93.8	69.5

Table 2: Ablation study of key components on RGBT234.

6	12	18	24	MPR $\uparrow$	MSR $\uparrow$
✓				92.2	67.9
✓	✓			92.5	68.5
✓	✓	✓		92.9	68.8
✓	✓	✓	✓	93.8	69.5

Table 3: Comparison of fusion positions on RGBT234.


Module	MPR $\uparrow$	MSR $\uparrow$	Params $\downarrow$
TBSI [ 17 ]	92.8	67.6	145.9M
BSI [ 15 ]	93.1	68.2	103.6M
DFM [ 42 ]	92.7	67.8	110.3M
ATF (Ours)	93.8	69.5	101.8M

Table 4: Comparison of fusion paradigms on RGBT234.


Module	MPR $\uparrow$	MSR $\uparrow$	Params $\downarrow$
Add	92.5	68.5	101.8M
Transformer	93.3	69.0	104.8M
Mamba	92.7	68.1	103.5M
Ours	93.8	69.5	101.8M

Table 5: Comparison of augmentation mechanisms on RGBT234.

Reasoning Token			Learnable Token
$N_{r}$	MPR $\uparrow$	MSR $\uparrow$	Length	MPR $\uparrow$	MSR $\uparrow$
1	93.8	69.5	0	92.8	68.4
2	93.5	69.4	2	93.8	69.5
4	93.2	68.9	4	92.6	68.7

Table 6: Comparison of token configurations on RGBT234.

$\mathbf{A}_{x2r}$	$\mathbf{A}_{x2h}$	$\mathbf{A}_{x2z}$	$\mathbf{A}_{x2x}$	MPR $\uparrow$	MSR $\uparrow$
✓				92.0	67.8
✓	✓			92.9	68.7
✓	✓	✓		93.7	69.0
✓	✓	✓	✓	93.8	69.5

Table 7: Comparison of attention scores on RGBT234.

Figure 5: Comparison with different hyper-parameters.

Figure 6: Visualization of attention maps.


Fusion Positions	MPR $\uparrow$	MSR $\uparrow$
$\left[{1,2,3,4}\right]$	92.4	67.5
$\left[{11,12,13,14}\right]$	92.8	67.9
$\left[{21,22,23,24}\right]$	93.1	68.7
$\left[{6,12,18,24}\right]$	93.8	69.5

Table 8: Comparison of different fusion positions.

Figure 7: Comparison with different fractions of missing text.

Figure 8: Visualization of dynamic token selection in ATF.

Figure 9: Qualitative results on the RGBT234 benchmark.

Figure 10: More qualitative results on the RGBT234 benchmark.

Explore similar work

RT-RMOT: A Dataset and Framework for RGB-Thermal Referring Multi-Object Tracking
Feb 25, 2026Yanqiu Yu, Zhifan Jin, Sijia Chen +4Visual Object Tracking Multi-Object Tracking
Huazhong University of Science and Technology, Wuhan, Hubei, China · South-Central Minzu University, Wuhan, Hubei, China
End-to-End Self-Supervised RGB-T Tracking without Modality Misleading
Sep 29, 2026Shenglan Li, Rui Yao, Kunyang Sun +4Multimodal Robustness Visual Object Tracking
School of Computer Science and Technology / School of Artificial Intelligence, China University of Mining and Technology, China. · Mine Digitization Engineering Research Center of the Ministry of Education, China. · University of Auckland, Auckland, New Zealand. +1
SARTM: Segment Any RGB Thermal Model with Language aided Distillation
May 4, 2025Dong Xing, Jinhe Zhang, Hang Yang +1Vision Foundation Model Adaptation Semantic Segmentation
Changchun Institute of Optics, Fine Mechanicsand Physics, Chinese Academy of Sciences University

Feb 25, 2026J/K move · Enter open · S save

RT-RMOT: A Dataset and Framework for RGB-Thermal Referring Multi-Object Tracking

Yanqiu Yu, Zhifan Jin, Sijia Chen, Tongfei Chu+3

Huazhong University of Science and Technology, Wuhan, Hubei, China · South-Central Minzu University, Wuhan, Hubei, China

Visual Object Tracking Multi-Object Tracking RL Fine-Tuning Multimodal Sensor Fusion

Open paper AskPDF arXiv

Referring Multi-Object Tracking has attracted increasing attention due to its human-friendly interactive characteristics, yet it exhibits limitations in low-visibility conditions, such as nighttime, smoke, and other challenging scenarios. To overcome this limitation, we propose a new RGB-Thermal RMOT task, named RT-RMOT, which aims to fuse RGB appearance features with the illumination robustness of the thermal modality to enable all-day referring multi-object tracking. To promote research on RT-RMOT, we construct the first Referring Multi-Object Tracking dataset under RGB-Thermal modality, named RefRT. It contains 388 language descriptions, 1,250 tracked targets, and 166,147 Language-RGB-Thermal (L-RGB-T) triplets. Furthermore, we propose RTrack, a framework built upon a multimodal large language model (MLLM) that integrates RGB, thermal, and textual features. Since the initial framework still leaves room for improvement, we introduce a Group Sequence Policy Optimization (GSPO) strategy to further exploit the model's potential. To alleviate training instability during RL fine-tuning, we introduce a Clipped Advantage Scaling (CAS) strategy to suppress gradient explosion. In addition, we design Structured Output Reward and Comprehensive Detection Reward to balance exploration and exploitation, thereby improving the completeness and accuracy of target perception. Extensive experiments on the RefRT dataset demonstrate the effectiveness of the proposed RTrack framework.

Abstract

Explore similar work

RT-RMOT: A Dataset and Framework for RGB-Thermal Referring Multi-Object Tracking

End-to-End Self-Supervised RGB-T Tracking without Modality Misleading

SARTM: Segment Any RGB Thermal Model with Language aided Distillation