Efficient Language-to-Vision Feature Injection for Referring Single-Object Tracking

Han Wang; Yuxuan Liu; Yuhan Sun; Jian Yang; Xiaotong Xu; Yixuan Lv; Zhuang Zhou; Shengyang Li

cs.CVAug 29, 2026

Efficient Language-to-Vision Feature Injection for Referring Single-Object Tracking

Authors: Han Wang, Yuxuan Liu, Yuhan Sun, Jian Yang, Xiaotong Xu, Yixuan Lv, Zhuang Zhou, Shengyang Li

Organizations: Technology and Engineering Center for Space Utilization, Chinese Academy of Sciences · University of Chinese Academy of Sciences

Abstract

Referring single-object tracking enables language-grounded target initialization and subsequent tracking by jointly leveraging semantic cues and visual templates. The core difficulty is to use language differently across stages: it is indispensable for grounding but can induce semantic drift during tracking when overemphasized. Meanwhile, current methods often require costly vision-language alignment training. We present LVTrack, a pure transformer framework that introduces a mode-conditioned Gated Feature Injector to adaptively regulate textual guidance and alleviate semantic drift. Together with targeted adaptations, it directly harnesses a frozen vision-language pretrained model, greatly reducing training cost and preserving strong language understanding. To further improve temporal localization, LVTrack integrates hybrid relative-absolute positional encodings with a lightweight memory mechanism and optimizes autoregressive box prediction using a Gaussian-smoothed KL loss. Extensive experiments on standard benchmarks demonstrate that LVTrack achieves strong performance.

Figures & tables

Figure 1: Architectural comparison between LVTrack and prior frameworks. LVTrack performs discrete-token autoregressive box prediction and relies on a fully frozen VLP backbone, injecting textual cues into visual features via a lightweight injector. Leveraging the pretrained backbone obviates costly vision–language alignment training, substantially reducing training cost.

Figure 2: Framework of proposed LVTrack. LVTrack is an end-to-end vision-language tracker that leverages a frozen VLP backbone and MGFI to enable multimodal reasoning. A position-enhanced encoder models template and search tokens, while a masked decoder autoregressively predicts discretized tokens for the object box’s relative coordinates. (Residual connections in MGFI are omitted for clarity.)

Figure 3: Comparison between cross-entropy and the proposed GS-KL loss for discretized coordinate prediction. GS-KL replaces the one-hot target with a Gaussian-smoothed soft target, assigning higher probability mass to bins closer to the ground truth and making token classification distance-aware.

Tracker	Initialize	TNL2K			LaSOT			OTB99
Tracker	Initialize	AUC	Prec	NPrec	AUC	Prec	NPrec	AUC	Prec	NPrec
TNL2K-II Wang et al. (2021b)	NL+BBOX	41.7	42.0	50.0	51.0	55.0	-	68.0	88.0	-
JointNLT Zhou et al. (2023)	NL+BBOX	56.9	58.1	73.6	60.4	63.6	69.4	65.3	85.6	79.5
QueryNLT Shao et al. (2024)	NL+BBOX	57.8	58.7	75.6	59.9	63.5	69.6	66.7	88.2	82.4
UVLTrack-B Ma et al. (2024)	NL+BBOX	63.1	66.7	-	69.4	75.9	-	69.3	89.9	-
DUTrack-256 Li et al. (2025)	NL+BBOX	64.9	70.6	82.9	73.0	81.1	83.8	70.9	93.9	-

Table 1: Comparison of our method with state-of-the-art approaches on TNL2k, LaSOT and OTB99 datasets. The best and second-best results are highlighted in bold and underlined respectively.

Metric	JointNLT	UVLTrack	SAVLT	Ours
Trainable parameters	153M	169M	129M	100M
Training epochs	300	300	300	80
Samples per epoch	60K	30K	60K	30K

Table 2: Comparison of training budget and efficiency.

Method	VLTVG	JointNLT	QueryNLT	UVLTrack	Ours
Acc.	72.98	70.07	72.00	73.86	74.88

Table 3: Visual grounding comparison on the RefCOCOg validation set.

Model	LaSOT	OTB99	RefCOCOg
LVTrack	58.7	63.3	74.88
w/o GS-KL Loss	56.4	59.6	70.92
w/o Amplitude Ratio Loss	58.3	60.2	71.46
w/o Dual Mode Fusion	57.7	58.7	71.14
w/o Memory Fusion	57.3	60.3	73.70
w/o 2D RoPE	55.3	58.1	71.72

Table 4: Ablation study on LaSOT (AUC), OTB99 (AUC), and RefCOCOg (Accuracy). Tracking is evaluated under the NL-initialized setting.

Figure 4: Comparison of GS-KL loss and cross-entropy loss across training epochs. The left panel reports grounding accuracy on RefCOCOg, and the right panel reports NL-initialized tracking performance on LaSOT.

Figure 5: Tracking visualizations under NL initialization. (a) shows the cattle-13 sequence and (b) shows the cat-3 sequence from LaSOT.

Figure 6: Feature map visualizations as language is progressively injected. The panel includes the input image and feature-magnitude heatmaps from layers 5, 7, and 10 of the visual backbone.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Experiment	Parameter	LaSOT	OTB99	RefCOCOg
ViT layers	9	56.3	60.8	71.18
	10	58.7	63.3	74.88
	11	56.6	58.7	71.64
Decay factor $\rho$ in $\mathcal{L}_{\text{ratio}}$	0.5	57.6	60.4	74.20
	0.6	58.7	63.3	74.88
	0.7	58.2	61.1	73.88

Appendix

Table 5: Hyperparameter analysis on LaSOT (AUC), OTB99 (AUC), and RefCOCOg (Accuracy). Tracking is evaluated under the NL-initialized setting. Data sampling ratio is reported as a three-number tuple, denoting the proportions of tracking, joint tracking, and grounding samples, respectively.

Backbone	RefCOCOg	LaSOT			OTB99
Backbone	Accuracy	AUC	Prec	NPrec	AUC	Prec	NPrec
PE Bolya et al. (2026)	74.88	58.7	61.9	68.3	63.3	82.9	75.9
SigLIP 2 Tschannen et al. (2025)	70.35	56.9	59.7	66.5	62.1	81.6	74.1
CLIP Radford et al. (2021)	70.86	56.7	58.8	66.0	61.6	80.3	73.7

Appendix

Table 6: Backbone comparison under an identical LVTrack setting on LaSOT (AUC), OTB99 (AUC), and RefCOCOg (Accuracy). We replace only the frozen VLP backbone (all base-scale) while keeping the architecture and training protocol unchanged.

Positional Encoding	RefCOCOg	LaSOT		OTB99
Positional Encoding	Accuracy	AUC	Prec	AUC	Prec
2D $\mathcal{P}_{\mathrm{abs}}$ , 2D $\mathcal{P}_{\mathrm{RoPE}}$	74.88	58.7	61.9	63.3	82.9
1D $\mathcal{P}_{\mathrm{abs}}$ , 2D $\mathcal{P}_{\mathrm{RoPE}}$	74.24	57.8	61.7	61.2	80.8
1D $\mathcal{P}_{\mathrm{abs}}$ , 1D $\mathcal{P}_{\mathrm{RoPE}}$	74.12	57.2	60.8	59.8	79.8

Appendix

Table 7: Ablation results of using different forms of positional encoding on RefCOCOg, LaSOT, and OTB99.

\mathrm{1D}\ \mathcal{P}_{\mathrm{abs}}

denotes a single 1D absolute positional embedding table with length equal to the total token sequence, i.e., the positional coordinates of template and search tokens are not defined under a shared coordinate system.

\mathrm{1D}\ \mathcal{P}_{\mathrm{RoPE}}

is defined analogously.

Tracker	Initialize	TNL2K			LaSOT			OTB99
Tracker	Initialize	AUC	Prec	NPrec	AUC	Prec	NPrec	AUC	Prec	NPrec
SiamRPN++ Li et al. (2019)	BBOX	41.3	41.2	48.0	49.6	49.1	56.9	-	-	-
AutoMatch Zhang et al. (2021)	BBOX	47.2	43.5	-	58.3	59.9	67.4	-	-	-
TrDiMP Wang et al. (2021a)	BBOX	52.3	52.8	-	63.9	61.4	-	-	-	-
TransT Chen et al. (2021)	BBOX	50.7	51.7	-	64.9	69.0	73.8	-	-	-
SwinTrack-B Lin et al. (2022)	BBOX	-	57.1	-	69.6	74.1	78.6	-	-	-

Appendix

Table 8: Comparison with state-of-the-art trackers under BBOX initialization on TNL2K, LaSOT, and OTB99. The best and second-best results are highlighted in bold and underlined, respectively.

Figure 7: Additional Visualizations under NL Initialization.

Visual Object Tracking

Visual object tracking follows objects across video frames, maintaining their locations and identities despite motion, occlusion, and changing appearance.

See all 171 papers

Dynamic Parsing and Updating Natural Language Specification using VLMs for Robust Vision-Language Tracking
Jun 28, 2026Xiao Wang, Liye Jin, Dan Xu +5Vision-Language Models Visual Object Tracking
School of Computer Science and Technology, Anhui University, Hefei 230601, China · School of Electronic and Information Engineering, Anhui University, Hefei 230601, China · Harbin Institute of Technology, Shenzhen, China +3
Learning to Track Instance from Single Nature Language Description
May 8, 2026Yaozong Zheng, Bineng Zhong, Qihua Liang +3Visual Object Tracking Vision-Language Alignment
Key Laboratory of Education Blockchain and Intelligent Technology, Ministry of Education Guangxi Normal University, Guilin 541004, China · University Engineering Research Center of Educational Intelligent Technology Guangxi Normal University, Guilin 541004, China · University of Southampton, Southampton, SO17 1BJ, United Kingdom
ReferTrack: Referring Then Tracking for Embodied Visual Tracking
Jul 22, 2026Hanjing Ye, Tianle Zeng, Jiazhao Zhang +6Visual Object Tracking Visual Grounding
1RCV Laboratory, SUSTech · 2Tencent Robotics X · 3Peking University +1

Jun 28, 2026J/K move · Enter open · S save

Dynamic Parsing and Updating Natural Language Specification using VLMs for Robust Vision-Language Tracking

Xiao Wang, Liye Jin, Dan Xu, Yuehang Li+4

School of Computer Science and Technology, Anhui University, Hefei 230601, China · School of Electronic and Information Engineering, Anhui University, Hefei 230601, China · Harbin Institute of Technology, Shenzhen, China+3

Vision-Language Models Visual Object Tracking

Open paper AskPDF arXiv

Vision-language tracking guided by natural language specifications leverages high-level semantic cues of target objects to substantially boost tracking accuracy and robustness. Existing studies have verified that adaptively optimizing textual descriptions throughout the tracking process can effectively mitigate the semantic-visual mismatch induced by dynamic variations in target appearance, position, and other inherent attributes. Nevertheless, mainstream methods that directly generate textual information via sequence models or large language models inevitably suffer from inherent defects, including erroneous target updating, excessive background distraction, and pervasive hallucination artifacts. To address the aforementioned limitations, this paper proposes a novel language dependency parsing mechanism to precisely distill core tracking principal components, encompassing target objects, semantic concepts, and background contextual information. On this basis, we perform component-aware adaptive textual description updates by exploiting the powerful cross-modal understanding capability of the pre-trained vision-language model Qwen-VL. By integrating the proposed elaborately designed modules into the baseline framework, our method achieves consistent and superior tracking performance on multiple large-scale vision-language tracking benchmarks, including TNL2K, LaSOT, TNLLT, and OTB-LANG. The source code and pre-trained models will be released at https://github.com/Event-AHU/Open_VLTrack.

Abstract

Visual Object Tracking

Explore similar work

Dynamic Parsing and Updating Natural Language Specification using VLMs for Robust Vision-Language Tracking

Learning to Track Instance from Single Nature Language Description

ReferTrack: Referring Then Tracking for Embodied Visual Tracking