VLX-Go: Robust Embodied Tracking through Multi-Stage Learning and On-Policy Refinement

Kyusong Lee; Heting Ying; Tiancheng Zhao

cs.CVOct 14, 2026

VLX-Go: Robust Embodied Tracking through Multi-Stage Learning and On-Policy Refinement

Authors: Kyusong Lee, Heting Ying, Tiancheng Zhao

Organizations: Om AI Research · Binjiang Institute of Zhejiang University

Abstract

Embodied visual tracking requires a robot to identify a designated target from visual observations and control its own motion to follow that target through an environment. Following an isolated person is a comparatively simple demonstration of embodied visual tracking, but following the intended person among distractors remains challenging. Existing methods exhibit a substantial drop in success from single-target to distractor tracking, motivating a focus on the shared spaces that person-following robots must navigate. We present VLX-GO, a vision-language waypoint policy built around the hypothesis that visual and language representations should be aligned before tracking-specific motion is learned. The policy combines complementary frozen visual encoders, multiscale temporal context, a transformer backbone, and a lightweight head for robot-centric waypoint prediction. Its training progresses through vision--language alignment, multimodal reasoning mid-training, and tracking specialization on 19,532 oracle episodes. Compared with 41.54% distractor-tracking success for our earlier OmTrackVLA model, the complete multi-stage recipe achieves 55.80%. We then explore on-policy refinement using filtered oracle labels for states visited by the policy. This preliminary study adds 2.42 percentage points of distractor-tracking success, reaching 58.22%, while reducing collision rate by 1.99 pts. The final policy records the highest distractor- and ambiguous-tracking success rates among the compared methods. These results support staged multimodal learning as a promising direction for tracking among distractors and motivate further study of on-policy refinement for person following in shared spaces. Code is available at https://github.com/om-ai-lab/VLX-Go.

Figures & tables

Figure 1: VLX-Go architecture and training workflow. (A) Frozen DINOv3 and SigLIP features are concatenated and pooled into 188 visual tokens: $31\times(2\times 2)$ for history and $8\times 8$ for the current frame. A vision–language adapter projects these features into a transformer backbone; temporal/view indicators, instruction tokens, and a learnable action query complete the input. A three-layer MLP decodes the action state into eight robot-centric waypoints. (B) Three offline stages train the adapter, then the backbone and adapter, and finally the backbone, adapter, and waypoint head. (C) DAgger collects policy rollouts, labels visited states with the oracle, filters the labels, and fine-tunes on aggregated rollouts and demonstrations. The updated policy supplies the next round of rollouts. Visual encoders remain frozen throughout. Scene and robot images are illustrative.

Stage	Objective	Training data	Parameter updates
1	Vision–language alignment	blip_laion_cc_sbu_558k alignment collection + human recognition dataset	Adapter only
2	Multimodal reasoning mid-training	Stage 1 alignment data + general-purpose VQA data	backbone + adapter
3	Tracking SFT from Stage 2 base	Oracle tracking collection	backbone + adapter + waypoint head
DAgger	On-policy fine-tuning from Stage 3	Filtered DAgger rollouts (20%) + oracle demonstrations (80%)	backbone + adapter + waypoint head

Table 1: Training workflow: three offline stages followed by on-policy DAgger fine-tuning (Section 6.3 ). Visual encoders remain frozen throughout; backbone updates in Stages 2 and 3 and DAgger use full-parameter fine-tuning.

Task	Episodes	Scenes
STT	6,906	675
DT	6,229	681
AT	6,397	684
Total	19,532	–

Table 2: Oracle episode collection used for tracking supervision in Stage 3.

	DT		AT
Method	SR $\uparrow$	CR $\downarrow$	SR $\uparrow$	CR $\downarrow$
IBVS $\dagger$ [ 4 ]	10.60	6.14	15.20	4.90
PoliFormer $\dagger$ [ 22 ]	2.62	44.50	3.04	41.50
EVT [ 27 ]	3.23	47.90	17.40	45.60
EVT $\ddagger$ [ 27 ]	15.70	53.30	18.30	44.90
Uni-NaVid (Vicuna-7B) [ 24 ]	11.30	43.50	8.26	43.70

Table 3: Comparison on the distractor and ambiguous tracking tasks of EVT-Bench. Scores for IBVS, PoliFormer, EVT, Uni-NaVid, and TrackVLA are taken from TrackVLA [ 19 ] . Both VLX-Go models are evaluated on 1,405 episodes per task.

	DT		AT
Model	SR $\uparrow$	CR $\downarrow$	SR $\uparrow$	CR $\downarrow$
Oracle (privileged)	86.74	5.17	84.92	5.00
VLX-Go (three-stage)	55.80	12.10	60.64	9.40
VLX-Go (three-stage + DAgger)	58.22	10.11	60.93	10.39

Table 4: Effect of DAgger, full DT and AT validation sets of EVT-Bench (1,405 episodes per task).

Task	$\Delta$ SR	$\Delta$ CR	$\Delta$ Successes	$\Delta$ Collisions
DT	+2.42	-1.99	+34	-28
AT	+0.29	+0.99	+4	+14

Table 5: Changes from VLX-Go (three-stage) to VLX-Go (three-stage + DAgger). Metric differences are percentage points computed from the displayed scores in Table 4 ; count differences are computed from the episode logs. Positive SR and negative CR indicate improvement.

Task	All collisions	Forward motion	Below threshold
DT	$\mathbf{170\rightarrow 142}$	$\mathbf{88\rightarrow 56}$	$82\rightarrow 86$
AT	$132\rightarrow 146$	$63\rightarrow 68$	$69\rightarrow 78$
Overall	$371\rightarrow 362$	$163\rightarrow 135$	$208\rightarrow 227$

Table 6: Collision counts before and after on-policy refinement. Each entry reports three-stage

\rightarrow

three-stage + DAgger. Forward: mean forward velocity

\geq 0.15

m/s before contact; below threshold: all remaining collisions.

Vision-Based Robot Control

Vision-based robot control uses camera images to guide robots’ navigation and manipulation through closed-loop perception and action.

See all 213 papers

ReferTrack: Referring Then Tracking for Embodied Visual Tracking
Jul 22, 2026Hanjing Ye, Tianle Zeng, Jiazhao Zhang +6Visual Object Tracking Visual Grounding
1RCV Laboratory, SUSTech · 2Tencent Robotics X · 3Peking University +1
USS: Unifying Spatial-Semantic Prompting for End to End Embodied Visual Tracking
Jun 24, 2026Yuchen Xie, Xinyu Zhou, Kuangji Zuo +4Visual Object Tracking Vision-Based Robot Control
Nanyang Technological University
UniTrackPLA: Unified Panorama-Language-Action Model for Instruction-Guided Navigation and Dynamic Person Tracking
Oct 1, 2026Pengfei Qi, Haoran Lin, Sizhuang Chen +7Visual Object Tracking Robot Navigation
College of Integrated Circuits, Hunan University, Changsha, China · School of Artificial Intelligence and Robotics and the National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University, Changsha, China · School of Advanced Technology, Xi’an Jiaotong-Liverpool University, Suzhou, China +1

Jul 22, 2026J/K move · Enter open · S save

ReferTrack: Referring Then Tracking for Embodied Visual Tracking

Hanjing Ye, Tianle Zeng, Jiazhao Zhang, Shaoan Wang+5

1RCV Laboratory, SUSTech · 2Tencent Robotics X · 3Peking University+1

Visual Object Tracking Visual Grounding VLMs for Robotics

Open paper AskPDF arXiv

Embodied visual tracking (EVT) requires a mobile agent to continuously follow a specific target described in natural language using only onboard vision. While recent vision-language-action (VLA) policies unify target identification and trajectory planning, their chain-of-thought (CoT) reasoning often operates in abstract spatial latents that are difficult to supervise and weakly aligned with explicit image-space detections. To address this, we introduce ReferTrack, a referring-then-tracking paradigm that grounds EVT using a single forward-facing camera. Our model first selects the target from an indexed set of bounding boxes, then decodes tracking waypoints conditioned on this image-grounded decision. To preserve target motion cues over time, ReferTrack maintains a sliding-window queue of previously selected bounding boxes, injecting their geometric features into the visual history via temporal-viewpoint-bbox indicator (TVBI) tokens. We further enhance target identification by co-training on a custom Refer-QA dataset. On EVT-Bench, ReferTrack achieves state-of-the-art single-view performance with success rates of 89.4%, 73.3%, and 74.1% on the single-target, distracted, and ambiguity tracking splits, respectively -- matching or even surpassing several multi-camera baselines on identification-heavy tasks. Finally, real-world deployments on legged and humanoid robots validate its robust sim-to-real transfer capabilities. Code is available at https://github.com/MedlarTea/referTrack.

Abstract

Vision-Based Robot Control

Explore similar work

ReferTrack: Referring Then Tracking for Embodied Visual Tracking

USS: Unifying Spatial-Semantic Prompting for End to End Embodied Visual Tracking

UniTrackPLA: Unified Panorama-Language-Action Model for Instruction-Guided Navigation and Dynamic Person Tracking