Language-guided embodied person tracking requires a mobile agent to identify a target from a referring description and consistently follow the target under egocentric observations. We present DobotTrack, a target-grounded vision-language-action (VLA) framework that integrates auxiliary grounding supervision with efficient temporal modeling. We systematically investigate five auxiliary tasks---person segmentation, bounding-box grounding, visual-spatial classification (VSC), future DINO feature prediction, and future-frame prediction---and adopt complementary mask and bounding-box supervision to enhance target delineation and localization. We introduce Q-identity, a target-conditioned historical visual compression module that leverages target-grounding predictions to selectively preserve identity-discriminative appearance features and spatial cues in compact historical tokens. On EVT-Bench, our 0.8B-parameter model achieves an average success rate of 77.9%, exceeding OmTrackVLA by 16.9 percentage points across the three tasks. We further construct a real-world person-tracking dataset collected using a quadruped robot, comprising 126 episodes and 596,323 mask-annotated frames. The target masks enable automatic derivation of bounding boxes and visual-spatial labels encoding horizontal position and apparent image scale, providing additional grounding supervision without separate manual annotation of these labels. Together, our framework and annotation pipeline support learning target-grounded tracking policies from simulated and real-world data. Code is available at https://github.com/Dobot-Team/dobot_track.
Figures & tables
Fig. 1: Challenges in language-guided embodied person tracking. (a) Distractors and occlusion complicate target grounding in the robot’s limited field of view. (b) Dense visual histories increase inference cost, while indiscriminate pooling can discard target-specific cues. (c) Median inference latency on an NVIDIA Jetson Orin Nano (FP16, batch size 1, including the visual encoder; 90 measurements per point) increases with the total input token count. Reducing the token count from 2048 to 256 lowers latency by approximately 52%, motivating target-conditioned history compression for on-device deployment.
Fig. 2: Overview of DobotTrack and Q-identity. Left: the VLA processes a target description, current-frame visual tokens, and compressed historical tokens to predict target grounding and future waypoints. Target-grounding predictions guide historical visual compression for subsequent decisions. Right: Q-identity aggregates target-region features and full-image context through joint query–target self-attention, full-image cross-attention, and feed-forward updates. The resulting compact historical tokens are fed back into the VLA’s visual input.
Fig. 3: Overview of the real-world tracking dataset. (A) Dataset composition. The dataset contains approximately 10.2 hours of real-world human-tracking data collected in indoor and outdoor environments under both STT and DT conditions. The annotations include RGB observations, target masks, bounding boxes, structured VSC labels, and future robot trajectories. The dataset contains 33 target appearance profiles, including 24 female and 9 male profiles. (B) Representative qualitative examples from indoor STT, outdoor STT, and outdoor DT sequences. Each example shows the natural-language instruction describing the target and two frames from the same tracking sequence. Green overlays denote the target segmentation masks, green rectangles indicate the corresponding bounding boxes, and the blue curve visualizes the 8-step ground-truth future robot trajectory.
Method
Size
RL
STT
DT
AT
SR↑
TR↑
CR↓
SR↑
TR↑
CR↓
SR↑
TR↑
CR↓
Multi-camera references
ABot-N0 ∗ [ 23 ]
4B
✗
86.9
87.6
8.5
66.7
75.4
11.6
67.3
79.5
7.1
CoMaTrack ∗ [ 24 ]
3B
✓
92.1
90.3
0.9
74.2
80.5
2.1
57.5
73.4
12.0
NavFoM ∗ [ 25 ]
7B
✗
88.4
80.7
–
62.0
67.9
–
–
–
–
TrackVLA++ [ 5 ]
7B
✗
90.9
82.7
1.5
74.0
73.7
3.5
55.9
63.8
15.1
TABLE I: EVT-Bench results (%). Multi-camera methods are external references and excluded from single-view ranking. Results and protocol labels follow the cited papers and the comparisons in ReferTrack and CoMaTrack. RL: ✓ indicates reinforcement learning, ✗ indicates no reinforcement learning; dashes denote unavailable entries. ∗ : co-trained with general navigation data; † : GroundingDINO detector; ‡ : SoM + GPT-4o perception. DobotTrack uses mask and box supervision, simulation and real-world training data, and 16 historical tokens per frame.
Auxiliary task
DT
SR↑
TR↑
CR↓
Trajectory only
55.9
83.6
16.5
+ Box
59.6
84.6
14.7
+ VSC
59.6
83.8
13.1
+ Mask
61.3
81.5
12.0
+ Box & Mask
63.7
84.5
13.2
TABLE II: Auxiliary-task ablation on DT (%) with the 0.8B model. All variants include trajectory supervision.
Auxiliary tasks
Target conditioned
STT
DT
AT
SR↑
TR↑
CR↓
SR↑
TR↑
CR↓
SR↑
TR↑
CR↓
Mask
✗
79.2
90.6
13.5
55.3
79.0
22.5
57.7
84.0
17.6
Mask
✓
84.8
90.0
4.5
67.6
82.0
10.3
65.6
83.6
7.6
Mask & Box
✓
87.6
91.4
5.6
69.0
83.4
13.1
67.8
85.6
11.5
TABLE III: Ablation of auxiliary supervision and target conditioning with the 0.8B model and 16-token Q-identity (%). The first two rows isolate target conditioning under mask supervision. Bold marks the highest SR values.
Compression
Tokens
STT
DT
AT
Avg. SR ↑
P95 latency (ms) ↓
SR↑
TR↑
CR↓
SR↑
TR↑
CR↓
SR↑
TR↑
CR↓
Pooling
256
86.6
89.7
4.7
61.5
77.5
13.2
59.7
82.4
12.7
69.3
203.9
Pooling
16
84.6
88.6
6.8
61.3
81.5
12.0
57.0
81.9
11.5
67.6
91.0
Q-identity
16
84.8
90.0
4.5
67.6
82.0
10.3
65.6
83.6
7.6
72.7
93.4
TABLE IV: Representation and token-budget comparison with the 0.8B mask-supervised model. Avg. SR is the mean over the three tasks. Bold marks the highest SR values. We report the 95th-percentile (P95) inference latency measured on Orin Nano.
Fig. 4: Real-world robotic platform and onboard sensors.
Fig. 5: Qualitative tracking comparisons in simulation. Rows show DT, STT, and AT episodes. Left: trajectory-only baseline observations from the start through occlusion to tracking failure. Right: DobotTrack observations at the baseline failure time and subsequent timesteps. Timestamps indicate elapsed episode time; TR values are episode-level results for the illustrated runs.
Fig. 6: Test on real-world system. DobotTrack follows the designated person in an outdoor pedestrian scene; the predicted mask, bounding box, and egocentric trajectory are overlaid on the robot-centric view.
Fig. 7: Real-world qualitative comparison between DobotTrack and the trajectory-only baseline in two challenging target-tracking sequences. For each sequence, the upper and lower rows show our method and the baseline, respectively, with frames sampled at 0.4 s intervals. Green masks and yellow bounding boxes indicate target segmentation and localization, while cyan curves represent predicted egocentric trajectories. All visualized predictions are generated by the models. The examples illustrate improved target localization and trajectory prediction under partial visibility and temporary target disappearance.
Training data
STT ↑
DT ↑
AT ↑
Simulation only
87.6
69.0
67.8
Simulation + real
90.2
72.5
70.9
TABLE V: Effect of real-world training data on EVT-Bench SR (%).
Embodied visual tracking (EVT) requires a mobile agent to continuously follow a specific target described in natural language using only onboard vision. While recent vision-language-action (VLA) policies unify target identification and trajectory planning, their chain-of-thought (CoT) reasoning often operates in abstract spatial latents that are difficult to supervise and weakly aligned with explicit image-space detections. To address this, we introduce ReferTrack, a referring-then-tracking paradigm that grounds EVT using a single forward-facing camera. Our model first selects the target from an indexed set of bounding boxes, then decodes tracking waypoints conditioned on this image-grounded decision. To preserve target motion cues over time, ReferTrack maintains a sliding-window queue of previously selected bounding boxes, injecting their geometric features into the visual history via temporal-viewpoint-bbox indicator (TVBI) tokens. We further enhance target identification by co-training on a custom Refer-QA dataset. On EVT-Bench, ReferTrack achieves state-of-the-art single-view performance with success rates of 89.4%, 73.3%, and 74.1% on the single-target, distracted, and ambiguity tracking splits, respectively -- matching or even surpassing several multi-camera baselines on identification-heavy tasks. Finally, real-world deployments on legged and humanoid robots validate its robust sim-to-real transfer capabilities. Code is available at https://github.com/MedlarTea/referTrack.
Hanjing Ye, Tianle Zeng, Jiazhao Zhang +6
1RCV Laboratory, SUSTech · 2Tencent Robotics X · 3Peking University +1
Embodied visual tracking requires a robot to identify a designated target from visual observations and control its own motion to follow that target through an environment. Following an isolated person is a comparatively simple demonstration of embodied visual tracking, but following the intended person among distractors remains challenging. Existing methods exhibit a substantial drop in success from single-target to distractor tracking, motivating a focus on the shared spaces that person-following robots must navigate. We present VLX-GO, a vision-language waypoint policy built around the hypothesis that visual and language representations should be aligned before tracking-specific motion is learned. The policy combines complementary frozen visual encoders, multiscale temporal context, a transformer backbone, and a lightweight head for robot-centric waypoint prediction. Its training progresses through vision--language alignment, multimodal reasoning mid-training, and tracking specialization on 19,532 oracle episodes. Compared with 41.54% distractor-tracking success for our earlier OmTrackVLA model, the complete multi-stage recipe achieves 55.80%. We then explore on-policy refinement using filtered oracle labels for states visited by the policy. This preliminary study adds 2.42 percentage points of distractor-tracking success, reaching 58.22%, while reducing collision rate by 1.99 pts. The final policy records the highest distractor- and ambiguous-tracking success rates among the compared methods. These results support staged multimodal learning as a promising direction for tracking among distractors and motivate further study of on-policy refinement for person following in shared spaces. Code is available at https://github.com/om-ai-lab/VLX-Go.
Kyusong Lee, Heting Ying, Tiancheng Zhao
Om AI Research · Binjiang Institute of Zhejiang University
Embodied Visual Tracking (EVT) requires an agent to continuously follow a designated target while moving through dynamic environments. Existing embodied tracking methods generally rely on either an implicit target-selection convention or a language description, leaving the target-specification interface largely fixed. However, different tracking scenarios naturally favor different forms of target specification: language can specify a target outside the robot's current view, whereas spatial prompts provide direct instance designation for visible targets and can be useful for selecting among similar looking people, designating hard-to-describe individuals, or specifying a target under time pressure. We therefore introduce unified spatial-semantic prompting for EVT, in which text, a point, a box, and a mask serve as complementary target specifications that can be selected according to different scenarios, and present USS, an end-to-end architecture that maps any of them to egocentric waypoints. A modality-specific prompt encoder feeds a common design comprising hybrid-attention fusion, temporal memory, cross-view aggregation, latent prediction, and waypoint decoding, with one policy instance trained for each interface under an identical recipe. Across 320 zero-shot real-robot trials with policies trained only in simulation, spatial prompts perform comparably to language in ordinary tracking scenarios while providing clear benefits when visually similar targets require precise instance designation, supporting our premise that different scenarios favor different target specifications. On simulation EVT-Bench under the standard language-prompt protocol, USS obtains the highest success rate among non-MLLM methods at 57 FPS, against the 4.8-10 FPS reported by MLLM trackers that are stronger on several metrics. Project site: https://arescheah.github.io/uss-project-page/.