Language-guided embodied person tracking requires a mobile agent to identify a target from a referring description and consistently follow the target under egocentric observations. We present DobotTrack, a target-grounded vision-language-action (VLA) framework that integrates auxiliary grounding supervision with efficient temporal modeling. We systematically investigate five auxiliary tasks---person segmentation, bounding-box grounding, visual-spatial classification (VSC), future DINO feature prediction, and future-frame prediction---and adopt complementary mask and bounding-box supervision to enhance target delineation and localization. We introduce Q-identity, a target-conditioned historical visual compression module that leverages target-grounding predictions to selectively preserve identity-discriminative appearance features and spatial cues in compact historical tokens. On EVT-Bench, our 0.8B-parameter model achieves an average success rate of 77.9%, exceeding OmTrackVLA by 16.9 percentage points across the three tasks. We further construct a real-world person-tracking dataset collected using a quadruped robot, comprising 126 episodes and 596,323 mask-annotated frames. The target masks enable automatic derivation of bounding boxes and visual-spatial labels encoding horizontal position and apparent image scale, providing additional grounding supervision without separate manual annotation of these labels. Together, our framework and annotation pipeline support learning target-grounded tracking policies from simulated and real-world data. Code is available at https://github.com/Dobot-Team/dobot_track.
Figures & tables
Fig. 1: Challenges in language-guided embodied person tracking. (a) Distractors and occlusion complicate target grounding in the robot’s limited field of view. (b) Dense visual histories increase inference cost, while indiscriminate pooling can discard target-specific cues. (c) Median inference latency on an NVIDIA Jetson Orin Nano (FP16, batch size 1, including the visual encoder; 90 measurements per point) increases with the total input token count. Reducing the token count from 2048 to 256 lowers latency by approximately 52%, motivating target-conditioned history compression for on-device deployment.
Fig. 2: Overview of DobotTrack and Q-identity. Left: the VLA processes a target description, current-frame visual tokens, and compressed historical tokens to predict target grounding and future waypoints. Target-grounding predictions guide historical visual compression for subsequent decisions. Right: Q-identity aggregates target-region features and full-image context through joint query–target self-attention, full-image cross-attention, and feed-forward updates. The resulting compact historical tokens are fed back into the VLA’s visual input.
Fig. 3: Overview of the real-world tracking dataset. (A) Dataset composition. The dataset contains approximately 10.2 hours of real-world human-tracking data collected in indoor and outdoor environments under both STT and DT conditions. The annotations include RGB observations, target masks, bounding boxes, structured VSC labels, and future robot trajectories. The dataset contains 33 target appearance profiles, including 24 female and 9 male profiles. (B) Representative qualitative examples from indoor STT, outdoor STT, and outdoor DT sequences. Each example shows the natural-language instruction describing the target and two frames from the same tracking sequence. Green overlays denote the target segmentation masks, green rectangles indicate the corresponding bounding boxes, and the blue curve visualizes the 8-step ground-truth future robot trajectory.
Method
Size
RL
STT
DT
AT
SR↑
TR↑
CR↓
SR↑
TR↑
CR↓
SR↑
TR↑
CR↓
Multi-camera references
ABot-N0 ∗ [ 23 ]
4B
✗
86.9
87.6
8.5
66.7
75.4
11.6
67.3
79.5
7.1
CoMaTrack ∗ [ 24 ]
3B
✓
92.1
90.3
0.9
74.2
80.5
2.1
57.5
73.4
12.0
NavFoM ∗ [ 25 ]
7B
✗
88.4
80.7
–
62.0
67.9
–
–
–
–
TrackVLA++ [ 5 ]
7B
✗
90.9
82.7
1.5
74.0
73.7
3.5
55.9
63.8
15.1
TABLE I: EVT-Bench results (%). Multi-camera methods are external references and excluded from single-view ranking. Results and protocol labels follow the cited papers and the comparisons in ReferTrack and CoMaTrack. RL: ✓ indicates reinforcement learning, ✗ indicates no reinforcement learning; dashes denote unavailable entries. ∗ : co-trained with general navigation data; † : GroundingDINO detector; ‡ : SoM + GPT-4o perception. DobotTrack uses mask and box supervision, simulation and real-world training data, and 16 historical tokens per frame.
Auxiliary task
DT
SR↑
TR↑
CR↓
Trajectory only
55.9
83.6
16.5
+ Box
59.6
84.6
14.7
+ VSC
59.6
83.8
13.1
+ Mask
61.3
81.5
12.0
+ Box & Mask
63.7
84.5
13.2
TABLE II: Auxiliary-task ablation on DT (%) with the 0.8B model. All variants include trajectory supervision.
Auxiliary tasks
Target conditioned
STT
DT
AT
SR↑
TR↑
CR↓
SR↑
TR↑
CR↓
SR↑
TR↑
CR↓
Mask
✗
79.2
90.6
13.5
55.3
79.0
22.5
57.7
84.0
17.6
Mask
✓
84.8
90.0
4.5
67.6
82.0
10.3
65.6
83.6
7.6
Mask & Box
✓
87.6
91.4
5.6
69.0
83.4
13.1
67.8
85.6
11.5
TABLE III: Ablation of auxiliary supervision and target conditioning with the 0.8B model and 16-token Q-identity (%). The first two rows isolate target conditioning under mask supervision. Bold marks the highest SR values.
Compression
Tokens
STT
DT
AT
Avg. SR ↑
P95 latency (ms) ↓
SR↑
TR↑
CR↓
SR↑
TR↑
CR↓
SR↑
TR↑
CR↓
Pooling
256
86.6
89.7
4.7
61.5
77.5
13.2
59.7
82.4
12.7
69.3
203.9
Pooling
16
84.6
88.6
6.8
61.3
81.5
12.0
57.0
81.9
11.5
67.6
91.0
Q-identity
16
84.8
90.0
4.5
67.6
82.0
10.3
65.6
83.6
7.6
72.7
93.4
TABLE IV: Representation and token-budget comparison with the 0.8B mask-supervised model. Avg. SR is the mean over the three tasks. Bold marks the highest SR values. We report the 95th-percentile (P95) inference latency measured on Orin Nano.
Fig. 4: Real-world robotic platform and onboard sensors.
Fig. 5: Qualitative tracking comparisons in simulation. Rows show DT, STT, and AT episodes. Left: trajectory-only baseline observations from the start through occlusion to tracking failure. Right: DobotTrack observations at the baseline failure time and subsequent timesteps. Timestamps indicate elapsed episode time; TR values are episode-level results for the illustrated runs.
Fig. 6: Test on real-world system. DobotTrack follows the designated person in an outdoor pedestrian scene; the predicted mask, bounding box, and egocentric trajectory are overlaid on the robot-centric view.
Fig. 7: Real-world qualitative comparison between DobotTrack and the trajectory-only baseline in two challenging target-tracking sequences. For each sequence, the upper and lower rows show our method and the baseline, respectively, with frames sampled at 0.4 s intervals. Green masks and yellow bounding boxes indicate target segmentation and localization, while cyan curves represent predicted egocentric trajectories. All visualized predictions are generated by the models. The examples illustrate improved target localization and trajectory prediction under partial visibility and temporary target disappearance.
Training data
STT ↑
DT ↑
AT ↑
Simulation only
87.6
69.0
67.8
Simulation + real
90.2
72.5
70.9
TABLE V: Effect of real-world training data on EVT-Bench SR (%).