WayTrace: A Map-to-Language-to-Trace Hierarchy for Embodied Urban Navigation
Organizations: Robotic Systems Lab (RSL), ETH Zurich, Switzerland · UC Berkeley, Berkeley, USA · NVIDIA, Zurich, Switzerland
Abstract
Web services provide walking directions for nearly every city, but following them reliably over long routes remains challenging for autonomous robots. Point-goal navigation policies rely heavily on accurate GNSS, while most vision-language navigation models struggle to execute even short instructions outdoors. We present WayTrace, a hierarchical system that connects web-map directions to robot motion through language. An off-the-shelf orchestrator vision-language model reads the turn-by-turn directions, a satellite map and recent camera images, and issues one atomic language command at a time, such as "cross the road to the left". An onboard trace predictor, trained on over one million automatically labelled samples from street-level images, draws the path for this command in the image, and a reinforcement learning policy tracks it. In long-range simulated missions, WayTrace remains robust to GNSS noise and rarely leaves crosswalks or sidewalks, while point-goal baselines degrade sharply with about ten times more violations. Deployed on a wheeled-legged robot, WayTrace completes real-world missions of up to 1.37 km at about 1.1 m/s through a busy city centre, rural trails and a university campus, handling crosswalks, staircases and construction sites. Project page: https://way-trace.github.io
Figures & tables
| Method | Guidance | Low level | SR [%] |
| CityWalker [ 2 ] | GNSS waypoints | released policy | 20.0 |
| VAMOS [ 3 ] | GNSS waypoints | released policy | 13.3 |
| OmniVLA [ 4 ] | GNSS waypoints | released policy | 0.0 |
| TIC-VLA [ 10 ] | tuned oracle | released policy | 6.7 |
| Foresight [ 11 ] | tuned oracle | path RL policy | 3.3 |
| VENTURA [ 13 ] | tuned oracle | path RL policy | 6.7 |
| Mission | Type | Characteristics | Length [m] | Speed [m/s] | Commands | Interventions | Autonomy [%] |
|---|---|---|---|---|---|---|---|
| Starbucks | city centre | stairs, crosswalks, tram tracks, construction sites | 426 | 1.12 | 26 | 0 | 100 |
| Snipes | city centre | crowded streets, construction site, crosswalk | 248 | 1.01 | 13 | 0 | 100 |
| Dieci | city centre | footbridge, tram crossing, sidewalks, crowded streets | 312 | 1.11 | 19 | 0 | 100 |
| Mendokoro | rural trail, campus | road, field paths, forest trails, plaza, ramp | 1368 | 1.19 | 21 | 2 | 97.2 |
| Human-Test (hand-labelled, 953 samples) | Walks-Test (auto-labelled, 31,012 samples) | |||||||||||
| NT | ADE | Side | NT | ADE | Side | |||||||
| Method | Full | Hor. | Full | Hor. | Full | Hor. | Full | Hor. | Full | Hor. | Full | Hor. |
| Ours | 83.2 | 85.8 | 5.0 | 4.0 | 97.9 | 98.3 | 70.4 | 72.5 | 8.6 | 7.5 | 92.6 | 94.8 |
| CityWalker † | 57.0 | 66.2 | 13.9 | 10.0 | 90.4 | 91.2 | 50.9 | 60.7 | 15.8 | 11.4 | 87.3 | 87.2 |
| VAMOS † | 56.6 | 42.6 | 16.4 | 19.8 | 97.7 | 95.5 | 48.8 | 33.7 | 18.5 | 22.3 | 92.5 | 91.6 |
| Foresight | 50.5 | 68.3 | 15.0 | 8.9 | 77.4 | 90.5 | 41.8 | 60.1 | 17.3 | 10.9 | 75.1 | 89.2 |
| Critical [%] | |||||
|---|---|---|---|---|---|
| Orchestrator VLM | all | 0 m | 5 m | s/call | $/1k |
| Qwen3.8-Flash-Next (ours) | 5.1 | 4.3 | 6.1 | 1.4 | — |
| Gemini 3.5 Flash-Lite | 5.9 | 4.0 | 8.2 | 5.1 | 2.9 |
| Gemini 3.8 Flash | 3.2 | 3.2 | 3.3 | 6.5 | 7.4 |
| GPT-6 Sol | 3.2 | 1.5 | 5.4 | 4.5 | 7.1 |
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Rule | Behaviour |
|---|---|
| Crossing rejection | Rejects a crossing command while the next crossing entry is more than 10 m ahead or on the other side. |
| Arrival rejection | Rejects “Go to the <destination>” while the destination is more than 15 m ahead. |
| Arrival lock | No further calls once the final command is accepted. |
| Turn display | A passed turn stays the next direction until the heading has turned by up to 45 ∘ or the robot is 15 m past it. |
| Command expiry | A command expires 12 s after its image capture, and the robot holds after 20 s without a valid command. |
| Off-route | More than 20 m from the route ends the run in simulation and pauses the robot. |
| Start | Destination | [m] | Manoeuvres | Zebras | Limit | |
|---|---|---|---|---|---|---|
| M1 | Neumarkt | Rheinfelder Bierhaus | 152 | 1 turn | 0 | 669 |
| M2 | Bellevue | Karl der Grosse | 223 | 1 turn | 0 | 858 |
| M3 | Mühlesteg | dieci pizza takeaway | 328 | 4 turns, 2 crossings | 2 | 1119 |
| M4 | Mühlesteg | Burger King | 361 | 2 turns, 3 crossings | 3 | 1195 |
| M5 | Seilergraben | Burger King | 393 | 1 fork, 2 crossings, stairs down | 3 | 1275 |
| M6 | Seilergraben | Café Zähringer | 406 | 2 crossings | 2 | 1308 |
| SR [ ] | Crosswalk viol. [ ] | Road viol. [ ] | |
| Method | [%] | [%] | [%] |
| CityWalker | 20.0 | 20.8 | 1.63 |
| VAMOS | 13.3 | 12.5 | 0.47 |
| CityWalker + RL policy | 36.7 | 54.8 | 7.58 |
| VAMOS + RL policy | 63.3 | 31.8 | 1.84 |
| WayTrace | 83.3 | 2.2 | 0.82 |
| Method | GNSS [m] | Runs | Off zebra | Water | No ground | Fell | Short | Overshot | Stuck | Lost | Other | Failures |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| CityWalker | 0 | 30 | 0 | 0 | 2 | 4 | 0 | 0 | 18 | 0 | 0 | 24 |
| VAMOS | 0 | 30 | 0 | 0 | 0 | 1 | 0 | 0 | 25 | 0 | 0 | 26 |
| CityWalker + RL policy | 0 | 30 | 16 | 0 | 0 | 3 | 0 | 0 | 0 | 0 | 0 | 19 |
| CityWalker + RL policy | 2 | 30 | 10 | 2 | 0 | 4 | 0 | 0 | 2 | 0 | 0 | 18 |
| CityWalker + RL policy | 5 | 30 | 8 | 1 | 0 | 3 | 6 | 2 | 3 | 1 | 0 | 24 |
| CityWalker + RL policy | 7 | 30 | 4 | 0 | 1 | 4 | 9 | 0 | 5 | 0 | 0 | 23 |
| Critical [%] | Critical per decision type | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Model | all | 0 m | 5 m | follow | turn | zebra entry | on zebra | zebra exit | arrival | s/call | $/1k |
| Qwen3.8-Flash-Next (ours, self-hosted) | 5.1 | 4.3 | 6.1 | 0 / 200 | 18 / 338 | 20 / 244 | 2 / 66 | 9 / 87 | 0 / 22 | 1.38 | local |
| Gemini 3.5 Flash-Lite | 5.9 | 4.0 | 8.2 | 4 / 200 | 19 / 338 | 5 / 244 | 0 / 66 | 28 / 87 | 0 / 22 | 5.12 | $2.9 |
| Gemini 3.8 Flash | 3.2 | 3.2 | 3.3 | 0 / 200 | 10 / 338 | 4 / 244 | 12 / 66 | 5 / 87 | 0 / 22 | 6.53 | $7.4 |
| GPT-6 Sol | 3.2 | 1.5 | 5.4 | 2 / 200 | 15 / 338 | 6 / 244 | 1 / 66 | 7 / 87 | 0 / 22 | 4.52 | $7.1 |
| Training data | Samples | NT | ADE | Side [%] |
|---|---|---|---|---|
| Hand-labelled + reconstructed walks | 327.8k | 79.33 | 6.07 | 95.9 |
| + 10k intent-conditioned | 337.8k | 80.66 | 5.67 | 97.0 |
| + 100k intent-conditioned | 427.8k | 82.57 | 5.14 | 97.5 |
| + 863k intent-conditioned | 1,190.6k | 83.32 | 4.94 | 98.1 |
| SR [%] | Repeated turn [%] | |||
|---|---|---|---|---|
| Trace predictor | all | turns | controls | turns |
| Current frame only | 74.7 | 56.0 | 93.3 | 56.0 |
| Temporal | 94.7 | 94.7 | 94.7 | 9.3 |
Embodied Navigation
Embodied navigation studies how robots perceive, map, and move through environments to reach goals and follow instructions.
See all 144 papers