Allocentric Memory, Egocentric Acting: A Dual-Frame VLA for Urban Aerial Navigation
Organizations: National Key Laboratory of Digital Intelligent Modeling and Simulation · College of Systems Engineering, National University of Defense Technology · Zhejiang University
Abstract
Language-goal aerial navigation requires a UAV to reach a destination specified in natural language using observations collected during flight. A key challenge is linking changing egocentric perception with spatial evidence anchored to stable locations for long-range navigation. We propose CityNav-VLA, a dual-frame vision-language-action model for urban navigation integrating dynamic egocentric context with persistent allocentric spatial memory. The UAV incrementally accumulates visual observations on a fixed north-up map initialized with geographic and landmark priors, preserving evidence across viewpoint changes. A dynamic query conditioned on the instruction, current observations, UAV state, and motion history retrieves relevant memory evidence to predict a map-frame target. A deterministic controller converts this target into body-frame commands using the UAV's current pose, connecting global reasoning with egocentric execution. To reduce target fluctuations over long horizons, early goal commitment (EGC) fixes the execution target to the coordinate-wise median of the first predictions without extra trainable parameters. On Test Unseen, CityNav-VLA reduces NE by 21.2% and improves SR and SPL by 15.8% and 30.9%, respectively. It also reduces NE by 6.2% and improves SR and SPL by 5.6% and 23.2%, respectively, relative to the former SOTA after dataset refinement. Cumulative ablations show that navigation-map features yield the largest gains, while metric coordinates further improve endpoint accuracy. With a 4B-scale backbone, inference averages 0.68 s per decision with 8.6 GB peak memory on a single A100, presenting remarkable model efficiency.
Figures & tables
| Method | Validation Seen | Validation Unseen | Test Unseen | |||||||||
| NE | SR | OSR | SPL | NE | SR | OSR | SPL | NE | SR | OSR | SPL | |
| CityNav | ||||||||||||
| FLAME [ 38 ] | 59.30 | 19.44 | 29.49 | 17.21 | 69.60 | 14.83 | 28.31 | 13.85 | 80.40 | 20.81 | 32.35 | 19.15 |
| FlightGPT [ 2 ] | 66.10 | 17.57 | 30.26 | 15.78 | 68.10 | 14.69 | 29.33 | 13.24 | 76.20 | 21.20 | 35.38 | 19.24 |
| GeoNav [ 34 ] | 58.60 | 21.80 | 39.98 | 12.37 | 64.10 | 16.99 | 32.41 | 9.96 | 73.50 | 25.91 | 41.68 | 16.09 |
| DroneNav [ 19 ] | 54.30 | 18.94 | 36.65 | 17.37 | 63.50 | 15.06 | 25.81 | 13.27 | 74.90 | 22.47 | 30.22 | 20.38 |
| Dataset | Difficulty | Episodes | NE | SR | OSR | SPL | ||||
| w/o | w/ | w/o | w/ | w/o | w/ | w/o | w/ | |||
| CityNav | Easy | 2,096 | 53.76 | \mathbf{48.30}_{{\color[rgb]{0.2148,0.4102,0.5898}\scriptsize\mathbf{(-5.46)}}} | 24.38 | \mathbf{29.29}_{{\color[rgb]{0.2148,0.4102,0.5898}\scriptsize\mathbf{(+4.91)}}} | 29.77 | \mathbf{31.39}_{{\color[rgb]{0.2148,0.4102,0.5898}\scriptsize\mathbf{(+1.62)}}} | 22.86 | \mathbf{28.12}_{{\color[rgb]{0.2148,0.4102,0.5898}\scriptsize\mathbf{(+5.26)}}} |
| Medium | 1,686 | 58.51 | \mathbf{54.79}_{{\color[rgb]{0.2148,0.4102,0.5898}\scriptsize\mathbf{(-3.72)}}} | 23.19 | \mathbf{25.50}_{{\color[rgb]{0.2148,0.4102,0.5898}\scriptsize\mathbf{(+2.31)}}} | 29.30 | \mathbf{31.02}_{{\color[rgb]{0.2148,0.4102,0.5898}\scriptsize\mathbf{(+1.72)}}} | 22.42 | \mathbf{24.82}_{{\color[rgb]{0.2148,0.4102,0.5898}\scriptsize\mathbf{(+2.40)}}} | |
| Hard | 1,529 | 64.47 | \mathbf{60.98}_{{\color[rgb]{0.2148,0.4102,0.5898}\scriptsize\mathbf{(-3.49)}}} | 17.99 | \mathbf{24.07}_{{\color[rgb]{0.2148,0.4102,0.5898}\scriptsize\mathbf{(+6.08)}}} | 24.26 | \mathbf{30.28}_{{\color[rgb]{0.2148,0.4102,0.5898}\scriptsize\mathbf{(+6.02)}}} | 17.40 | \mathbf{23.33}_{{\color[rgb]{0.2148,0.4102,0.5898}\scriptsize\mathbf{(+5.93)}}} | |
| Overall | 5,311 | 58.35 | \mathbf{54.01}_{{\color[rgb]{0.2148,0.4102,0.5898}\scriptsize\mathbf{(-4.34)}}} | 22.16 | \mathbf{26.59}_{{\color[rgb]{0.2148,0.4102,0.5898}\scriptsize\mathbf{(+4.43)}}} | 28.04 | \mathbf{30.95}_{{\color[rgb]{0.2148,0.4102,0.5898}\scriptsize\mathbf{(+2.91)}}} | 21.15 | \mathbf{25.70}_{{\color[rgb]{0.2148,0.4102,0.5898}\scriptsize\mathbf{(+4.55)}}} | |
| Refined CityNav | Easy | 2,081 | 37.00 | \mathbf{33.41}_{{\color[rgb]{0.2148,0.4102,0.5898}\scriptsize\mathbf{(-3.59)}}} | 27.73 | \mathbf{33.97}_{{\color[rgb]{0.2148,0.4102,0.5898}\scriptsize\mathbf{(+6.24)}}} | 51.61 | \mathbf{51.75}_{{\color[rgb]{0.2148,0.4102,0.5898}\scriptsize\mathbf{(+0.14)}}} | 24.46 | \mathbf{32.10}_{{\color[rgb]{0.2148,0.4102,0.5898}\scriptsize\mathbf{(+7.64)}}} |
| Variant | NE | SR | OSR | SPL | ||||
| F0 | 207.34 | 0.82 | 7.16 | 0.66 | ||||
| F1 | 58.81 | 12.27 | 25.14 | 11.28 | ||||
| F2 | 57.01 | 14.31 | 29.40 | 13.22 | ||||
| F3 | 52.42 | 15.76 | 28.03 | 14.68 |
| Map RGB evidence | NE | SR | OSR | SPL |
| Current view only | 47.13 | 18.80 | 24.40 | 17.38 |
| Accumulated views | 43.62 | 24.20 | 27.80 | 22.76 |
| Gain | -3.52 | +5.40 | +3.40 | +5.38 |
| Split | NE | SR | OSR | SPL |
| CityNav | ||||
| Validation Seen | -3.02 | +2.24 | +1.16 | +2.48 |
| Validation Unseen | -3.62 | +0.21 | -0.71 | +0.43 |
| Test Unseen | -4.34 | +4.42 | +2.92 | +4.55 |
| refined CityNav | ||||
| Validation Seen | -2.14 | +1.82 | -1.34 | +2.74 |
| Method | Params (B) | GPU Mem (GB) | Latency (ms) |
| GSM [ 16 ] | 34.96 | 75.3 | 4832.6 |
| FlightGPT [ 2 ] | 8.29 | 19.6 | 8655.9 |
| DroneNav [ 19 ] | 7.96 | 16.4 | 4254.7 |
| CityNav-VLA (Ours) | 4.46 | 8.56 | 679.7 |
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Setting | Symbol | Value |
| Spatial token grid | ||
| Feature dimension | ||
| Coordinate normalization | ||
| Vertical normalization | ||
| Loss error scale | ||
| Vertical loss weight |
| Subset | NE | SR | OSR | SPL |
| Easy | 60.66 | 21.00 | 49.00 | 11.72 |
| Medium | 76.44 | 20.00 | 43.75 | 13.64 |
| Hard | 76.44 | 28.33 | 43.33 | 22.30 |
| Overall | 69.87 | 22.50 | 45.83 | 15.00 |
| Backbone | NE | SR | OSR | SPL |
| Validation Unseen | ||||
| Raw Qwen3-VL-4B | ||||
| CityBot-4B | ||||
| Test Unseen | ||||
| Raw Qwen3-VL-4B | ||||
| CityBot-4B | ||||
| Backbone | NE | SR | OSR | SPL |
| Validation Unseen | ||||
| Raw Qwen3-VL-8B | 53.67 | 14.61 | 30.33 | 13.62 |
| CityBot-8B | 52.80 | 16.76 | 29.55 | 15.67 |
| Test Unseen | ||||
| Raw Qwen3-VL-8B | 43.58 | 22.21 | 37.63 | 20.65 |
| CityBot-8B | 42.21 | 23.04 | 35.01 | 21.74 |
| Representation | EGC | NE | SR | OSR | SPL |
| CityNav, three-dimensional evaluation | |||||
| No explicit metric coordinates | Off | 71.24 | 9.55 | 13.48 | 8.97 |
| Egocentric metric displacement | Off | 81.45 | 8.46 | 14.93 | 8.05 |
| Allocentric metric | Off | 69.78 | 9.27 | 14.51 | 8.76 |
| refined CityNav, planar evaluation | |||||
| No explicit metric coordinates | Off | 57.01 | 14.31 | 29.40 | 13.22 |
| Commitment step | NE | SR | OSR | SPL |
| 8 | 39.95 | 28.06 | 31.75 | 26.98 |
| 12 | 39.96 | 27.86 | 31.71 | 26.76 |
| 18 | 39.95 | 28.74 | 32.67 | 27.58 |
| 24 | 40.08 | 28.38 | 32.31 | 27.20 |
| 32 | 40.30 | 28.50 | 32.47 | 27.29 |
| 48 | 40.65 | 28.54 | 32.79 | 27.24 |
| Split | CityNav | refined CityNav | Removed |
| Training Seen | 22,002 | 21,878 | 124 |
| Validation Seen | 2,498 | 2,470 | 28 |
| Validation Unseen | 2,826 | 2,697 | 129 |
| Test Unseen | 5,311 | 5,281 | 30 |
| Property | CityNav discrete actions | Our continuous-control protocol |
| One environment decision | One forward, vertical, turning, or STOP action | One simultaneous forward/lateral/vertical displacement |
| Translation | Forward 5 m, or vertical m | Forward/lateral each in m; vertical in m |
| Heading | Turns of consume decisions | Constant; yaw command is zero |
| Largest horizontal / XYZ displacement | 5 / 5 m per decision | / 6 m per decision |
| 120-decision horizontal / XYZ cap | 600 / 600 m if all actions are forward | 678.82 / 720 m |
| Termination and SR | STOP within the target sphere | All 120 decisions; final distance at most 20 m in the evaluation coordinate space |
Aerial VLN
Aerial vision-and-language navigation (VLN) enables UAVs to follow natural-language goals using visual perception and flight control.
See all 71 papers