cs.ROApr 2, 2026
SaveAURA: Multimodal Shared Autonomy for Real-World Urban Navigation
Organizations: University of California, Los Angeles · Zhejiang University
Abstract
Long-horizon navigation in complex urban environments relies heavily on continuous human operation, which leads to fatigue, reduced efficiency, and safety concerns. Shared autonomy, where a Vision-Language AI agent and a human operator collaborate on maneuvering the mobile machine, presents a promising solution to address these issues. However, existing shared autonomy methods often require humans and AI to operate within the same action space, leading to high cognitive overhead. We present Assistive Urban Robot Autonomy (AURA), a new multi-modal framework that decomposes urban navigation into high-level human instruction and low-level AI control. AURA incorporates a Spatial-Aware Instruction Encoder to align various human instructions with visual and spatial context. To facilitate training, we construct MM-CoS, a large-scale dataset comprising teleoperation and vision-language descriptions. Experiments in simulation and the real world demonstrate that AURA effectively follows human instructions, reduces manual operation effort, and improves navigation stability, while enabling online adaptation. Moreover, under similar takeover conditions, our shared autonomy framework reduces the frequency of takeovers by more than 44%. Demo video and more detail are provided in the project page.
Figures & tables
Figure 1: Shared Autonomy for Urban Navigation. We introduce AURA, a dual-system VLA for shared autonomy in urban navigation. AURA not only follows instructions but also enables human users to guide and correct a robot in real time through various visual and language instructions.
Figure 2: Overview of the AURA shared autonomy framework. (a) AURA takes front-camera RGB observations and optional human guidance (e.g., texting, drafting, or arrowing). Observations are encoded by a ViT, while human inputs are processed by the SIE and tokenized; all tokens are fused in a pretrained LLM (with LoRA adapters) to produce context features. A diffusion-based action decoder then predicts a distribution over future trajectories via anchor proposals. (b) The SIE converts drafting/arrowing inputs into instruction tokens: it renders the human input as visual prompts, encodes the control points/vectors, and fuses them with visual features to produce instruction embeddings that are injected into the LLM via .
Figure 3: Samples from the auto-labeling pipeline. Each frame is annotated with three training labels produced by our auto-labeling pipeline: (1) the texting command expressed as a short verb phrase (e.g., “go straight”, “slow down”, “speed up”), (2) the drafting, visualized as a rendered path from the ground-truth future trajectory, and (3) the arrowing input, represented by instantaneous speed. The rightmost panel shows the reasoning traces used to supervise drafting and arrowing prediction.
Figure 4: Visualization of offline inference in MM-CoS . We illustrate three types of human instructions. The green polygon denotes the future trajectories predicted by AURA.
| minADE 1s | minFDE 1s | mAP | L2 1s | L2 2s | |
| GNM ‡ [ 49 ] | 0.594 | 0.988 | - | 0.988 | - |
| ViNT ‡ [ 51 ] | 0.638 | 1.056 | - | 1.056 | - |
| NoMaD ‡ [ 55 ] | 0.523 | 0.858 | 0.216 | 1.072 | 2.182 |
| MBRA [ 22 ] | 0.617 | 1.019 | - | 1.019 | 2.034 |
| CityWalker [ 35 ] | 0.648 | 1.125 | - | 1.125 | - |
| ViNT* [ 51 ] | 0.247 | 0.450 | - | 0.425 | 0.925 |
Table 1: Open-loop evaluation on MM-CoS. * denotes models re-trained on our dataset.
Figure 5: Human Cost Evaluation in Pseudo-simulation. We compare the human intervention cost of our model against prior methods [ 40 , 35 ] , as well as across different modes of instruction guidance within our own framework.
| Model | Finetune | Visual Prompt | SIE | ROUGE-L | Intent Score |
| InternVL3-2B | ✓ | 0.167 | 2.019 | ||
| InternVL3-8B | ✓ | 0.184 | 2.818 | ||
| InternVL3-2B | ✓ | ✓ | 0.532 | 4.885 | |
| AURA | ✓ | ✓ | 0.534 | 4.842 | |
| AURA | ✓ | ✓ | ✓ | 0.581 | 5.446 |
Table 2: Evaluation on instruction following.
Figure 6: Effectiveness of components on end-to-end planning.
| Method | HO (%) | NIR | ODR | TSR |
| NoMaD | 9.74 | 43.2 | 11.3 | 89.0 |
| CityWalker | 14.56 | 48.29 | 20.0 | 80.3 |
| Gemini | 16.9 | 255.7 | 32.0 | 63.2 |
| AURA | 1.73 | 16.99 | 10.5 | 89.3 |
Table 3: Real-world closed-loop navigation results. We report Human Operation Ratio (HO), Normalized Intervention Rate (NIR, emergency interventions per 100 meters), Off-track Distance Ratio (ODR), and Time Success Rate (TSR, ratio of valid autonomous time to total autonomous time). Lower is better for HO/NIR/ODR and higher is better for TSR.
Figure 7: Visualization of the teleoperation interface in real-world experiments. For each row, the center frame corresponds to the moment of human intervention (teleoperation takeover), with the other frames showing the surrounding context. The green polygon denotes the predicted trajectories, and the red lines on both sides indicate the width of the robot.
Figure 8: Overview of the real-world experiment setup. Left: the robot hardware platform; Right: the teleoperation interface.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Data distribution of our MM-CoS dataset (50h) . Behaviors are divided into six subclasses based on their outcomes and causes. Bars of the same color represent different specific actions within each subclass. The top of each bar indicates the number of occurrences of the behavior and its proportion of the total annotations.
Figure 10: A thumbnail montage showing a subset of the MM-CoS videos collected from real world environments for shared autonomy.
Figure 11: Data distribution across weather conditions and time of day. The dataset contains 3,040 scenes in total, with weather conditions primarily consisting of clear (48.1%) and sunny (36.4%) conditions, while cloudy (8.7%) and rainy (6.9%) conditions are less represented. Time-of-day distribution shows a strong bias toward daytime scenes (59.9%), followed by nighttime scenes (30.4%), with transitional periods of dawn (3.2%) and dusk (6.5%) being relatively scarce.
Figure 12: Auto labeling pipeline. We first select interesting clips based on behavioral cues and ego state information, focusing on frames that show rich interactions with objects or the environment. We then feed the front view images with visual prompts of the projected future trajectory and the speed of each frame into the VLM to generate textual instructions. Finally, we use the projected future trajectory on the front images to derive the drafting and arrowing instructions.
Figure 13: Prompt for behavior filter. We use this prompt to let the VLM determine whether the robot in the clip is interacting with objects or the environment.
Figure 14: Prompt for generating texting instructions. We expect the VLM to produce short, simple instructions that guide the robot at a high level.
Figure 15: Prompt for generating reasoning. We expect the VLM to generate a detailed description of the robot’s behavior in the video, and to provide supervision signals (drafting and arrowing) for interpreting human instructions.
Figure 16: Judgment module. After the policy updates its state and receives the observation, it produces a candidate action through a forward pass of the model, which is temporarily cached before execution. We then apply a judgment module that evaluates the action using both safety-based criteria and instruction-based constraints. Based on this evaluation, the system decides whether to (i) execute the action, (ii) inject an instruction token to guide the policy toward human instruction, or (iii) trigger a full human takeover when the candidate action violates critical safety conditions.
| HO% | TF | OF | HO 1s % | TF 1s | OF 1s | HO 2s % | TF 2s | OF 2s | HO 3s % | TF 3s | OF 3s | |
| MBRA [ 22 ] | 15.1 | 0.757 | 0.757 | 30.5 | 0.305 | 1.523 | 41.6 | 0.211 | 2.082 | 48.1 | 0.164 | 2.407 |
| CityWalker* [ 36 ] | 19.6 | 0.978 | 0.978 | 40.3 | 0.404 | 2.016 | 54.5 | 0.276 | 2.725 | 60.7 | 0.207 | 3.037 |
| AURA | 6.0 | 0.298 | 0.298 | 14.9 | 0.151 | 0.744 | 22.4 | 0.116 | 1.122 | 28.0 | 0.098 | 1.400 |
| AURA | 7.2 | 0.359 | 0.359 | 9.0 | 0.195 | 0.450 | 12.5 | 0.170 | 0.624 | 15.0 | 0.152 | 0.751 |
| AURA | 5.0 | 0.248 | 0.248 | 6.4 | 0.169 | 0.320 | 9.1 | 0.152 | 0.456 | 11.2 | 0.144 | 0.562 |
| AURA | 4.5 | 0.226 | 0.226 | 5.7 | 0.163 | 0.284 | 7.7 | 0.150 | 0.386 | 9.7 | 0.142 | 0.483 |
Table 4: Pseudo simulation evaluation on share control .
Figure 17: Prompt for the VLM judge. The judge module is required to output only yes or no. It checks whether the predicted trajectory matches the human-intended action. If the robot’s intent diverges from the human intention, human intervention is needed. For example, the robot may take an incorrect path or make unnecessary stops.
Figure 18: Qualitative results in the real world. The first three rows are guided by drafting, rows 4 to 6 by arrowing, and the last three rows by texting. The first column shows human instructions, while columns 2 to 5 show the actions executed by the robot.