Vision-Language-Action (VLA) models integrate high-level reasoning with low-level action execution for end-to-end robotic control. However, most VLA models target static manipulation in unconstrained environments, overlooking spatial constraints and dynamic obstacles. Fine-tuning VLAs on obstacle-avoidance data is a straightforward solution, but collecting such data is costly, especially for dynamic scenarios, and may degrade their original task capabilities. To address these limitations, we propose VLA-Stem, a lightweight, reinforcement-learning-based safety mechanism for VLA systems. Without fine-tuning the underlying VLA policy, VLA-Stem converts its action chunks into motion references and uses trajectory-level guidance to preserve their spatial intent while allowing flexible motion and timing adjustments. Conditioned on these references, robot proprioception, and local obstacle information, the Stem policy adaptively modifies execution at inference time, preserving the original task intent while enabling responsive adaptation to spatial constraints and dynamic obstacles. For robust and sample-efficient training, we combine teacher-student distillation with curriculum learning. We evaluate VLA-Stem on reference following, static spatial constraints, and dynamic obstacle avoidance. Across six constrained simulation settings, VLA-Stem achieves 52.8% average collision-free task success versus 23.7% for the strongest safety baseline, while preserving base-policy performance in obstacle-free settings; real-world experiments further show consistent collision reductions under static and dynamic obstacles. These results demonstrate that VLA-Stem couples high-level VLA task intent with rapid, reflex-like responses to immediate physical risks, providing a brainstem-inspired mechanism for robust robot execution.
Figures & tables
Fig. 1: This figure illustrates the biological motivation behind our method. The left side depicts the interaction between higher-level motor control and lower-level sensorimotor regulation: task-directed motor commands are continuously shaped by rapid responses to immediate physical conditions. In this functional analogy, higher-level task control and motor coordination correspond to the base VLA, while the brainstem-inspired lower-level regulation corresponds to our proposed VLA-Stem . The right side shows how VLA-Stem adapts the VLA-directed motion to bypass an obstacle while continuing toward the target object.
Fig. 2: VLA-Stem pipeline. The teacher is trained using curriculum learning, progressing from reference trajectory following to static obstacle avoidance and then to dynamic obstacle avoidance, with gradually increasing difficulty. The teacher policy takes as input the trajectory sequence produced by the base VLA policy, robot proprioception, and obstacle information, and additionally leverages privileged task information about the target object to accelerate training. Finally, the teacher policy is used to distill a student policy, removing the dependence on task-specific privileged information at deployment.
Method
Metric
PickCube
Object Stowing
StackCube
Normal
Static
Dynamic
Normal
Static
Dynamic
Normal
Static
Dynamic
π0
SUCC ↑
0.6884
0.0106
0.0044
0.4322
0.0006
0.0559
0.6506
0.1316
0.1453
COLL ↓
–
0.9422
0.9703
–
0.5363
0.6809
–
0.6478
0.5484
π0.5
SUCC ↑
0.7166
0.0325
0.0131
0.6138
0.0000
0.0256
0.7450
0.1725
0.1247
COLL ↓
–
0.8781
0.9328
–
0.5897
0.6156
–
0.7319
0.7012
VLA-finetune
SUCC ↑
0.7166
0.2847
0.1869
0.6138
0.0072
0.4784
0.7450
0.0062
0.0641
TABLE I: Experimental results on simulation tasks. Higher SUCC is better and lower COLL is better.
Fig. 3: Dynamic obstacle avoidance in simulation. Compared with π0.5 , VLA-Stem waits for the moving obstacle to pass before resuming task execution.
Fig. 4: Reference-conditioned execution on StackCube: trajectory visualization and quantitative results over 30 paired trials per direction.
Method
Normal
Static
Dynamic
SUCC ↑
COLL ↓
SUCC ↑
COLL ↓
SUCC ↑
COLL ↓
VLA-Stem w/o reference
0.0000
–
0.0000
0.1300
0.0000
0.0156
VLA-Stem
0.7325
–
0.5219
0.2316
0.4153
0.3122
TABLE II: VLA reference ablation on PickCube. Results are reported for student policies.
Fig. 5: Real-world evaluation of VLA-Stem. (a) Real-world experimental examples. (b–c) Quantitative results on Object Stowing and Stacking, respectively.
Method
Normal
Static
Dynamic
SUCC ↑
COLL ↓
SUCC ↑
COLL ↓
SUCC ↑
COLL ↓
VLA-Stem w/ Step-wise
0.2888
–
0.4778
0.0366
0.1453
0.2194
VLA-Stem
0.7061
–
0.5784
0.1756
0.5950
0.1203
TABLE III: Ablation results of Trajectory-level guidance on Object Stowing.
Method
Normal
Static
Dynamic
SUCC ↑
COLL ↓
SUCC ↑
COLL ↓
SUCC ↑
COLL ↓
VLA-Stem w/o distillation
0.2903
–
0.0525
0.3066
0.0900
0.4419
VLA-Stem
0.7061
–
0.5784
0.1756
0.5950
0.1203
TABLE IV: Ablation results of Teacher–student distillation on Object Stowing.
Stage
wtask
wcoll
wfollow
wstep
Following
0.6
0.0
0.4
0.2
Static
1.2
1.0
0.4
0.1
Dynamic
1.2
1.0
0.6
0.0
TABLE V: Reward weights used in the three curriculum stages.
Parameter
Following
Static
Dynamic
Parallel environments
256
192
192
Learning rate
2×10−4
1.5×10−4
1.5×10−4
PPO update epochs
8
12
12
Minibatch size
3,200
19,200
19,200
Rollout horizon
100
100
100
Discount factor γ
0.8
0.8
0.8
TABLE VI: Stage-wise PPO training configurations and simulation budgets. Transition counts are approximate and include both teacher and student training.
Vision-Language-Action (VLA) models demonstrate strong generalization in robotic manipulation and navigation, but existing fine-tuning methods provide limited safety guarantees. Current approaches primarily rely on Lagrangian optimization that enforces safety through soft penalties on expected cumulative cost, often resulting in residual constraint violations or overly conservative behavior. Moreover, learning safety in visual domains is challenging due to the absence of dense per-step safety annotations. We propose ShieldVLA, a safety-aligned fine-tuning framework for VLA models based on Hamilton-Jacobi (HJ) reachability. ShieldVLA learns a model-free approximation of the HJ reachability value function directly from visual observations to estimate the safe operating region. The learned safety critic gates policy optimization by separating reward maximization within feasible regions from recovery near unsafe states, avoiding persistent reward-cost trade-offs. To enable scalable supervision in visual environments, we introduce rubric-based VLM safety scores that convert semantic safety feedback into structured critic targets without requiring manual cost labels. Across five navigation and manipulation benchmarks spanning multiple VLA backbones, ShieldVLA reduces cumulative safety cost by 57% on average and improves task success rate by +0.13 over SafeVLA.
Manan Tayal, Akshay Nambi
Indian Institute of Science (IISc) · Microsoft Research
Vision-Language-Action (VLA) models have demonstrated remarkable capabilities in generalizing across diverse robotic manipulation tasks. However, deploying these models in unstructured environments remains challenging due to the critical need for simultaneous task compliance and safety assurance, particularly in preventing potential collisions during physical interactions. In this work, we introduce a Vision-Language-Safe Action (VLSA) architecture, named AEGIS, which contains a plug-and-play safety constraint (SC) layer formulated via control barrier functions. AEGIS integrates directly with existing VLA models to improve safety with theoretical guarantees, while maintaining their original instruction-following performance. To evaluate the efficacy of our architecture, we construct a comprehensive safety-critical benchmark SafeLIBERO, spanning distinct manipulation scenarios characterized by varying degrees of spatial complexity and obstacle intervention. Extensive experiments demonstrate the superiority of our method over state-of-the-art baselines. Notably, AEGIS achieves over 50% improvement in obstacle avoidance rate while substantially increasing the task success rate by nearly 10%. All benchmark datasets, code, and supplementary materials are publicly available at https://vlsa-aegis.github.io/.
Songqiao Hu, Zeyi Liu, Shuang Liu +5
Department of Automation, Tsinghua University, Beijing 100084, China. · Institute for Embodied Intelligence and Robotics, Tsinghua University, Beijing 100084, China. · 3TetraBOT. +1
Vision-Language-Action (VLA) models provide a natural language interface to robot control, but the mapping from language to behavior is often brittle and unintuitive: semantically similar instructions can induce drastically different behaviors, while some capabilities may not be elicitable through prompting alone. As a result, both human instructions and zero-shot language models can fail to reliably steer VLAs toward successful task execution. In this work, we propose a framework that interactively searches for language sequences that improve closed-loop VLA task performance, distills these sequences into a test-time language feedback policy (LFP), and learns an improvement head that predicts when language steering will improve performance. We conformalize this improvement head to prevent harmful steering interventions, where the LFP decreases task performance relative to the original instruction on out-of-distribution scenarios. Crucially, our approach operates on arbitrary frozen pre-trained VLAs, requiring neither access to the original training distribution nor fine-tuning of the underlying model. On seen environments, our conformalized LFP improves base VLA performance by 24.7% in simulation and 65.0% in hardware. On visual and semantic perturbations with calibration samples, our conformalized LFP has strong harmlessness guarantees, and produces recovery behaviors not observed with open-loop prompting.