Robot manipulation systems using Vision-Language-Action (VLA) model backbones typically use just one VLA for task execution. However, individual VLAs do not perform well across different task states and environments. We introduce a framework for dynamically composing multiple VLA policies during execution: StepWise Action Policy Routing (SWAP). SWAP formulates policy routing as an offline reinforcement learning problem, learning a routing critic that selects the most appropriate policy at each decision step given the current observation. SWAP enables robots to select new policies to execute online rather than committing to a single policy for the duration of an episode. We evaluate SWAP on both real-world DROID manipulation tasks and LIBERO simulation experiments. SWAP improves over fixed-policy execution and routing baselines, giving absolute improvements in real-world task success up to 33% while reducing successful trajectory robot action step length by 28.3%.
Figures & tables
Fig. 1 : Policy routing for robot manipulation. We introduce SWAP, an offline reinforcement learning framework for online policy routing. Given candidate robot policies, SWAP dynamically selects the best policy online for an instruction and current observation during task execution.
Fig. 2 : Overview of SWAP. (a) We construct an offline routing dataset by collecting trajectories from a set of candidate robot policies and chunking them into routing transitions labeled with sparse rewards and termination signals. (b) SWAP trains a routing critic offline using IQL, where visual observations and task instructions are encoded with a frozen vision-language model and combined with a candidate policy embedding to predict policy-specific Q-values. (c) SWAP evaluates all candidate policies under the current observation, selects the highest-valued policy, executes it in the environment for a short horizon, and repeats this process online throughout task execution.
Fig. 3 : Evaluation environments. We evaluate SWAP on both real-world DROID [ 1 ] robot manipulation tasks and LIBERO [ 48 ] simulation tasks. For simulation, we use the spatial subset of LIBERO-Plus [ 18 ] , which includes challenging out-of-distribution task variations.
Individual Policy
Routing
π0
π0.5
π0 -FAST
Random
SWAP (Ours)
Task
Succ.
Steps
Succ.
Steps
Succ.
Steps
Succ.
Steps
Succ.
Steps
Marker → Bowl
30%
274
60%
249
80%
209
60%
233
100%
200
Cube → Cup
50%
358
20%
420
40%
372
60%
336
90%
224
Cube on Cloth → Bowl
40%
288
50%
201
70%
227
80%
262
100%
155
Average
40%
307
43.3%
290
63.3%
269
66.7%
277
96.7%
193
TABLE I: Real Robot Policy Routing. We present real-world policy routing results on a Franka DROID setup across three manipulation tasks. SWAP achieves the strongest overall performance, improving success rate from 63.3% for the best fixed policy baseline to 96.7%, demonstrating that learned policy routing can effectively compose complementary behaviors across candidate policies. Additionally, among successful trajectories, SWAP completes tasks in 28.3% fewer steps than the best-performing individual policy (193 vs. 269 steps).
Segment
π0 -FAST
π0.5
First Third
51.0%
49.0%
Middle Third
22.6%
77.4%
Final Third
36.3%
63.7%
TABLE II: Policy preference shifts as execution progresses. Policy selection frequencies across normalized trajectory segments. SWAP begins with nearly equal use of both policies but concentrates on π0.5 during later stages of execution. This suggests that SWAP discovers policy strengths and exploits them at different phases of task completion.
Individual Policy
Routing
Task with Distractors
π0
π0.5
π0 -FAST
SWAP (Ours)
Marker → Bowl
10%
40%
40%
90%
Cube on Plate → Bowl
10%
30%
0%
80%
Average
10%
35%
20%
85%
TABLE IV : Real Robot Policy Routing under OOD Distractors. We add unseen distractor objects and evaluate each method over 10 trials per task. SWAP maintains high success despite the distribution shift.
Fig. 4 : Q-values and policy routing on the DROID marker-in-bowl task. SWAP scores π0 -FAST, π0 , and π0.5 based on the predicted Q-values of each policy throughout task execution. The images highlight key routing decisions in one SWAP rollout. The plot shows the corresponding Q-values over time, as well as the policy selected by SWAP at each decision step. Initially, SWAP favors π0 -FAST, which takes larger steps during free-space movement. As the robot approaches the object, it switches to π0.5 for the more precise grasp. After the marker is accidentally dropped, the Q-values collapse and SWAP again selects π0.5 to recover and reattempt the grasp. Once the marker is secured, SWAP switches back to π0 -FAST for faster motion to the goal.
Vision-Language-Action (VLA) policies leverage pretrained vision-language models (VLMs) to guide action generation for robot control. VLMs provide hierarchical visual-semantic representations that evolve across layers, from local visual geometry to abstract, language-aligned semantics; different manipulation tasks may therefore require different mixtures of layer representations. Meanwhile, the action module maintains intermediate representations that evolve throughout action computation and may provide useful information for subsequent decisions. However, existing VLA interfaces offer limited flexibility in representation access: VLM information is exposed through fixed layer assignments for each action layer, while intermediate action states are only propagated implicitly through residual streams without explicit reuse. We introduce LayerRoute, an action-conditioned representation routing interface that enables adaptive access to VLM layers and action representations. The Layer Mixture Router dynamically forms mixtures of cached VLM representations, while Action-State Reread reuses earlier action representations. Across diverse simulation and real-world benchmarks, LayerRoute consistently improves StarVLA-π and π0.5, achieving up to 7.2 gains on LIBERO Long with only 0.31% / 3.87% additional parameters. Ablation studies validate the benefit of action-conditioned layer routing, while routing analyses reveal structured allocation patterns across action layers and task settings.
Zheng Lu, Haoran Liao, Wanqi Zhong +9
Tsinghua University · Zhejiang University · Nanyang Technological University +2
Vision-language-action (VLA) policies commonly place a multi-billion-parameter model inside the feedback loop, repeatedly recomputing task context whenever they predict a new action chunk. Yet task understanding changes far more slowly than the robot state. We introduce Fast-VLA, which invokes a frozen 4B VLM once per episode to contextualize the instruction and initial views as fixed, multi-layer key-value task memory. During execution, a lightweight online actor fuses learned time- and arm-indexed queries with current visual and proprioceptive feedback to form action queries. These queries retrieve task context from the fixed memory and predict action chunks without re-running the foundation model. Fast-VLA achieves 92.46% and 92.40% success on RoboTwin 2.0 Clean and Randomized, and 95.95% and 79.11% on LIBERO and LIBERO-Plus evaluation. Against three continually conditioned baselines on the same hardware, it reduces RoboTwin amortized policy inference time by 10.9-17.1x and requires 1.22 GiB of post-prefill online resident memory. Task-memory swaps further show that the actor relies on episode-matched task memory. These results show that task context can be established once while closed-loop control remains responsive throughout execution. Project page: https://garfield-kh.github.io/FastVLA
Kehong Gong, Guanli Hou, Xiaoxu Zheng +6
Huawei Technologies Co., Ltd · Huawei Central Research Institute
Vision-language-action (VLA) models have become an important paradigm for robotic manipulation by unifying visual perception, language understanding, and action generation. Since reliable manipulation requires sufficient observation of objects, end-effectors, and their spatial relationships, multi-view inputs are widely used to provide complementary spatial information. However, our experiments show that existing VLA models do not fully exploit such multi-view complementarity. The policy tends to persistently rely on the wrist view, lacking the ability to dynamically adjust global-local view dependency across manipulation stages, which limits stage adaptability and robustness. To address this issue, we propose FocusVLA, a stage-aware multi-view routing framework. FocusVLA predicts future interaction intent from historical actions and uses it to guide dynamic view routing during action generation. By adaptively coordinating global and local observations across different task stages, FocusVLA improves multi-view utilization. On RoboTwin 2.0, FocusVLA improves the average success rate over the strongest baseline π0.5 by 10.10 pp. In real-world evaluation, FocusVLA improves the success rate from 71.7% to 85.0%, achieving a 13.3% gain over π0.5. Moreover, the intent prediction and dynamic routing modules of FocusVLA can be integrated into existing VLA frameworks without modifying their core architectures.
Huixin Xu, Ziya Yu, Peng Li
Nanjing Institute of Software Technology · Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences, Nanjing +2