Vision-language-action (VLA) policies can execute many tasks from standard initial states, yet a new instruction may fail after another task has altered the robot's physical state. We study instruction switching, where a new task is issued during or after the execution of a different one. We observe that a target task that is reliably completed from its standard initial states can become inaccessible from states produced by a preceding task. We call such states task islands. We propose Causeway, a training-free inference-time intervention. Given the current state and a re-entry pose for the target task, Causeway back-propagates through the frozen decoding computation and applies a state-directed write within the action-stream representation. The VLA decodes the return motion itself, without parameter updates, a new action head, or external action generation. Across 71 cross-object pairs, three switch timings, and three VLA architectures on LIBERO-Goal, Causeway raises bare-switch success from 3-26% to 47-65% and increases the rate of reaching the handoff neighborhood by 42-72 percentage points across models. Additional experiments on LIBERO-Object and a real xArm platform show that the recovery extends beyond the main LIBERO-Goal setting, both in simulation and on a robot.
Figures & tables
Fig. 1 : Causeway bridges task islands. (a) Demonstrations of task B define its demonstrated state set MB , with SB(g) denoting the valid re-entry region. (b) Executing task A can leave the robot in a state from which task B is no longer accessible, so a bare switch to ℓB fails. (c) Causeway steers the action-stream representation with a state-directed gradient (top) until the robot re-enters the basin, scheduled through four phases \raisebox{-0.9pt}{1}⃝–\raisebox{-0.9pt}{4}⃝ (bottom, Sec. IV-B ). Dashed: write active; solid: the frozen policy completes B on its own.
Fig. 2 : Entry-basin tolerance and terminal-state mismatch. Left three panels: success under component-wise perturbations around a re-entry pose; thin lines show individual tasks and bands show interquartile ranges. Right: bare-switch success after A completes vs. the source’s exit approach-axis error ( r=−0.80 ); A3 (red): low success despite moderate axis error—its hand exits inside the drawer.
Fig. 3 : Escape map (OpenVLA-OFT, LIBERO-Goal). 250, 180: hand–object distance, mm; pre: pre-grasp; far/pre for the two drawer tasks. Columns: the instruction issued at that state, grouped by target object. Cells: fraction of trials reaching the object of ℓ . Right: ∣F(s)∣ , the number of instructions with escape rate ≥0.5 ; ∣F(s)∣=1 (red) marks committed states.
Fig. 4 : Instruction influence attenuates along the approach, yet the gated representation remains steerable. (a) Action response to swapping ℓA→ℓB , relative to the start of the approach (median, IQR). (b) At post-approach states, a gradient direction computed on the current forward pass restores the response toward the open-gate range and aligns the decoded motion with the return direction.
Switch point
Re-entry to T⋆
Model
Method
Step 30
Step 60
After
Avg.
Reach
Fin ∣ R.
π0.5
Bare
29.9
22.9
25.1
26.0
18.7
58.7
CPG
35.2
21.1
26.5
27.6
–
–
CAA
26.9
21.7
26.4
25.0
–
–
Action
37.5
21.3
26.6
28.5
–
–
Ours
60.9
60.4
63.4
61.6
61.0
74.0
TABLE I : Switch success (%). ℓB issued at Step 30, Step 60, or after A completes. Reach : fraction reaching the handoff neighborhood of T⋆ ; Fin ∣ R. : completion given reach. Gray: references using external state manipulation. Best steering method per block in bold.
Fig. 5 : Per-target switch success, pooled over source tasks, switch timings, initial states, and rollouts. Filled: with the write; white inset: bare switch. Bottom: pooled average.
Approach
Pre-grasp
After A
A→B
bare
ours
bare
ours
bare
ours
button → shelf
100
100
0
100
40
80
shelf → button
0
100
0
80
0
60
button → plate
0
100
0
100
0
100
plate → button
0
100
0
80
0
100
corn → plate
0
100
0
100
80
100
TABLE II : Real-robot instruction switching (xArm, π0.5 fine-tuned). Success rate (%) over 5 trials per cell. ℓB is issued during the approach, at the pre-grasp pose, or after A completes. Tasks: press button , put the bowl on the plate / shelf , put the corn in the drawer ( corn ). † : same-object pairs (both move the bowl). Solo : the policy’s own success on B from the standard start.
Fig. 6 : Real-robot chained switching (xArm, π0.5 , fine-tuned then frozen): Top Row: Each instruction interrupts the previous task mid-execution; the final switch re-routes the held bowl from the plate to the shelf via the phase-conditioned target (closed-gripper branch, Sec. IV-B ); Bottom Row: Instructions issued after each task completes. Solid instruction chips mark the frame where a new instruction takes effect; insets show the bare switch failing from the same state. Quantitative pairwise results in Table II .
Vision-language-action (VLA) agents can execute natural-language instructions, yet deployed systems still lack an operational interface: deciding when the instruction is complete. This gap is acute in short composites ("do A, then B"), where mistimed handoffs cascade into downstream failures. Completion is inherently closed-loop because switching is an intervention that changes the instruction context and thus future actions and observations. We study completion under a deployable low-calibration regime motivated by open-ended instruction spaces, enforcing no test-time relearning and a single globally calibrated switching rule selected once on development set and reused unchanged on test set. Under this constraint, collapsing asymmetric boundary evidence into a single scalar can be brittle under polarity shifts across tasks. We propose Completion at the Boundary (CaB), which predicts an event-local completion object in the form of Boundary-Phase Tokens (Before/Hit/After), retaining two-sided boundary evidence under this discipline. CaB-When converts this completion object into a minimal, auditable switching decision (when), while CaB-How reuses the same completion object to condition action generation for boundary-stable control through handoffs (how). Using an intervention-aware E1/E2 protocol, we show that CaB improves composite execution and handoff quality on a first-person Minecraft VLA benchmark under matched capacity and deployability constraints.
Robot manipulation systems using Vision-Language-Action (VLA) model backbones typically use just one VLA for task execution. However, individual VLAs do not perform well across different task states and environments. We introduce a framework for dynamically composing multiple VLA policies during execution: StepWise Action Policy Routing (SWAP). SWAP formulates policy routing as an offline reinforcement learning problem, learning a routing critic that selects the most appropriate policy at each decision step given the current observation. SWAP enables robots to select new policies to execute online rather than committing to a single policy for the duration of an episode. We evaluate SWAP on both real-world DROID manipulation tasks and LIBERO simulation experiments. SWAP improves over fixed-policy execution and routing baselines, giving absolute improvements in real-world task success up to 33% while reducing successful trajectory robot action step length by 28.3%.
Vision-language-action (VLA) models can exceed 90% success on in-distribution tasks and withstand nuisance changes that preserve the required action, yet fail under counterfactual changes that demand a different action. Aggregate robustness scores can therefore conceal a more specific failure, in which a policy responds to both language and vision yet does not combine them to select the action the task requires. We call this failure instruction-action binding. Instructions cue familiar trajectory families, and visual feedback adjusts their execution. Behavioral analyses of fine-tuned π0.5 and GR00T-N1.7 policies reveal that failed rollouts often retain the source behavior or switch to another demonstrated task. These switches show that language is not simply ignored. Readouts and interventions connect these choices to task-conditioned internal states. Our analysis of the imitation objective shows how narrow conditional action support can leave grounded and instruction-keyed solutions indistinguishable on the demonstrations. This motivates Equivariant Counterfactual Training (ECT), which acts at two levels. ECT data supply valid demonstrations in which the same instruction requires different actions in distinguishable scenes, while the ECT loss trains each demonstration with its counterpart in the same update. In a controlled LIBERO-PRO comparison, full ECT raises π0.5's mean position-swap success from 36% to 59%. On CALVIN, where counterparts already occur in the original data, the ECT loss improves five-task completion without new demonstrations. On a real UR5e under a fixed demonstration budget, full ECT raises unseen-position success from 8% to 88%.
Hung-Jen Chen, Yu-Hsun Hou, Yan-Hong Chen +4
National Tsing Hua University · National Taiwan University