Causeway: Restoring Task Accessibility for Instruction Switching in VLA Policies

Qingzi Wang; Kaixi Feng; Guangyao Shi; Xiyang Wu; Ang Li; Dinesh Manocha

cs.ROSep 25, 2026

Causeway: Restoring Task Accessibility for Instruction Switching in VLA Policies

Authors: Qingzi Wang, Kaixi Feng, Guangyao Shi, Xiyang Wu, Ang Li, Dinesh Manocha

Organizations: University of Maryland, College Park, MD, USA. · University of Southern California, Los Angeles, CA, USA.

Abstract

Vision-language-action (VLA) policies can execute many tasks from standard initial states, yet a new instruction may fail after another task has altered the robot's physical state. We study instruction switching, where a new task is issued during or after the execution of a different one. We observe that a target task that is reliably completed from its standard initial states can become inaccessible from states produced by a preceding task. We call such states task islands. We propose Causeway, a training-free inference-time intervention. Given the current state and a re-entry pose for the target task, Causeway back-propagates through the frozen decoding computation and applies a state-directed write within the action-stream representation. The VLA decodes the return motion itself, without parameter updates, a new action head, or external action generation. Across 71 cross-object pairs, three switch timings, and three VLA architectures on LIBERO-Goal, Causeway raises bare-switch success from 3-26% to 47-65% and increases the rate of reaching the handoff neighborhood by 42-72 percentage points across models. Additional experiments on LIBERO-Object and a real xArm platform show that the recovery extends beyond the main LIBERO-Goal setting, both in simulation and on a robot.

Figures & tables

Fig. 1 : Causeway bridges task islands. (a) Demonstrations of task $B$ define its demonstrated state set $\mathcal{M}_{B}$ , with $\mathcal{S}_{B}(g)$ denoting the valid re-entry region. (b) Executing task $A$ can leave the robot in a state from which task $B$ is no longer accessible, so a bare switch to $\ell_{B}$ fails. (c) Causeway steers the action-stream representation with a state-directed gradient (top) until the robot re-enters the basin, scheduled through four phases \raisebox{-0.9pt}{1}⃝–\raisebox{-0.9pt}{4}⃝ (bottom, Sec. IV-B ). Dashed: write active; solid: the frozen policy completes $B$ on its own.

Fig. 2 : Entry-basin tolerance and terminal-state mismatch. Left three panels: success under component-wise perturbations around a re-entry pose; thin lines show individual tasks and bands show interquartile ranges. Right: bare-switch success after

A

completes vs. the source’s exit approach-axis error (

r{=}-0.80

); A3 (red): low success despite moderate axis error—its hand exits inside the drawer.

Fig. 3 : Escape map (OpenVLA-OFT, LIBERO-Goal). 250, 180: hand–object distance, mm; pre: pre-grasp; far/pre for the two drawer tasks. Columns: the instruction issued at that state, grouped by target object. Cells: fraction of trials reaching the object of $\ell$ . Right: $|F(s)|$ , the number of instructions with escape rate $\geq 0.5$ ; $|F(s)|{=}1$ (red) marks committed states.

Fig. 4 : Instruction influence attenuates along the approach, yet the gated representation remains steerable. (a) Action response to swapping

\ell_{A}{\to}\ell_{B}

, relative to the start of the approach (median, IQR). (b) At post-approach states, a gradient direction computed on the current forward pass restores the response toward the open-gate range and aligns the decoded motion with the return direction.

		Switch point				Re-entry to $T^{\star}$
Model	Method	Step 30	Step 60	After	Avg.	Reach	Fin $\mid$ R.
$\pi_{0.5}$	Bare	29.9	22.9	25.1	26.0	18.7	58.7
	CPG	35.2	21.1	26.5	27.6	–	–
	CAA	26.9	21.7	26.4	25.0	–	–
	Action	37.5	21.3	26.6	28.5	–	–
	Ours	60.9	60.4	63.4	61.6	61.0	74.0

TABLE I : Switch success (%).

\ell_{B}

issued at Step 30, Step 60, or after

A

completes. Reach : fraction reaching the handoff neighborhood of

T^{\star}

; Fin

\mid

R. : completion given reach. Gray: references using external state manipulation. Best steering method per block in bold.

Fig. 5 : Per-target switch success, pooled over source tasks, switch timings, initial states, and rollouts. Filled: with the write; white inset: bare switch. Bottom: pooled average.

	Approach		Pre-grasp		After $A$
$A\to B$	bare	ours	bare	ours	bare	ours
button $\to$ shelf	100	100	0	100	40	80
shelf $\to$ button	0	100	0	80	0	60
button $\to$ plate	0	100	0	100	0	100
plate $\to$ button	0	100	0	80	0	100
corn $\to$ plate	0	100	0	100	80	100

TABLE II : Real-robot instruction switching (xArm,

\pi_{0.5}

fine-tuned). Success rate (%) over 5 trials per cell.

\ell_{B}

is issued during the approach, at the pre-grasp pose, or after

A

completes. Tasks: press button , put the bowl on the plate / shelf , put the corn in the drawer ( corn ).

\dagger

: same-object pairs (both move the bowl). Solo : the policy’s own success on

B

from the standard start.

Fig. 6 : Real-robot chained switching (xArm, $\pi_{0.5}$ , fine-tuned then frozen): Top Row: Each instruction interrupts the previous task mid-execution; the final switch re-routes the held bowl from the plate to the shelf via the phase-conditioned target (closed-gripper branch, Sec. IV-B ); Bottom Row: Instructions issued after each task completes. Solid instruction chips mark the frame where a new instruction takes effect; insets show the bare switch failing from the same state. Quantitative pairwise results in Table II .

Vision-Language-Action Models

Vision-language-action models turn visual observations and natural-language instructions into actions for robots and other embodied agents.

See all 1,186 papers

Completion at the Boundary (CaB): Deployable Switching with Completion-Aware Control under Limited Calibration
May 29, 2026Yusuke Sano, Takeshi ItogaVision-Language-Action Models Embodied AI
Intelligent Systems Laboratory, SECOM Co., Ltd.
SWAP: Stepwise Action Policy Routing for Vision-Language-Action Models
Oct 3, 2026Mousumi Das, Aditeya Prajapati, Abrar Anwar +1Robotic Manipulation Vision-Language-Action Models
University of Southern California
When Instructions Retrieve Trajectories: Diagnosing and Mitigating Generalization Failures in VLA Models
Sep 30, 2026Hung-Jen Chen, Yu-Hsun Hou, Yan-Hong Chen +4Robotic Manipulation Vision-Language-Action Models
National Tsing Hua University · National Taiwan University

May 29, 2026J/K move · Enter open · S save

Completion at the Boundary (CaB): Deployable Switching with Completion-Aware Control under Limited Calibration

Yusuke Sano, Takeshi Itoga

Intelligent Systems Laboratory, SECOM Co., Ltd.

Vision-Language-Action Models Embodied AI Vision-Language Agents

Open paper AskPDF arXiv

Vision-language-action (VLA) agents can execute natural-language instructions, yet deployed systems still lack an operational interface: deciding when the instruction is complete. This gap is acute in short composites ("do A, then B"), where mistimed handoffs cascade into downstream failures. Completion is inherently closed-loop because switching is an intervention that changes the instruction context and thus future actions and observations. We study completion under a deployable low-calibration regime motivated by open-ended instruction spaces, enforcing no test-time relearning and a single globally calibrated switching rule selected once on development set and reused unchanged on test set. Under this constraint, collapsing asymmetric boundary evidence into a single scalar can be brittle under polarity shifts across tasks. We propose Completion at the Boundary (CaB), which predicts an event-local completion object in the form of Boundary-Phase Tokens (Before/Hit/After), retaining two-sided boundary evidence under this discipline. CaB-When converts this completion object into a minimal, auditable switching decision (when), while CaB-How reuses the same completion object to condition action generation for boundary-stable control through handoffs (how). Using an intervention-aware E1/E2 protocol, we show that CaB improves composite execution and handoff quality on a first-person Minecraft VLA benchmark under matched capacity and deployability constraints.

Abstract

Vision-Language-Action Models

Explore similar work

Completion at the Boundary (CaB): Deployable Switching with Completion-Aware Control under Limited Calibration

SWAP: Stepwise Action Policy Routing for Vision-Language-Action Models

When Instructions Retrieve Trajectories: Diagnosing and Mitigating Generalization Failures in VLA Models