RoboCasa-Shift: Stage-Aware Robustness Evaluation of Vision-Language-Action Policies in Long-Horizon Household Manipulation

Haocheng Wang; Youquan Liu; Weiyu Guo; Aokai Zhang; Donghang Wu; Hongyu Liu; Sha Zhang; Weidong Yang

Fig. 1: Overview of RoboCasa-Shift. RoboCasa-Shift evaluates VLA robustness on selected long-horizon RoboCasa tasks under controlled single-factor distribution shifts with multiple frozen severity levels. Strictly matched clean-perturbed rollouts enable counterfactual comparison under shared task conditions, while robustness is characterized jointly through end-to-end task outcomes and stage-aware semantic execution diagnostics. Representative findings illustrate that aggregate success rates can mask substantial episode-level outcome changes, similar end-to-end degradation can correspond to different failure locations, and severe visual corruption can shift failures toward earlier stages of task execution.

Fig. 2: Pipeline of RoboCasa-Shift. (a) Task Suite. Candidate RoboCasa tasks are filtered through clean-competence screening to construct the formal evaluation suite. (b) Controlled Shifts. Selected tasks are evaluated under four single-factor distribution shifts with calibrated and frozen severity levels. (c) Counterfactual Pairing. Clean and perturbed rollouts share the underlying task conditions and policy seed and differ only in the target shift. (d) Diagnosis. RoboCasa’s official task outcome is complemented by stage-aware semantic diagnostics of execution progress, first incomplete subgoal, and subgoal reach.

Task	Horizon	Mobility	Official Structure	Diagnostic Milestones
PreSoakPan	2400	Non-mobile	3	3
RinseSinkBasin	1350	Non-mobile	2	3
StackBowlsCabinet	2100	Mobile	2	2
WashLettuce	1650	Non-mobile	2	3

TABLE I: Formal task suite used in RoboCasa-Shift. The official horizon, mobility label, and task-structure value are obtained from RoboCasa365 task metadata. Diagnostic milestones denote the task-specific semantic execution milestones used for stage-aware analysis and are not used as task-complexity labels.

Component	Setting	Validation	Result
Environment	Shared episode config. + seed	State fingerprint	✓
Policy RNG	Uncontrolled	Action sequence	$\times$
Policy RNG	Shared episode-level seed	Action sequence	✓

TABLE II: Validation of the counterfactual pairing protocol. Environment-side pairing is verified through the initial-state fingerprint. The policy-side comparison shows that uncontrolled inference randomness breaks repeatability, whereas sharing the episode-level policy seed restores consistent action sequences.

Fig. 3: Overview of StageTracker. (a) A long-horizon task is represented as an ordered sequence of semantic subgoals, each associated with a simulator-state predicate. (b) During rollout, StageTracker evaluates the current predicate, records the completion step when it is first satisfied, and advances to the next subgoal. Completed subgoals are latched, allowing the tracker to report the deepest semantic progress reached and the first incomplete subgoal for failure localization.

Fig. 4: End-to-end robustness under controlled distribution shifts. Macro-averaged paired success-rate degradation

\Delta\mathrm{SR}

is shown for the four distribution-shift factors, with each factor evaluated at three frozen severity levels (

L_{1}

–

L_{3}

). Positive values indicate degradation relative to the matched clean condition. Error bars denote 95% confidence intervals obtained from paired bootstrap resampling. Severity levels are ordinal within each perturbation factor, and equal level indices across different factors do not imply physically equivalent perturbation magnitudes.

Fig. 5: Task-level robustness under controlled distribution shifts. Each cell reports the paired success-rate degradation

\Delta\mathrm{SR}

for one task and perturbation condition, computed from 20 matched episode pairs. Positive values indicate degradation relative to the matched clean condition, while negative values indicate a higher success rate under the shifted condition for the corresponding finite paired sample. The task-level breakdown reveals heterogeneous severity responses that can be obscured by macro aggregation.

Factor	Level	$\boldsymbol{\Delta\mathrm{SR}}$	$\mathbf{S}\!\rightarrow\!\mathbf{F}$	$\mathbf{F}\!\rightarrow\!\mathbf{S}$	Changed
Illumination	$L_{1}$	0.012	7	6	13
	$L_{2}$	-0.025	5	7	12
	$L_{3}$	0.025	10	8	18
Camera	$L_{1}$	0.000	4	4	8
	$L_{2}$	0.188	21	6	27
	$L_{3}$	0.138	18	7	25

TABLE III: Counterfactual outcome transitions across all perturbation conditions. Each condition contains 80 strictly matched clean–perturbed episode pairs. Changed denotes the number of pairs whose binary task outcome differs between the clean and perturbed executions.

Fig. 6: Stage-aware execution depth of perturbation-induced failures. Each marker summarizes strictly matched clean-success-to-perturbed-failure (

\mathrm{S}\!\rightarrow\!\mathrm{F}

) transitions under one perturbation factor and severity level. The vertical position indicates the mean normalized subgoal progress of the corresponding

\mathrm{S}\!\rightarrow\!\mathrm{F}

cohort, marker size represents the number of transition events, and marker color denotes the fraction of failures occurring before the first tracked semantic milestone. The annotated

n

gives the number of transitions in each condition. Gray lines connect

L_{1}

–

L_{3}

within the same perturbation factor only to visualize the within-factor severity trend and do not encode an additional quantity. Severity levels are ordinal within each perturbation factor and are not assumed to be physically equivalent across factors.

Fig. 7: Representative matched replays at different semantic failure depths. Each case compares a strictly matched clean–perturbed rollout pair that shares the same task instance, initial simulator state, episode metadata, and episode-level policy seed, while differing only in the target perturbation. The two rows show the clean and perturbed executions, respectively, and the columns present selected frames spanning the execution process. Checkmarks and crosses indicate whether the corresponding tracked semantic subgoals are completed, while the final annotation reports the official task outcome. These examples provide a qualitative view of how perturbations can change execution depth and failure location, complementing the aggregate stage-aware statistics in Figure 6 .

Task	Initial	Final $N$	Success	Clean SR	Wilson 95% CI	Status
DeliverStraw	2/10	10	2	0.20	[0.057, 0.510]	Floor
GetToastedBread	0/10	10	0	0.00	[0.000, 0.278]	Floor
KettleBoiling	5/10	20	9	0.45	[0.258, 0.658]	Below gate
LoadDishwasher	5/10	20	9	0.45	[0.258, 0.658]	Below gate
PackIdenticalLunches	1/10	10	1	0.10	[0.018, 0.404]	Floor
PreSoakPan	7/10	20	15	0.75	[0.531, 0.888]	Eligible

TABLE IV: Complete clean-competence screening results. The screening rule and eligibility threshold are defined in Section III-A . Tasks not extended beyond the initial 10 episodes are clean-floor cases.

Fig. 8: Representative severity calibration examples. Rows correspond to the four perturbation factors, and columns show the clean condition (

L_{0}

) and three calibrated severity levels (

L_{1}

–

L_{3}

). Illumination progressively darkens the scene, camera orientation changes the third-person viewpoint, robot base initialization perturbs the initial base position and yaw orientation, and Gaussian blur progressively reduces visual sharpness. For base initialization, the inset provides a schematic top-down view, where the black and orange markers represent the nominal and perturbed base poses, respectively, while the straight and curved arrows indicate planar displacement and yaw rotation. Severity levels are ordinal within each perturbation factor and are not assumed to be physically equivalent across factors.

Validation	Procedure	Result
Environment state	Pre-shift simulator fingerprint match	Pass
Policy stochasticity	Shared checkpoint and RNG state	Pass
Rollout metadata	Matched episode configuration	Pass

TABLE V: Pairing integrity validation. Environment-side checks verify consistency of the simulator configuration before perturbation injection, while policy-side checks control stochastic variation during paired execution.

Task	Milestone	Completion criterion
PreSoakPan	$g_{1}$	Pan in sink;
	$g_{2}$	Sponge in sink.
	$g_{3}$	Sink water is on.
RinseSinkBasin	$g_{1}$	Sink water is on.
	$g_{2}$	At least two basin zones rinsed.
	$g_{3}$	All three basin zones rinsed.

TABLE VI: Task-specific semantic subgoal specifications. Each

g_{k}

denotes a task-specific diagnostic milestone. The criteria were frozen before formal perturbation evaluation. For WashLettuce, thresholds refer to the simulator washed_time counter.

Task	Expected	Valid	Pairs	Status
PreSoakPan	260	260	20	PASS
RinseSinkBasin	260	260	20	PASS
StackBowlsCabinet	260	260	20	PASS
WashLettuce	260	260	20	PASS

TABLE VII: Integrity validation of the formal evaluation. Each task contains 20 fixed base episodes evaluated under 13 conditions, resulting in 260 expected rollouts per task.

Factor	Level	SR	$\Delta\mathrm{SR}$	95% CI of $\Delta\mathrm{SR}$
Clean	$L_{0}$	0.738	0.000	[0.000, 0.000]
Illumination	$L_{1}$	0.725	0.012	[-0.075, 0.100]
Illumination	$L_{2}$	0.762	-0.025	[-0.112, 0.062]
Illumination	$L_{3}$	0.713	0.025	[-0.075, 0.125]
Camera Orientation	$L_{1}$	0.738	0.000	[-0.063, 0.075]
Camera Orientation	$L_{2}$	0.550	0.188	[0.063, 0.313]

TABLE VIII: Complete macro-level end-to-end robustness results. Success rate (SR) and paired success-rate degradation

\Delta\mathrm{SR}

are averaged equally across the four formal tasks. Positive

\Delta\mathrm{SR}

indicates degradation relative to the matched clean condition. Confidence intervals are obtained from paired bootstrap resampling.

Factor	Level	$\mathrm{S}\!\rightarrow\!\mathrm{S}$	$\mathrm{S}\!\rightarrow\!\mathrm{F}$	$\mathrm{F}\!\rightarrow\!\mathrm{S}$	$\mathrm{F}\!\rightarrow\!\mathrm{F}$
Illumination	$L_{1}$	52	7	6	15
	$L_{2}$	54	5	7	14
	$L_{3}$	49	10	8	13
Camera Orientation	$L_{1}$	55	4	4	17
	$L_{2}$	38	21	6	15
	$L_{3}$	41	18	7	14

TABLE IX: Complete counterfactual outcome transitions across all perturbation conditions. Each row contains 80 strictly matched clean–perturbed episode pairs, categorized into four mutually exclusive outcomes: success-to-success (

\mathrm{S}\!\rightarrow\!\mathrm{S}

), success-to-failure (

\mathrm{S}\!\rightarrow\!\mathrm{F}

), failure-to-success (

\mathrm{F}\!\rightarrow\!\mathrm{S}

), and failure-to-failure (

\mathrm{F}\!\rightarrow\!\mathrm{F}

).

Model and Inference		Runtime Environment
Configuration	Value	Configuration	Value
Policy	GR00T N1.5	Base container	nvidia24.04-ubuntu22.04
Checkpoint	composite_seen/checkpoint-60000	Base Python / CUDA	Python 3.10 / CUDA 12.4
Evaluation split	Target	Base cuDNN / PyTorch	cuDNN 9.1 / PyTorch 2.3.0
Training / task scope	Composite-Seen	GPU	NVIDIA GeForce RTX 5090
Data configuration	panda_omron	GPU memory	32607 MiB

TABLE X: Model, inference, and runtime configuration used in the formal evaluation. The left half summarizes the GR00T policy and evaluation settings, while the right half reports the software and hardware environment. Runtime package versions are reported separately from the underlying container configuration.

Factor	Level	$N_{\mathrm{S}\rightarrow\mathrm{F}}$	Mean Progress	First-stage (%)
Illumination	$L_{1}$	7	0.595	0.0
	$L_{2}$	5	0.267	40.0
	$L_{3}$	10	0.333	50.0
Camera Orientation	$L_{1}$	4	0.458	25.0
	$L_{2}$	21	0.437	19.0
	$L_{3}$	18	0.417	27.8

TABLE XI: Condition-level statistics for perturbation-induced failures. Statistics are computed over strictly matched clean-success-to-perturbed-failure (

\mathrm{S}\!\rightarrow\!\mathrm{F}

) transitions.

N_{\mathrm{S}\rightarrow\mathrm{F}}

denotes the number of transition events. First-stage reports the percentage of these transitions whose first incomplete tracked milestone is

g_{1}

.

Task	GR00T N1.5	SmolVLA	$\pi_{0}$
PreSoakPan	15/20	0/10	0/10
RinseSinkBasin	14/20	0/10	0/10
StackBowlsCabinet	16/20	0/10	2/10
WashLettuce	14/20	1/10	0/10
Eligible tasks	4/4	0/4	0/4

TABLE XII: Clean-competence screening on the frozen four-task suite. GR00T N1.5 results are from the second-round final clean screening (

N=20

per task), whereas SmolVLA and

\pi_{0}

results are from the third-round initial screen (

N=10

per task). Neither additional policy reached the clean-screen extension threshold on any task.

Fig. 9: Task-wise distributions of first incomplete semantic milestones among perturbation-induced failures. Each subplot corresponds to one formal task. For each perturbation factor and severity level, the stacked bar shows the distribution of first incomplete milestones

g_{1},\ldots,g_{K}

among strictly matched

\mathrm{S}\!\rightarrow\!\mathrm{F}

transitions, where

K

depends on the task. Tracker-complete official failures, in which all tracked milestones are reached but RoboCasa’s official task-success criterion remains unsatisfied, are reported separately without assigning a first incomplete milestone. The proportions are calculated over all

\mathrm{S}\!\rightarrow\!\mathrm{F}

transitions in the corresponding task and condition. The annotation above each bar indicates the number of transitions

n

; conditions without such transitions are marked as

n=0

.

Fig. 10: Representative counterfactual outcome transitions under camera orientation at

L_{1}

. Case A (StackBowlsCabinet) shows a success-to-failure (

\mathrm{S}\!\rightarrow\!\mathrm{F}

) transition, whereas Case B (PreSoakPan) shows a failure-to-success (

\mathrm{F}\!\rightarrow\!\mathrm{S}

) transition. Within each case, the clean and perturbed rollouts share the same task instance, initial simulator state, and episode-level policy seed, differing only in the target camera-orientation perturbation. Columns show selected frames from the initial, intermediate, and final stages of execution, with the final outcomes determined by RoboCasa’s official task-success criterion. These examples illustrate opposite episode-level outcome changes under the same perturbation condition, despite zero aggregate success-rate degradation at

L_{1}

.

RoboCasa-Shift: Stage-Aware Robustness Evaluation of Vision-Language-Action Policies in Long-Horizon Household Manipulation

Abstract

Figures & tables

Robustness of VLA Models

VLA-Scope: Shift-Aware Failure Prediction for Vision-Language-Action Models

Retrieve to Act: Motion Primitive Graph Retrieval for Robust VLA Execution

Is Success All You Need? Investigating the Impact of Input Perturbations on VLA Behaviour in Tabletop Manipulation Tasks

Abstract

Robustness of VLA Models

Explore similar work

VLA-Scope: Shift-Aware Failure Prediction for Vision-Language-Action Models

Retrieve to Act: Motion Primitive Graph Retrieval for Robust VLA Execution

Is Success All You Need? Investigating the Impact of Input Perturbations on VLA Behaviour in Tabletop Manipulation Tasks