Organizations: Fudan University · Shenzhen Loop Area Institute, Shenzhen, China · The Chinese University of Hong Kong · University of Chinese Academy of Sciences · Shanghai Jiao Tong University
Vision-language-action (VLA) robustness is commonly summarized by aggregate task success, which obscures how individual executions change under distribution shifts and where failures emerge in long-horizon tasks. We introduce RoboCasa-Shift, a diagnostic robustness benchmark built on RoboCasa365 that combines controlled single-factor shifts, matched clean-perturbed rollouts in which each pair shares the same underlying settings and differs only in the target shift, and simulator-grounded stage-aware diagnostics. We evaluate GR00T N1.5 on four long-horizon tasks that pass a predefined clean-performance gate under four perturbation factors and three calibrated severity levels, including conditions that drive the evaluated policy to complete task failure, totaling 1,040 formal rollouts. Robustness responses vary substantially across factors and tasks, and increasing severity does not always produce monotonic degradation. For example, one camera condition yields ΔSR=0 despite 8 of 80 matched episodes changing outcome. Stage-aware analysis further shows that the same end-to-end degradation can correspond to different execution-depth profiles. Camera L2 and Gaussian blur L1 both yield ΔSR=0.188, yet their first-stage failure rates are 19.0% and 36.4%, respectively. Our findings suggest that VLA robustness should be viewed as a multi-dimensional property of execution rather than a single end-to-end performance measure, motivating evaluation protocols that jointly capture outcome degradation, paired outcome sensitivity, and failure progression. The code is publicly available at https://github.com/HaochengWang23/RoboCasa-Shift.
Figures & tables
Fig. 1: Overview of RoboCasa-Shift. RoboCasa-Shift evaluates VLA robustness on selected long-horizon RoboCasa tasks under controlled single-factor distribution shifts with multiple frozen severity levels. Strictly matched clean-perturbed rollouts enable counterfactual comparison under shared task conditions, while robustness is characterized jointly through end-to-end task outcomes and stage-aware semantic execution diagnostics. Representative findings illustrate that aggregate success rates can mask substantial episode-level outcome changes, similar end-to-end degradation can correspond to different failure locations, and severe visual corruption can shift failures toward earlier stages of task execution.
Fig. 2: Pipeline of RoboCasa-Shift. (a) Task Suite. Candidate RoboCasa tasks are filtered through clean-competence screening to construct the formal evaluation suite. (b) Controlled Shifts. Selected tasks are evaluated under four single-factor distribution shifts with calibrated and frozen severity levels. (c) Counterfactual Pairing. Clean and perturbed rollouts share the underlying task conditions and policy seed and differ only in the target shift. (d) Diagnosis. RoboCasa’s official task outcome is complemented by stage-aware semantic diagnostics of execution progress, first incomplete subgoal, and subgoal reach.
Task
Horizon
Mobility
Official Structure
Diagnostic Milestones
PreSoakPan
2400
Non-mobile
3
3
RinseSinkBasin
1350
Non-mobile
2
3
StackBowlsCabinet
2100
Mobile
2
2
WashLettuce
1650
Non-mobile
2
3
TABLE I: Formal task suite used in RoboCasa-Shift. The official horizon, mobility label, and task-structure value are obtained from RoboCasa365 task metadata. Diagnostic milestones denote the task-specific semantic execution milestones used for stage-aware analysis and are not used as task-complexity labels.
Component
Setting
Validation
Result
Environment
Shared episode config. + seed
State fingerprint
✓
Policy RNG
Uncontrolled
Action sequence
×
Policy RNG
Shared episode-level seed
Action sequence
✓
TABLE II: Validation of the counterfactual pairing protocol. Environment-side pairing is verified through the initial-state fingerprint. The policy-side comparison shows that uncontrolled inference randomness breaks repeatability, whereas sharing the episode-level policy seed restores consistent action sequences.
Fig. 3: Overview of StageTracker. (a) A long-horizon task is represented as an ordered sequence of semantic subgoals, each associated with a simulator-state predicate. (b) During rollout, StageTracker evaluates the current predicate, records the completion step when it is first satisfied, and advances to the next subgoal. Completed subgoals are latched, allowing the tracker to report the deepest semantic progress reached and the first incomplete subgoal for failure localization.
Fig. 4: End-to-end robustness under controlled distribution shifts. Macro-averaged paired success-rate degradation ΔSR is shown for the four distribution-shift factors, with each factor evaluated at three frozen severity levels ( L1 – L3 ). Positive values indicate degradation relative to the matched clean condition. Error bars denote 95% confidence intervals obtained from paired bootstrap resampling. Severity levels are ordinal within each perturbation factor, and equal level indices across different factors do not imply physically equivalent perturbation magnitudes.
Fig. 5: Task-level robustness under controlled distribution shifts. Each cell reports the paired success-rate degradation ΔSR for one task and perturbation condition, computed from 20 matched episode pairs. Positive values indicate degradation relative to the matched clean condition, while negative values indicate a higher success rate under the shifted condition for the corresponding finite paired sample. The task-level breakdown reveals heterogeneous severity responses that can be obscured by macro aggregation.
Factor
Level
ΔSR
S→F
F→S
Changed
Illumination
L1
0.012
7
6
13
L2
-0.025
5
7
12
L3
0.025
10
8
18
Camera
L1
0.000
4
4
8
L2
0.188
21
6
27
L3
0.138
18
7
25
TABLE III: Counterfactual outcome transitions across all perturbation conditions. Each condition contains 80 strictly matched clean–perturbed episode pairs. Changed denotes the number of pairs whose binary task outcome differs between the clean and perturbed executions.
Fig. 6: Stage-aware execution depth of perturbation-induced failures. Each marker summarizes strictly matched clean-success-to-perturbed-failure ( S→F ) transitions under one perturbation factor and severity level. The vertical position indicates the mean normalized subgoal progress of the corresponding S→F cohort, marker size represents the number of transition events, and marker color denotes the fraction of failures occurring before the first tracked semantic milestone. The annotated n gives the number of transitions in each condition. Gray lines connect L1 – L3 within the same perturbation factor only to visualize the within-factor severity trend and do not encode an additional quantity. Severity levels are ordinal within each perturbation factor and are not assumed to be physically equivalent across factors.
Fig. 7: Representative matched replays at different semantic failure depths. Each case compares a strictly matched clean–perturbed rollout pair that shares the same task instance, initial simulator state, episode metadata, and episode-level policy seed, while differing only in the target perturbation. The two rows show the clean and perturbed executions, respectively, and the columns present selected frames spanning the execution process. Checkmarks and crosses indicate whether the corresponding tracked semantic subgoals are completed, while the final annotation reports the official task outcome. These examples provide a qualitative view of how perturbations can change execution depth and failure location, complementing the aggregate stage-aware statistics in Figure 6 .
Task
Initial
Final N
Success
Clean SR
Wilson 95% CI
Status
DeliverStraw
2/10
10
2
0.20
[0.057, 0.510]
Floor
GetToastedBread
0/10
10
0
0.00
[0.000, 0.278]
Floor
KettleBoiling
5/10
20
9
0.45
[0.258, 0.658]
Below gate
LoadDishwasher
5/10
20
9
0.45
[0.258, 0.658]
Below gate
PackIdenticalLunches
1/10
10
1
0.10
[0.018, 0.404]
Floor
PreSoakPan
7/10
20
15
0.75
[0.531, 0.888]
Eligible
TABLE IV: Complete clean-competence screening results. The screening rule and eligibility threshold are defined in Section III-A . Tasks not extended beyond the initial 10 episodes are clean-floor cases.
Fig. 8: Representative severity calibration examples. Rows correspond to the four perturbation factors, and columns show the clean condition ( L0 ) and three calibrated severity levels ( L1 – L3 ). Illumination progressively darkens the scene, camera orientation changes the third-person viewpoint, robot base initialization perturbs the initial base position and yaw orientation, and Gaussian blur progressively reduces visual sharpness. For base initialization, the inset provides a schematic top-down view, where the black and orange markers represent the nominal and perturbed base poses, respectively, while the straight and curved arrows indicate planar displacement and yaw rotation. Severity levels are ordinal within each perturbation factor and are not assumed to be physically equivalent across factors.
Validation
Procedure
Result
Environment state
Pre-shift simulator fingerprint match
Pass
Policy stochasticity
Shared checkpoint and RNG state
Pass
Rollout metadata
Matched episode configuration
Pass
TABLE V: Pairing integrity validation. Environment-side checks verify consistency of the simulator configuration before perturbation injection, while policy-side checks control stochastic variation during paired execution.
Task
Milestone
Completion criterion
PreSoakPan
g1
Pan in sink;
g2
Sponge in sink.
g3
Sink water is on.
RinseSinkBasin
g1
Sink water is on.
g2
At least two basin zones rinsed.
g3
All three basin zones rinsed.
TABLE VI: Task-specific semantic subgoal specifications. Each gk denotes a task-specific diagnostic milestone. The criteria were frozen before formal perturbation evaluation. For WashLettuce, thresholds refer to the simulator washed_time counter.
Task
Expected
Valid
Pairs
Hash Mismatch
Seed Mismatch
Status
PreSoakPan
260
260
20
0
0
PASS
RinseSinkBasin
260
260
20
0
0
PASS
StackBowlsCabinet
260
260
20
0
0
PASS
WashLettuce
260
260
20
0
0
PASS
TABLE VII: Integrity validation of the formal evaluation. Each task contains 20 fixed base episodes evaluated under 13 conditions, resulting in 260 expected rollouts per task.
Factor
Level
SR
ΔSR
95% CI of ΔSR
Clean
L0
0.738
0.000
[0.000, 0.000]
Illumination
L1
0.725
0.012
[-0.075, 0.100]
Illumination
L2
0.762
-0.025
[-0.112, 0.062]
Illumination
L3
0.713
0.025
[-0.075, 0.125]
Camera Orientation
L1
0.738
0.000
[-0.063, 0.075]
Camera Orientation
L2
0.550
0.188
[0.063, 0.313]
TABLE VIII: Complete macro-level end-to-end robustness results. Success rate (SR) and paired success-rate degradation ΔSR are averaged equally across the four formal tasks. Positive ΔSR indicates degradation relative to the matched clean condition. Confidence intervals are obtained from paired bootstrap resampling.
Factor
Level
S→S
S→F
F→S
F→F
Illumination
L1
52
7
6
15
L2
54
5
7
14
L3
49
10
8
13
Camera Orientation
L1
55
4
4
17
L2
38
21
6
15
L3
41
18
7
14
TABLE IX: Complete counterfactual outcome transitions across all perturbation conditions. Each row contains 80 strictly matched clean–perturbed episode pairs, categorized into four mutually exclusive outcomes: success-to-success ( S→S ), success-to-failure ( S→F ), failure-to-success ( F→S ), and failure-to-failure ( F→F ).
Model and Inference
Runtime Environment
Configuration
Value
Configuration
Value
Policy
GR00T N1.5
Base container
nvidia24.04-ubuntu22.04
Checkpoint
composite_seen/checkpoint-60000
Base Python / CUDA
Python 3.10 / CUDA 12.4
Evaluation split
Target
Base cuDNN / PyTorch
cuDNN 9.1 / PyTorch 2.3.0
Training / task scope
Composite-Seen
GPU
NVIDIA GeForce RTX 5090
Data configuration
panda_omron
GPU memory
32607 MiB
TABLE X: Model, inference, and runtime configuration used in the formal evaluation. The left half summarizes the GR00T policy and evaluation settings, while the right half reports the software and hardware environment. Runtime package versions are reported separately from the underlying container configuration.
Factor
Level
NS→F
Mean Progress
First-stage (%)
Illumination
L1
7
0.595
0.0
L2
5
0.267
40.0
L3
10
0.333
50.0
Camera Orientation
L1
4
0.458
25.0
L2
21
0.437
19.0
L3
18
0.417
27.8
TABLE XI: Condition-level statistics for perturbation-induced failures. Statistics are computed over strictly matched clean-success-to-perturbed-failure ( S→F ) transitions. NS→F denotes the number of transition events. First-stage reports the percentage of these transitions whose first incomplete tracked milestone is g1 .
Task
GR00T N1.5
SmolVLA
π0
PreSoakPan
15/20
0/10
0/10
RinseSinkBasin
14/20
0/10
0/10
StackBowlsCabinet
16/20
0/10
2/10
WashLettuce
14/20
1/10
0/10
Eligible tasks
4/4
0/4
0/4
TABLE XII: Clean-competence screening on the frozen four-task suite. GR00T N1.5 results are from the second-round final clean screening ( N=20 per task), whereas SmolVLA and π0 results are from the third-round initial screen ( N=10 per task). Neither additional policy reached the clean-screen extension threshold on any task.
Fig. 9: Task-wise distributions of first incomplete semantic milestones among perturbation-induced failures. Each subplot corresponds to one formal task. For each perturbation factor and severity level, the stacked bar shows the distribution of first incomplete milestones g1,…,gK among strictly matched S→F transitions, where K depends on the task. Tracker-complete official failures, in which all tracked milestones are reached but RoboCasa’s official task-success criterion remains unsatisfied, are reported separately without assigning a first incomplete milestone. The proportions are calculated over all S→F transitions in the corresponding task and condition. The annotation above each bar indicates the number of transitions n ; conditions without such transitions are marked as n=0 .
Fig. 10: Representative counterfactual outcome transitions under camera orientation at L1 . Case A (StackBowlsCabinet) shows a success-to-failure ( S→F ) transition, whereas Case B (PreSoakPan) shows a failure-to-success ( F→S ) transition. Within each case, the clean and perturbed rollouts share the same task instance, initial simulator state, and episode-level policy seed, differing only in the target camera-orientation perturbation. Columns show selected frames from the initial, intermediate, and final stages of execution, with the final outcomes determined by RoboCasa’s official task-success criterion. These examples illustrate opposite episode-level outcome changes under the same perturbation condition, despite zero aggregate success-rate degradation at L1 .
Vision-language-action (VLA) models map visual observations and natural-language instructions to robotic actions, but distribution shifts can compromise their reliability. Because these models may still succeed under out-of-distribution (OOD) conditions, detecting OOD inputs alone is insufficient to predict execution failure. In this paper, we introduce VLA-Scope, a two-stage framework that combines input-shift characterization with execution history to predict failure during OOD rollouts. The first stage uses pooled image and language representations to detect OOD inputs and classify their shift categories. For inputs flagged as OOD, the second stage combines the predicted category, action-prefix features, and execution progress features. A logistic regression model shared across shift categories updates failure risk as execution proceeds. We evaluate the framework with OpenVLA on ten LIBERO-Spatial tasks using leave-one-group-out cross-validation. OOD detection achieves a ROC-AUC of 0.9454, and shift classification achieves 91% accuracy. Evaluated independently of the OOD gate on all 1,400 OOD rollouts, the failure predictor achieves a ROC-AUC of 0.8497 after 60 executed actions, compared with 0.7906 without execution progress features. It also achieves a higher ROC-AUC than the evaluated ActProbe and SAFE-MLP baselines. These results suggest that combining action features with temporally aggregated execution step representations improves failure prediction under input shifts.
Kaiwen Zhu, Dongfang Liu, Liangkai Liu
Department of Computer Science, Texas Tech University, Lubbock, TX, USA · Purdue University, West Lafayette, IN, USA
Vision-Language-Action (VLA) policies demonstrate remarkable semantic generalization, yet remain highly vulnerable to out-of-distribution (OOD) visual shifts during execution. While existing test-time methods attempt to suppress spurious visual cues, they fundamentally overlook the rich, uncorrupted kinematic priors embedded in the policy's own training experience. We introduce Retrieve to Act (R2A), a test-time scaling framework that formulates OOD robustness purely as a geometric motion primitive retrieval problem. R2A bypasses fragile visual features by structuring successful training demonstrations into an offline spatiotemporal motion primitive graph. During inference, a query-dependent Graph Neural Network dynamically localizes the robot's ongoing execution within this graph, retrieving structurally compatible future motion primitives. These retrieved geometric priors are then seamlessly harmonized with the frozen VLA's predicted actions via a learned, dimension-wise gating mechanism, adaptively rectifying erroneous spatial trajectories while preserving the policy's local reactivity. Extensive experiments across three state-of-the-art VLA architectures, GR00T N1.7, π0.5, and StableVLA, demonstrate that R2A consistently and significantly enhances robustness against diverse contextual perturbations. R2A achieves absolute success rate improvements of up to +12% in simulation benchmarks (LIBERO-Plus, RoboTwin2.0-Plus) and +40% in real-world deployments, without requiring any finetuning of the base policies. Project page: https://retrieve2act.github.io/r2a/
Pham Tri Quang, Bao-Ngoc Dao, Minh Duc Nguyen +15
Center for AI Research, VinUniversity · VinRobotics · Hanoi University of Science and Technology +6
Vision-Language-Action (VLA) models have achieved high task success rates on robot manipulation task benchmarks. More recently, there has been an emphasis on evaluating the robustness of VLA models to perturbations. However, this robustness is still predominantly measured through Task Success Rate (TSR). In this work, we propose a benchmark-agnostic evaluation framework to measure the behavioural robustness of models by characterising how successful trajectories are executed under perturbation. We implement this methodology by extending the widely-used LIBERO and LIBERO-Plus benchmarks. Across three state-of-the-art VLA models, four LIBERO task suites and seven perturbation conditions, we evaluate changes in both typical successful behaviour and its variability, including metrics of motion smoothness, efficiency and gripper behaviour. We find that perturbations can alter the behaviour of successful trajectories, a phenomenon which cannot necessarily be inferred from TSR alone. Across LIBERO suites, we identify cases where state-of-the-art VLA models achieve comparable TSR under the same perturbation condition, yet behaviour on successful trajectories diverges substantially. Therefore, to have a more robust assessment of task performance, we argue that suitable measures of robustness should capture not only whether a task is completed, but also how the robot behaves while completing it. When evaluating the robustness of VLA models, TSR may be complemented by behavioural evaluation metrics that characterise the nature and variability of successful task execution by robots.
Sophie Higham, Riccardo Andrea Izzo, Matteo Matteucci +1
School of Informatics, University of Edinburgh, UK · Department of Electronics, Informatics and Bioengineering, Politecnico di Milano, Italy