Robot policies are expected to generalize across deployment environments, including variations in the robot itself: cameras are bumped and remounted, calibrations drift, controllers run at different rates, and end-effector pads wear down. We present LIBERO-RIG, a benchmark that perturbs only the robot's hardware and setup while holding every LIBERO task fixed. The benchmark spans eleven perturbation axes, including camera mounting and optics, camera-to-arm calibration, actuation gain, control rate, proprioceptive noise, base pose, table height, and gripper pads. Each axis has five severity levels, yielding 210 conditions across the four LIBERO suites. We evaluate nine recent vision-language-action models using 500 episodes per condition and substantiate each finding with rigorous statistical tests. Pooled success falls from 0.932 to 0.767 at the middle severity and 0.509 at the highest severity. Performance degradation varies substantially across perturbation axes and action interfaces, and training on visual perturbations does not mitigate faults that leave camera images unchanged. We release the benchmark together with manifests specifying the fixed perturbation parameters for each episode.
Figures & tables
Figure 1: Same task, perturbed rig. The instruction, object identities, goal, initial state, and success predicate remain fixed while one of 11 rig properties varies across five severity levels. (a) Six observable axes (blue) alter camera views of the same arm and object configuration; changes in gripper-pad geometry (orange) are visible only in close-range wrist-camera views. (b) Four latent axes (green) alter command execution or reported state without changing the image. (c) Overall success at each severity level for the eight models evaluated on all four suites, with 95% intervals. Per-suite results appear in Fig. 3 , numerical values in Table 3 , and severity definitions in Table 1 . Circled numbers 1 – 11 identify the axes consistently across all tables and figures.
Table 1: The 11 perturbation axes. Circled numbers match those in Fig. 1 . Fault types are defined by the fault’s direct effect on the camera images at a fixed arm and object configuration: O observable, L latent, W weakly observable (visible only in close-range wrist views). Each severity ladder runs from s1 to s5; Appendix B lists the parameters at each level.
Table 2: Raw success rate per axis, 500 episodes per condition , Nominal is the unperturbed rate. Bold: best four-suite value per column. † LingBot-VA runs LIBERO-10 only and is not ranked.
Figure 2: Success rate by axis at severity 3 , pooled over the eight fully tested models (excluding LingBot-VA); dashed line, pooled nominal; observable axes blue, latent axes green, the weakly observable axis orange, with the axis numbers of Fig. 1 .
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Success rate by suite and severity level , level 1 light to level 5 dark, pooled over the eight four-suite models; the dashed line is each suite’s pooled nominal.
Figure 4: Rank stability at severity 3. Probability that each model lands at each rank over 5,000 task-resampled draws, paired across models.
Figure 5: What the severity ladders look like, axis by axis. Cosmos Policy, spatial suite, task “pick up the black bowl between the plate and the ramekin and place it on the plate”, episode 0: the first frame of the same episode under each perturbation, so panels differ only by the rig. Every axis a scene camera can show is included. The action-frame row is flat because the fault is latent. The wrist-camera row is flat only because these renders record the scene camera while that axis moves the wrist camera. On the camera-position row the workspace slides out of shot as the level rises.
Table 5: Tolerance per axis. The harshest severity level each model passes before its first failure on the axis, given as that level’s leading parameter (ladders in Table 1 , both parameters of every level in Appendix B ), read from the pooled success per level ( Tables 8 , 9 , 10 , 11 and 12 ). Criterion A passes a level when success is at least 0.80; criterion B when it is at least 0.90 of the model’s own nominal rate. ≥ marks a model that passes the harshest level, < one that fails the mildest. Control frequency 7 is not ordered by dose, so its cell gives the range of rates passed around the nominal 20 Hz. An asterisk marks a cell whose deciding rate lies within 0.02 of the bar. Levels are set per axis, so the table compares models within an axis, not axes with one another. † LingBot-VA is evaluated on LIBERO-10 only, where table height is not applied.
Model
libero-10 minus mean of other suites (s3)
NORA
−0.184
π0
−0.165
MolmoAct
−0.129
π0.5
−0.100
OpenVLA-OFT
−0.070
OFT-Plus
−0.067
Appendix
Table 7: Long-horizon gap per model at severity 3. Success on LIBERO-10 minus the mean over the other three suites, over the ten axes shared by all suites (table height excluded).
Figure 6: Success rate by axis at severities 4 (top) and 5 (bottom) , pooled over the eight four-suite models; dashed line is the pooled nominal; colors as in Fig. 2 .
Figure 7: Per-suite success with intervals (top) and rank stability (bottom) at severity 3. Top: raw success rate per suite, mean over the suite’s axes, with 95 percent intervals from a cluster bootstrap over the 10 tasks, the same task resamples for every model. Bottom: as Fig. 4 .
Figure 8: Per-suite success with intervals (top) and rank stability (bottom) at severity 4 , as in Fig. 7 . Cosmos Policy takes the object and LIBERO-10 suites.
Figure 9: Per-suite success with intervals (top) and rank stability (bottom) at severity 5 , as in Fig. 7 . First place in the object suite is a coin flip between OFT-Plus and π0.5 .
Figure 10: Success rate by suite on the ten shared axes at severities 3, 4, and 5 (top to bottom). Dots are the eight four-suite models.
Figure 11: Full results grid, severity 1. Rows are models, columns the four suites, 11 bars per panel in a fixed axis order. The dashed line is that model’s own unperturbed rate on that suite, so a bar reaching the line means the perturbation cost nothing. The numbers are in Tables 8 , 9 , 10 , 11 and 12 . This figure plots Table 8 ; Figs. 12 , 13 , 14 and 15 plot Tables 9 , 10 , 11 and 12 for severities 2 to 5.
Figure 12: Full results grid, severity 2 , as in Fig. 11 ; the numbers are in Table 9 .
Figure 13: Full results grid, severity 3 , as in Fig. 11 ; the numbers are in Table 10 .
Figure 14: Full results grid, severity 4 , as in Fig. 11 ; the numbers are in Table 11 .
Figure 15: Full results grid, severity 5 , as in Fig. 11 ; the numbers are in Table 12 .
Table 8: Raw success rate per model, suite, and axis at severity 1, plotted in Fig. 11 . Nom. is the model’s unperturbed rate on that suite.
Table 9: Raw success rate per model, suite, and axis at severity 2, plotted in Fig. 12 .
Table 10: Raw success rate per model, suite, and axis at severity 3, plotted in Fig. 13 .
Table 11: Raw success rate per model, suite, and axis at severity 4, plotted in Fig. 14 .
Table 12: Raw success rate per model, suite, and axis at severity 5, plotted in Fig. 15 .
Axis
Success at s3
Letters
camera position
0.448
a
camera orientation
0.596
b
robot-base pose shift
0.692
bc
action-frame rotation
0.752
bc
camera field of view
0.777
cde
control-frequency mismatch
0.786
bce
Appendix
Table 13: Axis ordering at severity 3 with compact letters. Raw success pooled over the eight four-suite models. Letters are the maximal sets of axes that the Holm-corrected paired sign-flip tests do not separate, so two axes share a letter exactly when they are not separated; 35 of the 55 pairs are separated. The groups overlap, so they are not tiers.
Model
Nominal
Harshest dose (s5)
Change
π0.5
0.969
0.968
−0.001
Cosmos Policy
0.973
0.958
−0.015
OFT-Plus
0.970
0.961
−0.009
OpenVLA-OFT
0.966
0.947
−0.019
π0
0.919
0.879
−0.040
X-VLA
0.976
0.777
−0.199
Appendix
Table 15: What each model does with a corrupted position report. Confirm tier. † single suite.
Robotic foundation models achieve impressive performance on standard manipulation benchmarks, yet these evaluations typically assume clean, timely, and consistent visual observations throughout execution. We introduce LIBERO-VPro, a benchmark for systematically evaluating the closed-loop visual robustness of robotic foundation models by perturbing the visual evidence available during execution. LIBERO-VPro covers four complementary dimensions, including Visual Evidence Degradation, Camera Staleness, Visual Source Consistency, and Task-Relevant Scene Variation, spanning 12 challenge categories, 96 experimental settings, and 3,296 task-condition cases. We evaluate three vision-language-action models and three world-action models over approximately 196,000 simulated episodes, complemented by 200 real-world rollouts on a Franka Research 3. Our results reveal that strong nominal performance can mask substantial weaknesses in visual grounding and adaptation. Models often remain successful despite severe object-level occlusion, yet degrade sharply when local interaction cues are disrupted or familiar spatial priors are violated. They are also highly sensitive to stale or missing observations and struggle when changed task preconditions require behavioral adaptation. Finally, VLAs and WAMs exhibit distinct robustness profiles, showing that visual robustness is multi-dimensional and architecture-dependent. LIBERO-VPro provides a systematic diagnostic framework for developing robotic foundation models that can more reliably ground and adapt their actions under challenging visual conditions.
Huiqiong Li, Zhiting Mei, Anirudha Majumdar +3
Singapore Management University · Princeton University · Fudan University
LIBERO has emerged as a widely adopted benchmark for evaluating robotic manipulation policies. However, strong performance on LIBERO does not necessarily indicate robust embodied capabilities, as models may exploit benchmark-specific regularities rather than generalize to diverse operating conditions. In this work, we introduce LIBERO-Pro2.0, a comprehensive benchmark for systematically evaluating the robustness of embodied foundation models under distribution shifts. LIBERO-Pro2.0 encompasses 22 static perturbations and 20 runtime interventions across six dimensions: Condition, Environment, Observation, Execution, Robot State Configuration, and Language, yielding 1,648 validated shifted task instances. To support robustness-oriented learning, we further collect 39,390 training trajectories through model-generated and human demonstrations. All shifted task instances undergo six rounds of iterative human review and correction to ensure physical plausibility and task validity. We evaluate 17 representative robot policies spanning vision-language-action models, world-action models, and robustness-oriented policies. Our results reveal substantial robustness gaps across all evaluated paradigms: under runtime distribution shifts, the evaluated models exhibit an average performance drop of 26.5 percentage points, with individual drops reaching 32.5 points. These findings highlight the gap between nominal benchmark performance and robustness under distribution shifts, underscoring the need for systematic robustness evaluation of embodied foundation models.
Xueyang Zhou, Yangming Xu, Qianjiang Li +10
Huazhong University of Science and Technology · Tsinghua University · University of Toronto +2
Vision-Language-Action (VLA) models have achieved high task success rates on robot manipulation task benchmarks. More recently, there has been an emphasis on evaluating the robustness of VLA models to perturbations. However, this robustness is still predominantly measured through Task Success Rate (TSR). In this work, we propose a benchmark-agnostic evaluation framework to measure the behavioural robustness of models by characterising how successful trajectories are executed under perturbation. We implement this methodology by extending the widely-used LIBERO and LIBERO-Plus benchmarks. Across three state-of-the-art VLA models, four LIBERO task suites and seven perturbation conditions, we evaluate changes in both typical successful behaviour and its variability, including metrics of motion smoothness, efficiency and gripper behaviour. We find that perturbations can alter the behaviour of successful trajectories, a phenomenon which cannot necessarily be inferred from TSR alone. Across LIBERO suites, we identify cases where state-of-the-art VLA models achieve comparable TSR under the same perturbation condition, yet behaviour on successful trajectories diverges substantially. Therefore, to have a more robust assessment of task performance, we argue that suitable measures of robustness should capture not only whether a task is completed, but also how the robot behaves while completing it. When evaluating the robustness of VLA models, TSR may be complemented by behavioural evaluation metrics that characterise the nature and variability of successful task execution by robots.
Sophie Higham, Riccardo Andrea Izzo, Matteo Matteucci +1
School of Informatics, University of Edinburgh, UK · Department of Electronics, Informatics and Bioengineering, Politecnico di Milano, Italy