LIBERO-RIG: A Hardware-Setup Robustness Benchmark for Vision-Language-Action Models

Andrea Yaoyun Cui; Hanyu Wang; Xinyi Wang; Bangzheng Li; Lin Zhang; Zhenggang Tang; Shyamal Buch; Dejia Xu

cs.ROOct 13, 2026

LIBERO-RIG: A Hardware-Setup Robustness Benchmark for Vision-Language-Action Models

Authors: Andrea Yaoyun Cui, Hanyu Wang, Xinyi Wang, Bangzheng Li, Lin Zhang, Zhenggang Tang, Shyamal Buch, Dejia Xu,

Organizations: Luma AI

Abstract

Robot policies are expected to generalize across deployment environments, including variations in the robot itself: cameras are bumped and remounted, calibrations drift, controllers run at different rates, and end-effector pads wear down. We present LIBERO-RIG, a benchmark that perturbs only the robot's hardware and setup while holding every LIBERO task fixed. The benchmark spans eleven perturbation axes, including camera mounting and optics, camera-to-arm calibration, actuation gain, control rate, proprioceptive noise, base pose, table height, and gripper pads. Each axis has five severity levels, yielding 210 conditions across the four LIBERO suites. We evaluate nine recent vision-language-action models using 500 episodes per condition and substantiate each finding with rigorous statistical tests. Pooled success falls from 0.932 to 0.767 at the middle severity and 0.509 at the highest severity. Performance degradation varies substantially across perturbation axes and action interfaces, and training on visual perturbations does not mitigate faults that leave camera images unchanged. We release the benchmark together with manifests specifying the fixed perturbation parameters for each episode.

Figures & tables

Figure 1: Same task, perturbed rig. The instruction, object identities, goal, initial state, and success predicate remain fixed while one of 11 rig properties varies across five severity levels. (a) Six observable axes (blue) alter camera views of the same arm and object configuration; changes in gripper-pad geometry (orange) are visible only in close-range wrist-camera views. (b) Four latent axes (green) alter command execution or reported state without changing the image. (c) Overall success at each severity level for the eight models evaluated on all four suites, with 95% intervals. Per-suite results appear in Fig. 3 , numerical values in Table 3 , and severity definitions in Table 1 . Circled numbers 1 – 11 identify the axes consistently across all tables and figures.

Panel of Table 1 — Table 1: The 11 perturbation axes. Circled numbers match those in Fig. 1 . Fault types are defined by the fault’s direct effect on the camera images at a fixed arm and object configuration: O observable, L latent, W weakly observable (visible only in close-range wrist views). Each severity ladder runs from s1 to s5; Appendix B lists the parameters at each level.

Panel of Table 2 — Table 2: Raw success rate per axis, 500 episodes per condition , Nominal is the unperturbed rate. Bold: best four-suite value per column. † LingBot-VA runs LIBERO-10 only and is not ranked.

Figure 2: Success rate by axis at severity 3 , pooled over the eight fully tested models (excluding LingBot-VA); dashed line, pooled nominal; observable axes blue, latent axes green, the weakly observable axis orange, with the axis numbers of Fig. 1 .

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Success rate by suite and severity level , level 1 light to level 5 dark, pooled over the eight four-suite models; the dashed line is each suite’s pooled nominal.

Figure 4: Rank stability at severity 3. Probability that each model lands at each rank over 5,000 task-resampled draws, paired across models.

Figure 5: What the severity ladders look like, axis by axis. Cosmos Policy, spatial suite, task “pick up the black bowl between the plate and the ramekin and place it on the plate”, episode 0: the first frame of the same episode under each perturbation, so panels differ only by the rig. Every axis a scene camera can show is included. The action-frame row is flat because the fault is latent. The wrist-camera row is flat only because these renders record the scene camera while that axis moves the wrist camera. On the camera-position row the workspace slides out of shot as the level rises.

Panel of Table 5 — Table 5: Tolerance per axis. The harshest severity level each model passes before its first failure on the axis, given as that level’s leading parameter (ladders in Table 1 , both parameters of every level in Appendix B ), read from the pooled success per level ( Tables 8 , 9 , 10 , 11 and 12 ). Criterion A passes a level when success is at least 0.80; criterion B when it is at least 0.90 of the model’s own nominal rate. $\geq$ marks a model that passes the harshest level, $<$ one that fails the mildest. Control frequency 7 is not ordered by dose, so its cell gives the range of rates passed around the nominal 20 Hz. An asterisk marks a cell whose deciding rate lies within 0.02 of the bar. Levels are set per axis, so the table compares models within an axis, not axes with one another. † LingBot-VA is evaluated on LIBERO-10 only, where table height is not applied.

Model	libero-10 minus mean of other suites (s3)
NORA	$-0.184$
$\pi_{0}$	$-0.165$
MolmoAct	$-0.129$
$\pi_{0.5}$	$-0.100$
OpenVLA-OFT	$-0.070$
OFT-Plus	$-0.067$

Appendix

Table 7: Long-horizon gap per model at severity 3. Success on LIBERO-10 minus the mean over the other three suites, over the ten axes shared by all suites (table height excluded).

Panel of Figure 6 — Figure 6: Success rate by axis at severities 4 (top) and 5 (bottom) , pooled over the eight four-suite models; dashed line is the pooled nominal; colors as in Fig. 2 .

Panel of Figure 7 — Figure 7: Per-suite success with intervals (top) and rank stability (bottom) at severity 3. Top: raw success rate per suite, mean over the suite’s axes, with 95 percent intervals from a cluster bootstrap over the 10 tasks, the same task resamples for every model. Bottom: as Fig. 4 .

Panel of Figure 8 — Figure 8: Per-suite success with intervals (top) and rank stability (bottom) at severity 4 , as in Fig. 7 . Cosmos Policy takes the object and LIBERO-10 suites.

Panel of Figure 9 — Figure 9: Per-suite success with intervals (top) and rank stability (bottom) at severity 5 , as in Fig. 7 . First place in the object suite is a coin flip between OFT-Plus and $\pi_{0.5}$ .

Panel of Figure 10 — Figure 10: Success rate by suite on the ten shared axes at severities 3, 4, and 5 (top to bottom). Dots are the eight four-suite models.

Figure 11: Full results grid, severity 1. Rows are models, columns the four suites, 11 bars per panel in a fixed axis order. The dashed line is that model’s own unperturbed rate on that suite, so a bar reaching the line means the perturbation cost nothing. The numbers are in Tables 8 , 9 , 10 , 11 and 12 . This figure plots Table 8 ; Figs. 12 , 13 , 14 and 15 plot Tables 9 , 10 , 11 and 12 for severities 2 to 5.

Figure 12: Full results grid, severity 2 , as in Fig. 11 ; the numbers are in Table 9 .

Figure 13: Full results grid, severity 3 , as in Fig. 11 ; the numbers are in Table 10 .

Figure 14: Full results grid, severity 4 , as in Fig. 11 ; the numbers are in Table 11 .

Figure 15: Full results grid, severity 5 , as in Fig. 11 ; the numbers are in Table 12 .

Panel of Table 8 — Table 8: Raw success rate per model, suite, and axis at severity 1, plotted in Fig. 11 . Nom. is the model’s unperturbed rate on that suite.

Panel of Table 9 — Table 9: Raw success rate per model, suite, and axis at severity 2, plotted in Fig. 12 .

Panel of Table 10 — Table 10: Raw success rate per model, suite, and axis at severity 3, plotted in Fig. 13 .

Panel of Table 11 — Table 11: Raw success rate per model, suite, and axis at severity 4, plotted in Fig. 14 .

Panel of Table 12 — Table 12: Raw success rate per model, suite, and axis at severity 5, plotted in Fig. 15 .

Axis	Success at s3	Letters
camera position	0.448	a
camera orientation	0.596	b
robot-base pose shift	0.692	bc
action-frame rotation	0.752	bc
camera field of view	0.777	cde
control-frequency mismatch	0.786	bce

Appendix

Table 13: Axis ordering at severity 3 with compact letters. Raw success pooled over the eight four-suite models. Letters are the maximal sets of axes that the Holm-corrected paired sign-flip tests do not separate, so two axes share a letter exactly when they are not separated; 35 of the 55 pairs are separated. The groups overlap, so they are not tiers.

Model	Nominal	Harshest dose (s5)	Change
$\pi_{0.5}$	0.969	0.968	$-0.001$
Cosmos Policy	0.973	0.958	$-0.015$
OFT-Plus	0.970	0.961	$-0.009$
OpenVLA-OFT	0.966	0.947	$-0.019$
$\pi_{0}$	0.919	0.879	$-0.040$
X-VLA	0.976	0.777	$-0.199$

Appendix

Table 15: What each model does with a corrupted position report. Confirm tier. † single suite.

Robustness of VLA Models

Robustness of vision-language-action (VLA) models concerns reliable robot behavior under changes to language, vision, sensors, tasks, and hardware.

See all 211 papers

LIBERO-VPro: Benchmarking Closed-Loop Visual Robustness of Robotic Foundation Models
Sep 21, 2026Huiqiong Li, Zhiting Mei, Anirudha Majumdar +3World Models for Robotics Robot Foundation Models
Singapore Management University · Princeton University · Fudan University
LIBERO-Pro2.0: A Benchmark for Evaluating the Robustness of Embodied Foundation Models
Oct 17, 2026Xueyang Zhou, Yangming Xu, Qianjiang Li +10Distribution Shift Robustness Embodied Agent Benchmarks
Huazhong University of Science and Technology · Tsinghua University · University of Toronto +2
Is Success All You Need? Investigating the Impact of Input Perturbations on VLA Behaviour in Tabletop Manipulation Tasks
Oct 1, 2026Sophie Higham, Riccardo Andrea Izzo, Matteo Matteucci +1Vision-Language-Action Models Robot Manipulation Benchmarks
School of Informatics, University of Edinburgh, UK · Department of Electronics, Informatics and Bioengineering, Politecnico di Milano, Italy

Sep 21, 2026J/K move · Enter open · S save

LIBERO-VPro: Benchmarking Closed-Loop Visual Robustness of Robotic Foundation Models

Huiqiong Li, Zhiting Mei, Anirudha Majumdar, Jingjing Chen+2

Singapore Management University · Princeton University · Fudan University

World Models for Robotics Robot Foundation Models Robot Manipulation Benchmarks Robustness of VLA Models

Open paper AskPDF arXiv

Robotic foundation models achieve impressive performance on standard manipulation benchmarks, yet these evaluations typically assume clean, timely, and consistent visual observations throughout execution. We introduce LIBERO-VPro, a benchmark for systematically evaluating the closed-loop visual robustness of robotic foundation models by perturbing the visual evidence available during execution. LIBERO-VPro covers four complementary dimensions, including Visual Evidence Degradation, Camera Staleness, Visual Source Consistency, and Task-Relevant Scene Variation, spanning 12 challenge categories, 96 experimental settings, and 3,296 task-condition cases. We evaluate three vision-language-action models and three world-action models over approximately 196,000 simulated episodes, complemented by 200 real-world rollouts on a Franka Research 3. Our results reveal that strong nominal performance can mask substantial weaknesses in visual grounding and adaptation. Models often remain successful despite severe object-level occlusion, yet degrade sharply when local interaction cues are disrupted or familiar spatial priors are violated. They are also highly sensitive to stale or missing observations and struggle when changed task preconditions require behavioral adaptation. Finally, VLAs and WAMs exhibit distinct robustness profiles, showing that visual robustness is multi-dimensional and architecture-dependent. LIBERO-VPro provides a systematic diagnostic framework for developing robotic foundation models that can more reliably ground and adapt their actions under challenging visual conditions.

Abstract

Robustness of VLA Models

Explore similar work

LIBERO-VPro: Benchmarking Closed-Loop Visual Robustness of Robotic Foundation Models

LIBERO-Pro2.0: A Benchmark for Evaluating the Robustness of Embodied Foundation Models

Is Success All You Need? Investigating the Impact of Input Perturbations on VLA Behaviour in Tabletop Manipulation Tasks