Organizations: Huazhong University of Science and Technology · Tsinghua University · University of Toronto · Wuhan University of Technology · Lehigh University
LIBERO has emerged as a widely adopted benchmark for evaluating robotic manipulation policies. However, strong performance on LIBERO does not necessarily indicate robust embodied capabilities, as models may exploit benchmark-specific regularities rather than generalize to diverse operating conditions. In this work, we introduce LIBERO-Pro2.0, a comprehensive benchmark for systematically evaluating the robustness of embodied foundation models under distribution shifts. LIBERO-Pro2.0 encompasses 22 static perturbations and 20 runtime interventions across six dimensions: Condition, Environment, Observation, Execution, Robot State Configuration, and Language, yielding 1,648 validated shifted task instances. To support robustness-oriented learning, we further collect 39,390 training trajectories through model-generated and human demonstrations. All shifted task instances undergo six rounds of iterative human review and correction to ensure physical plausibility and task validity. We evaluate 17 representative robot policies spanning vision-language-action models, world-action models, and robustness-oriented policies. Our results reveal substantial robustness gaps across all evaluated paradigms: under runtime distribution shifts, the evaluated models exhibit an average performance drop of 26.5 percentage points, with individual drops reaching 32.5 points. These findings highlight the gap between nominal benchmark performance and robustness under distribution shifts, underscoring the need for systematic robustness evaluation of embodied foundation models.
Figures & tables
Figure 1: Overview of LIBERO-Pro2.0. Built upon LIBERO, our benchmark introduces 22 static distribution shifts and 20 runtime interventions across six dimensions: condition, environment, observation, robot, execution, and language. The left and right panels illustrate representative static and runtime perturbations, respectively.
Figure 2: Failure patterns under task variations. Examples for the task “ Pick up the alphabet soup and put it in the basket. ” The policy behaves similarly despite changes to the target object, instruction, or object position.
Figure 3: Cases of LIBERO-Pro2.0.
Figure 4: Perturbation distribution in LIBERO-Pro2.0.
Model
Cond. S01–S05, S07–S09
Env. S06, S10–S14, S22
Obs. S15–S16, S19
Exec. S20
Robot S17–S18
Lang. S21
Avg.
Base
Δ
Vision-Language-Action Policies
OpenVLA ( Kim et al., 2025b )
25.2
37.6
34.4
0.0
54.0
67.5
33.9
64.6
−30.7
OpenVLA-OFT ( Kim et al., 2025a )
44.1
82.0
46.3
0.0
66.5
94.6
58.9
97.1
−38.2
OpenVLA-OFT w ( Kim et al., 2025a )
40.1
64.9
67.6
0.4
55.4
90.0
53.8
89.2
−35.4
OpenVLA-OFT m ( Kim et al., 2025a )
45.9
78.3
49.7
0.0
72.1
92.1
59.3
92.5
−33.2
OpenVLA-OFT+ ( Fei et al., 2026 )
47.2
78.5
45.0
0.0
73.8
90.0
59.2
92.1
−32.9
Table 1: Leaderboard of robot-policy robustness under static distribution shifts . Category scores and Avg. report mean success rates (%) over valid suite–perturbation entries across four LIBERO suites. Base denotes nominal success, and Δ=Avg.−Base is measured in percentage points. All calculations precede rounding. OpenVLA-OFT variants follow LIBERO-Plus ( Fei et al., 2026 ) : w excludes wrist-camera input, m uses joint fine-tuning across the four original suites, and + uses mixed fine-tuning on LIBERO-Plus data.
Model
Cond. D01–D06
Env. D07–D09
Obs. D10–D13
Exec. D14–D16
Robot D17–D19
Lang. D20
Avg.
Base
Δ
Vision-Language-Action Policies
OpenVLA ( Kim et al., 2025b )
20.6
41.9
41.3
53.6
29.9
63.8
36.4
64.6
−28.2
OpenVLA-OFT ( Kim et al., 2025a )
42.8
63.1
80.4
85.3
57.6
95.4
64.6
97.1
−32.5
OpenVLA-OFT w ( Kim et al., 2025a )
34.9
77.8
74.6
81.0
54.0
91.7
61.9
89.2
−27.3
OpenVLA-OFT m ( Kim et al., 2025a )
39.6
61.4
80.4
85.7
63.6
93.8
64.3
92.5
−28.2
OpenVLA-OFT+ ( Fei et al., 2026 )
41.3
56.9
79.8
84.2
64.0
93.8
63.8
92.1
−28.3
Table 2: Leaderboard of robot-policy robustness under runtime distribution shifts .
Figure 5: Comparison between matched static and runtime perturbations. Each pair contains a static shift occurring before execution and its runtime counterpart triggered during execution, including object pose changes (S01/D01, S02/D02), path constraints (S05/D05), occlusion (S12/D04), camera changes (S15/D07), robot-state variations (S17/D19), and language variations (S21/D20).
Model
Cond. D01–D06
Env. D07–D09
Obs. D10–D13
Exec. D14–D16
Robot D17–D19
Lang. D20
Avg.
OpenVLA-OFT
-26.6
-22.8
-15.0
-8.7
-27.3
-1.1
-19.8
π0.5
-3.5
-19.2
-3.2
-5.3
+7.5
+2.1
-4.1
Anchor-Align
-15.1
-19.5
-12.9
-10.8
-8.9
-7.9
-13.4
FastWAM
-15.6
-9.2
-1.4
-7.5
-6.8
-3.3
-8.7
Cosmos Policy
-11.8
-10.3
-6.4
-13.2
+1.1
-5.4
-8.4
Table 3: Feedback utilization gain Gfeedback (pp) across dynamic perturbation categories.
Figure 6: Effect of policy execution horizon on robustness to dynamic perturbations. Panels (a)–(e) report task success rates under five representative perturbation types; panel (f) shows the average across these perturbations.
Model
Continue
ΔHold
ΔRetry
ΔReset
OpenVLA-OFT
64.6
-15.7
-40.8
-31.6
π0.5
73.8
+18.9
-24.1
+0.7
Anchor-Align
61.6
+28.1
-16.8
+0.9
FastWAM
70.1
+24.3
-20.9
-0.4
Cosmos Policy
72.2
+26.0
-23.0
+2.5
Table 4: Average performance and intervention gains in RQ1-C. Δ denotes the change in success rate relative to normal continuation.
Figure 7: Weakest-single gap Δm(C) under composed perturbations. The heatmap compares static–static, static–dynamic, dynamic–dynamic, and three-way compositions across policies.
Model
S09
S13
D13
S22
P02: S09+S13
P06: S09+D13
T03: S09+S22+D13
OpenVLA-OFT
69.6
94.2
69.6
93.3
12.5
12.5
12.5
π0.5
70.0
92.5
83.8
94.2
12.5
12.1
12.9
Anchor-Align
72.5
86.7
79.6
87.9
10.0
11.3
11.7
FastWAM
80.3
91.2
83.7
88.3
9.2
11.7
10.0
Cosmos Policy
87.5
99.2
81.7
96.7
12.9
11.7
12.1
Table 5: Key evidence for non-compositional robustness. Although all models perform reasonably well on the constituent perturbations S09, S13, D13, and S22, they all collapse on the composed settings P02, P06, and T03.
Model
Object Pose
Receiver Pose
Path Obstacle
Camera Calibration
Robot State
Total
OpenVLA-OFT
20.8
24.4
14.8
26.9
52.9
28.0
π0.5
61.9
71.0
19.8
8.1
78.7
47.9
Anchor-Align
27.9
39.4
12.1
6.0
65.4
30.2
FastWAM
28.8
16.0
19.3
8.3
83.1
31.1
Cosmos Policy
42.9
39.2
11.7
57.1
82.9
46.7
Ours
68.8
73.2
96.3
20.8
91.7
70.1
Table 6: Transfer of robustness from training set to tasks held out from additional fine-tuning. Each column reports the average success rate over a static–dynamic perturbation pair. Gain denotes the improvement of Ours over π0.5 .
Appendix figures & tables26 assets
Supplementary material from the paper’s appendix.
Appendix
Model
S1
S2
S3
S4
S5
S6
S7
S8
S9
S10
S11
S12
S13
S14
S15
S16
S17
S18
S19
S20
S21
S22
OpenVLA
1.7
8.3
0.0
0.0
0.0
48.3
35.0
33.3
43.3
46.7
41.7
0.0
31.7
16.7
1.7
0.0
15.0
51.7
46.7
0.0
63.3
43.3
OpenVLA-OFT
33.3
0.0
0.0
0.0
0.0
86.7
58.3
83.3
81.7
91.7
85.0
55.0
90.0
93.3
0.0
8.3
0.0
81.7
90.0
0.0
88.3
88.3
OpenVLA-OFT w
20.0
1.7
0.0
0.0
0.0
76.7
55.0
86.7
81.7
86.7
90.0
0.0
78.3
75.0
43.3
0.0
40.0
76.7
86.7
0.0
85.0
83.3
OpenVLA-OFT m
40.0
1.7
0.0
0.0
0.0
76.7
55.0
73.3
71.7
80.0
56.7
53.3
85.0
86.7
0.0
1.7
0.0
75.0
86.7
0.0
80.0
85.0
OpenVLA-OFT+
28.3
1.7
0.0
5.0
0.0
60.0
45.0
60.0
56.7
55.0
63.3
65.0
76.7
88.3
0.0
0.0
0.0
83.3
78.3
0.0
80.0
81.7
UniVLA
3.3
5.0
5.0
8.3
0.0
61.7
81.7
76.7
85.0
76.7
76.7
0.0
73.3
78.3
26.7
0.0
40.0
85.0
83.3
0.0
83.3
83.3
Appendix
Table 7: Static and runtime perturbation results on LIBERO-Goal. All values are reported in percentages (%).
Model
S1
S2
S3
S4
S5
S6
S7
S8
S9
S10
S11
S12
S13
S14
S15
S16
S17
S18
S19
S20
S21
S22
OpenVLA
0.0
3.3
6.7
3.3
0.0
0.0
45.0
80.0
81.7
78.3
11.7
0.0
76.7
5.0
10.0
0.0
36.7
88.3
88.3
0.0
81.7
75.0
OpenVLA-OFT
8.3
1.7
0.0
16.7
1.7
0.0
95.0
91.7
60.0
98.3
80.0
70.0
98.3
100.0
10.0
20.0
51.7
100.0
96.7
0.0
96.7
100.0
OpenVLA-OFT w
0.0
0.0
1.7
0.0
0.0
0.0
95.0
98.3
100.0
100.0
61.7
0.0
98.3
86.7
50.0
45.0
96.7
15.0
100.0
0.0
100.0
98.3
OpenVLA-OFT m
43.3
6.7
1.7
3.3
0.0
0.0
76.7
93.3
40.0
98.3
66.7
98.3
86.7
93.3
3.3
5.0
86.7
100.0
98.3
0.0
98.3
100.0
OpenVLA-OFT+
48.3
63.3
0.0
0.0
0.0
0.0
81.7
93.3
83.3
100.0
68.3
100.0
95.0
100.0
0.0
3.3
91.7
100.0
100.0
0.0
100.0
98.3
UniVLA
5.0
3.3
3.3
35.0
0.0
0.0
78.3
91.7
100.0
86.7
26.7
0.0
100.0
51.7
0.0
0.0
66.7
98.3
100.0
0.0
96.7
93.3
Appendix
Table 8: Static and runtime perturbation results on LIBERO-Object. All values are reported in percentages (%).
Model
S1
S2
S3
S4
S5
S6
S7
S8
S9
S10
S11
S12
S13
S14
S15
S16
S17
S18
S19
S20
S21
S22
OpenVLA
15.0
5.0
3.3
N/A
0.0
70.0
88.3
70.0
68.3
36.7
71.7
0.0
73.3
38.3
40.0
28.3
83.3
83.3
88.3
0.0
75.0
80.0
OpenVLA-OFT
73.3
76.7
30.0
N/A
0.0
80.0
100.0
93.3
98.3
88.3
100.0
98.3
93.3
98.3
51.7
81.7
96.7
86.7
100.0
0.0
98.3
100.0
OpenVLA-OFT w
40.0
15.0
5.0
N/A
1.7
75.0
98.3
91.7
98.3
48.3
96.7
6.7
98.3
98.3
93.3
90.0
100.0
10.0
98.3
0.0
98.3
100.0
OpenVLA-OFT m
73.3
96.7
28.3
N/A
0.0
75.0
100.0
95.0
95.0
70.0
95.0
53.3
90.0
98.3
41.7
65.0
100.0
93.3
100.0
0.0
98.3
100.0
OpenVLA-OFT+
66.7
98.3
35.0
N/A
0.0
81.7
96.7
95.0
95.0
86.7
88.3
66.7
98.3
98.3
38.3
56.7
98.3
100.0
98.3
0.0
98.3
98.3
UniVLA
18.3
55.0
10.0
N/A
0.0
75.0
98.3
98.3
98.3
68.3
98.3
1.7
51.7
95.0
73.3
80.0
98.3
98.3
98.3
0.0
98.3
95.0
Appendix
Table 9: Static and runtime perturbation results on LIBERO-Spatial. All values are reported in percentages (%).
Model
S1
S2
S3
S4
S5
S6
S7
S8
S9
S10
S11
S12
S13
S14
S15
S16
S17
S18
S19
S20
S21
S22
OpenVLA
18.3
5.0
26.7
0.0
13.3
1.7
55.0
53.3
18.3
36.7
43.3
0.0
46.7
38.3
28.3
31.7
23.3
50.0
50.0
0.0
50.0
41.7
OpenVLA-OFT
30.0
26.7
36.7
0.0
46.7
33.3
91.7
95.0
38.3
80.0
96.7
41.7
95.0
68.3
0.0
6.7
23.3
91.7
90.0
0.0
95.0
85.0
OpenVLA-OFT w
35.0
8.3
33.3
3.3
61.7
11.7
78.3
85.0
46.7
70.0
66.7
0.0
75.0
70.0
73.3
60.0
41.7
63.3
71.7
1.7
76.7
66.7
OpenVLA-OFT m
31.7
30.0
56.7
0.0
66.7
15.0
95.0
100.0
46.7
90.0
95.0
90.0
91.7
73.3
10.0
88.3
43.3
78.3
96.7
0.0
91.7
88.3
OpenVLA-OFT+
36.7
35.0
48.3
0.0
68.3
10.0
90.0
86.7
43.3
83.3
85.0
91.7
85.0
88.3
0.0
75.0
41.7
75.0
90.0
0.0
81.7
85.0
UniVLA
36.7
28.3
36.7
6.7
30.0
35.0
63.3
68.3
5.0
66.7
75.0
0.0
86.7
26.7
73.3
78.3
61.7
88.3
83.3
0.0
88.3
81.7
Appendix
Table 10: Static and runtime perturbation results on LIBERO-10. All values are reported in percentages (%).
(a): D01–D10
Metric
D01
D02
D03
D04
D05
D06
D07
D08
D09
D10
OpenVLA-OFT
S0,live
97.1
97.1
97.1
97.1
97.1
97.1
97.1
97.1
97.1
97.1
S0,freeze
45.6
43.8
33.8
50.0
42.5
55.0
48.1
48.1
48.1
48.1
S1,live
5.4
24.3
17.6
95.8
16.8
95.3
38.3
56.7
94.2
87.9
S1,freeze
0.0
3.8
0.0
41.9
13.8
43.6
21.3
41.3
48.1
47.5
Appendix
Table 11: Complete feedback-utilization results. Success rates are percentages; 0/1 denotes absence/presence of a perturbation, and live/freeze denotes the visual-observation condition. Live allows the policy to receive updated observations and infer actions normally, whereas Freeze holds the visual observation fixed at its pre-event value for a specified interval. For each model, the freeze duration is set to one-eighth of the number of inference steps required to complete unperturbed tasks, with freezing initiated immediately before the perturbation event. The feedback gain Gfeedback=(S1,live−S1,freeze)−(S0,live−S0,freeze) is measured in percentage points. The two panels cover D01–D10 and D11–D20, respectively.
Figure 8: Recovery-intervention gains, ΔI=SI−Scontinue (percentage points), for Hold, Retry, and Restore (labeled Reset in the figure). Positive values indicate improvement over Continue, while negative values indicate degradation.
(a) D01–D10
Model
D01
D02
D03
D04
D05
D06
D07
D08
D09
D10
Hold
OpenVLA-OFT
48.1
48.8
46.3
47.5
49.4
55.7
48.8
48.8
48.8
48.8
π0.5
93.8
92.5
92.5
90.6
91.3
97.9
93.1
93.1
93.1
93.1
Anchor-Align
89.4
90.0
89.4
91.9
88.8
92.1
89.4
89.4
89.4
89.4
FastWAM
92.5
93.1
90.0
92.5
92.5
96.4
95.6
95.6
95.6
95.6
Appendix
Table 12: Success rates (%) under Hold, Retry, and Restore (Reset) interventions across all dynamic perturbations. Both panels retain the same intervention and model order.
Case
Composition
OpenVLA-OFT
π0.5
Anchor-Align
FastWAM
Cosmos Policy
Static + Static
P01
S11 + S15
30.4
0.0
0.0
5.0
50.8
P02
S09 + S13
12.5
12.5
10.0
9.2
12.9
P03
S19 + S21
82.9
86.3
82.5
5.8
85.4
P04
S08 + S22
82.5
80.8
73.8
76.3
80.4
Static + Dynamic
Appendix
Table 13: Success rates (%) under pairwise and three-way perturbation compositions. Compositions are grouped by perturbation type.
Figure 9: Static perturbations S01–S05. Each row contrasts BASE and the perturbed scene using the external and wrist views.
Figure 10: Static perturbations S06–S10.
Figure 11: Static perturbations S11–S14.
Figure 12: Static perturbations S15–S18.
Figure 13: Static perturbations S19–S22.
Figure 14: Dynamic perturbations D01–D05. Each row shows the temporal sequence around an intervention, using the views indicated in the panels.
Figure 15: Dynamic perturbations D06–D10.
Figure 16: Dynamic perturbations D11–D15.
Figure 17: Dynamic perturbations D16–D20.
Figure 18: Representative failure rollouts under perturbations; each row contains two five-frame sequences, read from left to right.
Category
Static perturbations
Runtime perturbations
Condition
S01–S05, S07–S09
D01–D06
Environment
S06, S10–S14, S22
D07–D09
Observation
S15–S16, S19
D10–D13
Execution
S20
D14–D16
Robot
S17–S18
D17–D19
Language
S21
D20
Appendix
Table 14: Mapping of static and runtime perturbations to evaluation categories.
ID
Perturbation
Intervention
Constraint
S01
Target planar pose
Translate the target in the horizontal plane and rotate it in yaw.
Preserve the target identity and intended goal.
S02
Receiver planar pose
Translate or rotate the receiver to change the placement destination and approach.
Preserve receiver identity and task goal.
S03
Receiver or support height
Change receiver or support elevation using a support platform or a coupled layout adjustment.
Retain a collision-free initial layout and feasible placement.
S04
Object position swap
Swap the planar positions of two objects; use a safe layout offset when a direct swap is unsuitable.
Keep object identities, task roles, and goal unchanged.
S05
Restricted approach or path
Add geometry that constrains approach or transport.
Retain valid inverse kinematics and a complete collision-free route.
S06
Instance replacement
Replace the target with a same-class or unseen asset specified by the task configuration.
Preserve semantic class and graspability.
Appendix
Table 15: Static perturbation definitions. Interventions are introduced at reset. Modified geometry or fault settings persist; initial-state changes do not prevent normal subsequent motion. Alternatives within a row are selected by the task configuration.
ID
Perturbation
Intervention
Trigger and lifetime
D01
Target pose update
Change target horizontal position or yaw, requiring updated localization and approach.
Approach/pre-grasp; retain the update as configured.
D02
Receiver pose update
Move or rotate the receiver without changing the task goal.
Post-grasp/pre-place; retain the update as configured.
D03
Receiver occupation
Introduce an occupying or blocking object near the receiving region.
Pre-place; occupation/release follows the case configuration.
D04
Temporary occlusion
Insert an occluder between camera and target, then remove it.
Task phase; enter, hold for a window, then exit.
D05
Path obstacle
Place an obstacle on the upcoming transport route while preserving an alternative collision-free path.
Transport; persistence follows the case configuration.
D06
Distractor rearrangement
Move irrelevant objects or rearrange local clutter; exclude the target and receiver.
Task phase; apply the configured scene update.
Appendix
Table 16: Dynamic perturbation definitions. Trigger phase and event lifetime are part of each case configuration. A restored sensor or control channel does not imply rollback of the complete scene. IDs match the paper visualizations.
Suite
Evaluation IDs
Training-pool IDs
Spatial
0, 8
1, 2, 3, 4, 5, 6, 7, 9
Object
1, 8
0, 2, 3, 4, 5, 6, 7, 9
Goal
3, 6
0, 1, 2, 4, 5, 7, 8, 9
Long
5, 8
0, 1, 2, 3, 4, 6, 7, 9
Appendix
Table 17: Task partition for the controlled eight-task studies. IDs are zero-based indices in the pinned release manifest. The training pool is the complement of the evaluation tasks within each suite; it does not specify which demonstrations were actually used in a training run.
Suite
ID
BASE task instruction
Spatial
0
Pick the akita black bowl between the plate and the ramekin and place it on the plate
Spatial
8
Pick the akita black bowl next to the plate and place it on the plate
Object
1
Pick the cream cheese and place it in the basket
Object
8
Pick the chocolate pudding and place it in the basket
Goal
3
Open the top layer of the drawer and put the bowl inside
Goal
6
Put the cream cheese on the bowl
Appendix
Table 18: The eight evaluation tasks. Instructions are taken directly from the BASE scene descriptions, with whitespace normalized.
Suite
ID
BASE task instruction
Spatial
1
Pick the akita black bowl next to the ramekin and place it on the plate
Spatial
2
Pick the akita black bowl from table center and place it on the plate
Spatial
3
Pick the akita black bowl on the cookies box and place it on the plate
Spatial
4
Pick the akita black bowl in the top layer of the wooden cabinet and place it on the plate
Spatial
5
Pick the akita black bowl on the ramekin and place it on the plate
Spatial
6
Pick the akita black bowl next to the cookies box and place it on the plate
Appendix
Table 19: The 32 training tasks. Instructions are taken directly from the BASE scene descriptions, with whitespace normalized.
Model
H
1/8H
1/4H
1/2H
1H
OpenVLA-OFT
8
1
2
4
8
π0.5
10
1
3
5
10
Anchor-Align
8
1
2
4
8
FastWAM
32
4
8
16
32
Cosmos Policy
16
2
4
8
16
Appendix
Table 20: Executed action lengths K for different policies and nominal execution horizon ratios in RQ1-B. The prediction horizon H remains fixed for each policy.
Robot policies are expected to generalize across deployment environments, including variations in the robot itself: cameras are bumped and remounted, calibrations drift, controllers run at different rates, and end-effector pads wear down. We present LIBERO-RIG, a benchmark that perturbs only the robot's hardware and setup while holding every LIBERO task fixed. The benchmark spans eleven perturbation axes, including camera mounting and optics, camera-to-arm calibration, actuation gain, control rate, proprioceptive noise, base pose, table height, and gripper pads. Each axis has five severity levels, yielding 210 conditions across the four LIBERO suites. We evaluate nine recent vision-language-action models using 500 episodes per condition and substantiate each finding with rigorous statistical tests. Pooled success falls from 0.932 to 0.767 at the middle severity and 0.509 at the highest severity. Performance degradation varies substantially across perturbation axes and action interfaces, and training on visual perturbations does not mitigate faults that leave camera images unchanged. We release the benchmark together with manifests specifying the fixed perturbation parameters for each episode.
Robotic foundation models achieve impressive performance on standard manipulation benchmarks, yet these evaluations typically assume clean, timely, and consistent visual observations throughout execution. We introduce LIBERO-VPro, a benchmark for systematically evaluating the closed-loop visual robustness of robotic foundation models by perturbing the visual evidence available during execution. LIBERO-VPro covers four complementary dimensions, including Visual Evidence Degradation, Camera Staleness, Visual Source Consistency, and Task-Relevant Scene Variation, spanning 12 challenge categories, 96 experimental settings, and 3,296 task-condition cases. We evaluate three vision-language-action models and three world-action models over approximately 196,000 simulated episodes, complemented by 200 real-world rollouts on a Franka Research 3. Our results reveal that strong nominal performance can mask substantial weaknesses in visual grounding and adaptation. Models often remain successful despite severe object-level occlusion, yet degrade sharply when local interaction cues are disrupted or familiar spatial priors are violated. They are also highly sensitive to stale or missing observations and struggle when changed task preconditions require behavioral adaptation. Finally, VLAs and WAMs exhibit distinct robustness profiles, showing that visual robustness is multi-dimensional and architecture-dependent. LIBERO-VPro provides a systematic diagnostic framework for developing robotic foundation models that can more reliably ground and adapt their actions under challenging visual conditions.
Huiqiong Li, Zhiting Mei, Anirudha Majumdar +3
Singapore Management University · Princeton University · Fudan University
Vision-Language-Action (VLA) or World Action (WAM) models have recently demonstrated remarkable performance in robotic manipulation. On LIBERO, SOTA method have achieved nearly 100% success rates, seemingly suggesting that the models are ready for deployment in real world. However, near perfect performance on existing benchmarks can be misleading: success under ideal conditions does not imply real world robustness. Existing benchmarks primarily evaluate task completion from predefined initial states, while real world interactions inevitably involve failures such as failed grasps, collisions, and unintended object movements. A robot must therefore not only execute tasks successfully, but also recognize and recover from failures to continue the task. Yet this capability remains largely unmeasured, revealing a critical gap between benchmark performance and real world reliability. To address this gap, we introduce LIBERO-Recover Benchmark, a large scale benchmark for failure recovery in robotic manipulation. Built upon LIBERO, we collect real execution failures from SOTA embodied models and construct 1,000+ scenarios across four recovery levels: (1) Action Retry, (2) Action Adaptation, (3) Object State Recovery, and (4) Environmental Recovery. We evaluate four core capabilities: spatial understanding, object structure reasoning, interaction understanding, and topological reasoning. As the first large-scale benchmark for embodied failure recovery, LIBERO-Recover shifts evaluation from Can the robot succeed?'' to Can the robot recover after failure?'', promoting robust and generalizable embodied agents. The project will be avaible in \textcolor{blue}{https://liulin815.github.io/LIBERO-Recovery/}.
Lin Liu, Zhicheng Bao, Lu Zhang +7
Beta Infinity · Beijing Jiaotong University · School of Information and Communication Engineering Dalian University of Technology +1