VLA-GAUGE: Systematically Evaluating Generalization and Robustness in Vision-Language-Action Models
Authors: Jun Xiao, Rui Ye, Dong Cao, Jinyu Gu, Ying Wang, Zhiliang Zhu
Organizations: Software College, Northeastern University, Shenyang, China · School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · Shanghai Syslong Information Technology Co., Ltd.
Vision-Language-Action models (VLAs) have achieved high success rates on familiar manipulation tasks, yet it remains unclear whether such scores reflect genuine zero-shot generalization and perturbation robustness. Existing evaluation frameworks provide limited capability-level diagnosis and combinatorial perturbation coverage, often without connecting diagnosis to targeted model refinement. We introduce VLA-GAUGE, a diagnostic framework that unifies both evaluations under a shared executable task representation. It combines (1) paired in-distribution and out-of-distribution tasks across six capability suites, (2) single-factor sensitivity analysis with combinatorial perturbation screening and evaluation, and (3) diagnosis-driven refinement through targeted data augmentation and model fine-tuning. We evaluate 12 VLAs spanning different model sizes and action-modeling paradigms under a unified protocol. We find that (1) The mean Zero-Shot Score on VLA-GAUGE is only 22.3%, with SimpleVLA-RL scoring 8.0% and ranking second to last among the evaluated models. (2) Under three representative combinatorial perturbation configurations, the mean R-Score is 9.3. (3) Diagnosis-driven fine-tuning improves π0.5's zero-shot generalization and robustness under a selected perturbation combination on held-out scenarios by 72.4 and 31.8 scoring points, respectively; the latter exceeds the 10.9-point gain from LIBERO-Plus single-factor fine-tuning despite fewer successful training trajectories. These results show that high reported success rates on familiar manipulation tasks do not guarantee strong zero-shot generalization to unseen tasks or robustness to combinatorial perturbations, while diagnostic evaluation provides actionable signals for targeted model improvement.
Figures & tables
TABLE I: A systematic comparison of the existing benchmarks.
Fig. 1: Examples Illustrating Challenges in VLA Generalization and Robustness Evaluation
Fig. 2: Overview of the VLA-Gauge Evaluation Framework
Fig. 3: Hierarchical Structure of the VLA-Gauge Perturbation Space
Models
Params
Generalization Capability
ZS
Vis.
Sem.
CW.
Spa.
Act.
Comp.
Large Models
UniVLA [ 22 ]
9B
29.3
18.7
23.6
10.2
35.9
23.6
23.6
OpenVLA-OFT [ 27 ]
7B
38.2
10.2
36.4
16.9
22.7
21.3
24.3
RIPT-VLA [ 28 ]
7B
30.2
6.7
28.4
5.3
15.3
23.1
18.2
SimpleVLA-RL [ 26 ]
7B
11.1
2.2
8.0
2.2
13.6
11.1
8.0
TABLE II: Zero-shot generalization performance of VLA models on VLA-Gauge .
Models
Vis.
Sem.
CW.
Spa.
Act.
Comp.
Average1.
ID
OOD
ID
OOD
ID
OOD
ID
OOD
ID
OOD
ID
OOD
ID
OOD
Large Models
UniVLA [ 22 ]
47.0
10.9
32.2
4.5
40.9
5.5
16.4
2.4
56.2
6.7
30.3
0.0
37.2
5.0
OpenVLA-OFT [ 27 ]
71.3
3.6
16.5
3.6
67.8
3.6
22.9
8.2
17.7
0.0
27.4
0.0
37.3
3.2
RIPT-VLA [ 28 ]
58.3
0.9
13.0
0.0
53.9
1.8
9.3
0.0
20.8
0.0
29.7
0.0
30.8
0.5
SimpleVLA-RL [ 26 ]
25.4
2.8
3.5
0.0
16.7
2.7
1.5
0.0
15.4
0.0
11.5
0.0
12.3
0.9
TABLE III: In-distribution and out-of-distribution zero-shot generalization performance across VLA models.
Fig. 4: Radar distributions of VLA models across six capability dimensions. (a) Small-scale models. (b) Large-scale models. (c) ID/OOD performance of small-scale models. (d) ID/OOD performance of large-scale models.
Fig. 5: Sensitivity of the two reference models to six selected perturbation attributes. (a) Robot initial state. (b) Action noise. (c) Image shift. (d) Target object scale. (e) Light diffuse. (f) Image rotation.
Models
Params
PFC(1,2)
PFC(2,3)
PFC(3,6)
Average1.
Large Models
UniVLA [ 22 ]
9B
9.3
8.9
0.8
6.4
OpenVLA-OFT [ 27 ]
7B
10.3
9.9
2.9
7.7
RIPT-VLA [ 28 ]
7B
8.2
4.4
1.6
4.8
SimpleVLA-RL [ 26 ]
7B
7.5
2.5
1.3
3.8
CronusVLA [ 29 ]
7B
17.3
3.2
1.8
7.4
TABLE IV: Overall robustness under representative PFCs across VLA models.
Fig. 7: Capability robustness under representative combinatorial perturbations.
Fig. 8: Diagnosis-driven fine-tuning gains in zero-shot generalization and perturbation robustness. (a) Overall comparison of the fine-tuning gains of VLA-Gauge on zero-shot generalization and on PFC(2,3) , together with the robustness gain from fine-tuning on Libero-Plus single-factor perturbation data. (b) Zero-shot generalization gains across capabilities. (c) Robustness gains under PFC(2,3) across capabilities.
While Vision-Language-Action models (VLAs) are rapidly advancing towards generalist robot policies, it remains difficult to quantitatively understand their limits and failure modes. To address this, we introduce a comprehensive benchmark called VLA-Arena. We propose a novel structured task design framework to quantify difficulty across three orthogonal axes: (1) Task Structure, (2) Language Command, and (3) Visual Observation. This allows us to systematically design tasks with fine-grained difficulty levels, enabling a precise measurement of model capability frontiers. For Task Structure, VLA-Arena's 170 tasks are grouped into four dimensions: Safety, Distractor, Extrapolation, and Long Horizon. Each task is designed with three difficulty levels (L0-L2), with fine-tuning performed exclusively on L0 to assess general capability. Orthogonal to this, language (W0-W4) and visual (V0-V4) perturbations can be applied to any task to enable a decoupled analysis of robustness. Our extensive evaluation of state-of-the-art VLAs reveals several critical limitations, including a strong tendency toward memorization over generalization, asymmetric robustness, a lack of consideration for safety constraints, and an inability to compose learned skills for long-horizon tasks. To foster research addressing these challenges and ensure reproducibility, we provide the complete VLA-Arena framework, including an end-to-end toolchain from task definition to automated evaluation and the VLA-Arena-S/M/L datasets for fine-tuning. Our benchmark, data, models, and leaderboard are available at https://vla-arena.github.io.
Borong Zhang, Jiahao Li, Jiachen Shen +7
Institute for Artificial Intelligence, Peking University. · Zhongguancun Academy. · PKU-PsiBot Joint Lab. +2
Vision-Language-Action (VLA) models have achieved high task success rates on robot manipulation task benchmarks. More recently, there has been an emphasis on evaluating the robustness of VLA models to perturbations. However, this robustness is still predominantly measured through Task Success Rate (TSR). In this work, we propose a benchmark-agnostic evaluation framework to measure the behavioural robustness of models by characterising how successful trajectories are executed under perturbation. We implement this methodology by extending the widely-used LIBERO and LIBERO-Plus benchmarks. Across three state-of-the-art VLA models, four LIBERO task suites and seven perturbation conditions, we evaluate changes in both typical successful behaviour and its variability, including metrics of motion smoothness, efficiency and gripper behaviour. We find that perturbations can alter the behaviour of successful trajectories, a phenomenon which cannot necessarily be inferred from TSR alone. Across LIBERO suites, we identify cases where state-of-the-art VLA models achieve comparable TSR under the same perturbation condition, yet behaviour on successful trajectories diverges substantially. Therefore, to have a more robust assessment of task performance, we argue that suitable measures of robustness should capture not only whether a task is completed, but also how the robot behaves while completing it. When evaluating the robustness of VLA models, TSR may be complemented by behavioural evaluation metrics that characterise the nature and variability of successful task execution by robots.
Sophie Higham, Riccardo Andrea Izzo, Matteo Matteucci +1
School of Informatics, University of Edinburgh, UK · Department of Electronics, Informatics and Bioengineering, Politecnico di Milano, Italy
Vision-Language-Action (VLA) models have achieved remarkable advances in robotic manipulation, yet their zero-shot generalization under out-of-distribution (OOD) conditions remains limited. These models often entangle task-relevant invariant structure with environment-specific non-invariant factors, causing policies to rely on spurious appearance cues during action prediction. In this work, we propose MixVLA, a model-agnostic training framework that improves the generalization of VLA models without requiring additional OOD data or architectural modifications. The key component of MixVLA is Adaptive Mixing of Non-Invariant Information (AMI). AMI stochastically mixes non-invariant representations to regularize distribution-specific variability while preserving complementary predictive cues. The mixed non-invariant features are then fused with invariant representations for final action prediction, resulting in improved robustness without sacrificing policy expressiveness. Extensive experiments across challenging manipulation settings, including LIBERO, LIBERO-Plus, the RoboTwin perturbation suite, and real-world tasks, demonstrate that MixVLA improves overall zero-shot robustness while retaining strong in-domain performance.
Pingrui Zhang, Yu Zhang, Pengyuan Wu +8
Fudan University · Shanghai AI Laboratory · Harbin Institute of Technology, Shenzhen, China +2