Rethinking the Practicality of Vision-language-action Model: A Comprehensive Benchmark and An Improved Baseline
Authors: Wenxuan Song, Jiayi Chen, Xiaoquan Sun, Huashuo Lei, Yikai Qin, Wei Zhao, Pengxiang Ding, Han Zhao, +7 more
Organizations: The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China. · Huazhong University of Science and Technology, Wuhan, China. · Westlake University, Hangzhou, China. · Zhejiang University, Hangzhou, China.
Vision-Language-Action (VLA) models have emerged as a generalist robotic agent. However, existing VLAs are hindered by excessive parameter scales, prohibitive pre-training requirements, and limited applicability to diverse embodiments. To improve the practicality of VLAs, we propose a comprehensive benchmark and an improved baseline. First, we propose CEBench, a new benchmark spanning diverse embodiments in both simulation and the real world with consideration of domain randomization. We collect 14.4k simulated trajectories and 1.6k real-world expert-curated trajectories to support training on CEBench. Second, using CEBench as our testbed, we study three critical aspects of VLAs' practicality and offer several key findings. Informed by these findings, we introduce LLaVA-VLA, a lightweight yet powerful VLA designed for practical deployment on consumer-grade GPUs. Architecturally, it integrates a compact VLM backbone with multi-view perception, proprioceptive tokenization, and action chunking. To eliminate reliance on costly pre-training, LLaVA-VLA adopts a two-stage training paradigm including post-training and fine-tuning. Furthermore, LLaVA-VLA extends the action space to unify navigation and manipulation. Experiments across embodiments demonstrate the capabilities of generalization and versatility of LLaVA-VLA , while real-world mobile manipulation experiments establish it as the first end-to-end VLA model for mobile manipulation. We will open-source all datasets, codes, and checkpoints upon acceptance to foster reproducibility and future research.
Figures & tables
Fig. 2: Real-world setup of the Cobot-Magic system for mobile bimanual manipulation (top view).
Fig. 3: Model architecture of our LLaVA-VLA.
Model
Param.
Success Rate (%)
Avg. Len.
1/5
2/5
3/5
4/5
5/5
ABC → D
LLaVA-VLA (ours)
0.5B
96.2
84.8
72.6
60.8
50.6
3.65
LLaVA-VLA (ours)
7B
97.4
86.2
73.4
64.6
53.4
3.75
RoboFlamingo [ 30 ]
3B
82.4
61.9
46.6
33.1
23.5
2.47
OpenVLA [ 15 ]
7B
91.3
77.8
62.0
52.1
43.5
3.27
TABLE I: Comparison among different versions of LLaVA-VLA in terms of success rates and average length on RoboTwin. Here, B denotes billions.
Method
Success Rate (%)
Avg. Len.
1/5
2/5
3/5
4/5
5/5
ABC → D
Concate Image Tokens
65.3
37.1
23.5
14.7
9.2
1.50
Merged Image
94.8
84.5
71.3
62.5
53.8
3.68
TABLE II: Comparison of different methods for integrating multi-view images on CALVIN.
Method
Success Rate (%)
Avg. Len.
1/5
2/5
3/5
4/5
5/5
ABC → D
MLP Projector
90.4
76.0
58.0
48.0
37.2
3.09
Prop. Tokenizer
94.8
84.5
71.3
62.5
53.8
3.68
TABLE III: Comparison of different methods for integrating proprioceptive information on CALVIN.
Chunk Size
1
5
12
20
Avg. Len.
2.25
3.68
3.35
0.70
TABLE IV: Comparison of different chunk sizes in terms of average length on CALVIN.
Training Curriculum
Open Laptop
Lift Pot
Pre-training
20.0%
18.0%
Post-training
38.0%
39.0%
TABLE V: Comparison of different training curricula on the seen tasks on RoboTwin.
Category
Method
Params
w/o
Success Rate (%)
Avg. Len.
Pre-training
1/5
2/5
3/5
4/5
5/5
ABC → D
Generative Methods
3D-VLA [ 43 ] ( ICML’24 )
1B
×
44.7
16.3
8.1
1.6
0
0.70
GR-1 [ 2 ] ( ICLR’24 )
195M
×
85.4
71.2
59.6
49.7
40.1
3.06
Vidman [ 44 ] ( NIPS’24 )
1B
×
91.5
76.4
68.2
59.2
46.7
3.42
Diffusion Policy
3D Diffuser Actor [ 45 ] ( CoRL’24 )
70M
✓
93.8
80.3
66.2
53.3
41.2
3.35
Large VLA Models
RoboFlamingo [ 30 ] ( ICLR’24 )
3B
✓
82.4
61.9
46.6
33.1
23.5
2.47
TABLE VI: Comparison with various manipulation baselines on CALVIN.
Small Model w/o Pre-training
Large Model
Simulation Task
ACT
DP
LLaVA-VLA (Ours)
RDT
Seen
DR
Seen
DR
Seen
DR
Seen
DR
Click Bell
4.0%
2.0%
54.0%
0.0%
81.0%
72.0%
80.0%
9.0%
Click Alarmclock
11.0%
0.0%
61.0%
5.0%
73.0%
65.0%
61.0%
12.0%
Lift Pot
7.0%
2.0%
31.0%
0.0%
39.0%
21.0%
72.0%
9.0%
Move Can Pot
0.0%
0.0%
39.0%
0.0%
28.0%
16.0%
25.0%
12.0%
TABLE VII: Evaluation on RoboTwin benchmark. Success rates for 8 tasks on the Seen and DR settings. Best result in each row highlighted in Bold .
Decoder
Success Rate (%)
Avg. Len.
Type
1/5
2/5
3/5
4/5
5/5
ABC → D
Discrete
94.8
84.5
71.3
62.5
53.8
3.68
Continuous
93.5
84.4
73.5
63.3
54.3
3.70
TABLE VIII: Comparison among different action spaces of LLaVA-VLA on CALVIN.
Bin
Success Rate (%)
Avg. Len.
Numbers
1/5
2/5
3/5
4/5
5/5
ABC → D
256
94.8
84.5
71.3
62.5
53.8
3.68
1024
96.4
86.5
72.1
60.2
48.6
3.65
TABLE IX: Comparison of different numbers of bins on CALVIN.
Unified
Move and
Move and
Action Space
fetch bottles
open the drawer
Action Value Token
2/10
1/10
Direction + Vaule Token
4/10
4/10
TABLE X: Comparison of unified action space on the mobile manipulation tasks in the real world.
Embodiments
Basic Single-arm Tasks
Dexterous Bimanual Tasks
Task
Stack Bowls
Restore Bottle
Click Bell
Place Vagetable
Pack Bottles
Lift Pot
Avg.
Success
Settings
Seen
DR
Seen
DR
Seen
DR
Seen
DR
Seen
DR
Seen
DR
Seen
DR
ACT [ 37 ]
30%
10%
15%
0%
30%
10%
20%
0%
10%
0%
7%
0%
18.6%
3.0%
TinyVLA [ 12 ]
25%
10%
10%
0%
25%
5%
15%
0%
20%
5%
10%
5%
17.5%
4.2%
LLaVA-VLA (Ours)
58%
40%
38%
27%
66%
54%
50%
32%
28%
16%
25%
15%
44.2%
30.7%
TABLE XI: Comparison of success rates on real-world bimanual tasks.
Fig. 4: Visualization of real-world tasks. The top two rows illustrate the seen tasks, while the bottom two rows correspond to settings with domain randomization. Out of the eight real-world tasks, we select two representative examples of single-arm manipulation (left) as well as two examples of bimanual collaboration (right).
Fig. 5: Evaluation in real-world mobile manipulation tasks.