Authors: Hanyi Zhang, Zihong Luo, Tianyu Li, Khang Nguyen, Basu Hela, Shreyas Kumar, Ngoc Duy Tran, Feng Dai, Charith Munasinghe, Jorge Peña Queralta, Giovanni Toffetti, Khoa Vo, Ngan Le, Ravi Prakash, Quan Vuong, Tung D. Ta, Long Hu, Anh Nguyen, Baoru Huang
Organizations: University of Liverpool, UK · Mohamed bin Zayed University of Artificial Intelligence, UAE · Indian Institute of Science, India · The University of Tokyo, Japan · Zürcher Hochschule für Angewandte Wissenschaften, Switzerland · University of Arkansas, USA · Physical Intelligence, USA · Huazhong University of Science and Technology, China
Vision language action models (VLAs) have advanced general purpose robotic grasping and manipulation by enabling robots to interpret visual observations and natural language instructions to generate executable action sequences. However, existing VLAs often implicitly assume gripper invariance, despite grasping strategies being inherently embodiment-dependent. Different gripper types, such as parallel-jaw and suction, usually require distinct interaction strategies to achieve the same grasping objective. Moreover, current datasets for VLAs predominantly rely on parallel-jaw grippers, limiting gripper-aware learning. To address this gap, we introduce MiGA, a multi-gripper-aware dataset spanning five distinct gripper types across multiple robots with 103,000 demonstrations, explicitly capturing strategy divergence under shared task objectives. We further propose GVLA, which combines a new multi-gripper tokenizer with adapter-based policy routing. Our new gripper encoding induces structured embedding information that balances parameter sharing and strategy differentiation, while layer-wise probing confirms meaningful gripper-conditioned representations for VLAs. Intensive experiments in both simulation and real-world robots show that our GVLA outperforms the current baselines across evaluated settings. Our method also improves zero-shot generalization or few-shot adaptation to new objects or unseen tasks, and enable more efficient gripper adaptation.
Figures & tables
Figure 1: We tackle the challenge of gripper-specific grasping in robotics, e.g., a suction cup gripper lifts a thin, flat, DVD-case-like box from above, while a parallel-jaw gripper pushes it to the table edge and grasps it from the side. We first introduce MiGA, a large-scale multi-gripper-aware dataset collected by diverse gripper types. Leveraging this dataset, we then develop a Gripper-aware Vision Language Action (GVLA) model that incorporates gripper conditioning through a new multi-gripper tokenization method and a dual Mixture-of-Adapter. Our approach improves task performance over gripper-agnostic baselines and enables cross-gripper adaptation.
Figure 2: Gripper embedding feature heatmap of different gripper representations. Our new multi-gripper tokenization method (d) produces well-clustered representations organized by gripper type. In contrast, other methods (a-c) fail to capture the underlying semantic structure of different gripper types. V1 and V2 denote the vacuum gripper: UR10 Suction Cup [ 23 ] and Cobot Pump [ 50 ] , F1 , and F2 denote finger-based gripper: Franka Panda Hand [ 16 ] and Detexterous Hand [ 4 ] .
Dataset
#Traj.
#Gripper Types
#Robots
Gripper-Specific Solution
Depth Image
Sub-step Decomposition
Data Domain
Collection
Open X-Embodiment [ 44 ]
1M+
1
22
✗
✗
✗
Real
Aggregation
DROID [ 26 ]
76K
1
1
✗
✗
✗
Real
Human
Bridge V2 [ 58 ]
60K
1
1
✗
✗
✗
Real
Human
RT-1 [ 6 ]
130K
1
1
✗
✗
✗
Real
Human
RH20T [ 12 ]
110K
1
7
✗
✓
✗
Real
Human
BridgeDataV2 [ 12 ]
60.1K
1
1
✗
✓
✗
Real
84% Human, 16% Scripted
Table 1: Comparison of existing datasets for VLAs. Our dataset provides a manually collected, gripper-aware data spanning multiple gripper types with gripper-specific manipulation strategies in both simulation and real-world settings.
Figure 3: Illustration of gripper-specific strategy variations across four task categories.
Figure 4: MiGA dataset statistics.
Figure 5: An overview of GVLA architecture. Our method extends the pretrained VLA backbone with two gripper-aware components. (Left) Multi-gripper tokenizer encodes gripper knowledge through three-level soft prompts: platform-specific, gripper type-specific, and instance-specific learnable tokens. (Right) A dual MoA routes action tokens through type-stratified expert pools using a cascaded domain and a gripper router, applying top-k selected adapters as residual connections to adapt to gripper-specific manipulation strategies.
Figure 6: Layer probing analysis.
Method
Flat
Stacked
Constrained
Semantic
Avg.(%)
AnyGrasp [ 13 ]
0.00
0.00
46.00
0.00
11.50
GraspMAS [ 41 ]
0.00
0.00
40.00
8.00
12.00
GraspVLA [ 9 ]
1.50
0.00
30.00
0.00
7.88
OpenVLA-OFT [ 27 ]
41.00
52.50
24.00
50.00
41.88
π0 [ 5 ]
30.00
21.50
36.00
65.00
38.13
π0.5 [ 22 ]
52.50
71.00
57.50
52.50
58.38
Table 2: Baseline comparison across task categories.
Table 9
Figure 8: Cross-object generalization results.
Figure 9: Adaptation results.
Configuration
PE ↓
Adapt. ↑
Gripper Repr.
w/o P(g)
0.037
0.86
w/o P(p)
0.035
0.54
w/o P(u)
0.032
0.90
Dual MoA
w/o MoA
0.037
0.52
w/o MoA(g)
0.033
0.56
w/o MoA(p)
0.034
0.54
Table 4: Component analysis.
Figure 10: Real-world robotic experiment setups and results.
In dynamic environments such as warehouses, hospitals, and homes, robots must seamlessly transition between gross motion and precise manipulations to complete complex tasks. However, current Vision-Language-Action (VLA) frameworks, largely adapted from pre-trained Vision-Language Models (VLMs), often struggle to reconcile general task adaptability with the specialized precision required for intricate manipulation. To address this challenge, we propose DAM-VLA, a dynamic action model-based VLA framework. DAM-VLA integrates VLM reasoning with diffusion-based action models specialized for arm and gripper control. Specifically, it introduces (i) an action routing mechanism, using task-specific visual and linguistic cues to select appropriate action models (e.g., arm movement or gripper manipulation), (ii) a dynamic action model that fuses high-level VLM cognition with low-level visual features to predict actions, and (iii) a dual-scale action weighting mechanism that enables dynamic coordination between the arm-movement and gripper-manipulation models. Across extensive evaluations, DAM-VLA achieves superior success rates compared to state-of-the-art VLA methods in simulated (SIMPLER, FurnitureBench) and real-world settings, showing robust generalization from standard pick-and-place to demanding long-horizon and contact-rich tasks.
Xiongfeng Peng, Jiaqian Yu, Dingzhe Li +8
Advanced Research Lab, Samsung R&D Institute China-Beijing (SRCB), China · Samsung AI Center, DS Division, South Korea · Hanyang University ERICA, South Korea
Human communication is inherently multimodal, where language is often accompanied by non-verbal cues such as gestures to convey intentions. However, current Vision-Language-Action (VLA) models treat robotic manipulation as a pure text-driven task, overlooking the important role of gestures in Human-Robot Interaction (HRI). This often leads to inaccurate intent grounding and unreliable manipulation when language instructions are ambiguous or underspecified. To address this challenge, we propose GIVE (Gesture Intent via Visual-Semantic Enhancement), an effective approach that enhances pre-trained VLA models with human gesture understanding without architectural modifications. Specifically, GIVE incorporates gesture information through two complementary pathways: a visual pathway that overlays hand skeletons and fingertip rays onto robot observations for explicit object grounding, and a semantic pathway that generates high-level descriptions of human gestures and task instructions for robust intent grounding. By jointly leveraging visual and semantic guidance, GIVE enables VLA policies to better associate gestures with manipulation behaviors and adapt to dynamic interaction intents. In real-world HRI experiments, GIVE substantially outperforms the baseline, improving target object recognition accuracy by 40% and overall task success rate by 80%, while demonstrating strong robustness and generalization to unseen spatial layouts and diverse participants.
Vision-Language-Action (VLA) models have shown strong potential for general-purpose robot manipulation by unifying perception and action. However, existing VLA systems primarily rely on textual instructions and struggle to resolve spatial ambiguity in complex scenes with multiple similar objects. To address this limitation, we introduce gesture as a parallel instruction modality and propose a Gesture-aware Vision-Language-Action model (GesVLA). Our approach encodes gesture features directly into the latent space, enabling them to participate in both high-level reasoning and low-level action generation, and adopts a dual-VLM architecture to achieve tight coupling between gesture representations and action policies. At the data level, we construct a scalable gesture data generation pipeline by rendering hand models onto real-world scene images. This reduces the sim-to-real visual gap while producing rich data with diverse motion patterns and corresponding pointing annotations. In addition, we employ a two-stage training strategy to equip the model with both gesture perception and action prediction capabilities. We evaluate our approach on multiple real-world robotic tasks, including a controlled block manipulation task for validation and more practical scenarios such as product and produce selection. Experimental results show that incorporating gesture consistently improves target grounding accuracy and human-robot interaction efficiency, especially in complex and cluttered environments. Project page: https://gwxuan.github.io/GesVLA/.