Organizations: School of Artificial Intelligence and Robotics, Hunan University, Changsha 410012, China. · National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University, Changsha 410082, China. · Key Laboratory of Advanced Manufacturing Technology of the Ministry of Education, Guizhou University, Guiyang 550025, China.
In unstructured environments, functional dexterous grasping calls for the tight integration of semantic understanding, precise 3D functional localization, and physically interpretable execution. Modular hierarchical methods are more controllable and interpretable than end-to-end VLA approaches, but existing ones still rely on predefined affordance labels and lack the tight semantic--pose coupling needed for functional dexterous manipulation. To address this, we propose BLaDA (Bridging Language to Dexterous Actions in 3DGS fields), an interpretable zero-shot framework that grounds open-vocabulary instructions as perceptual and control constraints for functional dexterous manipulation. BLaDA establishes an interpretable reasoning chain by first parsing natural language into a structured sextuple of manipulation constraints via a Knowledge-guided Language Parsing (KLP) module. To achieve pose-consistent spatial reasoning, we introduce the Triangular Functional Point Localization (TriLocation) module, which utilizes 3D Gaussian Splatting as a continuous scene representation and identifies functional regions under triangular geometric constraints. Finally, the 3D Keypoint Grasp Matrix Transformation Execution (KGT3D+) module decodes these semantic-geometric constraints into physically plausible wrist poses and finger-level commands. Extensive experiments on complex benchmarks demonstrate that BLaDA significantly outperforms existing methods in both affordance grounding precision and the success rate of functional manipulation across diverse categories and tasks. Code will be publicly available at https://github.com/PopeyePxx/BLaDA.
Figures & tables
Fig. 1: Comparison of existing pipelines: (a) end-to-end VLA is data-hungry, black-box, and poor generalization; (b) affordance-based methods rely on predefined labels, limited 2.5D localization, and primitive control; (c) BLaDA (ours) uses a structured intermediate S with 3D executable constraints for zero-shot, intent-conditioned execution.
Fig. 2: Overview of BLaDA. The top illustrates the construction of knowledge-guided functionality prompting and example demonstrations of “Handover me the drill” and “Use the drill”. The bottom shows the overall pipeline, which consists of three stages: (1) Language parsing (blue): the KLP module parses the input instruction L into structured manipulation primitive elements S=(ga,gr,gt,gf,κ,τ) ; (2) 3D Gaussian reconstruction and localization (green): TriLocation reconstructs a semantic 3D Gaussian field from multi-view RGB observations and localizes three functional keypoints Pi(x,y,z) , conditioned on (ga,gr,κ,τ) ; (3) Dexterous manipulation (yellow): FKGT3D(+) generates relative hand–object contact poses and produces fine-grained dexterous control actions under semantic constraints (gt,gf) , which are finally executed in the real world.
Fig. 3: Extract verbs, intent phrases, and tool types from human natural language instructions L , and classify them into specific task-intent priors and tool-topology priors.
Fig. 4: Overview of the TriLocation. a. We design the HSE module (highlighted in yellow), consisting of Select and Context-Aware Cropping units to decouple object/part regions and resolve semantic drift. This enables the construction of a clean, high-precision 3D Gaussian semantic field for keypoint localization. b. The module computes the CLIP [ 39 ] similarity between ga and each Gaussian feature fi to locate the semantic anchor p1 . c. A lightweight MLP predicts two relative offsets Δp2 and Δp3 , forming the graspable triangle structure {p1,p2,p3} , which is supervised by structure-aware losses based on edge lengths and internal angles.
Task Intent κ
Tool Topology τ
z^ (Primary Axis)
y^ (Hand Orientation)
Hold
τrod
Gravity Direction
Structural Axis
τhandle
Radial Direction
Handle Axis
τknob
Surface Normal
Tangential Direction
τsurface
Surface Normal
Major In-plane Axis
Press
τhandle
Tool Forward Axis
Handle Axis
Open
τknob
Rotation Axis
Tangential Direction
TABLE I: Definition of Local Coordinate Frames under Task-Tool Topology Constraints. The frame follows the right-hand rule where x^=y^×z^ .
Fig. 5: Real-world experiment setting and 6 typical scenarios demonstration.
Fig. 6: Relevance maps of given language instructions. We project the language-activated 3D Gaussian semantic features onto 2D images for visualization. The orange panels denote object-level relevance maps, while the green panels denote part-level relevance maps. Red and green rectangles highlight the erroneous or ambiguous responses of GraspSplats, and yellow ellipses indicate that our method produces more compact and complete response regions for local semantic parts.
Model
ga
gt
gf
gr
t
k
Avg.
ChatGPT4.0 [ 44 ] (w/)
0.65
0.81
0.74
0.74
0.72
0.86
0.753
ChatGPT4.0 [ 44 ] (w/o)
0.43
0.13
0.45
0.51
0.72
0.81
0.508
DeepSeekv3 [ 45 ] (w/)
0.62
0.80
0.74
0.79
0.73
0.78
0.743
DeepSeekv3 [ 45 ] (w/o)
0.42
0.18
0.46
0.62
0.77
0.79
0.540
Gemini2.5 [ 46 ] (w/)
0.70
0.85
0.64
0.80
0.66
0.82
0.745
Gemini2.5 [ 46 ] (w/o)
0.51
0.22
0.50
0.65
0.59
0.78
0.542
TABLE II: Comparison of LLMs w/ and w/o KLP on Language Reasoning Accuracy (LRA). Best results are in bold.
Fig. 7: Visualization of the effect of the local coordinate system on 3D functional keypoint localization. Each example presents, in order: the input image, a 3D visualization of the local coordinate system (red/green/blue indicate the x/y/z axes, where blue denotes the approach axis z and green denotes the grasp axis y ), the predicted three-point structure under the local-coordinate constraint, and the results without this constraint (red boxes).
Query Item
Model
MAE ↓
P_En ↑
KLD ↓
SIM ↑
NSS ↑
Hammer Handle
GraspSplats [ 28 ]
0.0141
0.2798
14.1642
0.1449
1.2916
Ours
0.0127
0.5962
13.0819
0.2040
2.7209
Improvement
(9.9%)
(113.1%)
(7.6%)
(40.8%)
(110.7%)
Spray Bottle Pink Button
GraspSplats [ 28 ]
0.0028
0.4001
12.7509
0.2529
7.9788
Ours
0.0028
0.4134
12.5954
0.2534
8.0849
Improvement
(0.0%)
(3.3%)
(1.2%)
(0.2%)
(1.3%)
TABLE III: Quantitative comparison of 2D Part Localization Performance. Our method shows significant improvements in localization accuracy and energy focus across different object scales.
Method
Hold
Press
Open
Click
Mean LSR
MKA [ 19 ]
50
20
10
10
22.5
w/o Local Coordinate
50
0
0
50
25
w/o HSE Module
62.5
50
50
50.0
53.13
TriLocation (Ours)
75
50
100
50
68.75
TABLE IV: Quantitative comparison of LSR (%) across 4 task categories and ablation variants.
Fig. 8: Dexterous grasping demonstration workflow based on 3D reconstructed points. Left of the dashed line: predicted reference points. Right of the dashed line, in sequence: initial state; pose alignment; coarse grasping based on gt ; tightening non- gf fingers; and gf fingers exerting gr force (Note that, except for the finger-force actuation, the post-grasp motions are completed by demonstration).
Fig. 9: Hyperparameter analysis of α and γ . The optimal configuration is highlighted.
Task-Tool
Ours
MKA [ 19 ]
DP* [ 47 ]
Hold Spraybottle
80
40
50
Press Spraybottle
30
10
10
TABLE V: FGS (%) on two real-world task-tool combinations (5 trials each).
We present ZeroDex, a zero-shot framework for long-horizon dexterous manipulation that grounds language instructions into executable 3D task plans from calibrated multi-view RGB images. Our framework builds on the grounding ability that vision-language models (VLMs) already exhibit in 2D and lifts their predictions into 3D through multi-view fusion, which extends VLM grounding beyond image-space keypoints to the metric 3D quantities that manipulation actually requires: contact points, placement targets, and tool-motion anchors. Because execution is driven by these 3D groundings rather than a learned policy, the system preserves the VLM's open-world generalization, without collecting demonstrations or post-training for a specific task or embodiment. The VLM decomposes a complex instruction into a sequence of atomic tasks, each executed by a primitive from a small reusable library that covers manipulation tasks including pick-and-place and tool-use: for tool-use, we retrieve an object-centric atomic action corresponding to the inferred skill category and align its stored 6D tool trajectory to the grounded keypoints; for dexterous execution, we expand the lifted grasp keypoint into a task-conditioned grasp affordance region and generate feasible grasp-motion pairs with an arm-hand motion generator. Real-world experiments show improved 3D grounding accuracy and higher task success than zero-shot visual-prompting and constraint-based baselines, both of which ground from a single RGB-D view, and our demonstration-free system outperforms VLA policies fine-tuned on 100 task demonstrations. We further demonstrate long-horizon manipulation, where execution verification and replanning recover from intermediate failures, and show that the system generalizes to unseen objects and tool-use tasks in novel scenes.
Embodied mobile manipulation requires language, visual observations, three-dimensional scene structure, and action feasibility to be aligned before execution. We study open-vocabulary target grounding with few-shot manipulation in local household workspaces and present an embodied multimodal grounding framework that integrates active multi-view Semantic 3D Gaussian Splatting (Semantic-3DGS), reachability-aware base positioning, and a diffusion-based vision-language-action policy. A task-driven local Semantic-3DGS serves as a shared interface across active sensing, language-conditioned 3D localization, obstacle-aware scene reasoning, base preparation, and semantic conditioning of the action model. To preserve pretrained action priors, the 3D semantic cues are injected only into the late action-expert blocks. In expanded 50-trial real-robot evaluations against representative vision-language-action (VLA) approaches, the full system achieves 60% long-horizon success compared with 40% for PointVLA and 28% for DexVLA, and reaches 74% success in heavily cluttered manipulation compared with 52% for the single-view variant and 46% for PointVLA. It also maintains 75% success under a 75 cm height shift and eliminates photo-induced false grasps. These results indicate that explicit, refreshable 3D semantic grounding can improve robustness under clutter, occlusion, viewpoint variation, and embodiment constraints.
Huosen Ou, Dongni Song, Yuncong Wang +2
The Hong Kong University of Science and Technology (Guangzhou) · Midea Group · The Hong Kong University of Science and Technology
Dexterous grasp synthesis has advanced rapidly in generating stable and physically plausible hand poses, but real-world manipulation requires grasps that preserve the function implied by the task. We study open-vocabulary task-oriented dexterous grasp generation, where a robot must infer functional intent from free-form language, ground it in multi-view visual observations and object geometry, and generate an executable high-degree-of-freedom grasp. We present OpenDexGrasp, a unified data and generative modeling framework for this setting. OpenDexVerse provides dual-source supervision organized by the Coverage-to-Alignment (C2A) Recipe: OpenDex-Scale offers large-scale semantic and geometric coverage through automatic grasp synthesis and vision-language annotation, while OpenDex-Align supplies high-quality embodied alignment through human teleoperation and category-level transfer. OpenDexGrasp learns a shared perception-action latent representation that couples open-vocabulary vision-language context with dexterous action generation. Affordance grounding and grasp generation provide complementary supervision over this latent space, enabling direct generation of task-consistent dexterous grasps without a separate affordance-to-pose inference stage. Extensive simulation and real-robot experiments demonstrate improved functional alignment, physical feasibility, generalization to unseen categories, and real-world execution success. Additional details and videos are available at https://opendexgrasp.github.io/.
Jiyao Zhang, Junhan Wang, Tianyu Wang +4
CFCS, School of CS, PKU, China · National Key Laboratory for Multimedia Information Processing, School of CS, PKU, China · PrimeBot +1