BLaDA: Bridging Language to Functional Dexterous Actions within 3DGS Fields

Fan Yang; Wenrui Chen; Guorun Yan; Ruize Liao; Wanjun Jia; Dongsheng Luo; Jiacheng Lin; Kailun Yang

cs.CVApr 9, 2026

BLaDA: Bridging Language to Functional Dexterous Actions within 3DGS Fields

Authors: Fan Yang, Wenrui Chen, Guorun Yan, Ruize Liao, Wanjun Jia, Dongsheng Luo, Jiacheng Lin, Kailun Yang,

Organizations: School of Artificial Intelligence and Robotics, Hunan University, Changsha 410012, China. · National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University, Changsha 410082, China. · Key Laboratory of Advanced Manufacturing Technology of the Ministry of Education, Guizhou University, Guiyang 550025, China.

Abstract

In unstructured environments, functional dexterous grasping calls for the tight integration of semantic understanding, precise 3D functional localization, and physically interpretable execution. Modular hierarchical methods are more controllable and interpretable than end-to-end VLA approaches, but existing ones still rely on predefined affordance labels and lack the tight semantic--pose coupling needed for functional dexterous manipulation. To address this, we propose BLaDA (Bridging Language to Dexterous Actions in 3DGS fields), an interpretable zero-shot framework that grounds open-vocabulary instructions as perceptual and control constraints for functional dexterous manipulation. BLaDA establishes an interpretable reasoning chain by first parsing natural language into a structured sextuple of manipulation constraints via a Knowledge-guided Language Parsing (KLP) module. To achieve pose-consistent spatial reasoning, we introduce the Triangular Functional Point Localization (TriLocation) module, which utilizes 3D Gaussian Splatting as a continuous scene representation and identifies functional regions under triangular geometric constraints. Finally, the 3D Keypoint Grasp Matrix Transformation Execution (KGT3D+) module decodes these semantic-geometric constraints into physically plausible wrist poses and finger-level commands. Extensive experiments on complex benchmarks demonstrate that BLaDA significantly outperforms existing methods in both affordance grounding precision and the success rate of functional manipulation across diverse categories and tasks. Code will be publicly available at https://github.com/PopeyePxx/BLaDA.

Figures & tables

Fig. 1: Comparison of existing pipelines: (a) end-to-end VLA is data-hungry, black-box, and poor generalization; (b) affordance-based methods rely on predefined labels, limited 2.5D localization, and primitive control; (c) BLaDA (ours) uses a structured intermediate $S$ with 3D executable constraints for zero-shot, intent-conditioned execution.

Fig. 2: Overview of BLaDA. The top illustrates the construction of knowledge-guided functionality prompting and example demonstrations of “Handover me the drill” and “Use the drill”. The bottom shows the overall pipeline, which consists of three stages: (1) Language parsing (blue): the KLP module parses the input instruction $L$ into structured manipulation primitive elements $\mathcal{S}=(g^{a},g^{r},g^{t},g^{f},\kappa,\tau)$ ; (2) 3D Gaussian reconstruction and localization (green): TriLocation reconstructs a semantic 3D Gaussian field from multi-view RGB observations and localizes three functional keypoints $P_{i}(x,y,z)$ , conditioned on $(g^{a},g^{r},\kappa,\tau)$ ; (3) Dexterous manipulation (yellow): FKGT3D(+) generates relative hand–object contact poses and produces fine-grained dexterous control actions under semantic constraints $(g^{t},g^{f})$ , which are finally executed in the real world.

Fig. 3: Extract verbs, intent phrases, and tool types from human natural language instructions $L$ , and classify them into specific task-intent priors and tool-topology priors.

Fig. 4: Overview of the TriLocation. a. We design the HSE module (highlighted in yellow), consisting of Select and Context-Aware Cropping units to decouple object/part regions and resolve semantic drift. This enables the construction of a clean, high-precision 3D Gaussian semantic field for keypoint localization. b. The module computes the CLIP [ 39 ] similarity between $g^{a}$ and each Gaussian feature $f_{i}$ to locate the semantic anchor $p_{1}$ . c. A lightweight MLP predicts two relative offsets $\Delta p_{2}$ and $\Delta p_{3}$ , forming the graspable triangle structure $\{p_{1},p_{2},p_{3}\}$ , which is supervised by structure-aware losses based on edge lengths and internal angles.

Task Intent $\kappa$	Tool Topology $\tau$	$\hat{z}$ (Primary Axis)	$\hat{y}$ (Hand Orientation)
Hold	$\tau_{\mathrm{rod}}$	Gravity Direction	Structural Axis
	$\tau_{\mathrm{handle}}$	Radial Direction	Handle Axis
	$\tau_{\mathrm{knob}}$	Surface Normal	Tangential Direction
	$\tau_{\mathrm{surface}}$	Surface Normal	Major In-plane Axis
Press	$\tau_{\mathrm{handle}}$	Tool Forward Axis	Handle Axis
Open	$\tau_{\mathrm{knob}}$	Rotation Axis	Tangential Direction

TABLE I: Definition of Local Coordinate Frames under Task-Tool Topology Constraints. The frame follows the right-hand rule where

\hat{x}=\hat{y}\times\hat{z}

Fig. 5: Real-world experiment setting and 6 typical scenarios demonstration.

Fig. 6: Relevance maps of given language instructions. We project the language-activated 3D Gaussian semantic features onto 2D images for visualization. The orange panels denote object-level relevance maps, while the green panels denote part-level relevance maps. Red and green rectangles highlight the erroneous or ambiguous responses of GraspSplats, and yellow ellipses indicate that our method produces more compact and complete response regions for local semantic parts.

Model	$g^{a}$	$g^{t}$	$g^{f}$	$g^{r}$	$t$	$k$	Avg.
ChatGPT4.0 [ 44 ] (w/)	0.65	0.81	0.74	0.74	0.72	0.86	0.753
ChatGPT4.0 [ 44 ] (w/o)	0.43	0.13	0.45	0.51	0.72	0.81	0.508
DeepSeekv3 [ 45 ] (w/)	0.62	0.80	0.74	0.79	0.73	0.78	0.743
DeepSeekv3 [ 45 ] (w/o)	0.42	0.18	0.46	0.62	0.77	0.79	0.540
Gemini2.5 [ 46 ] (w/)	0.70	0.85	0.64	0.80	0.66	0.82	0.745
Gemini2.5 [ 46 ] (w/o)	0.51	0.22	0.50	0.65	0.59	0.78	0.542

TABLE II: Comparison of LLMs w/ and w/o KLP on Language Reasoning Accuracy (LRA). Best results are in bold.

Fig. 7: Visualization of the effect of the local coordinate system on 3D functional keypoint localization. Each example presents, in order: the input image, a 3D visualization of the local coordinate system (red/green/blue indicate the $x/y/z$ axes, where blue denotes the approach axis $z$ and green denotes the grasp axis $y$ ), the predicted three-point structure under the local-coordinate constraint, and the results without this constraint (red boxes).

Query Item	Model	MAE $\downarrow$	P_En $\uparrow$	KLD $\downarrow$	SIM $\uparrow$	NSS $\uparrow$
Hammer Handle	GraspSplats [ 28 ]	0.0141	0.2798	14.1642	0.1449	1.2916
Hammer Handle	Ours	0.0127	0.5962	13.0819	0.2040	2.7209
Improvement		(9.9%)	(113.1%)	(7.6%)	(40.8%)	(110.7%)
Spray Bottle Pink Button	GraspSplats [ 28 ]	0.0028	0.4001	12.7509	0.2529	7.9788
Spray Bottle Pink Button	Ours	0.0028	0.4134	12.5954	0.2534	8.0849
Improvement		(0.0%)	(3.3%)	(1.2%)	(0.2%)	(1.3%)

TABLE III: Quantitative comparison of 2D Part Localization Performance. Our method shows significant improvements in localization accuracy and energy focus across different object scales.

Method	Hold	Press	Open	Click	Mean LSR
MKA [ 19 ]	50	20	10	10	22.5
w/o Local Coordinate	50	0	0	50	25
w/o HSE Module	62.5	50	50	50.0	53.13
TriLocation (Ours)	75	50	100	50	68.75

TABLE IV: Quantitative comparison of LSR (%) across 4 task categories and ablation variants.

Fig. 8: Dexterous grasping demonstration workflow based on 3D reconstructed points. Left of the dashed line: predicted reference points. Right of the dashed line, in sequence: initial state; pose alignment; coarse grasping based on $g^{t}$ ; tightening non- $g^{f}$ fingers; and $g^{f}$ fingers exerting $g^{r}$ force (Note that, except for the finger-force actuation, the post-grasp motions are completed by demonstration).

Fig. 9: Hyperparameter analysis of $\alpha$ and $\gamma$ . The optimal configuration is highlighted.

Task-Tool	Ours	MKA [ 19 ]	DP* [ 47 ]
Hold Spraybottle	80	40	50
Press Spraybottle	30	10	10

TABLE V: FGS (%) on two real-world task-tool combinations (5 trials each).

Robotic Manipulation

Robotic manipulation studies how robots perceive, plan, and control physical interactions to grasp, move, assemble, and use objects.

See all 1,685 papers

ZeroDex: Zero-Shot Long-Horizon Dexterous Manipulation via Multi-View 3D-Grounded VLM Reasoning
Jun 17, 2026Jisoo Kim, Sangwon Baik, Taeksoo Kim +6Dexterous Manipulation Zero-Shot Robotic Manipulation
Seoul National University · RLWRLD
Embodied Multimodal Grounding for Open-Vocabulary Mobile Manipulation via Semantic 3D Gaussian Splatting
Aug 11, 2026Huosen Ou, Dongni Song, Yuncong Wang +2Multimodal Grounding 3D Gaussian Splatting
The Hong Kong University of Science and Technology (Guangzhou) · Midea Group · The Hong Kong University of Science and Technology
OpenDexGrasp: Open-vocabulary Task-Oriented Dexterous Grasping
Sep 16, 2026Jiyao Zhang, Junhan Wang, Tianyu Wang +4Robotic Grasping Dexterous Manipulation
CFCS, School of CS, PKU, China · National Key Laboratory for Multimedia Information Processing, School of CS, PKU, China · PrimeBot +1

Jun 17, 2026J/K move · Enter open · S save

ZeroDex: Zero-Shot Long-Horizon Dexterous Manipulation via Multi-View 3D-Grounded VLM Reasoning

Jisoo Kim, Sangwon Baik, Taeksoo Kim, Sungjoo Kim+5

Seoul National University · RLWRLD

Dexterous Manipulation Zero-Shot Robotic Manipulation Long-Horizon Robotic Manipulation Language-Conditioned Robot Manipulation Generalization in Robotic Manipulation VLMs for Robotics+1

Open paper AskPDF arXiv

We present ZeroDex, a zero-shot framework for long-horizon dexterous manipulation that grounds language instructions into executable 3D task plans from calibrated multi-view RGB images. Our framework builds on the grounding ability that vision-language models (VLMs) already exhibit in 2D and lifts their predictions into 3D through multi-view fusion, which extends VLM grounding beyond image-space keypoints to the metric 3D quantities that manipulation actually requires: contact points, placement targets, and tool-motion anchors. Because execution is driven by these 3D groundings rather than a learned policy, the system preserves the VLM's open-world generalization, without collecting demonstrations or post-training for a specific task or embodiment. The VLM decomposes a complex instruction into a sequence of atomic tasks, each executed by a primitive from a small reusable library that covers manipulation tasks including pick-and-place and tool-use: for tool-use, we retrieve an object-centric atomic action corresponding to the inferred skill category and align its stored 6D tool trajectory to the grounded keypoints; for dexterous execution, we expand the lifted grasp keypoint into a task-conditioned grasp affordance region and generate feasible grasp-motion pairs with an arm-hand motion generator. Real-world experiments show improved 3D grounding accuracy and higher task success than zero-shot visual-prompting and constraint-based baselines, both of which ground from a single RGB-D view, and our demonstration-free system outperforms VLA policies fine-tuned on 100 task demonstrations. We further demonstrate long-horizon manipulation, where execution verification and replanning recover from intermediate failures, and show that the system generalizes to unseen objects and tool-use tasks in novel scenes.

Abstract

Robotic Manipulation

Explore similar work

ZeroDex: Zero-Shot Long-Horizon Dexterous Manipulation via Multi-View 3D-Grounded VLM Reasoning

Embodied Multimodal Grounding for Open-Vocabulary Mobile Manipulation via Semantic 3D Gaussian Splatting

OpenDexGrasp: Open-vocabulary Task-Oriented Dexterous Grasping