Structural Action Transformer for 3D Dexterous Manipulation

Xiaohan Lei; Min Wang; Bohong Weng; Wengang Zhou; Houqiang Li

cs.ROMar 4, 2026

Structural Action Transformer for 3D Dexterous Manipulation

Authors: Xiaohan Lei, Min Wang, Bohong Weng, Wengang Zhou, Houqiang Li

Organizations: MoE Key Laboratory of Brain-inspired Intelligent Perception and Cognition, University of Science and Technology of China · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center

Abstract

Achieving human-level dexterity in robots via imitation learning from heterogeneous datasets is hindered by the challenge of cross-embodiment skill transfer, particularly for high-DoF robotic hands. Existing methods, often relying on 2D observations and temporal-centric action representation, struggle to capture 3D spatial relations and fail to handle embodiment heterogeneity. This paper proposes the Structural Action Transformer (SAT), a new 3D dexterous manipulation policy that challenges this paradigm by introducing a structural-centric perspective. We reframe each action chunk not as a temporal sequence, but as a variable-length, unordered sequence of joint-wise trajectories. This structural formulation allows a Transformer to natively handle heterogeneous embodiments, treating the joint count as a variable sequence length. To encode structural priors and resolve ambiguity, we introduce an Embodied Joint Codebook that embeds each joint's functional role and kinematic properties. Our model learns to generate these trajectories from 3D point clouds via a continuous-time flow matching objective. We validate our approach by pre-training on large-scale heterogeneous datasets and fine-tuning on simulation and real-world dexterous manipulation tasks. Our method consistently outperforms all baselines, demonstrating superior sample efficiency and effective cross-embodiment skill transfer. This structural-centric representation offers a new path toward scaling policies for high-DoF, heterogeneous manipulators.

Figures & tables

Figure 1: Conceptual illustration of action chunk tokenization. (a) The conventional temporal-centric perspective, which structures actions as a sequence of

T

timesteps (chunk length), with each token having dimension

D_{a}

(action dim). (b) Our proposed structural-centric perspective, which reframes the action chunk as a sequence of

D_{a}

joints, where each token’s feature is its temporal trajectory over

T

. This

(D_{a},T)

view naturally handles heterogeneous embodiments as a variable-length, unordered sequence, which is a key feature of our approach.

Figure 2: Our proposed model architecture. The policy takes a history of $T_{o}$ raw 3D point clouds $\mathcal{P}_{t}=(\mathbf{P}_{t-T_{o}+1},\dots,\mathbf{P}_{t})$ and a language instruction $L$ as input. Observation Tokenizer : Each point cloud $\mathbf{P}_{k}$ in the history is processed via Farthest Point Sampling (FPS) and PointNets to extract local geometric tokens and a global scene context. The tokens from each time step are concatenated to form the final observation token sequence. Language is encoded by a T5 tokenizer [ 53 ] . Structural Action Tokenizer : Guided by the manipulator’s morphology, the Embodied Joint Codebook produces structural-centric embeddings aligned with the action dimension $D_{a}$ , which are added to the time-stepped noisy tokens $\mathbf{A}_{t}^{\tau}$ . Structural Action Transformer: A DiT [ 47 ] with causal masking predicts the action velocity field. This field is then integrated via an ODE solver to produce the final action chunk $\mathbf{A}_{t}$ .

Method	Params (M)	Modality	Adroit (3) [ 55 ]	DexArt (4) [ 2 ]	Bi-DexHands (4) [ 10 ]	Average Success
Diffusion Policy [ 12 ]	266.8	2D	$0.32\scriptstyle{\pm 0.03}$	$0.49\scriptstyle{\pm 0.04}$	$0.42\scriptstyle{\pm 0.05}$	$0.42\scriptstyle{\pm 0.04}$
HPT [ 68 ]	13.99	2D	$0.45\scriptstyle{\pm 0.02}$	$0.53\scriptstyle{\pm 0.05}$	$0.44\scriptstyle{\pm 0.04}$	$0.47\scriptstyle{\pm 0.04}$
UniAct [ 82 ]	1053	2D	$0.49\scriptstyle{\pm 0.01}$	$0.55\scriptstyle{\pm 0.03}$	$0.47\scriptstyle{\pm 0.07}$	$0.50\scriptstyle{\pm 0.05}$
3D Diffusion Policy [ 77 ]	255.2	3D	$0.68\scriptstyle{\pm 0.03}$	$0.69\scriptstyle{\pm 0.02}$	$0.55\scriptstyle{\pm 0.14}$	$0.63\scriptstyle{\pm 0.06}$
3D ManiFlow Policy [ 74 ]	218.9	3D	$0.70\scriptstyle{\pm 0.02}$	$0.70\scriptstyle{\pm 0.03}$	$0.59\scriptstyle{\pm 0.07}$	$0.66\scriptstyle{\pm 0.04}$
SAT (Ours)	19.36	3D	$\mathbf{0.75\scriptstyle{\pm 0.02}}$	$\mathbf{0.73\scriptstyle{\pm 0.03}}$	$\mathbf{0.67\scriptstyle{\pm 0.05}}$	$\mathbf{0.71\scriptstyle{\pm 0.04}}$

Table 1: Quantitative comparison of our method against 2D (image-based) and 3D (point cloud-based) baselines on 11 dexterous manipulation tasks from the Adroit [ 55 ] , DexArt [ 2 ] , and Bi-DexHands [ 10 ] simulation benchmarks.

Figure 3: Composition of the offline pre-training dataset. The pie chart illustrates the relative data scale of each of the constituent datasets [ 37 , 18 , 46 , 16 , 67 , 55 , 2 , 10 ] .

Token Dim.	Prams. (M)	1-NFE FLOPs (G)	Success
16	4.71	0.66	$0.66\scriptstyle{\pm 0.02}$
32	8.65	0.77	$\mathbf{0.71\scriptstyle{\pm 0.05}}$
64	19.36	0.99	$\mathbf{0.71\scriptstyle{\pm 0.04}}$
128	52.08	1.65	$\mathbf{0.71\scriptstyle{\pm 0.03}}$
256	162.74	3.63	$0.70\scriptstyle{\pm 0.04}$

Table 2: Ablation on temporal compression and token dimension. We vary the token dimension (

d_{feat}

), which is the embedding size after compressing the temporal trajectory (

T

) of each joint.

Human	Robot	Simulation	Scale	Success
✓	✓	✓	100%	$\mathbf{0.71\scriptstyle{\pm 0.04}}$
✓	✗	✗	100%	$0.68\scriptstyle{\pm 0.04}$
✗	✓	✗	100%	$0.66\scriptstyle{\pm 0.05}$
✗	✗	✓	100%	$0.70\scriptstyle{\pm 0.03}$
✓	✓	✓	10%	$0.68\scriptstyle{\pm 0.03}$

Table 3: Pre-training data ablation. We report the average success on simulation tasks after fine-tuning models pre-trained on different combinations of Human, Robot, and Simulation datasets.

Model Variant	Success
SAT	$\mathbf{0.71\scriptstyle{\pm 0.04}}$
SAT w.o. global point cloud token	$0.68\scriptstyle{\pm 0.05}$
SAT w.o. local point cloud token	$0.69\scriptstyle{\pm 0.03}$
SAT w.o. causal mask	$0.68\scriptstyle{\pm 0.04}$
SAT w.o. joint embedding	$0.01\scriptstyle{\pm 0.01}$
SAT w. temporal-centric action	$0.64\scriptstyle{\pm 0.05}$

Table 4: Model component ablation. We analyze the impact of removing key architectural components on the average success rate.

Figure 4: Few-shot adaptation efficiency. We plot the average success rate versus training epochs for our method and the UniAct [ 82 ] baseline, evaluated in few-shot settings using varying numbers of in-domain demonstrations.

Figure 5: Frequency analysis of joint types in our Embodied Joint Codebook, derived from a survey of 10 common dexterous hands.

Embodiment	Function	Rotation	Success (11 tasks)
✓	✓	✓	$\mathbf{0.71\scriptstyle{\pm 0.04}}$
✗	✓	✓	$0.57\scriptstyle{\pm 0.05}$
✓	✗	✓	$0.02\scriptstyle{\pm 0.02}$
✓	✓	✗	$0.62\scriptstyle{\pm 0.07}$

Table 5: Ablation study on the components of the Embodied Joint Codebook. We report the average success rate after fine-tuning models with different codebook components ablated.

Figure 6: T-SNE visualization of the learned Embodied Joint Codebook embeddings. These embeddings, derived from 10 dexterous manipulators, are colored by (a) Embodiment ID, (b) Functional Category, and (c) Rotation Axis.

Figure 7: Qualitative rollouts of our policy executing complex bimanual tasks in the real world.

Figure 8: Real-world experimental setup. (a) Our bimanual hardware setup. We collect demonstration data using a VR headset for teleoperation. (b) The set of diverse objects used in our real-world manipulation, requiring both precision and bimanual coordination.

	Success Rate
Task Description	HPT [ 68 ]	3DDP [ 77 ]	SAT (Ours)
Remove the pen cap	0.10	0.25	$\mathbf{0.30}$
Hand over Baymax	0.50	0.75	$\mathbf{0.85}$
Push then grab box	0.05	0.15	$\mathbf{0.35}$
Place block in plate	0.60	0.85	$\mathbf{0.90}$
Brush the cup	0.10	0.30	$\mathbf{0.45}$

Table 6: Quantitative results on 6 real-world bimanual manipulation tasks. We compare our model against baselines using in-domain demonstrations.

Explore similar work

Dexterous Point Policy: Learning Point-based Dexterous Hand Policies from Human Demonstrations
Jun 9, 2026Beomjun Kim, Seong Hyeon Park, Seunghoon Sim +3Dexterous Manipulation Cross-Embodiment Robot Learning
KAIST
Cross-Embodiment Robot Manipulation via a Unified Hand Action Space
Jul 3, 2026Luis Felipe Casas, Robert Teal, Keval Shah +3Dexterous Manipulation In-Hand Manipulation
Intelligent Robotics and Vision Lab, University of Texas at Dallas · Intelligent Robotics and Interactive Systems Lab, Arizona State University
One-Policy-Fits-All: Geometry-Aware Action Latents for Cross-Embodiment Manipulation
Mar 15, 2026Juncheng Mu, Sizhe Yang, Hojin Bae +5Cross-Embodiment Robot Learning Cross-Embodiment Transfer
Shanghai AI Laboratory · Tsinghua University · The Chinese University of Hong Kong +2

Jun 9, 2026J/K move · Enter open · S save

Dexterous Point Policy: Learning Point-based Dexterous Hand Policies from Human Demonstrations

Beomjun Kim, Seong Hyeon Park, Seunghoon Sim, Seungjun Moon+2

KAIST

Dexterous Manipulation Cross-Embodiment Robot Learning Robotic Manipulation Robot Imitation Learning

Open paper AskPDF arXiv

Robotic foundation models pre-trained on human demonstration videos have shown promise, but a significant embodiment gap remains when the resulting policies are deployed on real robots. A common remedy is to fine-tune these models on robot-specific demonstrations. However, robot data collection can be prohibitively expensive and time-consuming, which is particularly acute in dexterous manipulation, e.g., teleoperating a multi-fingered hand for even a single atomic task can take days. To address this, we introduce Dexterous Point Policy, a framework that learns dexterous manipulation policies directly from human videos and requires no robot demonstrations. Our core insight is that a unified 3D keypoint representation can bridge human and robot embodiments when used for both observations and actions. Specifically, we extract 3D keypoints of task-relevant objects and human hands from raw videos, and train an autoregressive transformer over these keypoints. We observe that at the keypoint level, specifically the wrist and fingertips, human and robot behaviors closely align, enabling direct policy transfer. On a suite of real-robot tasks spanning pick-and-place and tool use, Dexterous Point Policy attains 75.0% success, whereas a state-of-the-art VLA baseline reaches only 1.0%. Furthermore, our method generalizes strongly to unseen scenarios, including multi-object environments and novel object categories.

Abstract

Explore similar work

Dexterous Point Policy: Learning Point-based Dexterous Hand Policies from Human Demonstrations

Cross-Embodiment Robot Manipulation via a Unified Hand Action Space

One-Policy-Fits-All: Geometry-Aware Action Latents for Cross-Embodiment Manipulation