Structural Action Transformer for 3D Dexterous Manipulation
Authors: Xiaohan Lei, Min Wang, Bohong Weng, Wengang Zhou, Houqiang Li
Organizations: MoE Key Laboratory of Brain-inspired Intelligent Perception and Cognition, University of Science and Technology of China · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center
Achieving human-level dexterity in robots via imitation learning from heterogeneous datasets is hindered by the challenge of cross-embodiment skill transfer, particularly for high-DoF robotic hands. Existing methods, often relying on 2D observations and temporal-centric action representation, struggle to capture 3D spatial relations and fail to handle embodiment heterogeneity. This paper proposes the Structural Action Transformer (SAT), a new 3D dexterous manipulation policy that challenges this paradigm by introducing a structural-centric perspective. We reframe each action chunk not as a temporal sequence, but as a variable-length, unordered sequence of joint-wise trajectories. This structural formulation allows a Transformer to natively handle heterogeneous embodiments, treating the joint count as a variable sequence length. To encode structural priors and resolve ambiguity, we introduce an Embodied Joint Codebook that embeds each joint's functional role and kinematic properties. Our model learns to generate these trajectories from 3D point clouds via a continuous-time flow matching objective. We validate our approach by pre-training on large-scale heterogeneous datasets and fine-tuning on simulation and real-world dexterous manipulation tasks. Our method consistently outperforms all baselines, demonstrating superior sample efficiency and effective cross-embodiment skill transfer. This structural-centric representation offers a new path toward scaling policies for high-DoF, heterogeneous manipulators.
Figures & tables
Figure 1: Conceptual illustration of action chunk tokenization. (a) The conventional temporal-centric perspective, which structures actions as a sequence of T timesteps (chunk length), with each token having dimension Da (action dim). (b) Our proposed structural-centric perspective, which reframes the action chunk as a sequence of Da joints, where each token’s feature is its temporal trajectory over T . This (Da,T) view naturally handles heterogeneous embodiments as a variable-length, unordered sequence, which is a key feature of our approach.
Figure 2: Our proposed model architecture. The policy takes a history of To raw 3D point clouds Pt=(Pt−To+1,…,Pt) and a language instruction L as input. Observation Tokenizer : Each point cloud Pk in the history is processed via Farthest Point Sampling (FPS) and PointNets to extract local geometric tokens and a global scene context. The tokens from each time step are concatenated to form the final observation token sequence. Language is encoded by a T5 tokenizer [ 53 ] . Structural Action Tokenizer : Guided by the manipulator’s morphology, the Embodied Joint Codebook produces structural-centric embeddings aligned with the action dimension Da , which are added to the time-stepped noisy tokens Atτ . Structural Action Transformer: A DiT [ 47 ] with causal masking predicts the action velocity field. This field is then integrated via an ODE solver to produce the final action chunk At .
Method
Params (M)
Modality
Adroit (3) [ 55 ]
DexArt (4) [ 2 ]
Bi-DexHands (4) [ 10 ]
Average Success
Diffusion Policy [ 12 ]
266.8
2D
0.32±0.03
0.49±0.04
0.42±0.05
0.42±0.04
HPT [ 68 ]
13.99
2D
0.45±0.02
0.53±0.05
0.44±0.04
0.47±0.04
UniAct [ 82 ]
1053
2D
0.49±0.01
0.55±0.03
0.47±0.07
0.50±0.05
3D Diffusion Policy [ 77 ]
255.2
3D
0.68±0.03
0.69±0.02
0.55±0.14
0.63±0.06
3D ManiFlow Policy [ 74 ]
218.9
3D
0.70±0.02
0.70±0.03
0.59±0.07
0.66±0.04
SAT (Ours)
19.36
3D
0.75±0.02
0.73±0.03
0.67±0.05
0.71±0.04
Table 1: Quantitative comparison of our method against 2D (image-based) and 3D (point cloud-based) baselines on 11 dexterous manipulation tasks from the Adroit [ 55 ] , DexArt [ 2 ] , and Bi-DexHands [ 10 ] simulation benchmarks.
Figure 3: Composition of the offline pre-training dataset. The pie chart illustrates the relative data scale of each of the constituent datasets [ 37 , 18 , 46 , 16 , 67 , 55 , 2 , 10 ] .
Token Dim.
Prams. (M)
1-NFE FLOPs (G)
Success
16
4.71
0.66
0.66±0.02
32
8.65
0.77
0.71±0.05
64
19.36
0.99
0.71±0.04
128
52.08
1.65
0.71±0.03
256
162.74
3.63
0.70±0.04
Table 2: Ablation on temporal compression and token dimension. We vary the token dimension ( dfeat ), which is the embedding size after compressing the temporal trajectory ( T ) of each joint.
Human
Robot
Simulation
Scale
Success
✓
✓
✓
100%
0.71±0.04
✓
✗
✗
100%
0.68±0.04
✗
✓
✗
100%
0.66±0.05
✗
✗
✓
100%
0.70±0.03
✓
✓
✓
10%
0.68±0.03
Table 3: Pre-training data ablation. We report the average success on simulation tasks after fine-tuning models pre-trained on different combinations of Human, Robot, and Simulation datasets.
Model Variant
Success
SAT
0.71±0.04
SAT w.o. global point cloud token
0.68±0.05
SAT w.o. local point cloud token
0.69±0.03
SAT w.o. causal mask
0.68±0.04
SAT w.o. joint embedding
0.01±0.01
SAT w. temporal-centric action
0.64±0.05
Table 4: Model component ablation. We analyze the impact of removing key architectural components on the average success rate.
Figure 4: Few-shot adaptation efficiency. We plot the average success rate versus training epochs for our method and the UniAct [ 82 ] baseline, evaluated in few-shot settings using varying numbers of in-domain demonstrations.
Figure 5: Frequency analysis of joint types in our Embodied Joint Codebook, derived from a survey of 10 common dexterous hands.
Embodiment
Function
Rotation
Success (11 tasks)
✓
✓
✓
0.71±0.04
✗
✓
✓
0.57±0.05
✓
✗
✓
0.02±0.02
✓
✓
✗
0.62±0.07
Table 5: Ablation study on the components of the Embodied Joint Codebook. We report the average success rate after fine-tuning models with different codebook components ablated.
Figure 6: T-SNE visualization of the learned Embodied Joint Codebook embeddings. These embeddings, derived from 10 dexterous manipulators, are colored by (a) Embodiment ID, (b) Functional Category, and (c) Rotation Axis.
Figure 7: Qualitative rollouts of our policy executing complex bimanual tasks in the real world.
Figure 8: Real-world experimental setup. (a) Our bimanual hardware setup. We collect demonstration data using a VR headset for teleoperation. (b) The set of diverse objects used in our real-world manipulation, requiring both precision and bimanual coordination.
Success Rate
Task Description
HPT [ 68 ]
3DDP [ 77 ]
SAT (Ours)
Remove the pen cap
0.10
0.25
0.30
Hand over Baymax
0.50
0.75
0.85
Push then grab box
0.05
0.15
0.35
Place block in plate
0.60
0.85
0.90
Brush the cup
0.10
0.30
0.45
Table 6: Quantitative results on 6 real-world bimanual manipulation tasks. We compare our model against baselines using in-domain demonstrations.
Robotic foundation models pre-trained on human demonstration videos have shown promise, but a significant embodiment gap remains when the resulting policies are deployed on real robots. A common remedy is to fine-tune these models on robot-specific demonstrations. However, robot data collection can be prohibitively expensive and time-consuming, which is particularly acute in dexterous manipulation, e.g., teleoperating a multi-fingered hand for even a single atomic task can take days. To address this, we introduce Dexterous Point Policy, a framework that learns dexterous manipulation policies directly from human videos and requires no robot demonstrations. Our core insight is that a unified 3D keypoint representation can bridge human and robot embodiments when used for both observations and actions. Specifically, we extract 3D keypoints of task-relevant objects and human hands from raw videos, and train an autoregressive transformer over these keypoints. We observe that at the keypoint level, specifically the wrist and fingertips, human and robot behaviors closely align, enabling direct policy transfer. On a suite of real-robot tasks spanning pick-and-place and tool use, Dexterous Point Policy attains 75.0% success, whereas a state-of-the-art VLA baseline reaches only 1.0%. Furthermore, our method generalizes strongly to unseen scenarios, including multi-object environments and novel object categories.
Robot manipulation policies are typically tied to specific robotic hand embodiments, limiting the transfer of learned behaviors across platforms with different kinematic structures. In this work, we propose the Unified Hand Action Space (UHAS), a sphere-based unified action representation for cross-embodiment dexterous manipulation. UHAS represents robotic hand actions as geometric deformations of a canonical sphere and uses a Cascade Inverse Kinematics (CIK) algorithm to map the shared representation to embodiment-specific joint configurations. Using reinforcement learning, we train dexterous manipulation policies directly in the proposed action space for in-hand cube reorientation tasks. We evaluate our method in both simulation and real-world experiments across multiple robotic hands, including the Allegro Hand, LEAP Hand, Shadow Hand, and MANO Human Hand. Experimental results demonstrate effective dexterous manipulation, zero-shot transfer to unseen hands, rapid finetuning across embodiments, and successful real-world deployment. Our experiments show that the proposed UHAS representation enables stable dexterous control and cross-embodiment policy transfer across robotic hands.
Luis Felipe Casas, Robert Teal, Keval Shah +3
Intelligent Robotics and Vision Lab, University of Texas at Dallas · Intelligent Robotics and Interactive Systems Lab, Arizona State University
Cross-embodiment manipulation is crucial for enhancing the scalability of robot manipulation and reducing the high cost of data collection. However, the significant differences between embodiments, such as variations in action spaces and structural disparities, pose challenges for joint training across multiple sources of data. To address this, we propose One-Policy-Fits-All (OPFA), a framework that enables learning a single, versatile policy across multiple embodiments. We first learn a Geometry-Aware Latent Representation (GaLR), which leverages 3D convolution networks and transformers to build a shared latent action space across different embodiments. Then we design a unified latent retargeting decoder that extracts embodiment-specific actions from the latent representations, without any embodiment-specific decoder tuning. OPFA enables end-to-end co-training of data from diverse embodiments, including various grippers and dexterous hands with arbitrary degrees of freedom, significantly improving data efficiency and reducing the cost of skill transfer. We conduct extensive experiments across 11 different end-effectors. The results demonstrate that OPFA significantly improves policy performance in diverse settings by leveraging heterogeneous embodiment data. For instance, cross-embodiment co-training can improve success rates by more than 50% compared to single-source training. Moreover, by adding only a few demonstrations from a new embodiment (e.g., eight), OPFA can achieve performance comparable to that of a well-trained model with 72 demonstrations.
Juncheng Mu, Sizhe Yang, Hojin Bae +5
Shanghai AI Laboratory · Tsinghua University · The Chinese University of Hong Kong +2