PhysGraph: Physically-Grounded Graph-Transformer Policies for Bimanual Dexterous Hand-Tool-Object Manipulation
Authors: Runfa Blark Li, David Kim, Xinshuang Liu, Keito Suzuki, Dwait Bhatt, Nikola Raicevic, Xin Lin, Ki Myung Brian Lee, Nikolay Atanasov, Truong Nguyen
Organizations: University of California, San Diego
Bimanual dexterous manipulation for tool use remains a formidable challenge in robotics due to the high-dimensional state space and complicated contact dynamics. Existing methods naively represent the entire system state as a single configuration vector, disregarding the rich structural and topological information inherent to articulated hands. We present PhysGraph, a physically-grounded graph transformer policy designed explicitly for challenging bimanual hand-tool-object manipulation. Unlike prior works, we represent the bimanual system as a kinematic graph and introduce per-link tokenization to preserve fine-grained local state information. We propose a physically-grounded bias generator that injects structural priors directly into the attention mechanism, including kinematic spatial distance, dynamic contact states, geometric proximity, and anatomical properties. This allows the policy to explicitly reason about physical interactions rather than learning them implicitly from sparse rewards. Extensive experiments show that PhysGraph significantly outperforms baseline - ManipTrans in manipulation precision and task success rates while using only 51% of the parameters of ManipTrans. Furthermore, the inherent topological flexibility of our architecture shows qualitative zero-shot transfer to unseen tool/object geometries, and is sufficiently general to be trained on three robotic hands (Shadow, Allegro, Inspire).
Figures & tables
Fig. 2 : Overview of PhysGraph: (a) Physical Graph & Tokenization: The bimanual workspace is modeled as a graph where nodes represent links of the left/right hands, tools, and objects and edges represent kinematic (joints) or dynamic (contact) interactions. State-based multi-modal observations for each link are processed into parallel input tokens. (b) Physically-Grounded Bias Generator: This module computes four distinct biases (detailed in Fig. 3 ), aggregated into a composite bias matrix \mathchar29005 . The biases are applied via head-specific masking, allowing different attention heads to focus on specific physical relationships. (c) Graph Transformer Encoder: The tokenized inputs are processed by a transformer encoder where the Multi-Head Attention (MHA) is modulated by the generated bias in (b). (d) Output Heads: The globally encoded \delimiter67482370\mathchar29008\mathchar29007\mathchar29004\delimiter84267779 token is passed to MLP heads to predict the policy action distribution ( \mathchar28950 , \mathchar28955 ) and value function ( \mathchar29014 ).
Fig. 3 : Details of the Physically-Grounded Biases. (a) Spatial Bias: Encodes the topological structure of the hand graph by mapping the shortest path distance \mathchar29028\delimiter67273472\mathchar29045\mathchar24891\delimiter69640972\delimiter84054785 between nodes to learned scalar embeddings \mathchar29026\mathchar29043\mathchar29040\delimiter67273472\mathchar29032\delimiter84054785 , capturing the “hop” distance in the kinematic chain. (b) Edge Bias: Injects structural information by assigning distinct learned embeddings \mathchar29026\mathchar29029\mathchar29028\mathchar29031\mathchar29029\delimiter67273472\mathchar29032\delimiter84054785 indexed by the edge type \mathchar28956 connecting two nodes. (c) Geometric Bias: Incorporates spatial proximity in Cartesian space. It utilizes a Radial Basis Function (RBF) kernel \mathchar28948\delimiter67273472\mathchar29045\mathchar24891\delimiter69640972\delimiter84054785 to weight attention based on the Euclidean distance \delimiter69645069\mathchar29040\mathchar29045\mathchar8704\mathchar29040\delimiter69640972\delimiter69645069\mathchar28722 between node positions. (d) Anatomical Priors: Encodes knowledge regarding anatomical hand kinematics. The Serial Mask ( \mathchar29005\mathchar29043\mathchar29029\mathchar29042 ) highlights dependencies along the kinematic chain of a single finger, while the Synergy Mask ( \mathchar29005\mathchar29043\mathchar29049\mathchar29038 ) promotes attention between corresponding links across different fingers with the same anatomical levels.
Task ID
Description
Tool (R) / Object (L)
Methods
SR (%) \delimiter52568952
E_t (cm) \delimiter52573049
E_j (cm) \delimiter52573049
E_ft (cm) \delimiter52573049
083f7@0
Cut bread with knife
Knife / bread
Maniptrans
55.69
1.31
2.42
2.01
PhysGraph (No Bias)
82.84
1.02
2.26
1.92
PhysGraph (full)
90.05
0.69
2.17
1.42
97055@0
Scoop into bowl
Spoon / bowl
Maniptrans
74.99
1.76
3.10
2.16
PhysGraph (No Bias)
82.58
1.58
2.70
2.48
PhysGraph (full)
86.95
1.40
2.38
2.24
TABLE I : Quantitative results on bimanual tool-use tasks of Oakink2 dataset [ 23 ] . SR: Success Rate; E_t: Tool & Object Translation Error; E_j: Hand Joint Error; E_ft: Fingertip Error. ArtiMANO Hand. Pink indicates the best performance.
Fig. 4 : Qualitative results on bimanual tool-use tasks of Oakink2 dataset. Left to right: Ground truth, PhysGraph (ours), ManipTrans. Top to bottom: 0837f@0, e1fa6@0, 817fb@0, 1292e@0. Please refer to the corresponding videos on our website.
Fig. 5 : Zero-Shot Policy Transfer without finetuning . Left to right: Ground truth, PhysGraph (ours), ManipTrans. Top to bottom (sequence trained \mathchar12833 sequence deployed): 0837f@0 \mathchar12833 9fc3e@0 (chop knife \mathchar12833 fruit knife; bread \mathchar12833 apple), e1fa6@0 \mathchar12833 66c7f@0, 1292e@0 \mathchar12833 b9695@0.
Fig. 6 : Qualitative Results on Multi-Embodiments. Left to right: Ground truth, PhysGraph (ours), ManipTrans. Top to bottom: 0837f@0 (Shadow), e1fa6@0 (Allegro), 817fb@0 (Shadow), 1292e@0 (Shadow).
Task ID
Hand
Method
SR \delimiter52568952
E_t \delimiter52573049
E_j \delimiter52573049
E_ft \delimiter52573049
083f7@0
Inspire
ManipTrans
45.07
2.02
3.83
2.92
PhysGraph
62.14
1.62
2.67
2.3
Shadow
ManipTrans
22.34
1.58
3.54
2.67
PhysGraph
54.77
1.25
3.03
2.26
Allegro
ManipTrans
19.67
3.60
4.56
3.17
PhysGraph
29.34
2.95
4.30
2.51
TABLE II : Quantitative Results of Cross Embodiments. Pink indicates the best performance.
Dexterous grasp generation across robot hands is challenging because hands differ in kinematic topology, actuation dimensions, and native command spaces. We introduce GraspGraphNet, a topology-aware grasp generation framework that represents each hand as a URDF-derived kinematic graph and directly generates executable palm poses and joint configurations. GraspGraphNet combines hierarchical object surface encoding, differentiable forward kinematics, and dynamic world-edge message passing to model evolving robot-object interactions. It applies conditional flow matching directly in executable palm-pose and joint-state space, avoiding post-processing optimization, inverse kinematics, and retargeting. Using a shared model trained on Barrett Hand, Allegro Hand, and Shadow Hand, GraspGraphNet achieves an average success rate of 83.48% with 40ms inference time per grasp on a 40-object benchmark. Without retraining, the same model achieves 72.70% success on controlled finger-removal variants, demonstrating robustness to hand-topology variations. These results suggest that graph-structured hand representations can effectively support dexterous grasp generation across robot hands with different kinematic structures. Project: https://lysees.github.io/graspgraphnet-page
Yeonseo Lee, Taeyeop Lee, Hyosup Shin +2
Korea Advanced Institute of Science and Technology (KAIST), Daejeon, South Korea.
Human hand-object demonstrations provide dense reference motions for training dexterous manipulation reinforcement learning (RL) policies through reference tracking. However, to use such demonstrations for RL policy learning, retargeting must preserve hand pose and task-relevant hand-object contact structure. Otherwise, contact and feasibility artifacts can degrade downstream RL policy performance. We introduce TopoRetarget, an interaction-preserving retargeting framework that uses a single set of parameters across diverse retargeting conditions while maintaining task-relevant hand-object interaction and adapting human demonstrations to dexterous robot hands. The method constructs a sparse interaction graph over hand and object keypoints and optimizes distance-weighted Laplacian deformation with directional consistency, kinematic constraints, and penetration handling. Evaluations show that the generated references improve both interaction fidelity and policy learning: TopoRetarget achieves the best contact precision and alignment over all baselines on the ContactPose Dataset, improves Pen-Spin training success by 40.6 percentage points over the existing baseline methods, and enables zero-shot transfer to Wuji Hand hardware on cube reorientation and pen spinning.
Synthesizing physically grounded dexterous hand-object interactions is important for VR/AR, gaming, and robotics, yet remains challenging because subtle contact dynamics, diverse object geometries, and target trajectories must be jointly respected while maintaining natural grasps. Existing methods either track mocap trajectories with limited transfer to novel objects, or rely on hand-only generative priors that fail to capture coupled hand-object motion, sacrificing contact fidelity and sample efficiency. To tackle these challenges, we introduce PiHand, a Physically grounded Interaction-aware Hand motion synthesis framework. PiHand first trains expert policies that track mocap demonstrations in simulation, distills them into a unified hand-object interaction prior using a goal-conditioned VAE, and finally solve diverse dexterous manipulation tasks through bounded residual exploration in its latent space. With only a single reward and no complex curriculum training, this yields general tabletop manipulation policies that generalize to unseen objects and trajectories on GRAB and DexYCB. The same prior transfers to object catching and bimanual handover, two tasks that are rarely seen in its training data. Across all tasks, PiHand consistently outperforms baselines in all metrics. We will release the processed data and model checkpoints.
Niklas Berndt, Xianghui Xie, Enes Duran +2
University of Tübingen, Germany · Tübingen AI Center, Germany · Max Planck Institute for Informatics, Saarland Informatics Campus, Germany +1