Musculoskeletal robots provide superior advantages in flexibility and dexterity, positioning them as a promising frontier towards embodied intelligence. However, current research is largely confined to relative simple tasks, restricting the exploration of their full potential in multi-segment coordination. Furthermore, efficient learning remains a challenge, primarily due to the high-dimensional action space and inherent overactuated structures. To address these challenges, we propose Diff-Muscle, a musculoskeletal robot control algorithm that leverages differential flatness to reformulate policy learning from the redundant muscle-activation space into a significantly lower-dimensional joint space. Furthermore, we utilize the highly dynamic robotic table tennis task to evaluate our algorithm. Specifically, we propose a hierarchical reinforcement learning framework that integrates a Kinematics-based Muscle Actuation Controller (K-MAC) with high-level trajectory planning, enabling a musculoskeletal robot to perform dexterous and precise rallies. Experimental results demonstrate that Diff-Muscle significantly outperforms state-of-the-art baselines in success rates while maintaining minimal muscle activation. Notably, the proposed framework successfully enables the musculoskeletal robots to achieve continuous rallies in a challenging dual-robot setting.
Figures & tables
Fig. 1: Diff-Muscle. (a) Utilizing the inherent differential flatness, Diff-Muscle reformulates the learning problem in musculoskeletal systems from high-dimensional muscle space to low-dimensional joint space. (b) We evaluate Diff-Muscle in the highly dynamic robotic table tennis task to demonstrate its capability and efficiency in learning multi-segment coordination and rapid reactive behaviors.
Fig. 2: Hierarchical reinforcement learning framework. (a) In this framework, the policy takes as input current observation and high-level commands, and generates the target joint positions, which are subsequently translated into muscle signals through the Kinematics-based Muscle Actuation Controller (K-MAC). (b) We extend the framework and achieve successful consecutive dual-robot rallies. (c) Given the ball state, the planner predicts the desired racket position, orientation, and velocity at the predefined plane. (d) The Kinematics-based Muscle Actuation Controller integrates forward kinematics and a PD controller to translate target joint positions into muscle control signals.
Fig. 3: Musculoskeletal Model and Environment. (a) The musculoskeletal model consists of MyoArm (63 muscles, 27 DoF), MyoTorso (210 muscles, 3 DoF), and pelvis (2 actuators, 2DoF). (b) Single-robot environment to execute a single-serve ball return. (c) dual-robot rally environment.
Fig. 4: Learning curve. Diff-Muscle exhibits the most rapid improvement in reward and achieves the highest overall performance among all evaluated methods. Mean ± Std is computed across 5 random seeds.
Method
Hit(%)
Success(%)
Effort
Muscle-PPO
76.4
43.0
0.035
DynSyn [ 8 ]
84.5
62.6
0.0310
Muscle-PD [ 22 ]
90.5
72.9
0.0365
Diff-Muscle (ours)
95.1
80.3
0.0278
TABLE I: Comparison with existing methods on the single-robot environment.
Fig. 5: Ablation results. Diff-Muscle exhibits the most rapid improvement in reward and achieves the highest overall performance among all evaluated methods. Mean ± Std is computed across 5 random seeds.
Fig. 6: Distribution visualization. This scatter plot visualizes the y-z coordinates of 150 serves and corresponding returns.
Fig. 7: Visualization of dual-robot rally. In a round of continuous rally, the robot actively moves to the target hitting position and hits the ball at a certain speed, returning it to a position where the opponent can receive it.
Institute for AI Industry Research (AIR), Tsinghua University · The University of Hong Kong (HKU) · Institute for Interdisciplinary Information Sciences (IIIS), Tsinghua University +2