Humanoid robots have recently demonstrated promising capabilities in real-world ball sports. However, achieving professional motion styles while maintaining strong task performance remains challenging. In this work, we propose AdaPT, an Adaptive Motion Planning and Tracking framework that learns professional tennis serving and rally styles directly from broadcast videos. This hierarchical design is motivated by the key insight that the planner generates stylistic kinematic motions, while the tracker executes them with minimal interference with planning. Despite its effectiveness in simulation, a substantial sim-to-real gap emerges: tracking performance inevitably degrades on real robots, and this degradation is partially overlooked by autoregressive planning and further compounded by noisy perception. To address these issues, our adaptation mechanism improves tracking robustness by learning to track randomized execution speeds, while conditioning the planner on a learned motion-speed adapter to mitigate compounding errors. Real-world experiments on the Unitree G1 demonstrate the effectiveness of our adaptation mechanism in bridging the sim-to-real gap. We further deploy AdaPT policies on the full-size Dobot Atom humanoid robot (1.7m) and demonstrate in-the-wild serving without motion capture. Beyond these results, our real-world experiments reveal both algorithmic and engineering insights for future humanoid ball-sports systems. Videos and code are available on our project website.
Figures & tables
Figure 1: t-SNE visualization of motions and tracker rollouts.
Player
Total Time
# of Clips
Stroke Ratio
Spin Ratio
Forehand
Backhand
Flat & Top
Slice
Rafael Nadal
267min
5664
54%
46%
89%
11%
Roger Federer
179min
5264
48%
52%
84%
16%
Novak Djokovic
189min
5288
49%
51%
93%
7%
Mr. Black (MoCap)
219min
2987
57%
43%
76%
24%
Table 1: Statistics of motion data from broadcast videos of three players.
Figure 2: AdaPT learns stylized humanoid tennis skills through the pipeline: (a) broadcast videos are reconstructed, retargeted, and physically corrected to obtain tennis motions; (b) speed-adaptive policies learn to track serve and rally motions; (c) high-level planners select or generate style-consistent motions for serve and rally; (d) our robot localization or ball detection modules enable sim-to-real deployment, especially for serving without MoCap.
Figure 3: Serve hardware.
Stroke
Rafael Nadal
Roger Federer
Novak Djokovic
Succ. ↑
Ebo↓
EFID↓
Epow↓
Eacc↓
Succ. ↑
Ebo↓
EFID↓
Epow↓
Eacc↓
Succ. ↑
Ebo↓
EFID↓
Epow↓
Eacc↓
RL-Scratch [ 34 ]
97.4
1.6 ±0.6
297.3 ±16.3
92.7 ±20.0
123.1 ±8.9
98.8
2.5 ±1.8
299.0 ±29.8
130.8 ±27.9
145.0 ±8.3
98.8
2.5 ±1.8
307.7 ±18.4
130.8 ±27.9
145.0 ±8.3
AMP [ 27 ]
98.1
2.7 ±0.9
172.1 ±8.0
118.1 ±39.3
126.9 ±12.5
65.1
2.1 ±1.2
116.0 ±32.3
112.7 ±62.4
122.5 ±16.7
21.6
2.2 ±1.4
66.9 ±39.7
185.6 ±92.2
155.2 ±57.7
PULSE [ 21 ]
86.2
2.5 ±1.4
29.7 ±3.5
92.2 ±31.1
107.1 ±11.4
92.2
2.5 ±1.1
17.9 ±1.5
94.1 ±26.7
101.0 ±9.5
86.1
3.4 ±1.7
32.9 ±6.3
78.6 ±29.6
92.8 ±7.5
NCP [ 50 ]
85.6
4.8 ±2.3
9.2 ±3.5
33.0 ±13.2
55.3 ±5.6
89.1
3.6 ±1.5
6.8 ±4.7
24.2 ±15.3
48.6 ±5.2
69.0
4.9 ±2.3
24.0 ±2.8
22.8 ±6.2
56.6 ±4.6
Vid2Player3D [ 47 ]
82.6
4.8 ±2.0
5.1 ±5.2
15.1 ±5.6
41.1 ±5.6
81.5
2.9 ±1.2
13.4 ±3.2
18.5 ±8.3
41.9 ±7.9
83.2
3.3 ±1.9
6.3 ±3.2
24.0 ±9.7
40.6 ±7.3
Table 2: Simulation results. Evaluation of all candidate methods across different metrics for rally and serve tasks by learning the motion of three master-level players. ↑ denotes higher is better, while ↓ denotes the opposite. Overall, AdaPT consistently achieves strong motion fidelity while maintaining high task performance, demonstrating an effective balance between stylistic imitation and task performance across both rallying and serving.
Figure 4: Experimental setup for rally and serve task with motion capture.
Table 3: Real-robot evaluation of rally skills. The results demonstrate that both the Adaptive Tracker and Adaptive Planner contribute to robust real-world rally performance over all three players. The performance gap between simulation and real-world deployment is further analyzed in Sec. 4.5 .
Figure 5: Analysis of failure of RL-Scratch serving.
Player
Method
Succ. ↑
EBo3m↓
EBo6m↓
EBo9m↓
Edoftrack↓
Eacc↓
Rafael Nadal
DeepMimic
80.0
0.7 ±0.3
3.5 ±0.1
6.4 ±0.6
0.18 ±0.01
9.9 ±2.9
AdaPT (ours)
66.7
1.5 ±0.5
1.5 ±0.4
5.3 ±1.7
0.17 ±0.01
12.3 ±0.5
w/o-Planner
53.3
1.8 ±1.1
2.8 ±0.9
4.6 ±1.7
0.18 ±0.01
13.5 ±0.2
w/o-TossRew
60.0
1.3 ±0.4
2.5 ±1.4
–
0.18 ±0.01
11.7 ±0.7
Roger Federer
DeepMimic
53.3
1.6 ±1.0
4.2 ±1.5
7.8 ±0.0
0.36 ±0.03
28.4 ±0.9
AdaPT (ours)
73.3
1.4 ±1.7
2.8 ±1.2
6.4 ±1.4
0.17 ±0.01
22.0 ±1.0
Table 4: Real-robot evaluation of serve skills. The results demonstrate that both the Adaptive Tracker and Adaptive Planner contribute to robust real-world rally performance over all three players. Performance drops between simulation and real-world deployment are discussed in Sec. 4.5 .
Figure 6: Planner analysis. The planners successfully learn to adjust motion speed to (a, b) minimize tracking error and (c) adapt to varied ball-tossing height, (d) exhibiting better style than DeepMimic.
Figure 7: Sensitivity of planner-tracker methods to the horizon of the future ball trajectory.
Figure 8: Tossing guidance as a reward is crucial to imitate tossing style and control landing.
Figure 9: Impact of trajectory prediction errors on rally success. (a) Future ball trajectory prediction errors. (b,c) Correlation between pre- and post-bounce prediction errors and rally success, respectively.
Figure 14Figure 15
Figure 12: Large trajectory estimation noise with camera influence on real performance. But YOLO-based perception is sufficient for serve.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Term
Weight
Description
Racket–ball
5.0
Racket–ball contact and MVAE phase alignment; latched after a valid hit.
Opponent landing
100.0
Predicted opponent-court bounce; must clear the net ( ≥1.1m ).
Forward velocity
50.0
Deep forward ball speed after hit.
Spin selection
−1.0
MVAE spin-skill prediction should be consistent with the desired one.
Self-collision
−10.0
Robot self-collisions.
Facing backward
−20.0
Base facing away from opponent court.
Appendix
Table 7: Reward functions for rally training.
Term
Weight
Description
Wrist tracking
1.0
Tracks striking wrist DoF positions.
Tossing
5.0
Enforces accurate ball release position and velocity.
Palm-upward
1.5
Encourages an upward palm orientation before the toss.
Hitting
3.0
Binary reward for racket-ball strike.
Bounce
6.0
Penalizes error between predicted and target landing positions.
Net-crossing
1.0
Encourages the ball to pass over the net with height exceeding 1.1m .
Appendix
Table 8: Reward functions for serving training.
Domain Randomization
Sampling Distribution
Physical parameters
Foot tangential friction μ
U[0.3,1.2]
Encoder bias Δq
U[−0.01,0.01]
Base COM offset (Δx,Δy,Δz)
U[−0.08,0.08]
PD gain scale (kp,kd)
U[0.95,1.05]
Torque limit scale
U[0.9,1.0]
Appendix
Table 9: Domain randomization sampling distributions for rally training.
Domain Randomization
Sampling Distribution
Physical parameters
Foot tangential friction μ
U[0.3,1.2]
Encoder bias Δq
U[−0.01,0.01]
Base COM offset (Δx,Δy,Δz)
U[−0.08,0.08]
PD gain scale (kp,kd)
U[0.95,1.05]
Torque limit scale
U[0.9,1.0]
Appendix
Table 10: Domain randomization sampling distributions for serve training.
Figure 21Figure 22
Figure 13: Amateur serving motion using the Unitree Dex-3 end-effector.
Human athletes demonstrate versatile and highly-dynamic tennis skills to successfully conduct competitive rallies with a high-speed tennis ball. However, reproducing such behaviors on humanoid robots is difficult, partially due to the lack of perfect humanoid action data or human kinematic motion data in tennis scenarios as reference. In this work, we propose LATENT, a system that Learns Athletic humanoid TEnnis skills from imperfect human motioN daTa. The imperfect human motion data consist only of motion fragments that capture the primitive skills used when playing tennis rather than precise and complete human-tennis motion sequences from real-world tennis matches, thereby significantly reducing the difficulty of data collection. Our key insight is that, despite being imperfect, such quasi-realistic data still provide priors about human primitive skills in tennis scenarios. With further correction and composition, we learn a humanoid policy that can consistently strike incoming balls under a wide range of conditions and return them to target locations, while preserving natural motion styles. We also propose a series of designs for robust sim-to-real transfer and deploy our policy on the Unitree G1 humanoid robot. Our method achieves surprising results in the real world and can stably sustain multi-shot rallies with human players. Project page: https://zzk273.github.io/LATENT/
Zhikai Zhang, Haofei Lu, Yunrui Lian +12
Tsinghua University · Galbot Inc. · Shanghai AI Laboratory +2
Humanoid tennis requires reactive whole-body coordination under rapidly changing ball dynamics, yet collecting task-specific motion-capture demonstrations that cover diverse approach states, contact locations, and timing conditions is costly. Inspired by how humans acquire new skills by combining reusable motor experience with task-specific demonstrations, we present ATHLETE, a framework for learning reactive humanoid tennis from generic locomotion data and ordinary tennis videos. ATHLETE combines LAFAN1 locomotion with video-derived striking motions and uses Trajectory-Guided Motion Matching (TGMM) to construct diverse task-conditioned whole-body references. These references are transformed into a reactive policy through physics-based teacher--student learning with online DAgger distillation, followed by PPO fine-tuning for the tennis task. The resulting policy responds directly to robot and ball observations without requiring reference-motion inputs at deployment. On a real humanoid robot, ATHLETE achieves forehand and backhand hit rates of 90.00% and 86.67%, respectively, and enables multi-shot rallies with human players. These results demonstrate that ATHLETE can learn dynamic humanoid tennis skills from reusable locomotion data and video demonstrations. Result videos and code can be found at https://anonymous.4open.science/w/ATHLETE2027/.
Zhihao Tian, Zifan Wang, Enzheng Zhu +4
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China.
High-speed racket sports provide a demanding testbed for humanoid robots, requiring time-critical decisions, precise striking, and dynamic whole-body coordination. In badminton, fast-changing shuttle trajectories require timely contact decisions, while successful returns demand precise racket pose and velocity within a brief contact window and across a broad three-dimensional striking workspace. Human motion data provide valuable priors for such athletic skills, but usable badminton references are limited and imperfect. Direct tracking provides insufficient executable variation for diverse shuttle conditions, while purely task-driven optimization may produce unnatural motion. To address these challenges, we present a three-stage hierarchical reinforcement learning framework for dynamic humanoid badminton. First, task-randomized motion augmentation expands sparse annotated hitting events into executable target-conditioned stroke variations, forming a continuous latent skill space. Second, a high-level planner outputs continuous latent skill codes to compose these skills online according to the observed shuttle state. Third, a context-conditioned adversarial regularizer encourages more natural planner-level skill usage while preserving return performance. When deployed on a real humanoid robot, our system achieves sustained multi-skill rallies with human players, including forehand, backhand, and highly dynamic jump returns. This is the first real-world humanoid racket-sport system to demonstrate multi-skill human--robot rallies including highly dynamic jump returns.
Jingzhi Cui, Zhexiong Wang, Bangjie Xu +9
Tsinghua University · Hong Kong Embodied AI Lab · Zhejiang University +3