This paper presents a robust one-shot badminton shuttlecock detection framework for non-stationary robots. To address the lack of egocentric shuttlecock detection datasets, we introduce a dataset of 20,510 semi-automatically annotated frames captured across 11 distinct backgrounds in diverse indoor and outdoor environments, and categorize each frame into one of three difficulty levels. For labeling, we present a novel semi-automatic annotation pipeline, that enables efficient labeling from stationary camera footage. We propose a metric suited to our downstream use case and fine-tune a YOLOv8 network optimized for real-time shuttlecock detection, achieving an F1-score of 0.86 under our metric in test environments similar to training, and 0.70 in entirely unseen environments. Our analysis reveals that detection performance is critically dependent on shuttlecock size and background texture complexity. Qualitative experiments confirm their applicability to robots with moving cameras. Unlike prior work with stationary camera setups, our detector is specifically designed for the egocentric, dynamic viewpoints of mobile robots, providing a foundational building block for downstream tasks, including tracking, trajectory estimation, and system (re)-initialization.
Figures & tables
Fig. 1: Example detections of the fine-tuned model. Our method reliably detects shuttlecocks, even under challenging conditions.
Fig. 2: Dataset Overview and Difficulty Distribution: Representative frames from 11 backgrounds across 5 locations. Each frame shows the per-background difficulty distribution (top-right inset) with cropped shuttlecock examples (lower-right inset), both color-coded by difficulty: green ( easy ), orange ( medium ), red ( hard ). Sample sizes range from 642 to 3,407 frames per background. Bottom-right: Overall distribution across all 20,510 frames.
Fig. 3: Automated Shuttlecock Labeling Pipeline: Left: gmm -based background segmentation identifies foreground regions (red). Center: YOLOv8-seg detects and segments the opponent player (blue), whose region is subsequently excluded from shuttlecock candidates. Right: Final detection result after applying morphological operations, person removal, and spatial constraints (lower region shown in green represents the excluded zone), with the detected shuttlecock marked by a green bounding box.
Fig. 4: Cross-Validation Results Distribution: Performance metrics across individual backgrounds (left/green) and locations (right/blue). Each circle represents model performance when trained on all subsets except one and evaluated on the held-out subset. Background-based validation indicates how well the model generalizes to environments similar to those in the training set, while location-based validation indicates generalization to previously unseen environments. Diamonds indicate mean performance across all folds.
Metric
Overall
Easy
Medium
Hard
F1
0.703 / 0.864
0.788 / 0.920
0.414 / 0.688
0.238 / 0.576
Precision
0.970 / 0.954
0.977 / 0.961
0.934 / 0.943
0.772 / 0.810
Recall
0.552 / 0.789
0.661 / 0.882
0.266 / 0.542
0.140 / 0.447
TABLE I: Location-wise vs. Background-wise cross-validation by difficulty. Values are Loc / Bg .
Fig. 5: Distribution of Bounding Box Sizes and Corresponding Model Performance: The histogram shows the distribution of correct (green, n=15,561) and incorrect (red, n=4,915) predictions across different bounding box side lengths measured in input pixels and defined as the geometric mean w⋅h . The blue line represents model precision, the violet line represents recall (both calculated for bins with at least 50 samples), plotted on the secondary y-axis on the right.
Fig. 6: Predictions with moving camera: Example sequence from LEE_moving_1 showing correct shuttlecock detection across evenly sampled frames from a sequence of 60 frames.
Material
Description
Link
Dataset
Dataset in Ultralytics YOLO format, with separate subsets for each recording location and difficulty level, provided over Google Suite.
High-speed racket sports provide a demanding testbed for humanoid robots, requiring time-critical decisions, precise striking, and dynamic whole-body coordination. In badminton, fast-changing shuttle trajectories require timely contact decisions, while successful returns demand precise racket pose and velocity within a brief contact window and across a broad three-dimensional striking workspace. Human motion data provide valuable priors for such athletic skills, but usable badminton references are limited and imperfect. Direct tracking provides insufficient executable variation for diverse shuttle conditions, while purely task-driven optimization may produce unnatural motion. To address these challenges, we present a three-stage hierarchical reinforcement learning framework for dynamic humanoid badminton. First, task-randomized motion augmentation expands sparse annotated hitting events into executable target-conditioned stroke variations, forming a continuous latent skill space. Second, a high-level planner outputs continuous latent skill codes to compose these skills online according to the observed shuttle state. Third, a context-conditioned adversarial regularizer encourages more natural planner-level skill usage while preserving return performance. When deployed on a real humanoid robot, our system achieves sustained multi-skill rallies with human players, including forehand, backhand, and highly dynamic jump returns. This is the first real-world humanoid racket-sport system to demonstrate multi-skill human--robot rallies including highly dynamic jump returns.
Jingzhi Cui, Zhexiong Wang, Bangjie Xu +9
Tsinghua University · Hong Kong Embodied AI Lab · Zhejiang University +3
Humanoid robots have demonstrated strong capabilities for interacting with static scenes across locomotion and manipulation, yet dynamic real-world interactions remain challenging. As a step toward fast-moving object interactions, we present an RL training pipeline that yields a unified whole-body controller for humanoid badminton, coordinating footwork and striking without motion priors or expert demonstrations. In badminton, locomotion and striking are tightly coupled, making the final hitting objective difficult to optimize directly due to sparse rewards and conflicting gradients. We address this with an annealed curriculum that first stabilizes learning through auxiliary locomotion objectives, then progressively removes them to focus optimization on the final hitting objective. For deployment, we use an Extended Kalman Filter (EKF) to estimate and predict shuttlecock trajectories for target striking, and also develop a prediction-free variant that removes the EKF and explicit prediction. We validate the framework in simulation and on hardware. In simulation, two robots sustain a rally of 21 consecutive hits. In real-world tests with both machine-fed shuttles and human-robot rallies, the robot achieves outgoing shuttle speeds up to 19.1 m/s. Moreover, the prediction-free variant attains comparable performance to the EKF-based policy. Overall, our approach enables dynamic yet precise goal striking in humanoid badminton and suggests a path toward more dynamics-critical whole-body interaction tasks.
Robotic table tennis has emerged as a compelling benchmark for real-time robotic perception due to its fast ball dynamics and stringent timing requirements. Accurate, high-frequency, and low-latency ball state estimation is critical for reliable trajectory prediction and timely control. Traditional frame-based cameras face an inherent trade-off: low frame rates leave temporal blind spots that miss fast-moving objects and high frame rates raise data and computational cost. Event cameras instead offer microsecond temporal resolution and, under sufficient illumination, remain largely free of motion blur even at high ball speeds. However, the community lacks large-scale datasets to develop and benchmark event-based perception in realistic sports scenarios. We address this gap by introducing the first large-scale event-camera dataset for table tennis, comprising over 1000 rallies from a diverse group of players ranging from amateurs to elite-level athletes. Each recording captures the event stream alongside 14 synchronized high-speed frame-based cameras at 200 FPS, which we use to produce 1 kHz pseudo ground-truth labels for ball position, velocity, and spin. Building on this dataset, we train a convolutional neural network robust to background player motion that jointly estimates the ball's position and velocity in the image-plane from events. Treating the predicted velocity as an additional measurement in the Kalman filter reduces bounce-point prediction error by 36% relative to a position-only baseline. Finally, we close the perception-action loop by integrating the event-based system with a Stäubli robotic arm, enabling the first real-time human-robot table tennis rallies driven by event-based perception.