Perception and decision-making in high-speed dynamic scenarios remain challenging for current robots. In contrast, humans and animals can rapidly perceive and make decisions in such environments. Taking table tennis as a typical example, conventional frame-based vision sensors suffer from motion blur, high latency and data redundancy, which can hardly meet real-time, accurate perception requirements. Inspired by the human visual system, event-based perception methods address these limitations through asynchronous sensing, high temporal resolution, and inherently sparse data representations. However, current event-based methods are still restricted to simplified, unrealistic ball-only scenarios. Meanwhile, existing decision-making approaches typically require thousands of interactions with the environment to converge, resulting in significant computational costs. In this work, we present a biologically inspired approach for high-speed table tennis robots, combining event-based perception with sample-efficient learning. On the perception side, we propose an event-based ball detection method that leverages motion cues and geometric consistency, operating directly on asynchronous event streams without frame reconstruction, to achieve robust and efficient detection in real-world rallies. On the decision-making side, we introduce a human-inspired, sample-efficient training strategy that first trains policies in low-speed scenarios, progressively acquiring skills from basic to advanced, and then adapts them to high-speed scenarios, guided by a case-dependent temporally adaptive reward and a reward-threshold mechanism. With the same training episodes, our method improves return-to-target accuracy by 35.8%. These results demonstrate the effectiveness of biologically inspired perception and decision-making for high-speed robotic systems.
Figures & tables
Fig. 1: The mechanism of table tennis in humans and robots. Humans perceive the position of the ball through the retina, transmitting the information as spikes via the M pathway to the dorsal visual stream to support motion perception. During the process of learning table tennis skills, humans typically learn table tennis skills in a progression from easy to hard. Inspired by this, robots can train in a brain-inspired manner. They use event cameras to achieve perception by processing asynchronous event streams. In terms of decision learning, robots first train in low-speed scenarios, gradually acquiring basic skills before progressing to more advanced capabilities to achieve policy convergence, and then continue training in high-speed scenarios.
Fig. 2: Comparison between conventional cameras and event cameras. When imaging a rotating disk with black dots, conventional cameras capture image frames at a fixed frame rate. In contrast, event cameras record brightness changes in the form of an asynchronous event stream and generate events only when the intensity change exceeds a predefined threshold [ 1 ] .
Fig. 3: Overview framework. The simulation environment and the algorithm are connected and exchange data via the ZMQ communication protocol.
Fig. 4: Overall pipeline of the perception and decision-making modules.
Fig. 5: Ball detection pipeline illustration. (A) Raw event image. (B) Event Stream Denoising. (C) Density-Based Spatial Clustering, with green indicating negative polarity and red indicating positive polarity. (D) Polarity-based Event Filtering. (E) Geometric Verification. (F) Ball Localization and Adaptive ROI Update.
xmin=u−roi_expand,xmax=u+roi_expand
Algorithm 1 Event-based Ball Detection Method
Fig. 6: Stereo vision illustration. The system consists of two fixed event cameras whose optical center positions in the world coordinate frame are known and denoted as Oleft and Oright , respectively.
Fig. 7: Reward design illustration. The figure shows the reward assignments corresponding to four different ball landing cases. The boundary indicated in the figure is an artificially defined valid region; once the ball lands outside this boundary, the simulation is reset and a new serve is initiated.
Fig. 8: Dataset illustration. (A) Imaging examples of the left and right DVS cameras in the simulated dataset. (B–H) Examples from the real-world dataset, where B , C , and D correspond to scenes containing only the table tennis ball, and the remaining scenes include interference from moving humans. (I) Examples from the RGB dataset, illustrating that RGB-based detection is more susceptible to interference caused by motion blur and static circular objects in the background.
Fig. 9: High-fidelity table-tennis robot in a simulation environment.
Fig. 10: Comparison of detection performance between RGB and DVS across different scenes. Datasets marked with an asterisk (*) contain multiple moving objects, whereas datasets without an asterisk include only the ball.
Fig. 11: Comparison of different event-based methods on real-world datasets. As F-E Fusion and EBPP operate only in scenes where the ball is present, the comparison is limited to the first three ball-only scenes.
Dataset
Precision
Recall
Error (px)
real4*
1.000
0.928
2.61
real5*
1.000
0.985
2.77
real6*
1.000
0.902
2.59
real7*
1.000
0.794
2.89
TABLE I: Performance of our method in real-world scenes with multiple moving objects.
Fig. 12: Comparison against Baseline Algorithms in the High-speed Setting (5 m/s).
Fig. 13: Performance comparison against baseline algorithms over 1000 training iterations at 4 m/s.
Setting
Episodes
Simple
Case-based Reward
Ours
Rand pos., 3 m/s
350
46%
70%
78%
400
86%
80%
98%
600
92%
98%
100%
Rand pos., 4 m/s
600
46%
60%
72%
800
60%
80%
92%
900
60%
82%
98%
TABLE II: Successful return rates (%) under different reward designs.
Fig. 14: Learning Curves of Hitting Policies Trained with different reward designs over 1000 Iterations.
Fig. 15: Return-to-table rate comparison under the high-speed scenario (5 m/s) between policies trained from scratch and those adapted from a pre-trained model.
Fig. 16: Comparison of target-point return errors for three models adapted from pre-training under a high-speed (5 m/s) scenario.
Robotic table tennis has emerged as a compelling benchmark for real-time robotic perception due to its fast ball dynamics and stringent timing requirements. Accurate, high-frequency, and low-latency ball state estimation is critical for reliable trajectory prediction and timely control. Traditional frame-based cameras face an inherent trade-off: low frame rates leave temporal blind spots that miss fast-moving objects and high frame rates raise data and computational cost. Event cameras instead offer microsecond temporal resolution and, under sufficient illumination, remain largely free of motion blur even at high ball speeds. However, the community lacks large-scale datasets to develop and benchmark event-based perception in realistic sports scenarios. We address this gap by introducing the first large-scale event-camera dataset for table tennis, comprising over 1000 rallies from a diverse group of players ranging from amateurs to elite-level athletes. Each recording captures the event stream alongside 14 synchronized high-speed frame-based cameras at 200 FPS, which we use to produce 1 kHz pseudo ground-truth labels for ball position, velocity, and spin. Building on this dataset, we train a convolutional neural network robust to background player motion that jointly estimates the ball's position and velocity in the image-plane from events. Treating the predicted velocity as an additional measurement in the Kalman filter reduces bounce-point prediction error by 36% relative to a position-only baseline. Finally, we close the perception-action loop by integrating the event-based system with a Stäubli robotic arm, enabling the first real-time human-robot table tennis rallies driven by event-based perception.
Existing humanoid table tennis systems remain limited by their reliance on external sensing and their inability to achieve agile whole-body coordination for precise task execution. These limitations stem from two core challenges: achieving low-latency and robust onboard egocentric perception under fast robot motion, and obtaining sufficiently diverse task-aligned strike motions for learning precise yet natural whole-body behaviors. In this work, we present \methodname, a modular system for agile humanoid table tennis that unifies scalable whole-body skill learning with onboard egocentric perception, eliminating the need for external cameras during deployment. Our work advances prior humanoid table-tennis systems in three key aspects. First, we achieve agile and precise ball interaction with tightly coordinated whole-body control, rather than relying on decoupled upper- and lower-body behaviors. This enables the system to exhibit diverse strike motions, including explosive whole-body smashes and low crouching shots. Second, by augmenting and diversifying strike motions with a generative model, our framework benefits from scalable motion priors and produces natural, robust striking behaviors across a wide workspace. Third, to the best of our knowledge, we demonstrate the first humanoid table-tennis system capable of consecutive strikes using onboard sensing alone, despite the challenges of low-latency perception, ego-motion-induced instability, and limited field of view. Extensive real-world experiments demonstrate stable and precise ball exchanges under high-speed conditions, validating scalable, perception-driven whole-body skill learning for dynamic humanoid interaction tasks.
This paper addresses the challenge of automatic, frame-accurate event detection in table tennis videos. Current methods for estimating 3d ball trajectories and ball spin typically require that key events, such as ball-racket contacts, have already been identified in advance. This requirement makes it difficult to apply these methods to longer, unedited video recordings. To overcome this limitation, we propose EventNet, a two-stage pipeline to detect key events: (1) 2d keypoints are extracted of the upper-body poses for both players, table corners and ball center. A small keypoint transformer combines them into a compact representation that is robust to changes in viewpoint, lighting, and background clutter. (2) The temporal sequences of these frame-based representations are processed by a transformer encoder that predicts two time-to-event values for each frame, indicating how close the current frame is to the next and previous ball-racket contact. One novelty is a new, temporal cosine-like target signal. Furthermore, we introduce viewpoint augmentation via 3D reprojection and frame-rate augmentation to improve robustness and generalization. Our extensive ablation study gives deeper insights into the importance of various architectural and training aspects. Experimental results show that the proposed approach achieves an F1 score of 91.16% and a mean frame deviation between ground truth and predicted frame of 0.42 on the Latte-MV dataset and 73.08% / 1.16 on the challenging TTHQ dataset. Overall, our work demonstrates that 2d keypoint-based temporal modeling with our EventNet architecture is a promising and practical approach for automatic event detection in table tennis videos.
Rainer Lienhart, Daniel Kienzle, Shin'ichi Satoh +1
University of Augsburg Augsburg, Germany · National Institute of Informatics University of Tokyo Tokyo, Japan