Perceiving and Predicting Dynamic Sports: Spatiotemporal Intelligence in Vision Models for Egocentric Tennis Video
Authors: Tudor Braicu, Joseph DelPreto, Jared Boyer, Mónika Farsang, Maximilian Stölzle, Tim Niklas Seyde, Daniela Rus
Organizations: Massachusetts Institute of Technology (MIT), Cambridge, MA 02139 USA. · Cornell University, Ithaca, NY 14853 USA. · Technische Universität Wien (TU Wien), 1040 Vienna, Austria. · Liquid AI, Cambridge, MA 02142 USA.
Agile robots require spatiotemporal intelligence to interpret scenes, reason about interactions, and anticipate future events. Yet it remains unclear how well foundation vision models support these capabilities, particularly from a moving egocentric camera, which is a natural view for robots. To investigate this, we collect an annotated tennis dataset and train a suite of vision models on spatiotemporal tasks. Our dataset contains egocentric video synchronized with five allocentric videos, estimated global positions, labeled contact times, and annotated stroke types. We then evaluate seven prediction tasks from egocentric images: 2D ego player and 3D ball localization, future player and ball trajectories, forehand or backhand stroke anticipation, time to the next ego player ball contact, and 3D ball position at that contact. We compare image encoders, video encoders, and vision-language models with jointly trained prediction heads. The best performing models achieve mean current-frame localization errors of 0.14 m for the ego player and 0.54 m for the ball, future forecasting errors of 0.34 m for the ego player and 0.91 m for the ball over 1.1 s, and a mean contact-time prediction error of 0.14 s. Auxiliary depth information substantially improves 3D ball localization, suggesting that inferring metric depth remains a challenge. Longer image histories do not improve ball forecasting, indicating that representations exploit scene context rather than rely solely on inter-frame motion. Finally, we propose a conditional diffusion model to generate whole-body kinematic tennis motion from vision encoder predictions on egocentric video at real-time rates, adapting movements as predictions are refined. These results move towards a deeper understanding of how foundation vision models can support real-time dynamic robots using egocentric video.
Figures & tables
Fig. 1 : Visual models are trained on egocentric tennis videos to investigate spatiotemporal understanding, forecasting, and anticipation. The dataset will be publicly released, including synchronized allocentric videos and spatial and shot annotations.
Fig. 2 : The dataset includes 53,155 frames over 85 rallies. Spatial distributions of samples are shown for (a) the ball and (b) the ego player. The temporal distribution of samples relative to the ego player ball contact is shown in (c). Locations of the five static cameras are depicted in (a) and (b).
Model
3D Ball pos.
2D Player pos.
Shot type
Time to
Ball pos. at
Current (m) ↓
Future (m) ↓
Current (m) ↓
Future (m) ↓
(acc. %) ↑
contact (ms) ↓
contact (m) ↓
Image encoders
DINOv3
0.58±0.01
1.02±0.01
0.16±0.01
0.38±0.01
98.7±0.0
141.3±1.6
1.03±0.01
SigLIP 2
0.60±0.01
1.03±0.00
0.16±0.00
0.39±0.00
98.4±0.2
148.8±1.6
1.08±0.02
SAM 3
0.55±0.03
0.99±0.03
0.15±0.00
0.38±0.00
98.4±0.3
142.7±3.1
1.01±0.01
DA3
0.58±0.01
0.99±0.02
0.14±0.01
0.36±0.01
98.4±0.0
142.8±1.6
1.00±0.02
TABLE I: Prediction accuracy across probed visual representations. Task prediction heads are trained jointly on a shared encoder. “Current” predictions are at the final conditioning frame; “future” predictions are the mean over 12 subsequent frames with stride 0.1 s. Results reflect performance on the test split, using model checkpoints selected based on the validation split, across three seeds.
Fig. 3 : Prediction errors on approaching test-split shots versus time to contact: (a) current 3D ball position, (b) ball position at contact, and (c) time to contact. Lines show means over 0.1s intervals; shading shows ±1 SD over three seeds. (d) Ball 3D and player 2D future predictions over a time horizon of ∼ 1.10 s, across all models, averaged over three seeds.
Auxiliary Information Added
MLP † ↓
SigLIP 2 ↓
None (image only)
–
0.51±0.01
Ball
2.83±0.01
0.54±0.00
Player
4.28±0.03
0.51±0.01
Depth
2.39±0.01
0.21±0.01
Ball + player
2.66±0.03
0.53±0.01
Ball + depth
2.12±0.01
0.25±0.02
TABLE II: Current 3D ball position error with auxiliary inputs. Results reflect test-split errors over three seeds, using checkpoints selected based on the validation split. † Image-free baseline using only auxiliary inputs.
Frames
Span
GT MLP † ↓
SigLIP 2 ↓
V-JEPA 2 ↓
1
–
2.924±0.003
1.02±0.02
–
2
100 ms
0.297±0.001
1.02±0.05
1.22±0.01
2
300 ms
0.257±0.004
1.08±0.05
1.25±0.01
4
300 ms
0.238±0.003
1.08±0.02
1.18±0.02
TABLE III : Future 3D ball position error versus input history ( Frames : number of input frames; Span : history duration). Results reflect test-split errors over three seeds, using checkpoints selected based on the validation split. † Image-free baseline using only ground-truth 3D ball history.
Fig. 4 : LFM2.5-VL 450M results on approaching test-split shots until contact. (a) Predictions for a single frame horizon tend to improve as the ball nears the ego player. (b) Forecasted predictions tend to improve as the ball nears the ego player and degrade as the future horizon increases.
Fig. 5 : (a) LFM2.5-VL-450M forecasts ball and player positions and anticipates contact; crosshairs locate the recorded contact. (b) Recorded ego motion versus MDM rollouts conditioned on annotations (“rec. cond.”) or vision predictions (“vision cond.”). Trails fade toward the past; ball forecasts fade toward the future. t=0 denotes the recorded racket–ball contact.
Ball–racket proximity
Error relative to recorded motion
Approach
At hit (m) ↓
Min. (m) ↓
Center (m) ↓
Plane ( ∘ ) ↓
Velocity (m/s) ↓
Recorded motion
0.89 [-1pt] [ 0.76 , 1.02 ]
0.81 [-1pt] [ 0.69 , 0.92 ]
0.00
0.00
0.00
MDM (rec. cond.)
1.11 [-1pt] [ 1.04 , 1.19 ]
0.98 [-1pt] [ 0.93 , 1.04 ]
0.76 [-1pt] [ 0.68 , 0.84 ]
35.97 [-1pt] [ 33.10 , 38.90 ]
4.46 [-1pt] [ 3.91 , 5.07 ]
MDM (vision cond.)
1.28 [-1pt] [ 1.12 , 1.42 ]
1.17 [-1pt] [ 1.04 , 1.33 ]
0.95 [-1pt] [ 0.78 , 1.10 ]
41.31 [-1pt] [ 36.03 , 46.43 ]
5.38 [-1pt] [ 4.80 , 5.92 ]
TABLE IV : MDM rollouts on 16 test sequences: means over contacts with bootstrapped 95% confidence intervals in brackets. Racket errors are relative to recorded ego player motion; minimum distance uses a ±100ms window.
Egocentric video understanding studies how AI interprets first-person video to recognize actions, track interactions, reason about space and time, and assist users.
Touch is fundamental to dexterous manipulation, yet most egocentric human data increasingly used for robot learning lacks tactile information. Directly collecting large-scale tactile data is challenging due to sensor limitations, while human video data is abundant, contact-rich, and easily scalable. This motivates a natural question: can tactile signals be inferred purely from vision? To address this, we introduce EgoTac, a generalizable model that predicts rich tactile information directly from egocentric human videos. EgoTac is trained on a unified corpus of over 5.7M image-tactile pairs, covering both continuous force measurements and binary contacts. By learning from this diverse dataset, EgoTac captures nuanced touch dynamics across varied interactions. Experiments demonstrate strong performance: in-domain prediction achieves an average force error below 0.06N. On out-of-domain contact prediction benchmarks, EgoTac consistently outperforms the state-of-the-art contact estimator. It also captures the rise and fall patterns of real tactile data and enables zero-shot predictions on unconstrained real-world videos. Scaling analyses further reveal that both data diversity and volume improve performance steadily. Overall, EgoTac provides a scalable pathway to extract tactile priors from egocentric human videos, enabling broadly applicable tactile-aware robot learning. Related resources can be found at https://egotac.github.io/ .
Wenkang Zhang, Chengbo Yuan, Zicheng Zhang +2
Shanghai Qi Zhi Institute · Shanghai Jiao Tong University · Tsinghua University +1
Robotic table tennis has emerged as a compelling benchmark for real-time robotic perception due to its fast ball dynamics and stringent timing requirements. Accurate, high-frequency, and low-latency ball state estimation is critical for reliable trajectory prediction and timely control. Traditional frame-based cameras face an inherent trade-off: low frame rates leave temporal blind spots that miss fast-moving objects and high frame rates raise data and computational cost. Event cameras instead offer microsecond temporal resolution and, under sufficient illumination, remain largely free of motion blur even at high ball speeds. However, the community lacks large-scale datasets to develop and benchmark event-based perception in realistic sports scenarios. We address this gap by introducing the first large-scale event-camera dataset for table tennis, comprising over 1000 rallies from a diverse group of players ranging from amateurs to elite-level athletes. Each recording captures the event stream alongside 14 synchronized high-speed frame-based cameras at 200 FPS, which we use to produce 1 kHz pseudo ground-truth labels for ball position, velocity, and spin. Building on this dataset, we train a convolutional neural network robust to background player motion that jointly estimates the ball's position and velocity in the image-plane from events. Treating the predicted velocity as an additional measurement in the Kalman filter reduces bounce-point prediction error by 36% relative to a position-only baseline. Finally, we close the perception-action loop by integrating the event-based system with a Stäubli robotic arm, enabling the first real-time human-robot table tennis rallies driven by event-based perception.
Forecasting future 3D hand pose sequences from egocentric video is essential for understanding human intention and enabling embodied applications such as AR/VR assistance and human-robot interaction. However, this task remains a highly challenging problem because egocentric hand motion is driven by complex human intent, exhibits highly dexterous articulations, and is observed under drastic viewpoint shifts induced by ego-motion. In this work, we introduce EggHand, a foundation-model-based framework for egocentric hand pose forecasting that unifies multimodal semantic reasoning with dynamic motion modeling. Our approach couples an action decoder from a Vision-Language-Action (VLA) model, which captures the structured temporal dynamics of hand motion, with an egocentric video-text encoder that provides viewpoint-aware contextual information learned from large-scale first-person video. Together, these components overcome the brittleness of generic visual encoders under ego-motion and enable joint reasoning over motion, context, and high-level intent-without relying on body pose or external tracking. Experiments on the EgoExo4D dataset show that EggHand sets a new state of the art in forecasting accuracy, remains robust under severe ego-motion, and further enables controllable prediction via language-based task prompts. Project page: https://jyoun9.github.io/EggHand