Perceiving and Predicting Dynamic Sports: Spatiotemporal Intelligence in Vision Models for Egocentric Tennis Video
Authors: Tudor Braicu, Joseph DelPreto, Jared Boyer, Mónika Farsang, Maximilian Stölzle, Tim Niklas Seyde, Daniela Rus
Organizations: Massachusetts Institute of Technology (MIT), Cambridge, MA 02139 USA. · Cornell University, Ithaca, NY 14853 USA. · Technische Universität Wien (TU Wien), 1040 Vienna, Austria. · Liquid AI, Cambridge, MA 02142 USA.
Agile robots require spatiotemporal intelligence to interpret scenes, reason about interactions, and anticipate future events. Yet it remains unclear how well foundation vision models support these capabilities, particularly from a moving egocentric camera, which is a natural view for robots. To investigate this, we collect an annotated tennis dataset and train a suite of vision models on spatiotemporal tasks. Our dataset contains egocentric video synchronized with five allocentric videos, estimated global positions, labeled contact times, and annotated stroke types. We then evaluate seven prediction tasks from egocentric images: 2D ego player and 3D ball localization, future player and ball trajectories, forehand or backhand stroke anticipation, time to the next ego player ball contact, and 3D ball position at that contact. We compare image encoders, video encoders, and vision-language models with jointly trained prediction heads. The best performing models achieve mean current-frame localization errors of 0.14 m for the ego player and 0.54 m for the ball, future forecasting errors of 0.34 m for the ego player and 0.91 m for the ball over 1.1 s, and a mean contact-time prediction error of 0.14 s. Auxiliary depth information substantially improves 3D ball localization, suggesting that inferring metric depth remains a challenge. Longer image histories do not improve ball forecasting, indicating that representations exploit scene context rather than rely solely on inter-frame motion. Finally, we propose a conditional diffusion model to generate whole-body kinematic tennis motion from vision encoder predictions on egocentric video at real-time rates, adapting movements as predictions are refined. These results move towards a deeper understanding of how foundation vision models can support real-time dynamic robots using egocentric video.
Figures & tables
Fig. 1 : Visual models are trained on egocentric tennis videos to investigate spatiotemporal understanding, forecasting, and anticipation. The dataset will be publicly released, including synchronized allocentric videos and spatial and shot annotations.
Fig. 2 : The dataset includes 53,155 frames over 85 rallies. Spatial distributions of samples are shown for (a) the ball and (b) the ego player. The temporal distribution of samples relative to the ego player ball contact is shown in (c). Locations of the five static cameras are depicted in (a) and (b).
Model
3D Ball pos.
2D Player pos.
Shot type
Time to
Ball pos. at
Current (m) ↓
Future (m) ↓
Current (m) ↓
Future (m) ↓
(acc. %) ↑
contact (ms) ↓
contact (m) ↓
Image encoders
DINOv3
0.58±0.01
1.02±0.01
0.16±0.01
0.38±0.01
98.7±0.0
141.3±1.6
1.03±0.01
SigLIP 2
0.60±0.01
1.03±0.00
0.16±0.00
0.39±0.00
98.4±0.2
148.8±1.6
1.08±0.02
SAM 3
0.55±0.03
0.99±0.03
0.15±0.00
0.38±0.00
98.4±0.3
142.7±3.1
1.01±0.01
DA3
0.58±0.01
0.99±0.02
0.14±0.01
0.36±0.01
98.4±0.0
142.8±1.6
1.00±0.02
TABLE I: Prediction accuracy across probed visual representations. Task prediction heads are trained jointly on a shared encoder. “Current” predictions are at the final conditioning frame; “future” predictions are the mean over 12 subsequent frames with stride 0.1 s. Results reflect performance on the test split, using model checkpoints selected based on the validation split, across three seeds.
Fig. 3 : Prediction errors on approaching test-split shots versus time to contact: (a) current 3D ball position, (b) ball position at contact, and (c) time to contact. Lines show means over 0.1s intervals; shading shows ±1 SD over three seeds. (d) Ball 3D and player 2D future predictions over a time horizon of ∼ 1.10 s, across all models, averaged over three seeds.
Auxiliary Information Added
MLP † ↓
SigLIP 2 ↓
None (image only)
–
0.51±0.01
Ball
2.83±0.01
0.54±0.00
Player
4.28±0.03
0.51±0.01
Depth
2.39±0.01
0.21±0.01
Ball + player
2.66±0.03
0.53±0.01
Ball + depth
2.12±0.01
0.25±0.02
TABLE II: Current 3D ball position error with auxiliary inputs. Results reflect test-split errors over three seeds, using checkpoints selected based on the validation split. † Image-free baseline using only auxiliary inputs.
Frames
Span
GT MLP † ↓
SigLIP 2 ↓
V-JEPA 2 ↓
1
–
2.924±0.003
1.02±0.02
–
2
100 ms
0.297±0.001
1.02±0.05
1.22±0.01
2
300 ms
0.257±0.004
1.08±0.05
1.25±0.01
4
300 ms
0.238±0.003
1.08±0.02
1.18±0.02
TABLE III : Future 3D ball position error versus input history ( Frames : number of input frames; Span : history duration). Results reflect test-split errors over three seeds, using checkpoints selected based on the validation split. † Image-free baseline using only ground-truth 3D ball history.
Fig. 4 : LFM2.5-VL 450M results on approaching test-split shots until contact. (a) Predictions for a single frame horizon tend to improve as the ball nears the ego player. (b) Forecasted predictions tend to improve as the ball nears the ego player and degrade as the future horizon increases.
Fig. 5 : (a) LFM2.5-VL-450M forecasts ball and player positions and anticipates contact; crosshairs locate the recorded contact. (b) Recorded ego motion versus MDM rollouts conditioned on annotations (“rec. cond.”) or vision predictions (“vision cond.”). Trails fade toward the past; ball forecasts fade toward the future. t=0 denotes the recorded racket–ball contact.
Ball–racket proximity
Error relative to recorded motion
Approach
At hit (m) ↓
Min. (m) ↓
Center (m) ↓
Plane ( ∘ ) ↓
Velocity (m/s) ↓
Recorded motion
0.89 [-1pt] [ 0.76 , 1.02 ]
0.81 [-1pt] [ 0.69 , 0.92 ]
0.00
0.00
0.00
MDM (rec. cond.)
1.11 [-1pt] [ 1.04 , 1.19 ]
0.98 [-1pt] [ 0.93 , 1.04 ]
0.76 [-1pt] [ 0.68 , 0.84 ]
35.97 [-1pt] [ 33.10 , 38.90 ]
4.46 [-1pt] [ 3.91 , 5.07 ]
MDM (vision cond.)
1.28 [-1pt] [ 1.12 , 1.42 ]
1.17 [-1pt] [ 1.04 , 1.33 ]
0.95 [-1pt] [ 0.78 , 1.10 ]
41.31 [-1pt] [ 36.03 , 46.43 ]
5.38 [-1pt] [ 4.80 , 5.92 ]
TABLE IV : MDM rollouts on 16 test sequences: means over contacts with bootstrapped 95% confidence intervals in brackets. Racket errors are relative to recorded ego player motion; minimum distance uses a ±100ms window.
Egocentric video understanding studies how AI interprets first-person video to recognize actions, track interactions, reason about space and time, and assist users.