Gauging an individual's skill level is crucial, as it inherently shapes their behavior. Quantifying skill, however, is challenging because it is latent to the observed actions. To explore skill understanding in human behavior, we focus on dyadic sports -- specifically table tennis -- where skill manifests not just in complex movements, but in the subtle nuances of execution conditioned on game context. Our key idea is to learn a generative model of each player's tactical racket strokes and jointly embed them in a common latent space that encodes individual characteristics, including those pertaining to skill levels. By training these player models on a large-scale dataset of 3D-reconstructed professional matches and conditioning them on comprehensive game context -- including player positioning and opponent behaviors -- the models capture individual tactical identities within their latent space. We probe this learned player space and find that it reflects distinct play styles and attributes that collectively represent skill. By training a simple relative ranking network on these embeddings, we demonstrate that both relative and absolute skill predictions can be achieved. These results demonstrate that the learned player space effectively quantifies skill levels, providing a foundation for automated skill assessment in complex, interactive behaviors.
Figures & tables
Figure 1: We demonstrate skill level quantification using dyadic sports as a primary example. From match videos, we jointly learn player-specific generative models and a player space defined by their latent embeddings. We show that this learned player space contains rich information regarding individual proficiency, enabling the quantification of skill scores and both relative and absolute ranking predictions.
Figure 2: We model player semantics including their skill levels from observation of their plays. Our main model, HitFlow, jointly learns player embeddings and the distribution of racket hit vectors governing the ball trajectories in a play conditioned on the game context. The player embeddings encode semantics of each player including their skill levels. We demonstrate the extraction of such deeply latent characteristics of the players through probing and ranking of players and show their accuracy in match prediction.
Figure 3: The architecture of HitFlow. The velocity field estimation by Flow Matching is conditioned on the game context and player identity through FiLM [ 25 ] . Player embeddings are jointly learned with the model resulting in a player space encoding their latent characteristics including skill levels.
method
side
oblique
back
TT3D
12.4
17.1
29.8
HitFormer (w/o opt.)
10.3
16.3
18.8
HitFormer (w/ opt.)
10.1
16.2
18.5
Table 1: 3D ball trajectory reconstruction accuracy in Mean Absolute Error (MAE) in centimeters for various viewpoints. HitFormer consistently outperforms TT3D, particularly from the back viewpoint. Hit-vector optimization further refines accuracy.
method
HitFlow
LATTE-MV (i)
LATTE-MV (ii)
energy score (↓)
0.37
0.40
0.40
Table 2: Ball trajectory prediction accuracy comparison. HitFlow outperforms LATTE-MV for both ensemble settings.
Figure 4: Hit vector (ball trajectory) prediction. For each pair, the left shows LATTE-MV prediction (seed ensemble) and the right shows HitFlow prediction. The GT trajectories are shown in blue and the trajectories where the MAE takes median in the samples are shown in red as representatives. HitFlow outputs diverse trajectories and covers GT within the distribution.
Figure 5: Matthew Correlation Coefficient (MCC) ( left ) and t-SNE result for the player space by cosine distance. (a) MCC is sufficiently large even with a small number of labels, and tend to increase as the number of labels grow for sex, handedness, and height. Age appears to be unrelated to the player embeddings. (b) Player 14 , 15 and 43 , who are all female choppers, are particularly close to each other. Sex and handedness are reflected in the distributional bias simultaneously, showing that the embeddings encode sex, handedness and play style reflected in the observed plays.
Figure 8
sex
overall
rank gap >5
Male
77.3 %
87.4 %
Female
73.5 %
79.8 %
Table 3: Accuracy of rank prediction between unknown players. Overall accuracy for both male and female is over 70 %, and when the rank gap becomes larger than 5, the accuracy becomes higher.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: t-SNE visualization of hit vectors in SynthHit. Predefined shot types form distinct clusters, while random samples bridge the transition regions between them.
Figure 9: Number of rally samples available per player in the JTTA dataset.
Figure 10: Trajectory estimation from the observation viewpoint: (a) before and (b) after optimization. Green points denote ground truth, and orange points represent estimates. In the pre-optimized state, the estimation fails to capture the bounce discontinuity, resulting in a large reprojection error. Post-optimization, this deviation is resolved, accurately capturing the impact on the table.
The AI CUP 2025 Precise Analysis of Table Tennis Smart Racket Data Competition introduced smart table tennis rackets that collect extensive player swing data, enabling research on table tennis big data. These data support in-depth analysis of players' return techniques and swing-force consistency, improving the accuracy of player skill assessment. This study focuses on six-axis sensor data collected by smart table tennis rackets and proposes TTNet, a novel deep learning model with multitask learning capabilities, to advance table tennis data analysis and related applications. TTNet combines convolutional neural networks (CNNs), residual networks (ResNet), and self-attention mechanisms to simultaneously predict four player attributes: gender, playing hand, years of experience, and skill level. We adopt a two-stage training strategy that incorporates data augmentation and task-specific loss functions to improve generalization on imbalanced data. Our approach achieved second place on the official competition leaderboard.
Ko-Hsun Chen, Xiang-Wei Ke, Hsien-Cheng Huang +1
Department of Computer Science and Engineering, Yuan Ze University, Taoyuan, Taiwan
Human athletes demonstrate versatile and highly-dynamic tennis skills to successfully conduct competitive rallies with a high-speed tennis ball. However, reproducing such behaviors on humanoid robots is difficult, partially due to the lack of perfect humanoid action data or human kinematic motion data in tennis scenarios as reference. In this work, we propose LATENT, a system that Learns Athletic humanoid TEnnis skills from imperfect human motioN daTa. The imperfect human motion data consist only of motion fragments that capture the primitive skills used when playing tennis rather than precise and complete human-tennis motion sequences from real-world tennis matches, thereby significantly reducing the difficulty of data collection. Our key insight is that, despite being imperfect, such quasi-realistic data still provide priors about human primitive skills in tennis scenarios. With further correction and composition, we learn a humanoid policy that can consistently strike incoming balls under a wide range of conditions and return them to target locations, while preserving natural motion styles. We also propose a series of designs for robust sim-to-real transfer and deploy our policy on the Unitree G1 humanoid robot. Our method achieves surprising results in the real world and can stably sustain multi-shot rallies with human players. Project page: https://zzk273.github.io/LATENT/
Zhikai Zhang, Haofei Lu, Yunrui Lian +12
Tsinghua University · Galbot Inc. · Shanghai AI Laboratory +2
Understanding human skill is important for AI systems that collaborate with, coach, or assist people. Unlike typical latent variable estimation problems which rely on single observations, skill is a persistent, compositional, and behaviorally grounded construct that must be inferred from patterns over time. We introduce Skill Abstraction with Interpretable Latents (SAIL), a method for modeling human skill as an interpretable, multi-dimensional construct inferred from naturalistic behavior. Our approach produces a skill embedding that is robust to transient performance fluctuations and learns a transferable representation of human subskills. Furthermore, SAIL supports skill-informed behavior prediction that generalizes across a variety of in-domain contexts. We represent each individual with a persistent skill embedding that controls a blend between expert and novice bases and is trained using counterfactual subskill swaps for disentanglement. This design encourages representations that are both robust to performance variation and structured for interpretability. We demonstrate across racing and baseball that SAIL achieves strong predictive performance and consistently improves behaviorally grounded disentanglement over the evaluated baselines, while also improving downstream AI coaching performance.