SportSkills: Physical Skill Learning from Sports Instructional Videos

Kumar Ashutosh; Chi Hsuan Wu; Kristen Grauman

cs.CVMar 26, 2026

SportSkills: Physical Skill Learning from Sports Instructional Videos

Authors: Kumar Ashutosh, Chi Hsuan Wu, Kristen Grauman

Organizations: University of Texas at Austin

Abstract

Current large-scale video datasets focus on general human activity, but lack depth of coverage on fine-grained activities needed to address physical skill learning. We introduce SportSkills, the first large-scale sports dataset geared towards physical skill learning with in-the-wild video. SportSkills has more than 360k instructional videos containing more than 630k visual demonstrations paired with instructional narrations explaining the know-how behind the actions from 55 varied sports. Through a suite of experiments, we show that SportSkills unlocks the ability to understand fine-grained differences between physical actions. Our representation achieves gains of up to 4x with the same model trained on traditional activity-centric datasets. Crucially, building on SportSkills, we introduce the first large-scale task formulation of mistake-conditioned instructional video retrieval, bridging representation learning and actionable feedback generation (e.g., "here's my execution of a skill; which video clip should I watch to improve it?"). Formal evaluations by professional coaches show our retrieval approach significantly advances the ability of video models to personalize visual instructions for a user query.

Figures & tables

Panel of Figure 1 — Figure 1: Our proposed SportSkills , sourced from YouTube [ 90 ] , contains paired videos and instructional narrations describing the correct execution of a skill (top), totaling $638,399$ video clips. All the visuals are meant for skill-improvement. We propose this dataset to enable video models to learn physical skills. SportSkills contains $55$ popular sports (bottom), ranging from judo, sprinting, to tennis and handball.

Figure 2: SportSkills collection overview. We develop a curation pipeline to create a solid skill-learning video-language dataset sourced from in-the-wild coaching/how-to videos. We prompt an LLM [ 58 ] to generate a list of physical sports, and sport-specific skills. We use connecting words like ‘tutorial’, ‘drills’ to create search terms (left). Next, we query YouTube [ 90 ] with these queries and obtain narrations. We use an LLM [ 58 ] to filter out non-actionable instances, e.g . , “click the link to get discounts…” (middle). Finally, we use a VLM [ 13 ] to obtain pairs of $(v,t)$ with correct or incorrect demonstrations, filtering out cases without visual demonstrations (right).

Dataset	Total Hours	#Videos (#Participants)	#Skills	#Domains
JIGSAWS [ 5 ]	3.5	103 (8)	3	1
BEST [ 23 ]	26	500 (400)	5	5
MIT-Dive [ 66 ]	0.1	159	1	1
MIT-Skating [ 66 ]	7.3	150	1	1
UNLV-Dive [ 62 ]	0.4	370	1	1
MTL-AQA [ 65 ]	1.5	1,412	1	1

Table 1: Comparing SportSkills with other datasets in skill learning. Our dataset is

\mathbf{13\times}

longer and

\mathbf{12\times}

more videos than second longest dataset [ 59 ] . The number of skills and domains we consider are also significantly larger than prior datasets.

Figure 3: Overview of our method to retrieve visual feedback given a suboptimal execution by a learner (top left). We finetune a VLM [ 13 ] , in LoRA [ 37 ] setting, to predict if a given feedback candidate is relevant or not. Top right: overview of the expert annotation task where they see the learner video, actionable feedback in text form, and the candidate YouTube video as visual feedback, and they rate the relevance of the pair to create CoachGT. Bottom: our weakly supervised training data construction.

	Lin. Probe PR.
Sport	ViT-B/32	w/ Ours
Soccer	58.2	63.2
Basketball	60.2	64.9
Rock Cl.	56.5	61.2

Table 2: (Left) Linear probe PR-AUC of classifying a demonstration as correct or incorrect for soccer, basketball, and rock climbing. We compare CLIP ViT-B/32 with the same model trained with the SportSkills dataset. Our learned representation gives better results. (Right) Evaluating visual instructional feedback. We report average Cohen’s

d

and pairwise concordance for the same sports. We see that our method outperforms both trained (Video Cont., CrossTrainer [ 9 ] ) and zero-shot baselines (ZS VLM [ 13 ] ).

Figure 4: Average recall

@10

for 20 sports. We show the performance of image and video-based visual encoder w/ and w/o SportSkills as the training dataset (orange and blue shades, respectively). We see a clear increase in the encoders trained with our proposed dataset. See Appendix for the performance on the remaining sports.

Figure 5: Retrieval-based skill corrections. We show representative outputs from our proposed model that provides visual feedback on a suboptimal learner performance, given only their video (left column). We do not input the expert commentary (second column) during either training or testing; it is given here for illustration. We see that the retrievals by our method provide personalized corrective feedback. On the other hand, retrievals from the zero-shot retrieval baseline are not helpful in skill improvement. (Last row) We also include a few failure cases that showcase the difficulty of the task. Recall that the learner videos are sourced from [ 34 ] , and the feedback videos are from our proposed SportSkills .

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Screenshot of the annotation instructions for CoachGT. We provide a detailed description of the task to clarify the exact task to the experts.

Panel of Figure 7 — Figure 7: Examples of the annotation questions for CoachGT. We provide examples of the annotation questions for the experts to understand the task better.

Figure 8: Screenshot of the question. We ask the expert to watch the learner video, read the expert feedback, and watch each of the five candidate corrective instructional videos to rate each of them as very relevant, somewhat relevant, or not relevant.

Figure 9: Histogram plot of the number of videos in SportSkills per sport. We see that the number of videos per sport ranges from

2k

14k

with

6.7k

videos on average.

Figure 10: Average recall

@10

for all sports. We show the performance of image and video-based visual encoders w/ and w/o SportSkills as the training dataset (orange and blue shades, respectively). The light colors (blue and orange) represent image encodder performances, whereas the dark colors show the performance in the video encoders. We see a clear increase in the encoders trained with our proposed dataset. Note that even our image-encoder variant largely outperforms the video variant.

Sport	ViT-B/32	w/ Ours
Soccer	58.2	63.2
Basketball	60.2	64.9
Rock Cl.	56.5	61.2

Appendix

Table 3: Linear probe PR-AUC of classifying a demonstration as correct or incorrect for a random subset of

9

out of

55

sports. We compare CLIP ViT-B/32 with the same model trained with the SportSkills dataset. Our learned representation gives better results for all the chosen sports.

Sports Video Analysis

Sports video analysis uses computer vision and multimodal models to track athletes, locate events, and explain tactics in game footage.

See all 82 papers

ExpertEdit: Learning Skill-Aware Motion Editing from Expert Videos
Apr 12, 2026Arjun Somayazulu, Kristen GraumanVideo Editing
The University of Texas at Austin, USA
SLVMBench: Skill Learning from Video Memory
Jul 13, 2026Yudong Yang, Guangzhi Sun, Yixuan Li +1Memory-Augmented VLMs Video-Language Models
Tsinghua University · University of Cambridge
Video2Skill: From Streaming Experience to Reusable Embodied Skills
Sep 29, 2026Jianshu Zhang, Ce Zhang, Xiyuan Yang +6Agent Skill Learning Robot Skill Learning
Northwestern · CMU · UIUC +1

Apr 12, 2026J/K move · Enter open · S save

ExpertEdit: Learning Skill-Aware Motion Editing from Expert Videos

Arjun Somayazulu, Kristen Grauman

The University of Texas at Austin, USA

Video Editing

Open paper AskPDF arXiv

Visual feedback is critical for motor skill acquisition in sports and rehabilitation, and psychological studies show that observing near-perfect versions of one's own performance accelerates learning more effectively than watching expert demonstrations alone. We propose to enable such personalized feedback by automatically editing a person's motion to reflect higher skill. Existing motion editing approaches are poorly suited for this setting because they assume paired input-output data -- rare and expensive to curate for skill-driven tasks -- and explicit edit guidance at inference. We introduce ExpertEdit, a framework for skill-driven motion editing trained exclusively on unpaired expert video demonstrations. ExpertEdit learns an expert motion prior with a masked language modeling objective that infills masked motion spans with expert-level refinements. At inference, novice motion is masked at skill-critical moments and projected into the learned expert manifold, producing localized skill improvements without paired supervision or manual edit guidance. Across eight diverse techniques and three sports from Ego-Exo4D and Karate Kyokushin, ExpertEdit outperforms state-of-the-art supervised motion editing methods on multiple metrics of motion realism and expert quality. Project page: https://vision.cs.utexas.edu/projects/expert_edit/ .

Abstract

Sports Video Analysis

Explore similar work

ExpertEdit: Learning Skill-Aware Motion Editing from Expert Videos

SLVMBench: Skill Learning from Video Memory

Video2Skill: From Streaming Experience to Reusable Embodied Skills