Current large-scale video datasets focus on general human activity, but lack depth of coverage on fine-grained activities needed to address physical skill learning. We introduce SportSkills, the first large-scale sports dataset geared towards physical skill learning with in-the-wild video. SportSkills has more than 360k instructional videos containing more than 630k visual demonstrations paired with instructional narrations explaining the know-how behind the actions from 55 varied sports. Through a suite of experiments, we show that SportSkills unlocks the ability to understand fine-grained differences between physical actions. Our representation achieves gains of up to 4x with the same model trained on traditional activity-centric datasets. Crucially, building on SportSkills, we introduce the first large-scale task formulation of mistake-conditioned instructional video retrieval, bridging representation learning and actionable feedback generation (e.g., "here's my execution of a skill; which video clip should I watch to improve it?"). Formal evaluations by professional coaches show our retrieval approach significantly advances the ability of video models to personalize visual instructions for a user query.
Figures & tables
Figure 1: Our proposed SportSkills , sourced from YouTube [ 90 ] , contains paired videos and instructional narrations describing the correct execution of a skill (top), totaling 638,399 video clips. All the visuals are meant for skill-improvement. We propose this dataset to enable video models to learn physical skills. SportSkills contains 55 popular sports (bottom), ranging from judo, sprinting, to tennis and handball.
Figure 2: SportSkills collection overview. We develop a curation pipeline to create a solid skill-learning video-language dataset sourced from in-the-wild coaching/how-to videos. We prompt an LLM [ 58 ] to generate a list of physical sports, and sport-specific skills. We use connecting words like ‘tutorial’, ‘drills’ to create search terms (left). Next, we query YouTube [ 90 ] with these queries and obtain narrations. We use an LLM [ 58 ] to filter out non-actionable instances, e.g . , “click the link to get discounts…” (middle). Finally, we use a VLM [ 13 ] to obtain pairs of (v,t) with correct or incorrect demonstrations, filtering out cases without visual demonstrations (right).
Dataset
Total Hours
#Videos (#Participants)
#Skills
#Domains
JIGSAWS [ 5 ]
3.5
103 (8)
3
1
BEST [ 23 ]
26
500 (400)
5
5
MIT-Dive [ 66 ]
0.1
159
1
1
MIT-Skating [ 66 ]
7.3
150
1
1
UNLV-Dive [ 62 ]
0.4
370
1
1
MTL-AQA [ 65 ]
1.5
1,412
1
1
Table 1: Comparing SportSkills with other datasets in skill learning. Our dataset is 13× longer and 12× more videos than second longest dataset [ 59 ] . The number of skills and domains we consider are also significantly larger than prior datasets.
Figure 3: Overview of our method to retrieve visual feedback given a suboptimal execution by a learner (top left). We finetune a VLM [ 13 ] , in LoRA [ 37 ] setting, to predict if a given feedback candidate is relevant or not. Top right: overview of the expert annotation task where they see the learner video, actionable feedback in text form, and the candidate YouTube video as visual feedback, and they rate the relevance of the pair to create CoachGT. Bottom: our weakly supervised training data construction.
Lin. Probe PR.
Sport
ViT-B/32
w/ Ours
Soccer
58.2
63.2
Basketball
60.2
64.9
Rock Cl.
56.5
61.2
Table 2: (Left) Linear probe PR-AUC of classifying a demonstration as correct or incorrect for soccer, basketball, and rock climbing. We compare CLIP ViT-B/32 with the same model trained with the SportSkills dataset. Our learned representation gives better results. (Right) Evaluating visual instructional feedback. We report average Cohen’s d and pairwise concordance for the same sports. We see that our method outperforms both trained (Video Cont., CrossTrainer [ 9 ] ) and zero-shot baselines (ZS VLM [ 13 ] ).
Figure 4: Average recall @10 for 20 sports. We show the performance of image and video-based visual encoder w/ and w/o SportSkills as the training dataset (orange and blue shades, respectively). We see a clear increase in the encoders trained with our proposed dataset. See Appendix for the performance on the remaining sports.
Figure 5: Retrieval-based skill corrections. We show representative outputs from our proposed model that provides visual feedback on a suboptimal learner performance, given only their video (left column). We do not input the expert commentary (second column) during either training or testing; it is given here for illustration. We see that the retrievals by our method provide personalized corrective feedback. On the other hand, retrievals from the zero-shot retrieval baseline are not helpful in skill improvement. (Last row) We also include a few failure cases that showcase the difficulty of the task. Recall that the learner videos are sourced from [ 34 ] , and the feedback videos are from our proposed SportSkills .
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Screenshot of the annotation instructions for CoachGT. We provide a detailed description of the task to clarify the exact task to the experts.
Figure 7: Examples of the annotation questions for CoachGT. We provide examples of the annotation questions for the experts to understand the task better.
Figure 8: Screenshot of the question. We ask the expert to watch the learner video, read the expert feedback, and watch each of the five candidate corrective instructional videos to rate each of them as very relevant, somewhat relevant, or not relevant.
Figure 9: Histogram plot of the number of videos in SportSkills per sport. We see that the number of videos per sport ranges from 2k to 14k with 6.7k videos on average.
Figure 10: Average recall @10 for all sports. We show the performance of image and video-based visual encoders w/ and w/o SportSkills as the training dataset (orange and blue shades, respectively). The light colors (blue and orange) represent image encodder performances, whereas the dark colors show the performance in the video encoders. We see a clear increase in the encoders trained with our proposed dataset. Note that even our image-encoder variant largely outperforms the video variant.
Sport
ViT-B/32
w/ Ours
Soccer
58.2
63.2
Basketball
60.2
64.9
Rock Cl.
56.5
61.2
Appendix
Table 3: Linear probe PR-AUC of classifying a demonstration as correct or incorrect for a random subset of 9 out of 55 sports. We compare CLIP ViT-B/32 with the same model trained with the SportSkills dataset. Our learned representation gives better results for all the chosen sports.
Visual feedback is critical for motor skill acquisition in sports and rehabilitation, and psychological studies show that observing near-perfect versions of one's own performance accelerates learning more effectively than watching expert demonstrations alone. We propose to enable such personalized feedback by automatically editing a person's motion to reflect higher skill. Existing motion editing approaches are poorly suited for this setting because they assume paired input-output data -- rare and expensive to curate for skill-driven tasks -- and explicit edit guidance at inference. We introduce ExpertEdit, a framework for skill-driven motion editing trained exclusively on unpaired expert video demonstrations. ExpertEdit learns an expert motion prior with a masked language modeling objective that infills masked motion spans with expert-level refinements. At inference, novice motion is masked at skill-critical moments and projected into the learned expert manifold, producing localized skill improvements without paired supervision or manual edit guidance. Across eight diverse techniques and three sports from Ego-Exo4D and Karate Kyokushin, ExpertEdit outperforms state-of-the-art supervised motion editing methods on multiple metrics of motion realism and expert quality. Project page: https://vision.cs.utexas.edu/projects/expert_edit/ .
We introduce Skill Learning from Video Memory (SLVMBench), the first benchmark that jointly evaluates whether video large language models (video-LLMs) can learn skills from long video memory and apply them to real-time tasks. SLVMBench presents models with 2-3 hour video streams that contain a tutorial video embedded in a stream of arbitrary irrelevant videos, resembling real-world human learning practices. Video-LLMs are asked to apply the acquired skill to answer real-time questions about an ongoing video. Unlike long-video understanding benchmarks that emphasize passive comprehension and skill-learning benchmarks that rely on short, immediate demonstrations, SLVMBench tests the full pipeline of memorizing and extracting procedural knowledge, as well as transferring it to real-time tasks. Moreover, rigorous human annotations feature sub-second-level temporal calibration, manually engineered questions eliminating common-sense guessing, and collated tutorials to ensure coverage of the required skills. Evaluations on state-of-the-art proprietary and open-source video LLMs show that video-LLMs struggle substantially with learning and applying skill knowledge from videos. Moreover, performance degrades markedly when the skill knowledge is placed within a long video memory. These results reveal a key limitation of existing video LLMs and position SLVMBench as the first benchmark for studying real-time skill acquisition and application from long-context video memory.
Manipulation behaviors vary widely across objects and scenes, but they share a small set of reusable skills, and planning with these skills helps embodied agents generalize to new tasks. Yet an agent can only plan with skills it knows. Recovering skills from observed experience, the inverse of planning, builds this knowledge over time and yields skill data for training future agents. Vision-Language Models (VLMs) describe individual manipulation events well, but can they organize a stream of events into reusable skills? We formulate this problem as Streaming Embodied Skill Discovery (SESD): a model watches videos in sequence and maintains a persistent skill library that shapes its later decisions. To systematically measure this ability, we introduce Video2Skill, a benchmark that covers robot tabletop manipulation and human kitchen activity and tests three core capabilities: (i) locating manipulation events in time, (ii) grouping events of the same transformation, and (iii) deciding when to reuse an existing skill or create a new one. Across 19 open-source VLMs, many models group events at near-chance level, and scale does not consistently help. Their errors depend on how perception and library updates are coupled: joint models merge distinct transformations into one skill, while models that update the library from text descriptions duplicate recurring ones. Supervised fine-tuning, including our counterfactual library-state rebalancing (CLaRe), improves grouping but exposes a deeper bottleneck: trained models consolidate familiar skills yet rarely expand the library. Their libraries stall below half the reference size, and transformations unseen in training are located in time but almost never given a new skill. Recognizing when existing skills are insufficient thus emerges as the central challenge.