cs.CVAug 24, 2026
SaveTowards Comprehensive Basketball Understanding
Organizations: School of Mathematical Sciences, Peking University, China · School of Artificial Intelligence, Shanghai Jiao Tong University, China
Abstract
Understanding a basketball game requires recognizing events, localizing actions, identifying players, and relating these to structured game knowledge. Existing benchmarks primarily evaluate these abilities one at a time, leaving the interactions among these abilities under-explored. We introduce BasketballBench, a multimodal benchmark comprising 7,980 questions across ten tasks in text, image, and video. It is built from the 2025-2026 NBA season and includes official playby-play, rosters and profiles for 530 active players, and 2,501 possession-level broadcast clips. We further propose BasketballSkills, an agent that composes eight basketball-specific perception and retrieval tools under four reusable skills that specify tool order, evidence bindings, and stopping conditions. Experiments show that current MLLMs struggle particularly on questions requiring the integration of multiple capabilities, whereas BasketballSkills outperforms them, highlighting the effectiveness of explicitly composing domain-specific capabilities for comprehensive basketball understanding.
Figures & tables
Figure 1: Representative examples of the ten BasketballBench tasks. The benchmark covers different aspects with outputs ranging from multiple-choice answers and timestamps to structured event sequences.
| Index | Task | Input | Samples | Output | Primary capability |
|---|---|---|---|---|---|
| Q1 | Basketball Knowledge QA | Text | 1,200 | Multiple Choice | Domain Knowledge Grounding |
| Q2 | Match-Situation QA | Text | 1,200 | Multiple Choice | Contextual Retrieval |
| Q3 | Player Image Knowledge QA | Image | 400 | Multiple Choice | Domain Knowledge Grounding |
| Q4 | Scorebug Reading QA | Image | 200 | Structured Text | Broadcast Understanding |
| Q5 | Action Localization QA | Video | 600 | Multiple Choice | Spatial Understanding |
| Q6 | Action Identification QA | Video | 500 | Multiple Choice | Player Identification |
Table 1: BasketballBench comprises 7,980 examples across 10 tasks, covering text, image, and video inputs, and assesses basic perceptual abilities, basketball domain knowledge, and reasoning.
Figure 2: BasketballSkills Architecture Overview. BasketballSkills selectively loads the appropriate skills for each task and, following their guidance, composes and invokes specialized tools to complete the task.
Figure 3: Representative BasketballSkills execution examples. Each example visualizes the selected skill, invoked tool sequence, intermediate observations, and final prediction. Both the successful and failed cases further illustrate how evidence is composed and how upstream errors may propagate to the answer.
| Model | Q5 | Q6 | ||||
|---|---|---|---|---|---|---|
| Raw | Target | Raw | Target | |||
| GPT-5.4 | 57.2 | 61.5 | +4.3 | 85.4 | 97.0 | +11.6 |
| Claude Sonnet 5 | 23.5 | 33.7 | +10.2 | 55.8 | 84.4 | +28.6 |
| Gemini 3.5 Flash | 41.5 | 47.8 | +6.3 | 78.2 | 94.6 | +16.4 |
| Qwen3.5-9B | 19.7 | 36.3 | +16.7 | 47.2 | 85.8 | +38.6 |
| Molmo2-8B | 18.8 | 21.5 | +2.7 | 32.8 | 45.0 | +12.2 |
Table 4: Effect of target-player grounding on Q5 and Q6. Raw uses the original video, whereas Target highlights the ground-truth event-relevant player with a bounding box in every frame. denotes the absolute improvement in percentage points, computed from unrounded accuracies.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Data Category | Coverage | Scale |
|---|---|---|
| Teams | All current NBA teams | 30 teams |
| Players | Player profiles, including a small number of box-score-only stub records | 633 players |
| Seasonal schedules | All 2025–2026 games across PRE, REG, IST, PIT, and PST | 1,423 games |
| Detailed game records | Team-, player-, and period-level box scores for all regular-season games | 1,230 games |
| Player seasons | Player–team affiliations by season and competition type, covering season years 2012–2025 | 7,940 records |
| Player-season statistics | Season totals and per-game averages across the collected statistical fields | 15,880 rows |
Table A.1: Coverage of the structured basketball knowledge base.
| Model | Basketball AP@50:95 | Player AP@50:95 |
|---|---|---|
| Original RF-DETR Medium | 0.1734 | 0.6029 |
| Fine-tuned RF-DETR Medium | 0.5122 | 0.7886 |
| Improvement | +0.3388 | +0.1857 |
Table B.1: Class-wise RF-DETR Medium detection performance on the held-out test set. Both metric columns report AP@50:95. For the original COCO model, sports ball and person are mapped to basketball and player, respectively.
| Class ID | Shot zone | Training examples | Weight |
|---|---|---|---|
| 0 | Above the Break 3 | 4,592 | 0.5069 |
| 1 | Restricted Area | 3,471 | 0.6706 |
| 2 | Left Corner 3 | 755 | 3.083 |
| 3 | Right Corner 3 | 695 | 3.3492 |
| 4 | Mid-Range | 1,602 | 1.453 |
| 5 | In The Paint (Non-RA) | 2,851 | 0.8164 |
Table B.2: Training-set class frequencies and loss weights for Shot-Zone Classification.
| Model | Team Acc. | Score Acc. | Clock Acc. |
|---|---|---|---|
| GPT-5.4 | 95.5 | 99.0 | 97.0 |
| Claude Sonnet 5 | 94.5 | 99.0 | 97.0 |
| Gemini 3.5 Flash | 95.5 | 98.0 | 97.0 |
| Qwen2.5-VL-7B | 72.0 | 96.0 | 95.0 |
| Qwen3.5-4B | 76.5 | 98.0 | 91.0 |
| Qwen3.5-9B | 85.5 | 96.0 | 59.0 |
Table C.1: Detailed Scorebug Reading results (%). Team and Score are all-example slot accuracies, and Clock requires an exact match of both the period and normalized clock value. An unparseable or missing output is evaluated as incorrect for every corresponding field.
| Ground truth | RA | Paint | ATB3 | LC3 | RC3 | Mid | U |
|---|---|---|---|---|---|---|---|
| Restricted Area | 74.0 | 6.0 | 1.5 | 1.0 | 0.0 | 9.0 | 8.5 |
| Paint (Non-RA) | 30.5 | 44.0 | 0.0 | 0.0 | 0.0 | 21.0 | 4.5 |
| Above Break 3 | 1.0 | 0.0 | 64.0 | 15.0 | 4.0 | 16.0 | 0.0 |
| Left Corner 3 | 2.5 | 2.0 | 0.0 | 90.0 | 0.0 | 4.0 | 1.5 |
| Right Corner 3 | 3.0 | 0.0 | 1.0 | 2.0 | 83.0 | 8.0 | 3.0 |
| Mid-Range | 8.5 | 16.0 | 4.0 | 3.0 | 3.0 | 64.0 | 1.5 |
Table C.2: Element-wise average of the row-normalized confusion matrices from two independent BasketballSkills runs on Action Localization QA. Rows are ground-truth zones and columns are predictions; U denotes an unparsed response.
| Model | Video Timestamp | Game Clock | ||||||
|---|---|---|---|---|---|---|---|---|
| P-Acc@1s | Acc@1s | Parse | MAE | P-Acc@1s | Acc@1s | Parse | MAE | |
| GPT-5.4 | 69.2 | 69.2 | 100.0 | 1.67 | 86.2 | 85.7 | 99.4 | 1.52 |
| Claude Sonnet 5 | 36.8 | 36.8 | 100.0 | 2.54 | 65.8 ∗ | 65.7 ∗ | 99.9 ∗ | 11.44 ∗ |
| Gemini 3.5 Flash | 57.6 | 55.9 | 97.2 | 1.30 | 74.4 | 70.4 | 94.6 | 2.92 |
| Qwen2.5-VL-7B | 1.1 | 1.1 | 100.0 | 5.96 | 45.0 ∗ | 44.9 ∗ | 99.9 ∗ | 16.33 ∗ |
| Qwen3.5-4B | 52.5 | 51.0 | 97.2 | 1.40 | 46.3 ∗ | 45.8 ∗ | 99.0 ∗ | 30.55 ∗ |
Table C.3: Detailed Temporal Localization QA (Q7) results using the corrected temporal annotations. P-Acc@1s denotes accuracy over parseable outputs, while Acc@1s is computed over all examples, treating an unparseable output as incorrect. Parse denotes the output parse rate, and MAE is measured in seconds. For game-clock predictions, Acc@1s additionally requires the predicted period to match the reference period; game-clock MAE is computed only over parsed predictions with the correct period. ∗ indicates that the reported value differs from strict evaluation at the displayed precision after model-agnostic Period-Alias Normalization ( 1ST / 2ND / 3RD / 4TH to 1 / 2 / 3 / 4 ). The primary Q7 score in Table 2 is the macro-average of the two all-example Acc@1s columns. For VideoLLaMA3-7B, all Q7 outputs are schema-invalid and therefore cannot be parsed. Its all-example Acc@1s is reported as 0.0, while parsed accuracy and MAE are undefined and denoted by “–”.
| Model | Type+Participant | Type | PA | Parse | ||||
|---|---|---|---|---|---|---|---|---|
| P | R | F1 | P | R | F1 | |||
| GPT-5.4 | 26.1 | 34.8 | 29.8 | 52.2 | 69.5 | 59.6 | 46.7 | 99.9 |
| Claude Sonnet 5 | 11.3 | 19.5 | 14.3 | 34.6 | 60.0 | 43.9 | 24.5 | 95.5 |
| Gemini 3.5 Flash ∗ | 22.5 | 33.7 | 27.0 | 43.2 | 64.7 | 51.8 | 49.4 | 97.8 |
| Qwen2.5-VL-7B ∗ | 1.3 | 2.1 | 1.6 | 22.7 | 34.9 | 27.5 | 5.1 | 100.0 |
| Qwen3.5-4B ∗ | 6.5 | 11.1 | 8.2 | 30.9 | 53.0 | 39.1 | 20.6 | 100.0 |
Table C.4: Detailed Play Event QA (Q8) results after Conservative JSON Recovery (%). Under Type + Participant , a match requires both the event type and the participant to be correct; under Type , only the event type must match. P, R, and F1 denote LCS-based micro-averaged precision, recall, and F1, respectively. PA denotes Participant Accuracy over event-type-aligned event pairs, and Parse denotes the post-recovery schema-validity rate. ∗ indicates that Conservative JSON Recovery changes at least one output from the corresponding model. All VideoLLaMA3-7B outputs fail the required Q8 output schema. Accordingly, its all-example sequence metrics and Parse Rate are reported as 0.0. PA is undefined because no event-type-aligned prediction–reference pairs can be formed.
More in this topic
Sports Video Analysis
Sports video analysis uses computer vision and multimodal models to track athletes, locate events, and explain tactics in game footage.
See all 80 papers