TennisExpert: Towards Expert-Level Analytical Sports Video Understanding

Zhaoyu Liu; Xi Weng; Lianyu Hu; Zhe Hou; Kan Jiang; Jin Song Dong; Yang Liu

cs.CVMar 11, 2026

TennisExpert: Towards Expert-Level Analytical Sports Video Understanding

Authors: Zhaoyu Liu, Xi Weng, Lianyu Hu, Zhe Hou, Kan Jiang, Jin Song Dong, Yang Liu

Organizations: National University of Singapore · Nanyang Technological University · Griffith University

Abstract

Tennis is one of the most widely followed sports, generating extensive broadcast footage with strong potential for professional analysis, automated coaching, and real-time commentary. However, automatic tennis understanding remains underexplored due to two key challenges: (1) the lack of large-scale benchmarks with fine-grained annotations and expert-level commentary, and (2) the difficulty of building accurate yet efficient multimodal systems suitable for real-time deployment. To address these challenges, we introduce TennisVL, a large-scale tennis benchmark comprising over 200 professional matches (471.9 hours) and 40,000+ rally-level clips. Unlike existing commentary datasets that focus on descriptive play-by-play narration, TennisVL emphasizes expert analytical commentary capturing tactical reasoning, player decisions, and match momentum. Furthermore, we propose TennisExpert, a multimodal tennis understanding framework that integrates a video semantic parser with a memory-augmented model built on Qwen3-VL-8B. The parser extracts key match elements (e.g., scores, shot sequences, ball bounces, and player locations), while hierarchical memory modules capture both short- and long-term temporal context. Experiments show that TennisExpert consistently outperforms strong proprietary baselines, including GPT-5, Gemini, and Claude, and demonstrates improved ability to capture tactical context and match dynamics. Our dataset and code are publicly available at https://github.com/LZYAndy/TennisExpert.

Figures & tables

Figure 1: Existing sports commentary datasets primarily provide descriptive narration of immediate actions and visual states (top and middle rows). In contrast, TennisVL provides expert-level analytical commentary (bottom row), emphasizing tactical intent, match momentum, and performance evaluation beyond surface-level descriptions.

Dataset	Sport	Video (h)	# Clips	Event Seq.	ASR Trans.	Analytic
SoccerNet-Caption [ 26 ]	Soccer	715.9	36,894	✓	✗	✗
MatchTime [ 32 ]	Soccer	715.9	32,743	✓	✓	✗
GOAL [ 30 ]	Soccer	25.0+	8,981	✗	✗	$\bigcirc$
SVN [ 45 ]	Basketball	7.7	9,632	✗	✗	✗
NSVA [ 42 ]	Basketball	84.8	32,019	✓	✗	✗
Tennis Dataset [ 44 ]	Tennis	1.5	633	✗	✗	✗

Table 1: Comparison of TennisVL with existing sports video commentary datasets. “#Clips”: annotated video–text pairs; “Event Seq.”: action sequences; “ASR Trans.”: aligned audio transcripts; “Analytic”: analytic/tactical reasoning (

\bigcirc

= partial).

Panel of Figure 2 — Figure 2: Dataset statistics of clip duration, commentary length, and tournaments.

Figure 3: Overall architecture of TennisExpert . Given a sequence of rallies, the current rally $V_{t}$ is processed by a video semantic parser to obtain structured metadata $M_{t}$ , including scoreboard state ( $s_{t}$ ), fine-grained event sequence ( $e_{t}$ ), and spatial object detections ( $o_{t}$ ). A hierarchical memory mechanism maintains match context: short-term memory ( $S$ ) stores recent rally representations in a FIFO buffer, while long-term memory ( $L$ ) consolidates past events into cumulative match statistics. The tactic-aware MLLM (Qwen3-VL) integrates visual tokens, structured metadata, and memory context to generate expert-level commentary $C_{t}$ , which is fed back into short-term memory to preserve match momentum for subsequent rallies.

Open-Weight Models (Zero-Shot)
Model	Traditional Metrics				LLM-based Metrics
Model	BLEU-4	METEOR	ROUGE-L	CIDEr	Acc	Coh	Exc	Pro	Pac	Total
InternVL3.5-8B	0.75	20.20	13.61	0.08	0.90	5.10	6.69	2.86	3.47	19.02
InternVL3.5-38B	0.84	20.58	13.22	0.11	1.45	8.98	10.56	4.16	5.25	30.40
Qwen3-VL-8B	1.20	18.90	14.90	3.89	1.70	10.56	11.78	5.05	8.59	37.68
Qwen3-VL-32B	1.24	20.42	15.46	2.61	2.19	10.74	10.81	5.50	8.74	37.97

Table 2: Quantitative evaluation results on traditional metrics and LLM-based metrics. “Acc”, “Coh”, “Exc”, “Pro”, and “Pac” denote “Accuracy”, “Coherence”, “Excitement”, “Professionalism”, and “Pacing”. Our TennisExpert achieves SOTA performance.

Model Setup	Input	Token Count	VRAM (GB)	Latency (s)
Baselines (Dense video)	Full raw video up to $t$	$\mathcal{O}(T)$	OOM	High
TennisExpert (Ours)	Structured $V_{t}+M_{t}+H_{t}$	$\mathcal{O}(1)$ ( $\sim$ 14k)	$\sim$ 20 GB	< 2 s

Table 3: Computational efficiency during inference. Our approach significantly reduces VRAM and Latency compared to processing full raw videos.

Zero-Shot (Qwen3-VL-8B)
Input	Memory	Traditional Metrics				LLM-based Metrics
Input	Memory	BLEU-4	METEOR	ROUGE-L	CIDEr	Acc	Coh	Exc	Pro	Pac	Total
$V_{t}$	–	1.20	18.90	14.90	3.89	1.70	10.56	11.78	5.05	8.59	37.68
$V_{t}+M_{t}$	–	1.58	21.17	16.39	4.40	7.86	14.62	14.10	11.20	13.75	61.53
Fine-Tuned (Qwen3-VL-8B)
$V_{t}$	–	1.95	16.30	16.15	7.86	4.10	15.05	6.63	6.49	10.46	42.74

Table 4: Ablation study on input structure and hierarchical memory.

V_{t}

M_{t}

\mathcal{S}_{t}

, and

\mathcal{L}_{t}

denote video, metadata, short-term memory, and long-term memory, respectively.

Figure 4: Semantic parser visualization on various court surfaces. Court corners (red), ball (green), bounces (yellow), and player boxes are projected onto broadcast frames.

Figure 5: Qualitative comparison of commentary generation. Our method produces expert-level commentary across multiple dimensions: (a) tactical depth, (b) temporal momentum, (c) professional terminology, and (d) strategic prediction.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 6: More qualitative results on commentary generation.

Sports Video Analysis

Sports video analysis uses computer vision and multimodal models to track athletes, locate events, and explain tactics in game footage.

See all 83 papers

TennisVAR: A Stroke-Evidence-Grounded Multimodal Large Language Model for Tactical Reasoning in Tennis Videos
Aug 13, 2026Yifan Mei, Qingling Shi, Changli Wu +3Sports Video Analysis Multimodal Large Language Models
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University · Shanghai Innovation Institute · School of Artificial Intelligence, Shanghai Jiao Tong University
Prompt, Probe, Train, or Annotate? Single-camera sports video understanding in amateur settings
Sep 23, 2026Sai Varun Kodathala, Prashanth Pollishetty, Jaylen CargillSports Video Analysis Video Action Recognition
Research and Development Sports Vision, Inc. Minnetonka, MN
Multi-Task Tennis Stroke Biomechanics Analysis Using MediaPipe Pose
Jun 14, 2026Jigyashman HazarikaHuman Activity Recognition Sports Video Analysis
Department of Artificial Intelligence and Machine Learning, BMS Institute of Technology and Management, Bengaluru, India

Aug 13, 2026J/K move · Enter open · S save

TennisVAR: A Stroke-Evidence-Grounded Multimodal Large Language Model for Tactical Reasoning in Tennis Videos

Yifan Mei, Qingling Shi, Changli Wu, Jiayuan Rao+2

Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University · Shanghai Innovation Institute · School of Artificial Intelligence, Shanghai Jiao Tong University

Sports Video Analysis Multimodal Large Language Models Multimodal Reasoning

Open paper AskPDF arXiv

Sports-video understanding is moving beyond event recognition toward explaining how actions collectively shape match progression, however, existing tennis-video methods either perceive individual strokes without modeling their tactical dependencies or generate high-level analyses without grounding them in the underlying events. To bridge this perception-to-understanding gap, we formulate stroke-evidence-grounded tactical reasoning, a new rally-level task that requires models to jointly predict an open-ended answer, a hierarchical tactic label, an ordered sequence of supporting strokes, and decisive key actions, with each evidence stroke anchored to its racket-ball contact frame. We further introduce TRACE (Tactical Reasoning with Action-Chain Evidence in Tennis), a large-scale expert-annotated benchmark containing 11,189 rally videos, 41,485 stroke events, 25,429 tactical units, and 11,189 question-answer pairs, which unifies fine-grained stroke attributes, cross-stroke tactical relations, hierarchical tactic annotations, and evidence-grounded questions across factual perception, tactical understanding, and decision reasoning. Building on TRACE, we propose TennisVAR (Tennis Video Action-chain Reasoner), an evidence-grounded multimodal large language model that follows an "event-relation-evidence-tactic" reasoning paradigm, where an Event Parsing Module converts continuous rallies into explicit stroke-event sequences while a Tactical Graph-Guided Temporal Reasoner jointly models rally progression and same-player decision dependencies to identify question-relevant evidence and decisive actions.

Abstract

Sports Video Analysis

Explore similar work

TennisVAR: A Stroke-Evidence-Grounded Multimodal Large Language Model for Tactical Reasoning in Tennis Videos

Prompt, Probe, Train, or Annotate? Single-camera sports video understanding in amateur settings

Multi-Task Tennis Stroke Biomechanics Analysis Using MediaPipe Pose