Audio for Sports Highlight Detection: A Comparative Empirical Study

Hao Xu; Meenakshi Sarkar; Vishnu Raj; David Gunawan

cs.CVSep 15, 2026

Audio for Sports Highlight Detection: A Comparative Empirical Study

Authors: Hao Xu, Meenakshi Sarkar, Vishnu Raj, David Gunawan

Organizations: Deakin University Melbourne, Australia · Dolby Laboratories Inc. Bangalore, India · Dolby Laboratories Inc. Sydney, Australia

Abstract

Sports highlight detection aims to identify the most exciting and meaningful moments from long sports videos. While existing methods often emphasize visual or visual-language representations, sports videos contain rich audio cues, including commentator speech, crowd reactions, whistles, ball impacts, and referee calls. In this work, we revisit the role of audio in sports highlight detection and ask a simple question: how far can audio alone go? We construct lightweight audio-only baselines using pretrained audio representations and compare them with visual-only and audio-visual methods on the SV-Highlights benchmark. Surprisingly, our audio-only GRU baseline achieves strong performance and outperforms several existing audio-visual methods under our supervised evaluation setting. Furthermore, a simple audio-visual fusion baseline achieves the best performance across all metrics, indicating that audio and visual cues provide complementary information. To better understand the contribution of audio, we conduct source-separated analysis and show that vocal/commentary audio is more informative than background-only audio, while their combination performs best. We also analyze interpretable audio cues and find that highlight clips exhibit higher RMS loudness, peak loudness, and mid-frequency energy than non-highlight clips, although substantial distribution overlap indicates that loudness alone is insufficient. Our findings suggest that audio is an underexplored but highly informative modality for sports highlight detection and should be treated as a primary signal rather than merely an auxiliary cue.

Figures & tables

Figure 1. Motivation for studying audio in sports highlight detection. Both audio-only and visual perception can convey sufficient event and affective information for audiences to recognize and respond to important sports moments. This motivates our investigation of how effectively audio alone can support highlight detection.

Figure 2. Architecture overview of the Audio GRU baseline.

Model	Input	mAP	Hit@1	Hit@K	IoU
Vision (MLP)	V	$19.98\pm 0.65$	$35.59\pm 2.11$	$24.19\pm 0.98$	$14.07\pm 0.61$
Vision (GRU)	V	$37.99\pm 0.39$	$79.57\pm 4.21$	$39.68\pm 0.58$	$25.48\pm 0.38$
Audio (MLP)	A	$33.55\pm 1.14$	$76.96\pm 3.96$	$36.43\pm 1.07$	$22.68\pm 0.81$
Audio (GRU)	A	$41.79\pm 0.68$	$86.90\pm 3.35$	$42.37\pm 0.62$	$27.49\pm 0.51$
Fusion (GRU)	V+A	$\mathbf{46.92\pm 0.71}$	$\mathbf{91.09\pm 1.91}$	$\mathbf{46.07\pm 0.57}$	$\mathbf{30.67\pm 0.43}$

Table 1. Modality ablation on SV-Highlights. We compare visual-only, audio-only, and audio-visual fusion baselines.

Model	Input	mAP	Hit@1	Hit@K	IoU
JAV ( Badamdorj et al., 2021 )	V+A	$32.03\pm 0.49$	$71.20\pm 5.47$	$34.82\pm 0.39$	$21.48\pm 0.27$
TTA ( Islam et al., 2026 )	V+A	$32.37\pm 1.32$	$71.20\pm 9.45$	$35.62\pm 1.23$	$22.09\pm 0.91$
UMT ( Liu et al., 2022 )	V+A	$21.08\pm 5.28$	$49.27\pm 15.74$	$24.54\pm 5.90$	$14.35\pm 3.80$
QD-DETR ( Moon et al., 2023 )	V+A	$35.42\pm 0.58$	$58.62\pm 6.07$	$38.28\pm 1.01$	$24.25\pm 0.77$
MS-DETR ( Ma et al., 2025 )	V+A	$36.43\pm 1.62$	$64.39\pm 2.47$	$39.26\pm 1.47$	$25.03\pm 1.13$
TR-DETR ( Sun et al., 2024 )	V+A	$37.77\pm 0.25$	$74.36\pm 3.78$	$40.02\pm 0.12$	$25.63\pm 0.11$

Table 2. Comparison with existing methods on SV-Highlights. We report mean and standard deviation over three runs.

Model	Audio Input	mAP	Hit@1	Hit@K	IoU
Audio baseline (MLP)	Vocal Only	$29.83\pm 0.83$	$73.30\pm 0.24$	$33.53\pm 0.68$	$20.44\pm 0.49$
Audio baseline (GRU)	Vocal Only	$37.39\pm 0.95$	$83.78\pm 3.54$	$38.99\pm 0.34$	$24.67\pm 0.28$
Audio baseline (MLP)	Background Only	$26.67\pm 1.32$	$61.27\pm 7.29$	$29.95\pm 1.36$	$18.01\pm 0.95$
Audio baseline (GRU)	Background Only	$33.52\pm 0.17$	$68.03\pm 8.81$	$35.68\pm 0.41$	$22.28\pm 0.32$
Audio baseline (MLP)	Combined	$33.55\pm 1.14$	$76.96\pm 3.96$	$36.43\pm 1.07$	$22.68\pm 0.81$
Audio baseline (GRU)	Combined	$\mathbf{41.79\pm 0.68}$	$\mathbf{86.90\pm 3.35}$	$\mathbf{42.37\pm 0.62}$	$\mathbf{27.49\pm 0.51}$

Table 3. Source-separated audio analysis on SV-Highlights. We compare vocal-only, background-only, and original combined audio settings.

Sport	Model	GT (%)	PPR (%)	Over-pred.	Region Ratio
NFL	MS-DETR	10.27	28.41	2.77	3.01
	Vision GRU	10.27	30.26	2.95	2.34
	Audio GRU	10.27	25.62	2.50	1.83
	Fusion GRU	10.27	26.99	2.63	2.17
Baseball	MS-DETR	4.33	19.95	4.61	2.62
	Vision GRU	4.33	26.06	6.02	2.17

Table 4. Sport-specific temporal prediction behaviour at a decision threshold of 0.5. GT denotes the ground-truth positive ratio, and PPR denotes the predicted positive ratio. Over-prediction is the ratio between PPR and GT, while Region Ratio is the ratio between the numbers of predicted and ground-truth highlight regions. The overall results are computed across the complete test set. Best results are shown in bold , and second-best results are underlined .

Sports Video Analysis

Sports video analysis uses computer vision and multimodal models to track athletes, locate events, and explain tactics in game footage.

See all 83 papers

SVHighlights: Towards Extremely Long Sport Video Highlight Detection
Jun 5, 2026Donggyu Lee, Youngbin Ki, Jeonghun Kang +1Sports Video Analysis Long-Video Understanding
Ulsan National Institute of Science and Technology Ulsan, Republic of Korea
SGWIB:Sliced Gromov-Wasserstein Information Bottleneck for Video Highlight Detection
Sep 12, 2026Hanjuan Huang, Yung-Chieh Yeh, Hsing-Kuo PaoTemporal Video Understanding Information Bottleneck
Key Laboratory of Agricultural Machinery Intelligent Control and Manufacturing of Fujian Province University, College of Mechanical and Electrical Engineering, Wuyi University, Wuyishan 354300, China · National Taiwan University of Science and Technology, No. 43, Sec. 4, Keelung Rd., Taipei, Taiwan
Zero-shot video highlight detection based on text descriptions and synthetic images
Sep 13, 2026Michal Byra, Alberto Presta, Grzegorz Stefanski +1Video Understanding Temporal Video Understanding
Samsung AI Center, Warsaw, Poland · Institute of Fundamental Technological Research, Polish Academy of Sciences, Warsaw, Poland

Jun 5, 2026J/K move · Enter open · S save

SVHighlights: Towards Extremely Long Sport Video Highlight Detection

Donggyu Lee, Youngbin Ki, Jeonghun Kang, Taehwan Kim

Ulsan National Institute of Science and Technology Ulsan, Republic of Korea

Sports Video Analysis Long-Video Understanding Temporal Video Understanding

Open paper AskPDF arXiv

While highlight detection for long-form videos is of great practical importance, most existing methods remain limited to short-form content, largely due to the absence of a suitable benchmark. To bridge this gap, we introduce SVHighlights, to the best of our knowledge, the first benchmark for highlight detection in extremely long sports videos, each exceeding one hour in duration, across multiple sports categories. SVHighlights is constructed from pairs of full-length sports videos and their corresponding official highlight videos using a dataset generation pipeline, enabling scalable label generation without conventional per-clip saliency annotation. The benchmark comprises 320 videos with an average duration of 2.00 hours and a total of 640.18 hours, substantially exceeding previous datasets. Existing methods also face fundamental challenges on long videos: models trained on short clips fail to generalize to hour-long content, and their clip-level scoring lacks the broader context needed to identify highlights. To address this and provide a strong baseline, we present TF-SELECTOR, a training-free segment-based approach that divides each video into context-aware segments by merging adjacent shots sharing the same semantic content, and predicts segment-level saliency scores using a large language model with multimodal inputs including visual captions, transcripts, and audio volume. Experiments demonstrate that TF-SELECTOR achieves superior performance across most metrics compared to Video Temporal Grounding (VTG)-tuned baselines, with improvements of +2.50 in HIT@1, +4.04 in HIT@K, and +2.95 in IoU. These results establish SVHighlights as a challenging testbed for long-form highlight detection and demonstrate that a simple segment-based strategy can effectively scale to hour-long videos.

Abstract

Sports Video Analysis

Explore similar work

SVHighlights: Towards Extremely Long Sport Video Highlight Detection

SGWIB:Sliced Gromov-Wasserstein Information Bottleneck for Video Highlight Detection

Zero-shot video highlight detection based on text descriptions and synthetic images