SGWIB:Sliced Gromov-Wasserstein Information Bottleneck for Video Highlight Detection
Authors: Hanjuan Huang, Yung-Chieh Yeh, Hsing-Kuo Pao
Organizations: Key Laboratory of Agricultural Machinery Intelligent Control and Manufacturing of Fujian Province University, College of Mechanical and Electrical Engineering, Wuyi University, Wuyishan 354300, China · National Taiwan University of Science and Technology, No. 43, Sec. 4, Keelung Rd., Taipei, Taiwan
Video highlight detection aims to identify temporally important segments that capture the most informative or engaging events in a video. Reliable prediction therefore requires not only discriminative segment representations but also preservation of the temporal relationships among neighboring and distant segments. The information bottleneck principle has proven effective for learning compact and task-relevant representations, yet it has not been explored for video highlight detection, and applying conventional formulations directly would overlook inter-segment relational structure and distort highlight relevant temporal organization during compression. We therefore introduce the Sliced Gromov-Monge Gap (SGMG), a structure aware regularizer that measures the excess relational distortion induced by a prescribed source-to-bottleneck mapping relative to an optimal sliced structural correspondence. Building on SGMG, we develop SGWIB, an information-bottleneck framework for single-modal video highlight detection that learns compact bottleneck representations while preserving inter-segment temporal structure. We further introduce Home-Away-Related Contextual Pseudo-Labels and a contextual disentanglement module that reduce sports-specific contextual bias by separating highlight oriented information from contextual patterns. Experiments on MrHiSum and MoSu show that SGWIB attains the best Kendall's tau, Spearman's rho, mAP@50, and mAP@30 among the compared single-modal methods on both datasets. On MrHiSum, the visual model improves the strongest previous results by 0.031, 0.031, 0.87, and 0.75 on these four metrics, respectively. These results show that structure-aware information-bottleneck regularization combined with contextual disentanglement improves segment-level highlight prediction.
Figures & tables
Symbol
Description
m∈M={v,a}
Modality index for visual ( v ) and audio ( a ).
Xm∈RL×dm
Padded source sequence; also the detached source-side geometry reference of SGMG.
M,I,n
Temporal validity mask, valid index set, and n=∣I∣ .
y,ym
Ground-truth and predicted highlight-score sequences.
Xm
Backbone representation.
Hm,Xspum
Task-oriented and context-oriented representations from HAR-CDM.
TABLE I: Summary of the principal notation.
Fig. 1: Overview of the proposed SGWIB framework. The visual and audio branches follow the same formulation but are optimized independently.
Dataset
Videos
Categories
Train/Val/Test
MrHiSum
31,892
3,509
80%/10%/10%
MoSu
52,678
3,406
60%/20%/20%
TABLE II: Summary of the datasets used in the experiments.
Dataset
Method
τ↑
ρ↑
mAP@50 ↑
mAP@30 ↑
mAP@15 ↑
MrHiSum
VASNet [ fajtl2019summarizing ]
0.068
0.100
58.67
39.67
25.21
PGL-SUM [ apostolidis2021combining ]
0.097
0.141
53.77
35.25
21.52
iPTNet [ jiang2022joint ]
0.020
0.029
55.53
–
22.74
SL-module [ xu2021crosscategory ]
0.062
0.091
58.27
38.88
24.44
SUM-GAN [ mahasseni2017unsupervised ]
0.067
0.095
56.62
–
23.56
AC-SUM-GAN [ apostolidis2021ac ]
0.012
0.018
55.35
–
21.88
TABLE III: Visual-only comparison with state-of-the-art methods on MrHiSum and MoSu. Best and second-best results within each dataset are highlighted in bold and underlined, respectively.
Fig. 2: Qualitative comparison of temporal highlight prediction on (a) MrHiSum, (b) MoSu, and (c) an additional YouTube video. The ground-truth highlight intervals of the YouTube example are manually annotated for qualitative evaluation.
Dataset
Modality
HAR-CDM
SGWIB
γm
βm
τ↑
ρ↑
mAP@50 ↑
mAP@30 ↑
mAP@15 ↑
MrHiSum
Visual
✗
✗
–
–
0.157
0.210
64.15
45.87
28.95
✓
✗
0.05
–
0.180
0.242
65.38
45.88
28.30
✗
✓
–
10−8
0.181
0.242
65.66
46.79
29.40
✓
✓
0.01
10−6
0.192
0.255
66.11
47.22
30.08
Audio
✗
✗
–
–
0.141
0.191
63.39
45.87
28.88
✓
✗
0.01
–
0.173
0.230
65.22
46.86
29.74
TABLE IV: Component ablation of HAR-CDM and SGWIB under the best validation-selected hyperparameters. For configurations that include HAR-CDM or SGWIB, the selected γm and βm are reported. The best and second-best results within each dataset and modality are shown in bold and underlined, respectively.
Dataset
Modality
IB Formulation
Optimal β
τ↑
ρ↑
mAP@50 ↑
mAP@30 ↑
mAP@15 ↑
MrHiSum
Visual
KLIB
10−8
0.181
0.242
65.52
47.02
29.67
SGWIB
10−6
0.192
0.255
66.11
47.22
30.08
Audio
KLIB
10−8
0.166
0.220
64.78
46.05
28.99
SGWIB
10−8
0.179
0.239
65.45
46.52
29.98
MoSu
Visual
KLIB
10−3
0.244
0.326
68.40
50.03
32.21
SGWIB
10−5
0.249
0.331
68.67
50.35
32.65
TABLE V: Task-level comparison of KLIB and SGWIB. All network components and training settings are identical except for the information-bottleneck regularizer. The optimal β is selected independently on the validation set.
Method
Time
Iters.
it/s
Cost
KLIB
18m 40s
1,744
1.56
1.00×
SGWIB
20m 34s
1,744
1.41
1.10×
GWIB
2h 15m 58s
1,744
0.21
7.28×
TABLE VI: Average per-epoch training cost on the MrHiSum visual branch. Each epoch contains 1,744 iterations, and results are averaged over 10 epochs. Relative cost is normalized to KLIB.
Fig. 3: Structural changes induced by KLIB and SGWIB under different bottleneck coefficients βv on the MrHiSum visual branch. Lower values indicate better structural preservation.
Fig. 4: Joint t-SNE visualization of the input and bottleneck visual representations produced by KLIB and SGWIB on MrHiSum under (a) low ( βv=10−10 ), (b) validation-selected ( βv=10−6 ), and (c) high ( βv=10−1 ) compression strengths. Open and filled circles denote xtv,std and ztv,std , respectively, and each line connects the representations of the same valid temporal segment before and after compression.
Fig. 5: Sensitivity of the independently trained visual and audio SGWIB models to the bottleneck coefficient βm on MrHiSum. Each panel compares the two modalities under the same evaluation metric. Red circles indicate the best value of each modality for the corresponding metric.
Fig. 6: Sensitivity to the HAR-CDM loss weight γm on MrHiSum. Visual and audio denote independently trained single-modal models. The bottleneck coefficients are fixed to βv⋆=10−6 and βa⋆=10−8 , respectively. Red circles indicate the best value of each modality for the corresponding evaluation metric.
Temporal video understanding studies how models track actions, events, and changes over time to answer questions, locate moments, and predict what happens next.
While highlight detection for long-form videos is of great practical importance, most existing methods remain limited to short-form content, largely due to the absence of a suitable benchmark. To bridge this gap, we introduce SVHighlights, to the best of our knowledge, the first benchmark for highlight detection in extremely long sports videos, each exceeding one hour in duration, across multiple sports categories. SVHighlights is constructed from pairs of full-length sports videos and their corresponding official highlight videos using a dataset generation pipeline, enabling scalable label generation without conventional per-clip saliency annotation. The benchmark comprises 320 videos with an average duration of 2.00 hours and a total of 640.18 hours, substantially exceeding previous datasets. Existing methods also face fundamental challenges on long videos: models trained on short clips fail to generalize to hour-long content, and their clip-level scoring lacks the broader context needed to identify highlights. To address this and provide a strong baseline, we present TF-SELECTOR, a training-free segment-based approach that divides each video into context-aware segments by merging adjacent shots sharing the same semantic content, and predicts segment-level saliency scores using a large language model with multimodal inputs including visual captions, transcripts, and audio volume. Experiments demonstrate that TF-SELECTOR achieves superior performance across most metrics compared to Video Temporal Grounding (VTG)-tuned baselines, with improvements of +2.50 in HIT@1, +4.04 in HIT@K, and +2.95 in IoU. These results establish SVHighlights as a challenging testbed for long-form highlight detection and demonstrate that a simple segment-based strategy can effectively scale to hour-long videos.
Donggyu Lee, Youngbin Ki, Jeonghun Kang +1
Ulsan National Institute of Science and Technology Ulsan, Republic of Korea
Existing multimodal video highlight detectors typically assume that visual, audio, and textual streams are continuously available. In practice, however, inputs may suffer from localized frame missingness or complete-stream outage. We formulate this robustness challenge along two dimensions: temporal missingness, where frames are missing independently in each modality, and stream-level missingness, where one modality is unavailable throughout a video. Moreover, we find that the mean squared error (MSE) loss is misaligned with both the evaluation metrics and the peak-driven nature of highlights. Therefore, we propose Temporal-Stream Modality Dropout (TSMD), which combines structured missingness simulation with a joint objective comprising pointwise MSE, per-video Pearson correlation, and peak-oriented RankNet loss terms. TSMD has three variants: temporal, stream-level, and mixed dropout. On the MoSu and Mr. HiSum datasets, TSMD-Temporal improves mAP@15 by 7.06 and 3.41 points over TripleSumm under 50% independent temporal removal, whereas TSMD-Stream performs the best under complete-stream removal. TSMD-Mix retains most of these complementary benefits and ranks the best or the second-best across the evaluated temporal and stream-level conditions.
Bo-Yuan Cheng, Kuan-Yu Chen, Po-Han Huang +2
AI Research Center, Inventec Corporation, Taiwan · Graduate Institute of Communication Engineering, National Taiwan University, Taiwan
Traditional video highlight detection relies on a narrow, event-centric definition of saliency, which often fails to generalize to unconstrained personal videos where highlights are heterogeneous and perspective-dependent. To address this, we introduce TRINITY, a multi-perspective benchmark that decomposes highlight saliency into three complementary dimensions, Event, Emotion, and Nature, within a unified temporal framework. Leveraging this multi-faceted view, we propose a shared-backbone multi-branch architecture designed for parallel multi-perspective prediction via view-specific experts. Comprehensive experiments demonstrate that our method significantly outperforms state-of-the-art baselines, achieving gains of +7.15/+3.62 mAP (rho=15%/50%) on Mr. HiSum and +10.82 mAP on YouTube Highlights. These results validate that multi-perspective modeling provides a more robust and comprehensive formulation of video saliency, especially for complex real-world scenarios. The benchmark and relevant codes will be released upon acceptance. The benchmark is available at https://huggingface.co/datasets/vanilladucky/TRINITY and the code is available at https://github.com/vanilladucky/TRINITY.
Qianqian Chen, Hyun Bin Kim, Denzel Elden Wijaya +3