TRINITY: A Multi-Perspective Benchmark for Personal-Style Video Highlight Detection

Qianqian Chen; Hyun Bin Kim; Denzel Elden Wijaya; Yang Yi; Bo Liu; Yangkai Ding

cs.CVAug 30, 2026

TRINITY: A Multi-Perspective Benchmark for Personal-Style Video Highlight Detection

Authors: Qianqian Chen, Hyun Bin Kim, Denzel Elden Wijaya, Yang Yi, Bo Liu, Yangkai Ding

Organizations: Huawei Technologies Co., Ltd.

Abstract

Traditional video highlight detection relies on a narrow, event-centric definition of saliency, which often fails to generalize to unconstrained personal videos where highlights are heterogeneous and perspective-dependent. To address this, we introduce TRINITY, a multi-perspective benchmark that decomposes highlight saliency into three complementary dimensions, Event, Emotion, and Nature, within a unified temporal framework. Leveraging this multi-faceted view, we propose a shared-backbone multi-branch architecture designed for parallel multi-perspective prediction via view-specific experts. Comprehensive experiments demonstrate that our method significantly outperforms state-of-the-art baselines, achieving gains of +7.15/+3.62 mAP (rho=15%/50%) on Mr. HiSum and +10.82 mAP on YouTube Highlights. These results validate that multi-perspective modeling provides a more robust and comprehensive formulation of video saliency, especially for complex real-world scenarios. The benchmark and relevant codes will be released upon acceptance. The benchmark is available at https://huggingface.co/datasets/vanilladucky/TRINITY and the code is available at https://github.com/vanilladucky/TRINITY.

Figures & tables

Figure 1: Visualization of highlight distributions from three perspectives— Event , Emotion , and Nature —for two example videos. The horizontal axis denotes time (in seconds), and the vertical axis represents the highlight score.

Panel of Figure 2 — Figure 2: Overview of the annotation pipelines used to construct the TRINITY dataset. The Emotion pipeline detects temporally stable facial expressions and verifies candidate segments via multimodal reasoning. The Nature pipeline identifies scenic moments through semantic filtering and aesthetic assessment, retaining segments with both landscape relevance and strong visual appeal.

Perspective	#Videos	#Peaks	Avg. Peak Dur. (s)
TRINITY-Nature	15,540	15,540	10.34
TRINITY-Emotion	16,257	32,046	4.87
TRINITY-Event	27,845	55,649	9.90

Table 1: Global statistics of the three highlight perspectives in the full TRINITY dataset.

Figure 3: Left. Overview of the proposed architecture with a shared backbone and perspective-specific experts. Right. Visualization of the learned embedding space, where private experts form distinct clusters while the shared backbone captures global temporal context. Compared with a single-task event-only model (bottom-right), the proposed design yields a structured and separable representation space across perspectives.

Mr. HiSum			YouTube Highlights								VEATIC
Method	$\mathrm{mAP}_{\rho=15\%}$	$\mathrm{mAP}_{\rho=50\%}$	Method	Dog	Gym	Park.	Skate.	Skii.	Surf.	Avg.	Method	SAGR $\uparrow$	RMSE $\downarrow$
SAVS-Net [ 1 ]	-	57.12	Temp.-Cue [ 54 ]	55.38	62.66	70.88	69.06	60.05	59.76	62.97	RB-SFP [ 36 ]	-	0.2423
SL-Module [ 52 ]	24.95	58.63	Mr. HiSum [ 46 ]	50.80	67.20	78.30	42.00	61.80	71.60	62.00	VEATIC [ 35 ]	0.7637	0.2410
PGL-SUM [ 23 ]	27.45	61.60	RASL [ 27 ]	64.20	75.80	70.90	45.60	66.50	66.70	65.10	VDFS [ 38 ]	0.804	0.182
HierSum [ 4 ]	32.60	63.80	C2 [ 58 ]	-	-	-	-	-	-	67.20	-	-	-
Aha [ 6 ]	32.66	64.19	AV-Recur. [ 15 ]	-	-	-	-	-	-	68.30	–	–	–

Table 2: Fine-tuned highlight detection results on Mr. HiSum, YouTube Highlights, and VEATIC. Bold indicates the best result and underlined values indicate the second best.

Method	TRINITY-Nature		TRINITY-Emotion		TRINITY-Event
Method	$\mathrm{mAP}_{\rho=15\%}$	$\mathrm{mAP}_{\rho=50\%}$	$\mathrm{mAP}_{\rho=15\%}$	$\mathrm{mAP}_{\rho=50\%}$	$\mathrm{mAP}_{\rho=15\%}$	$\mathrm{mAP}_{\rho=50\%}$
SL-Module [ 52 ]	38.68	62.23	50.55	67.57	31.46	62.71
PGL-SUM [ 23 ]	46.40	61.84	62.47	73.81	33.61	61.84
CSTA [ 13 ]	36.47	39.67	64.96	67.29	35.97	40.77
SumDiff [ 21 ]	35.68	59.63	37.17	61.38	26.21	57.45
Aha † [ 6 ]	38.69	40.02	21.02	28.29	32.66	64.19

Table 3: State-of-the-art comparison under the three TRINITY highlight perspectives (Nature, Emotion, and Event), reflecting the multi-perspective formulation of TRINITY. Bold indicates the best result.

Setting	Arch. Config.					Event		Nature		Emotion		Avg.
Setting	#B.B.	Sha.	Pri.	#Head	MTL	15%	50%	15%	50%	15%	50%	15%	50%
S1: Single-Task Baseline	1	✓	×	1	×	32.85	63.06	43.16	65.57	61.05	75.56	45.69	68.06
S2: Shared-Backbone MTL	1	✓	×	3	✓	37.97	66.86	44.41	64.35	65.20	75.71	49.19	68.97
S3: Independent Experts	3	×	×	1	×	37.54	67.51	44.78	65.26	63.27	76.34	48.53	69.70
S4: Shared–Private Experts	4	✓	✓	3	✓	40.98	69.06	58.74	68.05	67.42	71.32	55.71	69.48

Table 4: Architecture ablation on TRINITY. We analyze the impact of shared backbones, private experts, and multi-task learning. Results are reported using

\mathrm{mAP}_{\rho=15\%}

and

\mathrm{mAP}_{\rho=50\%}

(denoted as 15% and 50% in the header).

Setting	Event		Nature		Emotion		Avg.
Setting	$mAP_{\rho=15\%}$	$mAP_{\rho=50\%}$	$mAP_{\rho=15\%}$	$mAP_{\rho=50\%}$	$mAP_{\rho=15\%}$	$mAP_{\rho=50\%}$	$mAP_{\rho=15\%}$	$mAP_{\rho=50\%}$
D1: Event	40.25	69.45	31.25	65.58	40.83	65.76	37.44	66.93
D2: Event + Nature	39.98 $\downarrow$ 0.27	68.70 $\downarrow$ 0.75	57.89 $\uparrow$ 26.64	67.71 $\uparrow$ 2.13	45.98	61.06	48.25	64.32
D3: Full (Event + Nature + Emotion)	40.98 $\uparrow$ 1.00	69.06 $\uparrow$ 0.36	58.74 $\uparrow$ 0.85	68.05 $\uparrow$ 0.34	67.42 $\uparrow$ 21.44	71.32 $\uparrow$ 10.26	55.71	69.48

Table 5: Dataset contribution ablation under a fixed architecture (S4: Shared–Private Experts). Models are trained with progressively richer supervision.

Panel of Figure 4 — Figure 4: Comparison of gradient cosine similarity during training with and without Aligned-MTL.

Figure 5: Attention visualization of a personal vlog video. The visualization shows temporal localization across different semantic perspectives. The Emotion backbone highlights human-centric contentment, the Event backbone captures conversational interaction segments, and the Nature backbone focuses on environmental landmarks (Niagara Falls), illustrating the specialization of perspective-specific backbones.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Official Name	#Videos
QuerYD [ 33 ]	2,858
TACoS [ 37 ]	273
Mr. HiSum [ 46 ]	27,845
DiDeMo [ 12 ]	10,463
InternVid-VTime [ 14 ]	45,911
CosMo-Cap [ 49 ]	68,057

Appendix

Table 6: Public datasets forming the source video pool used for constructing TRINITY.

Dataset	Pair	Overlap	$N_{\text{vids}}$	Peaks (#)	Dur. (s)	Cov. (%)
Mr. HiSum	Ev–Em	0.034	3096	3.0/2.3	10.84/4.35	15.5/5.0
	Ev–Na	0.052	2814	3.1/3.0	9.65/8.65	15.5/14.1
	Em–Na	0.002	78	1.5/2.3	3.8/7.5	3.0/8.9
	Ev–Em–Na	N/A	78	2.9/1.5/2.3	10.52/3.76/7.47	15.5/3.0/8.9

Appendix

Table 7: Statistical analysis of highlight structures and temporal overlap across three perspectives on the Mr. HiSum dataset pool. Pair denotes the compared perspectives (Ev: Event, Em: Emotion, Na: Nature). Overlap reports the temporal IoU between highlight segments.

N_{\text{vids}}

denotes the number of videos containing both annotations. Peaks , Dur. , and Cov. denote the average number of peaks, duration (seconds), and temporal coverage per video, respectively. Values are reported as A/B format.

Emotion	#Peaks	Ratio (%)
Happiness	27,737	86.55
Sadness	3,408	10.63
Surprise	452	1.41
Anger	356	1.11
Disgust	67	0.21
Fear	26	0.08

Appendix

Table 8: Emotion distribution of extracted peaks (by peak count).

$\theta_{v}^{\text{land}}$	False Positives	False Negatives	Accuracy
0.1	2	1	97%
0.2	2	1	97%
0.3	2	1	97%
0.4	2	1	97%
0.5	2	1	97%
0.6	2	16	82%

Appendix

Table 9: Influence of landscape threshold

\theta_{v}^{\text{land}}

on scenic video classification (

\theta_{v}^{\text{per}}

is fixed to 0.3). The evaluation subset contains 80 scenic and 20 non-scenic videos. Accuracy is computed over all 100 videos.

Method	TRINITY-Nature		TRINITY-Emotion		TRINITY-Event		Average
Method	$\mathrm{mAP}_{15}$	$\mathrm{mAP}_{50}$	$\mathrm{mAP}_{15}$	$\mathrm{mAP}_{50}$	$\mathrm{mAP}_{15}$	$\mathrm{mAP}_{50}$	$\mathrm{mAP}_{15}$	$\mathrm{mAP}_{50}$
Uncertainty [ 20 ]	50.48	68.45	74.67	61.75	38.15	68.56	54.43	66.25
Large event weight	58.43	67.65	67.26	71.77	40.46	69.55	55.38	69.66
Small event weight	58.91	69.37	66.46	72.08	40.57	69.56	55.31	70.34
Uniform	58.74	68.05	67.42	71.32	40.98	69.06	55.71	69.48

Appendix

Table 10: Comparison of various weight loss strategies.

Method	TRINITY-Nature		TRINITY-Emotion		TRINITY-Event		Average
Method	$\mathrm{mAP}_{15}$	$\mathrm{mAP}_{50}$	$\mathrm{mAP}_{15}$	$\mathrm{mAP}_{50}$	$\mathrm{mAP}_{15}$	$\mathrm{mAP}_{50}$	$\mathbf{mAP}_{15}$	$\mathbf{mAP}_{50}$
Single Backbone (Seq.)	46.47	64.76	66.34	72.35	33.86	64.66	48.89	67.26
Single Backbone (R.R.)	44.41	64.35	65.20	75.71	37.97	66.86	49.19	68.97
Ours (Sequential)	58.71	69.20	68.30	71.15	39.67	68.97	55.56	69.77
Ours (Original)	58.74	68.05	67.42	71.32	40.98	69.06	55.71	69.48

Appendix

Table 11: Comparison of various weight loss strategies with average performance across datasets.

Setting	Eval. Set	TRINITY-Nature		TRINITY-Emotion		TRINITY-Event
Setting	Eval. Set	$\mathrm{mAP}_{15}$	$\mathrm{mAP}_{50}$	$\mathrm{mAP}_{15}$	$\mathrm{mAP}_{50}$	$\mathrm{mAP}_{15}$	$\mathrm{mAP}_{\rho=50\%}$
Ours	Full test	58.74	68.05	67.42	71.32	40.98	69.06
Ours	78-video subset	46.66	66.36	47.50	62.87	36.70	67.89
Ours w/ MixBatch	Full test	42.19	63.54	63.61	75.40	39.06	67.60

Appendix

Table 12: Co-occurrence and training-strategy ablations on TRINITY.

Hyperparameters	VEATIC	YouTubeHighlights
learning rate	3e-4	1e-4
l2 regularization	3.9	1.8
batch size	4	16
dropout ratio	0.43	0.42
epochs	100	100

Appendix

Table 13: Hyperparameter settings for VEATIC and YouTubeHighlights datasets.

Panel of Figure 6 — Figure 6: Case 1: Skateboarding practice video. The video is dominated by human actions rather than scenic content. The Event branch captures the temporal structure of skateboarding attempts, while the Emotion branch produces peaks around visible facial reactions. The Nature branch remains largely inactive due to the absence of natural scenery. Predicted highlight scores are shown as curves, and shaded regions indicate ground-truth highlights.

Panel of Figure 7 — Figure 7: Case 2: Leaf-flower crafting vlog. The video records the process of making decorative leaf flowers. Since no faces appear in the video, the Emotion branch remains inactive. The Nature branch highlights the aesthetically pleasing opening segment showing the completed leaf flower, while the Event branch captures the key crafting actions. Predicted highlight scores are shown as curves, and shaded regions indicate ground-truth highlights.

Panel of Figure 8 — Figure 8: Case 3: Indoor dance competition performance. The video takes place entirely indoors and therefore contains no natural scenery. The Event branch captures key dance movements and performance highlights, while the Emotion branch responds to expressive moments and audience reactions. The Nature branch remains inactive throughout the timeline. Predicted highlight scores are shown as curves, and shaded regions indicate ground-truth highlights.

Panel of Figure 9 — Figure 9: Case 4: Tool-making tutorial. The video mainly consists of close-up manual operations during the construction of a tool. Since there are no visible faces or natural scenery, both the Emotion and Nature branches remain inactive. The Event branch instead captures the key procedural steps of the crafting process. Predicted highlight scores are shown as curves, and shaded regions indicate ground-truth highlights.

Video Saliency Prediction

See all 12 papers

Zero-shot video highlight detection based on text descriptions and synthetic images
Sep 13, 2026Michal Byra, Alberto Presta, Grzegorz Stefanski +1Video Understanding Temporal Video Understanding
Samsung AI Center, Warsaw, Poland · Institute of Fundamental Technological Research, Polish Academy of Sciences, Warsaw, Poland
SVHighlights: Towards Extremely Long Sport Video Highlight Detection
Jun 5, 2026Donggyu Lee, Youngbin Ki, Jeonghun Kang +1Sports Video Analysis Long-Video Understanding
Ulsan National Institute of Science and Technology Ulsan, Republic of Korea
Moving Forward with Video Saliency: A New Dataset and Benchmark where Motion Matters
Oct 2, 2026Susmit Agrawal, Rebecca Wanner, Juliane Verwiebe +3Video Saliency Prediction Visual Saliency
Tübingen AI Center, University of Tübingen · IMPRS-IS · Optocycle +2

Sep 13, 2026J/K move · Enter open · S save

Zero-shot video highlight detection based on text descriptions and synthetic images

Michal Byra, Alberto Presta, Grzegorz Stefanski, Krzysztof Arendt

Samsung AI Center, Warsaw, Poland · Institute of Fundamental Technological Research, Polish Academy of Sciences, Warsaw, Poland

Video Understanding Temporal Video Understanding

Open paper AskPDF arXiv

Detecting video highlights, the most informative or engaging moments in a video, is important for applications such as video summarization and content recommendation. We propose a zero-shot framework that combines CLIP, large language models (LLMs), and diffusion models. Given lightweight video metadata, such as a title or category, an LLM generates textual descriptions of likely highlight events. These descriptions are further converted into synthetic visual prototypes using a diffusion model. Textual and visual representations are matched to video frames using CLIP, enabling frame-level highlight detection without highlight annotations or dataset-specific training. Experiments on TVSum and SumMe demonstrate strong zero-shot performance, with particularly favorable results on TVSum. The proposed approach provides an effective framework for metadata-conditioned zero-shot video highlight detection.

Abstract

Video Saliency Prediction

Explore similar work

Zero-shot video highlight detection based on text descriptions and synthetic images

SVHighlights: Towards Extremely Long Sport Video Highlight Detection

Moving Forward with Video Saliency: A New Dataset and Benchmark where Motion Matters