Automatically extracting engaging and high-quality humorous scenes from cinematic titles is pivotal for creating captivating video previews and snackable content, boosting user engagement on streaming platforms. Long-form cinematic titles, with their extended duration and complex narratives, challenge scene localization, while humor's reliance on diverse modalities and its nuanced style add further complexity. This paper introduces an end-to-end system for automatically identifying and ranking humorous scenes from long-form cinematic titles, featuring shot detection, multimodal scene localization, and humor tagging optimized for cinematic content. Key innovations include a novel scene segmentation approach combining visual and textual cues, improved shot representations via guided triplet mining, and a multimodal humor tagging framework leveraging both audio and text. Our system achieves an 18.3% AP improvement over state-of-the-art scene detection on the OVSD dataset and an F1 score of 0.834 for detecting humor in long text. Extensive evaluations across five cinematic titles demonstrate 87% of clips extracted by our pipeline are intended to be funny, while 98% of scenes are accurately localized. With successful generalization to trailers, these results showcase the pipeline's potential to enhance content creation workflows, improve user engagement, and streamline snackable content generation for diverse cinematic media formats.
Figures & tables
Figure 1: Funny scene extraction pipeline overview. The pipeline consists of three main blocks: shot detection, shot representation extraction, and scene merging, followed by downstream binary tagging and humor scoring.
Figure 2: Scene detection module overview : Triplets generated from MovieNet-SSeg ground-truth boundaries; spatial and temporal augmentations applied during contrastive pretraining with triplet loss; for each video, we extract shot-level video caption and use shot text-encoder to extract shot-level text features; fine-tuning aggregates neighboring shot features and trains MLP layers via supervised learning.
Figure 3: Multi-modal humor tagging pipeline overview . (1) Extract audio and detect laughter, (2) transcribe audio and analyze funny conversations, (3) filter improper humor, then score and and output the ranked funny scene videos.
Shot-Encoder Variants
V1
V2
V3
final
NMI -overall
0.606
0.604
0.611
0.632
Table 1: Scene clustering quality evaluated using normalized mutual information (NMI) across various triplet mining strategies.
Version
Augmentations
best AP
best F1
No.
Resize
Color
Horizon
Random
Uniform
Jitter
tal Flip
Masking
Masking
A1
✓
✓
-
-
-
41.7
35.1
A2
✓
✓
✓
✓
-
47.2
41.8
A3
✓
✓
✓
-
8%-15%
38.1
16.8
A4
✓
✓
✓
-
16%-25%
34.3
15.2
Table 2: Impact of different augmentations on Movienet-Sseg.
Model
best AP
ShotCoL ( Chen et al. 2021 )
25.02
BASSL ( Mun et al. 2022 )
28.68
Ours (Visual)
33.94
Table 3: Comparison between our method and other shot-level pre-training baselines on OVSD, utilizing visual signals (V) only.
Input-modality
best AP
best F1
Visual (V)
47.2
41.8
Visual (V) + Text (T)
51.5
46.4
Table 4: Impact of input modalities on the scene detection performance on Movienet-Sseg test set.
Train Set
Test Set
Model
Acc
F1
UR-FUNNY
MHD
Fine-tuned Transformer
0.591
0.719
ColBERT-base-5
0.694
0.820
ColBERT-base-10
0.728
0.834
ColRoBERTa-base-5
0.720
0.830
ColRoBERTa-base-10
0.690
0.812
MHD
MHD
MASM ( Patro et al. 2021 )
0.720
0.810
Table 5: Our text-based humor tagging model performance.
Movies
Trailers
Is the clip intended to be funny?
87%
100%
Do curators find the clip funny?
74%
88%
Proper Scene Ending?
98%
53%
Table 6: curators’ evaluation on movies & trailers.
Figure 4: Example extracted humorous scenes and instances of improper humor from various long-form cinematic titles. Observed laughter is marked with orange boxes, while laugh-worthy dialogue or humorous context is highlighted in violet .
Temporal video understanding studies how models track actions, events, and changes over time to answer questions, locate moments, and predict what happens next.