Where is the multimodal goal post? On the Ability of Foundation Models to Recognize Highlights in Videos

Aditya K Surikuchi; Raquel Fernández; Sandro Pezzelle

cs.CVJan 22, 2026

Where is the multimodal goal post? On the Ability of Foundation Models to Recognize Highlights in Videos

Authors: Aditya K Surikuchi, Raquel Fernández, Sandro Pezzelle

Organizations: Institute for Logic, Language and Computation, University of Amsterdam

Abstract

Foundation models are used for many real-world applications involving language understanding and generation from temporally-ordered multimodal events. In this work, we study the ability of models to identify the most important sub-events in a video, which is a fundamental prerequisite for summarizing, reasoning over, or narrating multimodal events. Specifically, we focus on football games and evaluate models on their ability to distinguish between important and non-important sub-events in a game. To this end, we construct a new dataset by leveraging publicly available football highlight reels containing the most important moments of a game according to professional experts, without any additional annotation costs. Using our dataset, we compare several state-of-the-art multimodal models and show that they fail to reliably capture contextual importance. Analyses of models beyond standard evaluation metrics reveal their tendency to rely on a single dominant modality and their ineffectiveness in synthesizing necessary information from multiple sources. Our findings underline the importance of modular architectures that dynamically adapt to multimodal heterogeneity and the need for complementary training procedures that maximize cross-modal synergy.

Figures & tables

Figure 1: The clip on the top, showing a shot-on-target, was featured in the highlights of the football game to which the clip belongs. In contrast, the one on the bottom, showing a corner kick, was not. A logistic regression classifier trained on the transcriptions of commentaries identifies the text in blue as driving the prediction toward the ‘important’ class, and the text in orange as driving it toward ‘non-important’. The confidence of a foundation model, Qwen-2.5-Omni, in correctly identifying this clip featured in the highlights reel as ‘important’ is highest under vision-only input ( V=3.81 , L=-0.18, LV=-0.93), whereas for the ‘non-important’ clip multimodal information (language in the commentary complementing the visual) results in the highest model confidence (V=0.5, L=0.87, LV=1.34 ). More examples from our dataset are provided in Figure 11 of Appendix C .

Figure 2: MCC and accuracy for all models under various combinations of modalities. We compute uncertainties (line indicators on each bar) representing the 95% confidence interval, estimated via bootstrap resampling Berg-Kirkpatrick et al. (2012) . Under each modality combination,

\smallstar

denotes the highest scoring model and

\diamond

denotes the highest scoring open-source model.

\filledstar

represents the best performing model and combination of modalities.

Panel of Figure 3 — Figure 3: Contribution scores of the three modalities for: (a) Qwen2.5-Omni-7B (b) Gemini-2.5-Flash, separated for important ( IM ) and non-important ( NIM ) moments.

Figure 4: For each prototypical and contextual moment (important or non-important), we visualize model confidence (

{\Delta}Z

) under optimal unimodal vs. multimodal settings. The diagonal (

y=x

) indicates equivalent confidence, while points on and below it signal unimodal dominance and cross-modal interference, respectively. Absence of points above the diagonal highlights limited multimodal synergy in open models.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: The duration distributions of non-important moments (NIM) reflect the distributions of important ones (IM), with the average duration (

\bar{t}

) of the audio modality extending beyond the average of corresponding video frames for both classes.

Figure 6: On the top, we compare frames from an example <H,G> pair both pertaining to the exact same instance of the game (accounting for frame rates). Differences in terms of scorecards and advertisement overlays are penalized appropriately by the SSIM metric (0.86); On the bottom, the frame from G pertains to an instance in the same game that is half a second apart from the instance depicted by the H frame. In this case, SSIM (0.44) accurately reflects all valid visual differences in terms of player formations and ball position.

Figure 7: Among the 127 < H , G > pairs we process, more than half of the frames in 100 highlight ( H ) videos are accurately localized in their corresponding full-game ( G ) videos after step 2.

System Prompt: You are an expert of soccer games, who is well versed with the knowledge of the sport, and capable of recognizing game highlights.

Generation Prompt 1: Classify whether this is an important moment of the game that should be included in the highlights. Here is the commentary associated with this moment: “{< COMMENTARY >}”. Respond only with a “YES” or a “NO”.

Generation Prompt 2: Is this an important moment of the game that is worthy of inclusion in the highlights reel? This is the transcription of the commentary: “{< COMMENTARY >}”. Respond only with a “YES” or a “NO”.

Appendix

Table 1: System prompt and paraphrased generation prompts used to evaluate all the models. When the commentary is not among the input modalities, the sentence enclosing it is omitted.

	A	L	V	AL	AV	LV	ALV
Gemini-2.5-Flash	42 8 16 34	43 7 19 31	43 7 17 33	42 8 17 33	44 6 14 36	46 4 21 29	45 5 16 34
Qwen2.5-Omni-7B	33 17 14 36	38 12 18 32	43 7 22 28	41 9 20 30	45 5 25 25	39 11 13 37	44 6 22 28
Qwen2.5-VL-7B-Instruct		2 48 0 50	42 8 17 33			44 6 24 26

Appendix

Table 2: Confusion matrices with percentages for three high-scoring models. In each matrix, clockwise from the top left are values for: True Important , False Non-important , False Important , and True Non-important predictions. The off-diagonal of matrices represent error percentages with a stronger shade of red indicating a higher percentage. We observe that models struggle more with non-important compared to important moments.

Figure 8: MCC, accuracy, ROC AUC, and F1 scores for all models under various combinations of modalities. Under each modality combination,

\smallstar

denotes the highest scoring model and

\diamond

denotes the highest scoring open source model.

\filledstar

represents the overall best performing model and combination of modalities.

Panel of Figure 10 — Figure 10: Modality-level contribution scores for all the models we considered in this study, separated for important ( IM ) and non-important ( NIM ) moments.

Figure 11: Examples from our MOMENTS dataset: (a) shows a ‘goal’ scoring moment, (b) shows a ‘shot-on-target’ saved by a goalkeeper from the opposing team, (c) shows a non-important ‘corner kick’ that did not result in a goal, and (d) shows another non-important moment in which players are passing the ball in the midfield.

Multimodal Foundation Models

Multimodal foundation models learn shared capabilities across text, images, audio, video, and other signals for understanding, generation, and action.

See all 354 papers

Zero-Shot Multimodal Event Spotting in Soccer Broadcasts
Oct 27, 2026Sadia Kamal, Mahmudul Hasan, Tony BraskichSports Video Analysis Soccer Analytics
Department of Computer Science and Engineering, University of Maryland Baltimore County, Maryland, USA · Comcast Applied AI, Comcast, Philadelphia, USA
HAVEN: Hierarchically Aligned Multimodal Benchmark for Unified Video Understanding
May 19, 2026Mengqi Shi, Haopeng ZhangMultimodal Grounding Video Summarization
Department of Information and Computer Sciences · University of Hawaii at Manoa
Cut to the Chase: Training-free Multimodal Summarization via Chain-of-Events
Mar 6, 2026Xiaoxing You, Qiang Huang, Lingyu Li +2Video Summarization Multimodal Generation
School of Computer Science, Hangzhou Dianzi University · School of Intelligence Science and Engineering, Harbin Institute of Technology (Shenzhen) · School of Information Science and Technology, University of Science and Technology of China

Oct 27, 2026J/K move · Enter open · S save

Zero-Shot Multimodal Event Spotting in Soccer Broadcasts

Sadia Kamal, Mahmudul Hasan, Tony Braskich

Department of Computer Science and Engineering, University of Maryland Baltimore County, Maryland, USA · Comcast Applied AI, Comcast, Philadelphia, USA

Sports Video Analysis Soccer Analytics Multimodal Learning

Open paper AskPDF arXiv

Automatic soccer event spotting supports highlight generation, video indexing, and sports analytics, but existing methods depend on large annotated datasets and soccer-specific training. We present a zero-shot framework that detects, classifies, and temporally localizes events in full-length soccer broadcasts without task-specific parameter training. Each match is segmented into overlapping clips represented by a compact visual storyboard and the aligned commentary transcript, which a general-purpose multimodal model processes jointly. On three manually annotated matches containing 130 events, the best configuration achieves 72.0% F1@5s and outperforms SoccerNet-trained CALF and NetVLAD++ in a cross-dataset setting, while the storyboard representation reduces inference cost by 72% compared with individual frames. We further analyze the effects of model family, input representation, and prompt optimization, and introduce an evidence-grounded excitement-scoring stage that ranks detected events with natural-language explanations for highlight selection.

Abstract

Multimodal Foundation Models

Explore similar work

Zero-Shot Multimodal Event Spotting in Soccer Broadcasts

HAVEN: Hierarchically Aligned Multimodal Benchmark for Unified Video Understanding

Cut to the Chase: Training-free Multimodal Summarization via Chain-of-Events