cs.MMOct 24, 2026
SaveAdShot: Benchmarking Multimodal Large Language Models for Video Advertisement Clipping
Organizations: Northeastern University, Boston, MA 02115, USA · Southern Methodist University, Dallas, TX 75205, USA
Abstract
Advertisers routinely create multiple duration variants of ads to meet marketing budgets and viewer preferences, a labor-intensive and costly process. While multimodal large language models (MLLMs) excel at general video understanding, their ability to perform specialized editorial tasks remains unexplored. We introduce AdShot, a benchmark for evaluating MLLMs on shot selection for ad clipping. It contains 823 professionally edited ad pairs (30-sec sources and 15-sec edits) across 194 brands and 17 industries, annotated with shot-level boundaries and content features (action, emotion, information, imagery). We evaluate 13 state-of-the-art video-only MLLMs and 2 audio-visual MLLMs (7B-72B parameters) on shot selection accuracy (Precision, Recall, F1), duration constraints (15s), and content preservation, including an audio ablation. We find: (1) model architecture matters more than scale, with smaller models often matching or exceeding larger ones; performance is strongest for target ads with 6-10 shots; (2) models exhibit a mean absolute duration error of 5.39s, with most tending to overshoot the 15s target, though they can still substantially reduce human editing effort; (3) all models exhibit systematic content biases, favoring information-rich and emotion-focused shots while under-selecting action and imagery. We publicly release our benchmark to facilitate research at the intersection of multimodal AI and computational advertising.
Figures & tables
Figure 1: Example of the shot selection task: From a 31-sec Subaru ad from YouTube (shots indexed in red on the right panel; we show the first frame of each shot for better representation), professional editors select and order shots (blue indices) to create a 16-sec version promoting the same product (CrossTrek).
| Benchmark | Domain | Task | Videos | Length | Key Distinction |
| VideoAds Zhang et al. (2025) | Ads | Open-ended QA | 3,153 | 15–60s | QA on ad content |
| AdsQA Long et al. (2025) | Ads | Open-ended QA | 600 | 5–90s | Reasons about implicit ad meanings |
| AdSum Xie et al. (2026) | Ads | Shot Selection | 204 | 15s, 30s | Supervised audio-visual model; no MLLM evaluation |
| Video-MME Fu et al. (2025) | General | Multiple-choice QA | 900 | 11s–1h | Broad multimodal understanding |
| MVBench Li et al. (2023) | General | Open-ended QA | 4,000 | 10s–10min | Temporal reasoning across events |
| MLVU Zhou et al. (2025) | General | Multiple-choice QA | 1,500+ | Hours | Long-form video reasoning |
Table 1: Comparison between AdShot and existing video understanding benchmarks.
Figure 2: Methodological overview of long-short ad pair construction. Given videos from a brand’s YouTube channel, we first apply AutoShot ( Zhu et al., 2023 ) for shot boundary detection, then extract shot embeddings using Swin3D ( Liu et al., 2022 ) . We cluster the embeddings using DBSCAN to match each shot in the 15-sec ad to its corresponding shot in the 30-sec ad. Throughout the figure, blue (red) elements (e.g., digits and dots) represent the 15-sec (30-sec) ad.
Figure 4: MLLM evaluation pipeline for ad shot selection. Keyframes from shots of a 30-sec ad, along with metadata (e.g., durations) and task instructions specifying the clipping objective, are fed to the model. The model outputs selected shots with reasoning, which are evaluated for accuracy and content preservation.
| Model | Family | Params | Modality |
| LLaVA-OneVision ( Li et al., 2024a ) | LLaVA | 7B, 72B | V |
| Qwen2.5-VL ( Bai et al., 2025b ) | Qwen | 7B, 32B | V |
| Qwen3-VL ( Bai et al., 2025a ) | Qwen | 8B, 32B | V |
| InternVL2.5 ( Chen et al., 2024 ) | InternVL | 8B, 26B | V |
| InternVL3.5 ( Zhu et al., 2025 ) | InternVL | 8B, 38B | V |
| Aria ( Li et al., 2024b ) | MoE | 25.3B | V |
Table 3: Evaluated models, parameter scales, and input modalities. V = video-only; A+V = audio + video.
| Model | Precision | Recall | F1 | MDE | MDE | MDE |
| Random baseline | 0.569 | 0.541 | 0.503 | 8.18 | 9.01 | 7.16 |
| Video-only inputs | ||||||
| Qwen2.5-VL 7B | 0.613 | 0.649 | 0.582 | 7.82 | 5.92 | 1.90 |
| LLaVA-OneVision 7B | 0.578 | 0.601 | 0.539 | 8.68 | 6.41 | 2.27 |
| Qwen3-VL 8B | 0.618 | 0.580 | 0.576 | 4.08 | 2.99 | 1.10 |
| InternVL2.5 8B | 0.594 | 0.698 | 0.613 | 8.30 | 7.98 | 0.32 |
Table 4: Performance on shot selection (1 FPS). Random baseline samples a random subset size , then picks shots uniformly at random (Appendix D ). Bold marks the best per column across all evaluated models. / indicate higher/lower is better.
| Model | F1 | MDE (s) | ||||
| SC 6 | 6 SC 10 | SC 10 | SC 6 | 6 SC 10 | SC 10 | |
| Video-only inputs | ||||||
| Qwen2.5-VL 7B | 0.609 | 0.706 | 0.719 | 9.95 | 12.44 | 14.05 |
| LLaVA-OneVision 7B | 0.534 | 0.467 | 0.381 | 4.46 | 3.40 | 5.21 |
| Qwen3-VL 8B | 0.595 | 0.614 | 0.640 | 7.39 | 8.11 | 10.40 |
| InternVL2.5 8B | 0.480 | 0.450 | 0.360 | 3.32 | 4.06 | 6.45 |
Table 5: Shot selection performance by targeted shot count (SC) at fps=1. F1 and MDE for 15-sec ads with 6, 6–10, and 10 shots. Shaded cells highlight per-model best F1 and lowest MDE across bins; bold marks the best per column.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Category | Question |
| Action | Q1: Is there a call to find out more information about the product or the brand (e.g., shop online, visit the Web)? |
| Q2: Is there online contact information provided (e.g., URL, website)? | |
| Q3: Is there a visual or verbal call to purchase (e.g., buy now, order now)? | |
| Q4: Does the ad portray a sense of urgency to act (e.g., buy before sales ends, order before ends)? | |
| Q5: Is there an incentive to buy (e.g., a discount, a coupon, a sale or "limited time offer")? | |
| Q6: Is there offline contact information provided (e.g., phone, mail, store location)? |
Table 6: Annotation Questions of Ad Content Feature by Category
Figure 5: Content Feature Divergence. MLLMs over-select emotion- and information-focused shots while under-selecting action- and imagery-driven content, indicating systematic biases relative to professional edits.
Figure 6: Ablation on temporal sampling. Using 3 or 5 frames per shot instead of 1 FPS does not consistently improve MLLM performance, indicating limited benefit from denser temporal input.
| Model | fps=1 | fps=3 | fps=5 | |||
| F1 | MDE | F1 | MDE | F1 | MDE | |
| Video-only inputs | ||||||
| Qwen2.5-VL 7B | 0.582 | 7.82 | 0.576 | 9.79 | 0.643 | 10.91 |
| LLaVA-OneVision 7B | 0.539 | 8.68 | 0.540 | 8.80 | 0.541 | 8.41 |
| Qwen3-VL 8B | 0.576 | 4.08 | 0.573 | 4.46 | 0.586 | 4.96 |
| InternVL2.5 8B | 0.613 | 8.30 | 0.390 | 11.09 | — | — |
Table 7: Ablation on FPS. Shot selection performance under different temporal sampling densities. / indicate higher/lower is better. Shaded cells highlight, for each model, the best performance across the three FPS values; bold marks the best across all models. Some models (e.g., InternVL2.5) are excluded from 5 FPS due to token-length constraints.
| Industry | Prec. | Rec. | F1 |
| Food & Beverages | 0.707 | 0.674 | 0.674 |
| Cosmetic | 0.618 | 0.699 | 0.634 |
| Restaurants | 0.664 | 0.631 | 0.634 |
| Services | 0.619 | 0.657 | 0.625 |
| Health | 0.638 | 0.641 | 0.622 |
| Telecommunications | 0.615 | 0.661 | 0.621 |
Table 8: Per-industry shot-selection performance of Qwen3-VL 32B (1 FPS), sorted by F1. Shaded cells are above the overall average (Precision 0.636, Recall 0.618, F1 0.611). Industry labels follow Table 2 .
| Model | Input | Prec. | Rec. | F1 | MDE | MDE | MDE |
| MiniCPM-o 2.6 8B | Audio+Video | 0.581 | 0.841 | 0.669 | 10.06 | 1.72 | 10.50 |
| Video-only | 0.588 | 0.778 | 0.644 | 8.38 | 2.28 | 9.60 | |
| (audio) | |||||||
| Qwen2.5-Omni 7B | Audio+Video | 0.615 | 0.838 | 0.668 | 11.18 | 3.16 | 12.68 |
| Video-only | 0.595 | 0.660 | 0.585 | 7.75 | 3.47 | 9.81 | |
| (audio) |
Table 9: Audio ablation: same model, same 823 AdShot pairs, with and without the audio channel. Audio+Video rows are from Table 4 ; Video-only rows are this ablation. is the audio contribution (Audio+Video Video-only). MDE / MDE are the mean under-/over-shoot in seconds.
| Industry | F1 | F1 | F1 | Rec | Rec | Rec |
| Retail | 0.659 | 0.586 | 0.825 | 0.639 | ||
| Legal | 0.644 | 0.602 | 0.822 | 0.689 | ||
| Services | 0.648 | 0.567 | 0.883 | 0.716 | ||
| Government | 0.669 | 0.563 | 0.825 | 0.611 | ||
| Health | 0.640 | 0.575 | 0.817 | 0.637 | ||
| Automotive | 0.727 | 0.571 | 0.920 | 0.604 |
Table 10: Per-industry audio ablation for Qwen2.5-Omni 7B across all 17 industries, sorted by volume. “ A” is audio+video, “ A” is video-only; is the audio contribution. Shaded F1 cells are positive: audio improves F1 in 14 of 17 industries and recall in 16 of 17.
Figure 7: Example 1 — Camping World (Automotive / Vehicles), 22 shots. Blue: Qwen3-VL 32B selected; yellow: golden shot (ground truth).
Figure 8: Example 2 — Cologuard (Health), 16 shots. Blue: Qwen3-VL 32B selected; yellow: golden shot (ground truth).
Figure 9: Example 3 — La-Z-Boy (Retail), 10 shots. Blue: Qwen3-VL 32B selected; yellow: golden shot (ground truth).
| Category | # Questions | Agreement (%) |
| Action | 7 | 93.96 |
| Information | 3 | 74.47 |
| Emotion | 5 | 76.44 |
| Imagery | 3 | 70.63 |
| Overall | 18 | 81.96 |
Table 11: Agreement between Gemini 3.1 Pro and AdNotator on a random sample of 1,000 AdShot shots, by content-feature category and overall. Each shot is labeled on the same 18 binary questions (7 Action, 3 Information, 5 Emotion, 3 Imagery), giving 18,000 matched question–shot decisions.