RoughCut-Bench: A Benchmark for Automatic Video Rough-Cut with Large Vision-Language Models
Organizations: ReLER, CCAI Zhejiang University, ByteDance Hangzhou, Zhejiang, China · ByteDance Beijing, China · Peking University, ByteDance Beijing, China · School of Artificial Intelligence Zhejiang University Hangzhou, Zhejiang, China ·
Abstract
Video rough-cut is a core stage in media production workflows, organizing raw media materials into an initial video cut. Despite recent advances in large vision-language models (LVLMs) for video-related tasks, automatic rough-cut remains underexplored, without a comprehensive benchmark for evaluation or mature workflows for producing executable cut plans. To address the evaluation gap, we introduce RoughCut-Bench, which covers two practical settings: (1) a visually driven setting over video cases, covering highlight detection and selection, source-video ordering, and adherence to user requirements when provided; and (2) a speech-driven setting over automatic speech recognition (ASR) transcripts, covering filler removal, expression deduplication, and low-value content trimming. Across these settings, the benchmark contains 101 visually driven cases, including 831 raw videos with 1,476 highlight annotations. It also includes 84 speech-driven cases, each consisting of an ASR transcript with annotated cleanup spans. The evaluation compares cut plans with these annotations in both settings and additionally uses a human-aligned agentic evaluator to assess rendered-cut quality in the visual setting. To apply LVLMs to automatic rough-cut, we further propose agentic workflows that guide them to produce structured cut plans from input materials. Experiments show that current LVLMs remain limited in producing expert-aligned cut plans. Our benchmark, code, and prompts are released at https://whalesong-zrs.github.io/RoughCut-Bench-Project-Page.
Figures & tables
| Model\Metric | Visual Set-1 | Visual Set-2 | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Precision | Recall | F1 | Avg. IoU | Kendall’s | Precision | Recall | F1 | Avg. IoU | Kendall’s | |
| Gemini 2.5 Pro | 0.467 | 0.556 | 0.508 | 0.437 | 0.556 | 0.344 | 0.341 | 0.342 | 0.354 | 0.654 |
| Gemini 3.0 Pro | 0.490 | 0.568 | 0.526 | 0.440 | 0.565 | 0.345 | 0.342 | 0.343 | 0.358 | 0.692 |
| GPT-5.4 | 0.443 | 0.597 | 0.509 | 0.428 | 0.512 | 0.265 | 0.301 | 0.282 | 0.315 | 0.610 |
| GPT-5.5 | 0.446 | 0.604 | 0.513 | 0.420 | 0.542 | 0.311 | 0.343 | 0.326 | 0.336 | 0.702 |
| Qwen3.5-Plus | 0.468 | 0.571 | 0.514 | 0.441 | 0.471 | 0.288 | 0.316 | 0.301 | 0.351 | 0.678 |
| Model\Metric | Visual Set-1 | Visual Set-2 | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Ordering | Redundancy | Overall | Human Score | Ordering | Requirement | Redundancy | Overall | Human Score | |
| Gemini 2.5 Pro | 3.33 | 2.52 | 3.11 | 3.46 | 3.66 | 4.31 | 2.97 | 3.60 | 3.76 |
| Gemini 3.0 Pro | 3.30 | 2.58 | 3.13 | 3.52 | 3.46 | 4.09 | 2.87 | 3.42 | 3.50 |
| GPT-5.4 | 3.15 | 2.07 | 2.82 | 2.70 | 3.69 | 4.15 | 2.68 | 3.47 | 3.60 |
| GPT-5.5 | 3.30 | 2.05 | 2.88 | 2.88 | 3.43 | 4.16 | 2.68 | 3.45 | 3.42 |
| Qwen3.5-Plus | 3.18 | 2.32 | 2.95 | 3.22 | 3.35 | 4.14 | 2.68 | 3.36 | 3.35 |
| Model\Metric | IM | RM R | D | ||||
|---|---|---|---|---|---|---|---|
| P | R | F1 | Succ. | P | R | F1 | |
| Gemini 2.5 Pro | 0.430 | 0.891 | 0.581 | 0.491 | 0.707 | 0.609 | 0.654 |
| Gemini 3.0 Pro | 0.416 | 0.898 | 0.569 | 0.497 | 0.692 | 0.609 | 0.648 |
| GPT-5.4 | 0.332 | 0.837 | 0.475 | 0.357 | 0.651 | 0.693 | 0.671 |
| GPT-5.5 | 0.403 | 0.913 | 0.559 | 0.340 | 0.670 | 0.610 | 0.639 |
| Qwen3.5-Plus | 0.511 | 0.785 | 0.617 | 0.535 | 0.732 | 0.444 | 0.543 |
| Input | Visual Set-1 | Visual Set-2 | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Precision | Recall | F1 | Avg. IoU | Kendall’s | Precision | Recall | F1 | Avg. IoU | Kendall’s | |
| Time + Frame + Desc | 0.490 | 0.568 | 0.526 | 0.440 | 0.565 | 0.345 | 0.342 | 0.343 | 0.358 | 0.692 |
| Time + Frame | 0.474 | 0.566 | 0.516 | 0.440 | 0.518 | 0.348 | 0.340 | 0.344 | 0.365 | 0.474 |
| Time + Desc | 0.480 | 0.556 | 0.515 | 0.450 | 0.556 | 0.295 | 0.303 | 0.299 | 0.358 | 0.618 |
| FPS | Visual Set-1 | Visual Set-2 | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Precision | Recall | F1 | Avg. IoU | Kendall’s | Precision | Recall | F1 | Avg. IoU | Kendall’s | |
| 0.5 | 0.466 | 0.537 | 0.499 | 0.436 | 0.547 | 0.284 | 0.289 | 0.286 | 0.348 | 0.651 |
| 1.0 | 0.490 | 0.568 | 0.526 | 0.440 | 0.565 | 0.345 | 0.342 | 0.343 | 0.358 | 0.692 |
| 2.0 | 0.486 | 0.576 | 0.527 | 0.445 | 0.614 | 0.357 | 0.364 | 0.360 | 0.357 | 0.732 |
| Strategy\Metric | IM | RM R | D | ||||
|---|---|---|---|---|---|---|---|
| P | R | F1 | Succ. | P | R | F1 | |
| CoT zero-shot | 0.416 | 0.898 | 0.569 | 0.497 | 0.692 | 0.609 | 0.648 |
| CoT one-shot | 0.385 | 0.910 | 0.542 | 0.491 | 0.671 | 0.688 | 0.680 |
| w/o CoT zero-shot | 0.466 | 0.831 | 0.597 | 0.451 | 0.693 | 0.388 | 0.497 |
| w/o CoT one-shot | 0.512 | 0.741 | 0.605 | 0.398 | 0.702 | 0.332 | 0.451 |
Video Editing Benchmarks
Video editing benchmarks measure whether systems follow edit instructions while preserving content, temporal consistency, and audiovisual quality.
See all 36 papers