RoughCut-Bench: A Benchmark for Automatic Video Rough-Cut with Large Vision-Language Models
Authors: Ruisi Zhao, You Li, Sihan Chen, Aosong Cheng, Chixiang Ma, Sen Pei, Zhen Yang, Hehe Fan, Dongliang He, Yi Yang
Organizations: ReLER, CCAI Zhejiang University, ByteDance Hangzhou, Zhejiang, China · ByteDance Beijing, China · Peking University, ByteDance Beijing, China · School of Artificial Intelligence Zhejiang University Hangzhou, Zhejiang, China · ReLER, CCAI Zhejiang University Hangzhou, Zhejiang, China
Video rough-cut is a core stage in media production workflows, organizing raw media materials into an initial video cut. Despite recent advances in large vision-language models (LVLMs) for video-related tasks, automatic rough-cut remains underexplored, without a comprehensive benchmark for evaluation or mature workflows for producing executable cut plans. To address the evaluation gap, we introduce RoughCut-Bench, which covers two practical settings: (1) a visually driven setting over video cases, covering highlight detection and selection, source-video ordering, and adherence to user requirements when provided; and (2) a speech-driven setting over automatic speech recognition (ASR) transcripts, covering filler removal, expression deduplication, and low-value content trimming. Across these settings, the benchmark contains 101 visually driven cases, including 831 raw videos with 1,476 highlight annotations. It also includes 84 speech-driven cases, each consisting of an ASR transcript with annotated cleanup spans. The evaluation compares cut plans with these annotations in both settings and additionally uses a human-aligned agentic evaluator to assess rendered-cut quality in the visual setting. To apply LVLMs to automatic rough-cut, we further propose agentic workflows that guide them to produce structured cut plans from input materials. Experiments show that current LVLMs remain limited in producing expert-aligned cut plans. Our benchmark, code, and prompts are released at https://whalesong-zrs.github.io/RoughCut-Bench-Project-Page.
Figures & tables
Figure 1. Dataset statistics of RoughCut-Bench (Sec. 3.2 ). (a) Distribution of the 101 visual cases across six creation scenarios; the outer bars distinguish cases with and without user requirements. (b) Duration distribution of the 831 raw videos. (c) Counts and annotated span-length distributions of the IM, RM → R, and D speech-cleanup labels.
Figure 2. Proposed agentic workflows and complementary evaluation in RoughCut-Bench (Secs. 3.3 and 3.4 ). The visual branch detects candidate highlights and provides corresponding reasons and visual descriptions, then performs highlight selection and source-video ordering. The speech-driven branch processes each ASR transcript and produces structured cleanup records, reasons, and a cleaned transcript. Annotation-based metrics compare cut plans with professional annotations, while the human-aligned agentic evaluator assesses rendered cuts using the input materials and available user requirements.
Model\Metric
Visual Set-1
Visual Set-2
Precision
Recall
F1
Avg. IoU
Kendall’s τ
Precision
Recall
F1
Avg. IoU
Kendall’s τ
Gemini 2.5 Pro
0.467
0.556
0.508
0.437
0.556
0.344
0.341
0.342
0.354
0.654
Gemini 3.0 Pro
0.490
0.568
0.526
0.440
0.565
0.345
0.342
0.343
0.358
0.692
GPT-5.4
0.443
0.597
0.509
0.428
0.512
0.265
0.301
0.282
0.315
0.610
GPT-5.5
0.446
0.604
0.513
0.420
0.542
0.311
0.343
0.326
0.336
0.702
Qwen3.5-Plus
0.468
0.571
0.514
0.441
0.471
0.288
0.316
0.301
0.351
0.678
Table 1. Annotation-based results for the visual setting (Sec. 4.2 ). Visual Set-1 evaluates general rough-cut, while Visual Set-2 evaluates requirement-constrained rough-cut. The best and second-best scores are shown in bold and underlined, respectively.
Model\Metric
Visual Set-1
Visual Set-2
Ordering
Redundancy
Overall
Human Score
Ordering
Requirement
Redundancy
Overall
Human Score
Gemini 2.5 Pro
3.33
2.52
3.11
3.46
3.66
4.31
2.97
3.60
3.76
Gemini 3.0 Pro
3.30
2.58
3.13
3.52
3.46
4.09
2.87
3.42
3.50
GPT-5.4
3.15
2.07
2.82
2.70
3.69
4.15
2.68
3.47
3.60
GPT-5.5
3.30
2.05
2.88
2.88
3.43
4.16
2.68
3.45
3.42
Qwen3.5-Plus
3.18
2.32
2.95
3.22
3.35
4.14
2.68
3.36
3.35
Table 2. Human-aligned agentic evaluation for rendered cuts (Sec. 4.3 ). The agentic evaluator scores ordering and redundancy control, together with requirement following for Visual Set-2 , on a 1–5 scale. Overall denotes the visual quality score assigned by jointly considering the evaluated dimensions. The best and second-best scores are shown in bold and underlined, respectively.
Model\Metric
IM
RM → R
D
P
R
F1
Succ.
P
R
F1
Gemini 2.5 Pro
0.430
0.891
0.581
0.491
0.707
0.609
0.654
Gemini 3.0 Pro
0.416
0.898
0.569
0.497
0.692
0.609
0.648
GPT-5.4
0.332
0.837
0.475
0.357
0.651
0.693
0.671
GPT-5.5
0.403
0.913
0.559
0.340
0.670
0.610
0.639
Qwen3.5-Plus
0.511
0.785
0.617
0.535
0.732
0.444
0.543
Table 3. Annotation-based results for the speech-driven setting (Sec. 4.4 ). The best and second-best scores are shown in bold and underlined, respectively.
Figure 3. Agreement between the agentic evaluator and professional assessments over four feedback rounds (Sec. 4.3 ).
Input
Visual Set-1
Visual Set-2
Precision
Recall
F1
Avg. IoU
Kendall’s τ
Precision
Recall
F1
Avg. IoU
Kendall’s τ
Time + Frame + Desc
0.490
0.568
0.526
0.440
0.565
0.345
0.342
0.343
0.358
0.692
Time + Frame
0.474
0.566
0.516
0.440
0.518
0.348
0.340
0.344
0.365
0.474
Time + Desc
0.480
0.556
0.515
0.450
0.556
0.295
0.303
0.299
0.358
0.618
Table 4. Ablation results of input components for the Cut-Planning Agent (Sec. 4.5 ). Time, Frame, and Desc denote timestamps, representative key frames, and segment descriptions, respectively. We use Time + Frame + Desc as the default setting.
FPS
Visual Set-1
Visual Set-2
Precision
Recall
F1
Avg. IoU
Kendall’s τ
Precision
Recall
F1
Avg. IoU
Kendall’s τ
0.5
0.466
0.537
0.499
0.436
0.547
0.284
0.289
0.286
0.348
0.651
1.0
0.490
0.568
0.526
0.440
0.565
0.345
0.342
0.343
0.358
0.692
2.0
0.486
0.576
0.527
0.445
0.614
0.357
0.364
0.360
0.357
0.732
Table 5. Ablation results of FPS in the Video-Understanding Agent (Sec. 4.5 ). We use 1 FPS as the default frame sampling rate.
Strategy\Metric
IM
RM → R
D
P
R
F1
Succ.
P
R
F1
CoT zero-shot
0.416
0.898
0.569
0.497
0.692
0.609
0.648
CoT one-shot
0.385
0.910
0.542
0.491
0.671
0.688
0.680
w/o CoT zero-shot
0.466
0.831
0.597
0.451
0.693
0.388
0.497
w/o CoT one-shot
0.512
0.741
0.605
0.398
0.702
0.332
0.451
Table 6. Ablation results of reasoning strategies for the Transcript-Cleanup Agent (Sec. 4.5 ). We use CoT zero-shot as the default setting.