AutoCut: End-to-end advertisement video editing based on multimodal discretization and controllable generation

Milton Zhou; Sizhong Qin; Yongzhi Li; Quan Chen; Peng Jiang

cs.CVMar 30, 2026

AutoCut: End-to-end advertisement video editing based on multimodal discretization and controllable generation

Authors: Milton Zhou, Sizhong Qin, Yongzhi Li, Quan Chen, Peng Jiang

Organizations: Tsinghua University · Kuaishou Technology

Abstract

Short-form videos have become a primary medium for digital advertising, requiring scalable and efficient content creation. However, current workflows and AI tools remain disjoint and modality-specific, leading to high production costs and low overall efficiency. To address this issue, we propose AutoCut, an end-to-end advertisement video editing framework based on multimodal discretization and controllable editing. AutoCut employs dedicated encoders to extract video and audio features, then applies residual vector quantization to discretize them into unified tokens aligned with textual representations, constructing a shared video-audio-text token space. Built upon a foundation model, we further develop a multimodal large language model for video editing through combined multimodal alignment and supervised fine-tuning, supporting tasks covering video selection and ordering, script generation, and background music selection within a unified editing framework. Finally, a complete production pipeline converts the predicted token sequences into deployable long video outputs. Experiments on real-world advertisement datasets show that AutoCut reduces production cost and iteration time while substantially improving consistency and controllability, paving the way for scalable video creation.

Figures & tables

Figure 1: Overview of the AutoCut framework. Low–fps frames are tokenized for efficient multimodal reasoning, while high–fps frames are kept for accurate visual matching and clip retrieval. Given optional inputs, AutoCut predicts video, text, and audio tokens, which are decoded or retrieved to compose the final advertisement video.

Figure 2: Overview of the proposed AutoCut framework. Multimodal tokenization converts scripts, frames, and audio into unified discrete tokens, which are organized into an alignment input sequence. Stage 1 performs multimodal alignment on large-scale data to align the added token embeddings with the LLM backbone. Stage 2 applies task-specific SFT for video selection, video sorting, script generation, and BGM selection.

Figure 3: Dataset statistics for the multimodal alignment (top) and SFT (bottom) stages. The alignment dataset is substantially larger but exhibits more diverse and irregular distributions, while the SFT dataset is smaller yet more balanced and of higher annotation quality.

Model	Video			Script		Audio
Model	CSA	CRA	VSC	SQ	WCD $\downarrow$	MSS
Qwen3–8B [ 32 ] (Caption)	0.1374	0.01648	0.9308	79.9947	5.2648	–
Qwen3–8B (Caption + SFT)	0.5687	0.03022	1.1227	59.1592	6.8230	–
Qwen3–32B [ 32 ] (Caption)	0.1731	0.01373	0.9342	80.7747	7.1023	–
Qwen2.5-VL–7B Instruct [ 4 ]	0.2418	0.01648	0.9649	79.4867	6.4709	–
Qwen2.5-VL–32B Instruct [ 4 ]	0.6648	0.02472	0.9980	78.3180	12.5071	–

Table 1: Quantitative results across all tasks. The best results are boldfaced and the second best results are underlined.

Method	CSA	CRA	VSC	SQ	WCD $\downarrow$
sft only	0.4780	0.08242	1.0043	83.1898	4.4346
emb+full+sft	0.7170	0.05770	0.9669	78.9644	4.4984
emb+sft (ours)	0.6593	0.10714	1.0360	84.6255	3.0182

Table 2: Ablation on training settings.

Figure 4: User study results: win–loss ratios against GPT-4o across five evaluation dimensions.

RQ-VAE Configs	Video	Audio
Input feature size	128	2048
Quantization heads	8	8
Codebook size	256	256
Codebook dim	128	256
Shared codebook	False	False
Encoder MLP size	[512, 512]	[1024, 512]

Table 3: Hyperparameters for the video and audio RQ-VAE quantizers.

Config	Alignment Stage	SFT Stage
Base model	Qwen3-8B	Aligned model
Template	qwen3	qwen3
Cutoff length	8096	4000
Packing	Enabled	Disabled
Per-device batch size	2	1
Grad. accumulation	1	1

Table 4: Hyperparameters used during the multimodal alignment and SFT stages.

Figure 5: Qualitative case studies of our automatic ad video editing pipeline. For each case (top: wireless earbuds, bottom: children’s face cream), we show the provided product information, selected video clips, and the aligned script sentences. Each frame strip visualizes the model’s clip selection and ordering, together with the corresponding sentence and clip-level timestamps, illustrating how the system produces a coherent, time-aligned ad from raw footage and product metadata.

Figure 6: Additional human study comparing AutoCut with Gemini 3 on 50 advertisement videos. A large proportion of cases are judged as Tie across all five dimensions, indicating that AutoCut achieves performance highly comparable to this stronger and more recent baseline.

Figure 7: Overview of our dataset construction pipeline. Top : multimodal alignment data construction. Raw ad videos are paired with sentence-level scripts and product information, then segmented into short clips using punctuation-aligned timestamps, producing aligned video tokens and audio tokens. Bottom-left : data filtering. We keep videos whose clip number and duration fall within preset ranges and whose visual–script relevance score (estimated by Qwen2.5-VL-32B) exceeds an 80% threshold. Bottom-right : task-oriented SFT dataset construction. From the filtered aligned data we instantiate four supervision signals: video selection, video sorting, script generation, and BGM selection, each conditioned on product information and/or the full script.

Criterion	Positive indicators	Negative indicators
Visual Smoothness	• Transitions feel natural and continuous. • No obvious stutter, jitter, or dropped frames. • Motion and camera movement appear stable.	• Noticeable lag, stutter, or frame skipping. • Abrupt cuts that break visual flow. • Shaky or inconsistent motion that distracts the viewer.
Logical Consistency	• Shot order follows a clear story or explanation. • The same product and context are maintained throughout. • Scene changes support a coherent narrative or demo.	• Scenes feel random or out of order. • Products or contexts change without explanation. • Transitions break the story or confuse the viewer.
Script–Visual Alignment	• The script accurately describes what appears on screen. • Key selling points are shown at the right visual moments. • No hallucinated objects, brands, or functions.	• The script mentions things not visible in the video. • Important visual content is not reflected in the script. • Timing between narration and visuals is clearly off.
BGM–Visual Compatibility	• Music tempo matches motion and editing rhythm. • Overall mood fits the product and scenario. • Volume and energy support, rather than overshadow, the content.	• Music is too fast/slow compared with visual rhythm. • Style or emotion of the music feels inappropriate. • Sudden changes in volume or style that distract the viewer.
Overall Attractiveness	• The ad is engaging and easy to follow. • Product value and advantages are clearly conveyed. • You would be more willing to consider buying the product.	• The ad feels dull, confusing, or unconvincing. • Product value is unclear or poorly communicated. • You would not be motivated to learn more or purchase.

Table 5: Human evaluation criteria and decision guidelines used in the pairwise comparison study.

Video Editing

Video editing uses AI to alter, compose, relight, or reorganize moving footage while controlling how changes persist across frames.

See all 178 papers

CutClaw: Agentic Hours-Long Video Editing via Music Synchronization
Mar 31, 2026Shifang Zhao, Yihan Hu, Ying Shan +2Agentic Video Editing Video Editing
Beijing Jiaotong University · GVC Lab, Great Bay University · ARC Lab, Tencent
Auto-Director: Leveraging Multi-Agent Collaboration for Autonomous Video Editing
Oct 20, 2026Yuxia Chen, Wenbo Zhu, Yangguang Ji +3Video Editing Agentic Video Editing
University of Electronic Science and Technology of China, China · University of California, Berkeley · Nanyang Technological University +3
VideoAgent: All-in-One Framework for Video Understanding and Editing
Jun 22, 2026Hengji Zhou, Lingxuan Huang, Jian Wang +4Video Editing Benchmarks Agentic Video Editing
1Harbin Institute of Technology, Shenzhen · 2South China University of Technology · 3The University of Hong Kong +1

Mar 31, 2026J/K move · Enter open · S save

CutClaw: Agentic Hours-Long Video Editing via Music Synchronization

Shifang Zhao, Yihan Hu, Ying Shan, Yunchao Wei+1

Beijing Jiaotong University · GVC Lab, Great Bay University · ARC Lab, Tencent

Agentic Video Editing Video Editing Audio-Visual Alignment

Open paper AskPDF arXiv

Editing the video content with audio alignment forms a digital human-made art in current social media. However, the time-consuming and repetitive nature of manual video editing has long been a challenge for filmmakers and professional content creators alike. In this paper, we introduce CutClaw, an autonomous multi-agent framework designed to edit hours-long raw footage into meaningful short videos that leverages the capabilities of multiple Multimodal Language Models~(MLLMs) as an agent system. It produces videos with synchronized music, followed by instructions, and a visually appealing appearance. In detail, our approach begins by employing a hierarchical multimodal decomposition that captures both fine-grained details and global structures across visual and audio footage. Then, to ensure narrative consistency, a Playwriter Agent orchestrates the whole storytelling flow and structures the long-term narrative, anchoring visual scenes to musical shifts. Finally, to construct a short edited video, Editor and Reviewer Agents collaboratively optimize the final cut via selecting fine-grained visual content based on rigorous aesthetic and semantic criteria. We conduct detailed experiments to demonstrate that CutClaw significantly outperforms state-of-the-art baselines in generating high-quality, rhythm-aligned videos. The code is available at: https://github.com/GVCLab/CutClaw.

Abstract

Video Editing

Explore similar work

CutClaw: Agentic Hours-Long Video Editing via Music Synchronization

Auto-Director: Leveraging Multi-Agent Collaboration for Autonomous Video Editing

VideoAgent: All-in-One Framework for Video Understanding and Editing