Edit in Sync: Unified End-to-End Audio-Visual Editing for Human-Centric Videos

Ziye Yuan; Minghong Cai; Xu He; Ruchang Yao; Xiao Fu; Kaixin Ding; Ming Zhang; Xiangyu Du

cs.CVOct 20, 2026

Edit in Sync: Unified End-to-End Audio-Visual Editing for Human-Centric Videos

Authors: Ziye Yuan, Minghong Cai, Xu He, Ruchang Yao, Xiao Fu, Kaixin Ding, Ming Zhang, Xiangyu Du,

Organizations: Peking University · Kling AI, Kuaishou Technology · The Chinese University of Hong Kong · Tsinghua University ·

Abstract

Human-centric audio-visual editing requires precise alignment of lip movements, gestures, and expressions with speech. Existing methods process inputs through independent video and audio pipelines, preventing cross-modal interaction between visual editing and audio dubbing. Many of these pipelines further rely on mask-based inpainting. As a result, they suffer from poor lip-audio synchronization, cumulative pipeline errors, and an inherent mask-size dilemma. We present AVE, the first unified end-to-end framework that jointly performs reference-guided person swapping, insertion, and deletion across both audio and video modalities without requiring spatial masks. To overcome data scarcity, we design a synthesis pipeline and introduce AVE-Dataset, the first audio-visual editing dataset to provide 100K pairs with appearance and voice references. By combining Synthetic-to-Real reconstruction with preliminary audio warm-up, our framework transcends synthetic pipeline limits and mitigates audio-video token imbalance. AVE outperforms modality-separate audio-visual editing pipelines in editing realism, temporal consistency, and audio-visual alignment.

Figures & tables

Figure 2: Separate editing pipelines versus AVE

Figure 3: Automated data construction pipeline of AVE-Dataset. The workflow consists of three phases: (1) raw corpus collection and quality pre-filtering, (2) video-audio synthesis for editing pair creation, (3) identity retrieval and multimodal augmentation for cross-pair reference construction.

Figure 4: Overview of the AVE framework. To prevent token ambiguity, our Modality-Aware Sync-RoPE introduces role-specific 3D coordinate offsets that explicitly distinguish source, target, and reference tokens while maintaining temporal correspondence along a shared timeline.

Figure 5: Qualitative comparisons across person swapping , insertion , and removal . Please zoom in for more details.

Swap
	VBench (%)					Audio Quality & AV Sync			Ref Similarity			BG Fidelity
Task / Method	Subj. Cons. $\uparrow$	BG Cons. $\uparrow$	Temp. Flick. $\uparrow$	Motion Sm. $\uparrow$	Imaging Q. $\uparrow$	PQ $\uparrow$	Sync-C $\uparrow$	Sync-D $\downarrow$	CAM++ $\uparrow$	Arc $\uparrow$	CLIP $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$
VACE+PA	95.26	94.30	96.48	98.11	68.13	6.586	4.86	9.29	0.665	0.614	0.705	35.37	0.0481
Wan2.2+PA	94.09	96.12	97.76	97.71	66.41	6.586	5.02	8.76	0.665	0.634	0.710	33.27	0.0505
MoCha+PA	96.52	93.16	97.97	97.30	65.11	6.586	5.05	8.72	0.665	0.639	0.712	32.42	0.0584
InstructAV2AV	95.86	93.88	95.56	98.10	66.60	6.255	3.87	9.65	–	–	–	31.84	0.0640

Table 1: Quantitative comparison on AVE-Dataset-Bench across three editing tasks. Reconstruction metrics are computed over the background region. Best results are shown in bold. “+PA” denotes cascaded baselines paired with our audio post-processing pipeline.

Swap
	VBench (%)					Audio Quality & AV Sync
Task / Variant	Subj. Cons. $\uparrow$	BG Cons. $\uparrow$	Temp. Flick. $\uparrow$	Motion Sm. $\uparrow$	Imaging Q. $\uparrow$	PQ $\uparrow$	Sync-C $\uparrow$	Sync-D $\downarrow$
w/o Audio Pretrain	96.60	95.34	97.94	97.95	68.58	6.107	4.71	9.25
w/o Multistage	95.90	94.57	97.28	98.21	67.88	6.580	5.28	8.66
Ours	97.35	95.69	97.67	98.54	69.74	6.709	5.39	8.53
Delete

Table 2: Ablation study on AVE-Dataset-Bench across tasks. Best results are shown in bold.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Configuration	Stage 0 Audio Warm-up	Stage 1 Bidirectional Joint Adaptation	Stage 2 Quality Refinement
Training data	200K audio examples	120K editing pairs	70K editing pairs
Task mixture	30% voice conversion; 40% text-to-speech; 30% vocal separation	40K Swap ; 37K Insert ; 41K Delete	25K Swap ; 15K Insert ; 30K Delete
Reference source	–	Retrieved + generated	Retrieved only
VOR real-captured subset	–	20K Delete	20K Delete
Optimizer	AdamW	AdamW	AdamW
Learning rate	$5\times 10^{-5}$	$1\times 10^{-5}$	$1\times 10^{-5}$

Appendix

Table 3: Stage-wise training configuration. Dataset sizes and task mixtures are approximate. The VOR subset is included in, not added to, the reported Delete counts.

Swap
	VBench (%)					Audio Quality & AV Sync
Task / Variant	Subj. Cons. $\uparrow$	BG Cons. $\uparrow$	Temp. Flick. $\uparrow$	Motion Sm. $\uparrow$	Imaging Q. $\uparrow$	PQ $\uparrow$	Sync-C $\uparrow$	Sync-D $\downarrow$
w/o Modality-Aware Sync-RoPE	96.46	96.01	95.38	98.43	60.96	6.050	3.14	9.84
Ours	97.35	95.69	97.67	98.54	69.74	6.709	5.39	8.53
Delete
w/o Modality-Aware Sync-RoPE	94.24	93.73	96.09	97.80	62.47	6.235	–	–

Appendix

Table 4: Ablation study of Modality-Aware Sync-RoPE on AVE-Dataset-Bench across tasks. Best results are shown in bold.

Video Editing

Video editing uses AI to alter, compose, relight, or reorganize moving footage while controlling how changes persist across frames.

See all 178 papers

SpongeBob: Sync-Aware Harmonious Audio-Visual Generative Editing
May 24, 2026Sen Liang, Cong Wang, Fengbin Guan +6Video Editing Audio-Visual Synchronization
University of Science and Technology of China · 2Tencent Hunyuan
CrossEdit: Cross-Modal Training Enables Rich Audio-Visual Editing
Oct 7, 2026William Chen, Prem Seetharaman, Ke Chen +10Instruction-Based Video Editing Unified Multimodal Models
Adobe Research · Carnegie Mellon University
AVE-Compass: Towards Holistic Evaluation for Audio-Video Editing Abilities
Jul 17, 2026Yuqing Wen, Yukai Huang, Qianqian Xie +6Video Editing Benchmarks Agentic Video Editing
National University of Singapore · Beijing University of Posts and Telecommunications · NJU-LINK Team, Nanjing University +3

May 24, 2026J/K move · Enter open · S save

SpongeBob: Sync-Aware Harmonious Audio-Visual Generative Editing

Sen Liang, Cong Wang, Fengbin Guan, Zhentao Yu+5

University of Science and Technology of China · 2Tencent Hunyuan

Video Editing Audio-Visual Synchronization Audio-Visual Alignment

Open paper AskPDF arXiv

Visual and acoustic events in the physical world are inherently coupled, yet existing video editing methods typically adopt decoupled pipelines, lacking bidirectional modality interaction. This results in two key limitations: (i) audio-visual desynchronization and (ii) contextual conflicts between generated audio and preserved content. To address these, we propose SpongeBob, the first end-to-end audio-visual joint editing framework featuring bidirectional cross-modal interaction. For synchronization, a Sync-Aware Mechanism aligns visual edits with sound events via bidirectional attention, temporal alignment, and spatial constraints. For contextual consistency, a Context-Aware Module leverages acoustic and visual context attention to prevent semantic clashes. Additionally, we introduce Sync-Preserving Training and Guidance (SPTG) to enhance alignment without degrading quality. Due to the scarcity of paired data, we construct a scalable data pipeline and a large-scale subject-level dataset. We also propose SpongeBob-Bench for systematic evaluation. Experiments show SpongeBob significantly outperforms existing baselines, improving Sync-C by 30% and Ctx-F1 by 12.5%. Our project page is available at: https://hy-spongebob.github.io/.

Abstract

Video Editing

Explore similar work

SpongeBob: Sync-Aware Harmonious Audio-Visual Generative Editing

CrossEdit: Cross-Modal Training Enables Rich Audio-Visual Editing

AVE-Compass: Towards Holistic Evaluation for Audio-Video Editing Abilities