AVENUE: Audio-Video EditiNg Understanding and Evaluation

Hayeon Kim; Yoojin Jang; Jaejun Yoo

cs.MMAug 31, 2026

AVENUE: Audio-Video EditiNg Understanding and Evaluation

Authors: Hayeon Kim, Yoojin Jang, Jaejun Yoo

Organizations: Ulsan National Institute of Science and Technology (UNIST)

Abstract

Audio-video (AV) editing aims to modify audio and video content according to a target prompt. Unlike single-modality editing, AV editing requires models to infer a modality-selective edit scope from the prompt alone: determining not only what should change, but also which modality should be preserved. Faithfully evaluating such models therefore requires both (i) benchmarks that span diverse edit types and modality categories, and (ii) evaluation that is itself modality-aware and sample-specific. However, existing AV editing benchmarks provide limited coverage of edit types and modality combinations, while current evaluation systems are often modality-blind and sample-agnostic, making it difficult to assess whether models faithfully preserve the unintended modality. To address these gaps, we introduce AVENUE, Audio-Video EditiNg Understanding and Evaluation, comprising two contributions: (1) a benchmark of 1,291 source clips and 7,957 editing instructions across audio-targeted, video-targeted, and AV-coupled edit types, curated and human-verified from VGGSound; and (2) a sample-specific, modality-aware evaluation framework that specifies, for each sample, both the intended change and the content that must remain intact. We evaluate representative AV editing models spanning three editing paradigms : joint, sequential, and separate, providing the first systematic analysis of modality-selectivity across paradigms. Our findings reveal a fundamental open challenge: when editing one modality, existing models frequently induce unintended changes in the other, regardless of paradigm. AVENUE provides a benchmark and modality-aware evaluation framework to drive progress toward more controllable AV editing models. Our dataset is publicly available on Hugging Face: https://huggingface.co/datasets/AVENUE-dataset/AVENUE.

Figures & tables

Figure 1: Overview of AVENUE. (a) AVENUE evaluates this capability through a modality-aware and sample-specific framework with four metrics. (b) Existing AV editing models—joint, sequential, and separated—struggle to infer modality-selective edit-scope from the prompt alone.

Benchmark	# Editing Instances	Edit Types	Edit Modality	Data Access
OAVE [ 20 ]	1,100	add, adjust	AV	✗
Object-AVEdit [ 27 ]	-	add, remove, replace	AV	✗
VGG-Edit [ 21 ]	450	add, remove, replace	AV	✗
AvED [ 1 ]	110	replace	AV	✓
AVENUE	7,957	12 edit-types (add, remove, replace, change …)	Audio Video AV	✓

Table 1: (Left) Comparison with existing audio-video editing benchmarks. We compare each benchmark in terms of the number of editing instances, supported edit types, edit modality, and data accessibility. A, V, and AV denote audio-targeted, video-targeted, and audio-video coupled editing, respectively. (Right) AVENUE taxonomy. AVENUE defines a three-part edit taxonomy: Ⓐ audio-targeted , Ⓑ video-targeted , and Ⓒ av-coupled edits . The outer sectors further categorize the 12 fine-grained editing types considered across the three taxonomy groups. The blue -highlighted portion denotes AvED’s coverage, primarily sounding-object replacement.

Figure 2: AVENUE Bench. The pipeline consists of three components: (a) category-aware instruction generation using Gemini-3-flash with human verification, producing structured edit prompts across 12 taxonomy categories spanning audio-targeted, video-targeted, and AV-coupled edits; (b) instruction examples with explicit target prompt; and (c) MLLM-as-a-Judge evaluation via QWEN3-Omni, assessing four dimensions—Edit Accuracy, Modality Selectivity, Perceptual Quality, and AV Consistency.

Model Type	Model		Video Preservation $\uparrow$		Audio Preservation $\uparrow$
	Video	Audio	MS (Video)	V-CLIP	MS (Audio)	CLAP
Separated	RAVE $+$ SDEdit		70.7	0.956	48.3	0.721
	RAVE $+$ ZETA				76.3	0.859
Sequential	RAVE $\rightarrow$ CAVE				58.7	0.736
	Source $v\rightarrow$ CAVE		–	–	56.3	0.731
Joint	AvED		76.3	0.995	85.0	0.886

Table 2: Self-prompt preservation across AV editing model types. We evaluate preservation under a null-edit condition where the target prompt matches the source prompt. We report our MLLM-based Modality Selectivity (MS) together with representative embedding-based preservation metrics [ 31 , 23 ] . Higher scores indicate better preservation. Shared video outputs (RAVE) are reported once across corresponding pipelines, and “–” denotes metrics that are not applicable. Bold and underlined indicate the best and second-best results, respectively.

Figure 3: Evaluation of AV-coupled editing under Cat.C. All scores are reported out of 100, where higher is better. (a) Edit Accuracy (EA) across five sub-categories of Category C. (b) Comparison of joint, sequential, and separate editing paradigms across EA, Perceptual Quality (PQ), and AV Consistency (AV-C). We report results on the full Category C benchmark as well as two representative cases: Scene Change (C1) and Sounding Object Replacement (C3).

Model Type	Model			Edit Accuracy	Percept Quality	Modal Selectivity (Audio)
Model Type	Video		Audio	(Cat A) $\uparrow$	(Cat A) $\uparrow$	(Cat B) $\uparrow$
Separate	RAVE	$+$	ZETA	79.7	88.2	93.0
Separate	RAVE	$+$	SDEdit	78.7	82.4	51.4
Sequential	RAVE	$\rightarrow$	CAVE	79.6	89.3	58.0
	Source $v$	$\rightarrow$	CAVE	79.8	89.7	–
	RAVE	$\rightarrow$	MMAudio	75.8	89.5	39.7

Table 3: Audio modality evaluation overview. All scores are scaled to 100. Edit Accuracy and Perceptual Quality are measured on Cat.A (audio-targeted) samples, and Modality Selectivity on Cat.B (video-targeted) samples to assess unintended audio change. Higher is better for all metrics.

MLLM-Judge	Pairwise agreement $\uparrow$	Leave-one-out $\uparrow$	Kendall $\tau$ -b $\uparrow$
Human reference	71.1%	79.2%	0.546
Qwen3-Omni [ 29 ]	67.0%	72.5%	0.353
Gemini-3.1-pro [ 32 ]	66.2%	73.6%	0.440
Gemini-3.6-Flash [ 33 ]	64.0%	67.7%	0.359
Qwen2.5-Omni [ 34 ]	60.4%	64.8%	0.092
Audio-Visual-Flamingo [ 35 ]	56.8%	58.7%	0.117

Table 4: Judge agreement with human judgments. We report individual-level pairwise agreement, leave-one-out agreement, and Kendall’s

\tau_{b}

for five automated judges. Human reference denotes the corresponding human–human agreement, averaged across the five judge-specific human references. Bold and underline denote the best and second-best results among automated judges, respectively.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Metric	AVENUE( $N{=}110$ )	AvED ( $N{=}110$ )
TTR ( $\uparrow$ )	0.305	0.078
distinct-2 ( $\uparrow$ )	0.640	0.127
Unique-prompt ratio ( $\uparrow$ )	$\sim$ 100%	20.9%
self-BLEU ( $\downarrow$ )	0.240	0.867
Mean pairwise cosine ( $\downarrow$ )	0.160	0.235

Appendix

Table A 1 : Size-controlled comparison of editing-prompt diversity between AVENUE and AvED. AVENUE is randomly subsampled to

N=110

to match the size of AvED, and results are averaged over 200 random subsets.

\uparrow

\downarrow

indicate whether a higher or lower value corresponds to greater diversity.

Panel of Figure 9 — Figure A 1 : Word clouds of content words added in editing prompts, grouped by editing modality: audio-targeted (A), visual-targeted (B), and AV-coupled (C). Words are extracted by aligning source descriptions and target prompts via sequence matching, then filtered to retain nouns and adjectives targeted, excluding domain-generic terms.

Figure A 2 : Overview of AVENUE Data Curation (1) VGGSound clips are categorized into 8 semantic categories. (2) A two-stage filtering pipeline retains high-quality, edit-suitable clips based on audio-visual alignment and edit compatibility criteria (semantic consistency, ROI consistency, person density, non-static). (3) Human annotators verify the retained clips and confirm which edit types are applicable per clip, producing a curated set with category-aware edit type assignments. (4) Valid clips are paired with structured instruction templates to generate per-sample annotations including target prompt, edit sub-type, what changed and what preserved, which are again verified by human annotators.

Figure A 3 : Statistics for semantic categories

Figure A 4 : Illustration of the user interface used for human verification.

Group	Edit Type	Instruments	Animals	Human	Vehicles	Domestic	Sports	Nature	Explosions
A. Audio-targeted	A1: Non-visual Sound Addition	–	✓	–	–	✓	✓	✓	–
	A2: Non-visual Sound Removal	✓	✓	–	✓	✓	✓	✓	–
	A3: Non-visual Sound Replacement	✓	✓	–	–	–	–	✓	–
	A4: Sound Intensity/Pitch Change	✓	–	–	✓	–	–	✓	✓
B. Video-targeted	B1: Attribute/Color Change	✓	✓	✓	✓	✓	✓	–	–
	B2: Style Transfer	✓	✓	✓	✓	–	✓	✓	–

Appendix

Table A 2 : Category-aware editing taxonomy. ✓indicates that the edit type is applicable to the given category. Edit types are grouped by modality scope: A (Audio-targeted), B (Video-targeted), and C (AV-coupled).

Figure A 5 : User interface for the human–judge alignment study.

Model Type	Model		Video Preservation				Audio Preservation
	Video	Audio	V-CLIP $\uparrow$	IB $\uparrow$	DINO $\uparrow$	LPIPS $\downarrow$	CLAP $\uparrow$	IB $\uparrow$	LPAPS $\downarrow$
Separated	RAVE $\hskip 8.50012pt+\hskip 8.50012pt$ SDEdit		0.956	0.911	0.939	0.204	0.721	0.534	4.111
	RAVE $\hskip 8.50012pt+\hskip 8.50012pt$ ZETA						0.859	0.757	2.765
Sequential	RAVE $\hskip 8.50012pt\rightarrow\hskip 8.50012pt$ CAVE						0.736	0.589	4.323
	Source $v$ $\hskip 8.50012pt\rightarrow\hskip 8.50012pt$ CAVE		-	-	-	-	0.731	0.580	4.320
Joint	AvED		0.995	0.939	0.967	0.110	0.886	0.846	3.126

Appendix

Table A 3 : Self-prompt preservation evaluation across AV editing model types. We evaluate source preservation when the target prompt is identical to the source prompt and therefore requires no semantic edit. Higher V-CLIP [ 31 ] , ImageBind [ 24 ] , DINO [ 42 ] , and CLAP [ 23 ] scores and lower LPIPS [ 43 ] and LPAPS scores indicate stronger preservation. The RAVE + SDEdit, RAVE + ZETA, and RAVE → CAVE pipelines share the same RAVE-generated video output; therefore, their video-preservation scores are identical and are reported once using merged cells spanning the corresponding rows. Audio-preservation scores are evaluated separately for each complete pipeline. For Source → CAVE, the original source video is directly retained without video editing, and "–" therefore denotes that video-preservation metrics are not separately evaluated. Bold and underlined values indicate the best and second-best results, respectively.

Category	Metric	Joint	Separated
Audio-targeted	EA (non-cond.) $\uparrow$	63.4	81.6
	EA $\uparrow$	70.0	87.0
	CMTG $\uparrow$	28.6	24.6
Video-targeted	EA (non-cond.) $\uparrow$	72.6	77.4
	EA $\uparrow$	74.0	68.0
	CMTG $\uparrow$	22.0	20.0

Appendix

Table A 4 : Cross-modality-conditioned targeted editing (pilot study). Each edit is anchored to a trigger event in the untargeted modality. EA (non-cond.) reports Edit Accuracy on the original, non-conditioned prompts of the same clips as a reference point, while EA is measured on the cross-modal-conditioned prompts. CMTG denotes Cross-Modal Temporal Grounding. All scores are scaled to 100; higher is better. Bold indicates the better result between the two paradigms.

Figure A 6 : Mean Spearman $\rho$ between our MLLM-based evaluation scores and embedding based metrics across all evaluation dimensions and edit categories. Error bars indicate the min–max range across generation models. All reported correlations are statistically significant $(p<0.05)$ . Higher $\rho$ indicates stronger agreement between MLLM-based scores and embedding-based metrics. The suffixes ta , tv , and av denote target text–target audio, target text–target video, and target audio–target video alignment, respectively. The single-modality suffixes a and v instead denote preservation metrics, measuring source audio–target audio and source video–target video alignment, respectively.

Figure A 7 : Category C overall performance

Model	Scene change (C1)	Action change (C2) $\uparrow$	Object replacement (C3) $\uparrow$	Mood change (C4) $\uparrow$	Speed change (C5) $\uparrow$
RAVE $\quad+\quad$ ZETA	64.6	59.7	51.5	70.2	77.9
RAVE $\quad+\quad$ SDEdit	63.4	62.1	54.1	73.9	82.9
CAVE	64.4	61.4	48.3	71.4	80.8
AVIEdit	39.2	60.4	50.5	39.2	77.0
AvED	48.9	59.4	53.5	56.5	70.8

Appendix

Table A 5 : AV-coupled (C) Edit Accuracy. Bold denotes the best result in each column.

Model Type	Model			Edit Accuracy	Percept Quality	Modal Selectivity (Video)
Model Type	Video		Audio	(Cat B) $\uparrow$	(Cat B) $\uparrow$	(Cat A) $\uparrow$
Separate	RAVE	$+$	ZETA	79.1	69.5	86.2
Sequential	ZETA	$\rightarrow$	AVIEdit	65.0	73.9	84.6
Joint	AvED			77.3	53.2	87.5

Appendix

Table A 6: Video modality evaluation overview. All scores are scaled to 100. Edit Accuracy and Perceptual Quality are measured on Cat.B (video-targeted) samples, and Modality Selectivity on Cat.A (audio-targeted) samples to assess unintended video change. Higher is better for all metrics.

Model	Paradigm	A(Audio)	B(Video)	C(AV)
AvED [ 1 ]	Joint	✓	✓	✓
CAVE [ 19 ]	Sequential	✓	✓ ∗	✓
MMAudio [ 30 ]	Sequential	✓	✓ ∗	✓
RAVE $\hskip 8.50012pt+\hskip 8.50012pt$ SDEdit [ 14 , 13 ]	Separate	✓	✓ ∗	✓
RAVE $\hskip 8.50012pt+\hskip 8.50012pt$ ZETA [ 14 , 12 ]	Separate	✓	✓ ∗	✓

Appendix

Table A 7 : Existing AV editing models evaluated across our proposed edit taxonomies. ∗ Video editing via shared RAVE [ 14 ] backbone.

Video Editing Benchmarks

Video editing benchmarks measure whether systems follow edit instructions while preserving content, temporal consistency, and audiovisual quality.

See all 32 papers

Jul 17, 2026

AVE-Compass: Towards Holistic Evaluation for Audio-Video Editing Abilities

While instruction-based video editing has advanced rapidly, real-world videos contain tightly coupled audio and visual signals, and editing one modality often requires coordinated changes in the other. Existing benchmarks primarily evaluate visual transformations on silent clips or isolated audio editing, leaving complex audio-visual editing and cross-modal consistency underexplored. We introduce AVE-Compass, a comprehensive benchmark with 145 curated source videos, 196 audio-visually coupled editing instructions, and 2,688 fine-grained checklist items. It evaluates Instruction Following, Fidelity Preserving, Realism, and Editing Intent through checklist-based MLLM judging and a dedicated realism rubric, complemented by automated cross-modal, video, and audio metrics. Extensive evaluation shows that state-of-the-art models still struggle to execute cross-modal instructions while preserving non-target content. We further propose AVE-Agent, a modular agent framework that decomposes complex instructions into dependent subtasks and iteratively improves editing results through self-reflection and evaluator feedback. AVE-Agent improves instruction execution, Fidelity Preserving, and audio-visual alignment in joint editing while maintaining competitive perceptual quality.

Yuqing Wen, Yukai Huang, Qianqian Xie +6

National University of Singapore · Beijing University of Posts and Telecommunications · NJU-LINK Team, Nanjing University +3

Video Editing Benchmarks Agentic Video Editing Video Editing Audio-Visual Alignment Instruction-Based Video Editing

Ask PDF

Oct 7, 2026

CrossEdit: Cross-Modal Training Enables Rich Audio-Visual Editing

Multimedia editing requires generative models to interpret complex, compositional instructions and modify only the desired elements across modalities, a capability largely beyond existing systems. Current approaches are typically limited to simple, single-attribute edits within one modality, since diverse, high-quality editing pairs are hard to obtain, especially for cross-modal tasks where aligned audiovisual (AV) data is scarce. We present CrossEdit, a unified omni-modal editing model for images, audio, and video that performs AV movie scene edits zero-shot through cross-modal transfer. Our key observation is that complex instruction-following, once learned in any modality, generalizes to others. We exploit the additive nature of acoustic signals to procedurally generate large-scale audio editing pairs with compositional instructions, and show that fine-tuning on this synthetic data and self-supervised AV masked reconstruction, alongside a targeted set of curated cross-modal tasks, induces instruction-following that transfers zero-shot to unseen modality and instruction combinations. To evaluate this capability, we release CrossEditBench, a human-annotated benchmark of AV edits on movie scenes, and propose AV-FES, a metric that jointly scores instruction following and consistency. We show that our proposed techniques improve performance on zero-shot AV movie scene editing, while maintaining or improving performance on lip-synced speech editing and standard image, video, and audio editing benchmarks. Demos are available at https://wanchichen.github.io/crossedit/.

William Chen, Prem Seetharaman, Ke Chen +10

Adobe Research · Carnegie Mellon University

Instruction-Based Video Editing Unified Multimodal Models Cross-Modal Learning Multimodal Instruction Tuning Video Editing Benchmarks

Ask PDF

Jun 2, 2026

JAVEDIT: Joint Audio-Visual Instruction-Guided Video Editing with Agentic Data Curation

While instruction-based video editing has seen significant progress, joint audio-visual editing remains constrained by the absence of dedicated datasets and benchmarks. To bridge this gap, we present JAVEdit-100k, the first large-scale, high-quality dataset tailored for instruction-guided joint audio-visual editing. Focusing on human-centric videos, JAVEdit-100k comprises approximately 100K editing triplets spanning five distinct categories, including subject editing and speech editing. This dataset is rigorously constructed via four meticulously designed generation pipelines, seamlessly paired with an agent-in-the-loop quality control mechanism. Furthermore, to address the lack of standardized evaluation within the field, we introduce JAVEditBench, a comprehensive benchmark featuring curated source videos and human-aligned instructions across all editing categories. Finally, we propose JAVEdit, a pioneering baseline model for instruction-guided joint audio-visual editing. Experiments show that \model\ outperforms all baselines on five of six evaluation metrics.

Yinan Chen, Chuming Lin, Zhennan Chen +12

1Zhejiang University · 2Tencent Youtu Lab · 3Nanjing University +3

Video Editing Benchmarks Training Data Curation Video Editing Instruction-Based Video Editing

Ask PDF

Abstract

Video Editing Benchmarks

Explore similar work

AVE-Compass: Towards Holistic Evaluation for Audio-Video Editing Abilities

CrossEdit: Cross-Modal Training Enables Rich Audio-Visual Editing

JAVEDIT: Joint Audio-Visual Instruction-Guided Video Editing with Agentic Data Curation