PRIMEdit: Probability Redistribution for Instance-Aware Multi-Object Video Editing With a New Benchmark Dataset

Samuel Teodoro; Agus Gunawan; Soo Ye Kim; Jihyong Oh; Munchurl Kim

cs.CVDec 17, 2024

PRIMEdit: Probability Redistribution for Instance-Aware Multi-Object Video Editing With a New Benchmark Dataset

Authors: Samuel Teodoro, Agus Gunawan, Soo Ye Kim, Jihyong Oh, Munchurl Kim

Organizations: KAIST · Adobe Research · Chung-Ang University

Abstract

Recent Artificial Intelligence (AI)-based video editing has enabled users to edit videos through simple text prompts, significantly simplifying the editing process. However, recent zero-shot video editing techniques focus on global or single-object edits, which can lead to unintended changes in other parts of the video. When editing multiple objects only within their localized regions, existing methods face challenges, such as unfaithful editing, editing leakage, and lack of suitable evaluation datasets and metrics. To overcome these limitations, we propose $\textbf{P}$ robability $\textbf{R}$ edistribution for $\textbf{I}$ nstance-aware $\textbf{M}$ ulti-object Video $\textbf{Edit}$ ing ( $\textbf{PRIMEdit}$ ). PRIMEdit is a zero-shot framework that introduces two key modules: (i) Instance-centric Probability Redistribution (IPR) to ensure precise localization and faithful editing and (ii) Disentangled Multi-instance Sampling (DMS) to prevent editing leakage. Additionally, we present the MIVE dataset for multi-instance video editing, including its subsets categorized by instance counts and sizes for controlled analysis across different levels of scene complexity. We also introduce the Cross-Instance Accuracy (CIA) Score that is firstly proposed to evaluate the editing leakage in multi-instance video editing tasks, which exhibits strong correlation with human evaluation as measured by Spearman's $ρ= 0.86$ . Our extensive qualitative, quantitative, and user study evaluations demonstrate that PRIMEdit significantly outperforms recent state-of-the-art methods in terms of editing faithfulness, accuracy, and leakage prevention, setting a new benchmark for multi-instance video editing.

Figures & tables

Fig. 1: Limitations of previous SOTA methods. (a) ControlVideo [ 15 ] relies on single global captions, and (b) GAV [ 20 ] depends on bounding box conditions that can sometimes overlap. Both are susceptible to unfaithful editing ( red arrow ) and attention leakage ( blue arrow ). Note that the colored words in Prompt correspond to the instances to be edited in Input .

Fig. 2: Given a video, instance masks, and target instance captions, our PRIMEdit enables faithful and disentangled edits in (a) single- and (b)-(c) multi-instance levels, as well as an applicability to more fine-grained (d) partial instance level without additional training. Unlike previous methods, our PRIMEdit does not rely on global edit captions, but leverages individual instance captions. Object masks are color-coded to match its corresponding edit caption.

Panel of Table 3 — TABLE I: Comparison of our PRIMEdit against previous multi-instance video editing state-of-the-art methods.

Fig. 3: The overall framework of our PRIMEdit. For a given $M$ multi-instances, each instance has a caption $c_{i}$ and instance masks $\boldsymbol{m_{i}}$ for editing. Our Disentangled Multi-instance Sampling (DMS, Sec. III-D ) consists of series noise sampling (SNS, yellow box), latent fusion (green box) to fuse different instance latents, re-inversion (pink box) to harmonize the latents after fusion, and parallel noise sampling (PNS, blue box). In addition, our Instance-centric Probability Redistribution (IPR, Sec. III-C ) provides better spatial control.

Symbol / Acronym	Description
PRIMEdit	Proposed multi-instance video editing framework.
LDM	Latent Diffusion Model.
$\mathcal{E}$ / $\mathcal{D}$	VAE encoder / decoder of the LDM.
$\epsilon_{\theta}$	U-Net noise prediction network.
$f^{1:N}$	Input video frames ( $N$ total frames).
$M$	Number of instances in the video.

TABLE II: Key Notations and Acronyms in PRIMEdit.

Fig. 4: A comparative illustration of our IPR versus Vanilla Cross-Attention [ 5 ] and DenseDiffusion [ 27 ] (top). The attention manipulation details of our IPR is shown at the bottom.

Fig. 5: An illustration of the four-step video caption generation pipeline of our MIVE dataset. (1) Image Captioning - The caption generation process starts by prompting LLaVA [ 71 ] to generate caption for each video that includes all instances in the video. Since LLaVa can only accept images, we perform the prompting for each frame and select one representative caption that includes the most instances. (2) Text Summarization - We utilize Llama 3 [ 72 ] to summarize the initial caption generated by LLaVa. (3) Manual Fix and Annotation - We manually include all of the instances of interest that are not included in the caption and manually add tags to map the instance captions to corresponding segmentation masks. (4) Target Caption Generation - We utilize Llama 3 to generate target captions by retexturing or swapping instances similar to [ 11 ] for each instance.

Dataset Name	Number	Number of	Number of	Number of Object	Number of Instances	Instance	Instance	Range of Average Instance
Dataset Name	of Clips	Frames per Clip	Objects per Clip	Classes	per Object Class	Captions	Masks	Mask Size Per Video (%)
TGVE [ 73 ] & TGVE+ [ 55 ]	76	$32-128$	1-2	No Info	1-2	$\checkmark$	$\times$	No Masks
VideoGrain [ 24 ]	13	$16-32$	1-5	7	1-2	✓	✓	$0.01\sim 92.34$
MIVE Dataset ( Ours )	200	$12-46$	1-9	54	1-17	✓	✓	$0.02\sim 77.35$
$\text{MIVE}_{E}$	200	12-46	1	37	1	$\checkmark$	$\checkmark$	$0.32\sim 77.35$
$\text{MIVE}_{M}$	116	12-46	1-3	45	1-2	$\checkmark$	$\checkmark$	$0.02\sim 77.35$

TABLE III: Comparison of our MIVE dataset with existing text-guided video editing datasets (top), and the subset categorization of MIVE dataset according to difficulty levels (bottom).

\text{MIVE}_{E}

\text{MIVE}_{M}

\text{MIVE}_{H}

and

\text{MIVE}_{X}

indicate Easy Video , Medium Video , Hard Video and eXtreme Video subsets, respectively.

Fig. 6: An illustration of Cross-Instance Accuracy (CIA) Score computation. We calculate the Local Textual Faithfulness (LTF) between each cropped instance and all the instance captions. For each row, we assign 1 to the maximum LTF (shown in red ) and 0 to the rest. The CIA Score is calculated as the mean of the diagonal elements (shown in green ).

Metric	Full Name	What It Measures
GTC [ 24 , 17 , 16 , 18 ]	Global Temporal Consistency	Temporal consistency across consecutive frames (overall video smoothness).
GTF [ 24 , 17 , 16 , 19 ]	Global Textual Faithfulness	Alignment between edited frames and the global edit prompt.
FA [ 18 ]	Frame Accuracy	Percentage of frames more aligned with the global target prompt than the global source prompt.
LTC	Local Temporal Consistency	Temporal consistency of each cropped instance across consecutive frames.
LTF [ 28 , 62 ]	Local Textual Faithfulness	Alignment between each cropped instance and its corresponding target instance caption.
IA	Instance Accuracy	Percentage of instances more aligned with the target instance caption than the source instance caption.

TABLE IV: Evaluation metrics. Acronyms, full names, and concise descriptions of the metrics used in our experiments.

Fig. 7: Qualitative comparison for three videos (with increasing difficulty from left to right) in our MIVE dataset. (a) shows the color-coded masks overlaid on the input frames to match the corresponding instance captions. (b)-(d) use global target captions for editing. (e) uses global source caption, instance source captions, global target caption, and instance target captions along with bounding boxes (omitted in (a) for better visualization). (f) uses global source caption, instance source captions, global target caption, and instance target captions, along with instance masks. Our PRIMEdit in (g) uses instance captions and masks. Unfaithful editing examples are shown in red arrow and attention leakage are shown in green arrow .

Methods	Venue	Editing	Global Scores			Local Scores			Leakage Scores			User Study
Methods	Venue	Scopes	GTC $\uparrow$	GTF $\uparrow$	FA $\uparrow$	LTC $\uparrow$	LTF $\uparrow$	IA $\uparrow$	CIA ( Ours ) $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	TC $\uparrow$	TF $\uparrow$	Leakage $\uparrow$
ControlVideo [ 15 ]	ICLR’24	Global	0.9745	0.2738	0.8850	0.9548	0.1960	0.4944	0.4963	0.5327	0.4242	0.0167	0.0656	0.0208
FLATTEN [ 16 ]	ICLR’24	Global	0.9678	0.2389	0.2627	0.9507	0.1881	0.2463	0.5109	0.8220	0.1314	0.3260	0.0594	0.1646
RAVE [ 17 ]	CVPR’24	Global	0.9675	0.2728	0.5769	0.9551	0.1869	0.3504	0.4950	0.7699	0.2188	0.0667	0.0354	0.0458
TokenFlow [ 19 ]	ICLR’24	Global	0.9686	0.2571	0.5617	0.9477	0.1868	0.3499	0.5319	0.7991	0.1807	0.0125	0.0521	0.0531
FreSCo [ 18 ]	CVPR’24	Global	0.9543	0.2528	0.4202	0.9324	0.1860	0.2960	0.5172	0.7567	0.2427	0.0823	0.0073	0.0104

TABLE V: Quantitative results for multi-instance video editing. The best and second best scores are shown in red and blue , respectively.

Fig. 8: Our user study interface and questionnaire form. Participants are shown an input video with a source caption, an annotated video with a target caption, and 8 randomly ordered edited videos from PRIMEdit and 7 other SOTA VE methods. Each instance mask is color-coded to correspond with its target instance caption. Participants select the video with the Best Temporal Consistency , Best Textual Faithfulness , and Least Editing Leakage .

Methods		LTC $\uparrow$	LTF $\uparrow$	IA $\uparrow$	CIA $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Core Comp.	DMS only	0.9564	0.2018	0.4920	0.6497	0.9007	0.0600
	IPR + SNS only	0.9503	0.2047	0.5288	0.6743	0.9063	0.0500
	IPR + PNS only	0.9542	0.2044	0.5311	0.6718	0.8993	0.0585
	Ours, Full	0.9552	0.2048	0.5369	0.6705	0.9008	0.0586
IPR	No Modulation [ 5 ]	0.9564	0.2018	0.4920	0.6497	0.9007	0.0600
	Dense Diffusion [ 27 ]	0.9530	0.2078	0.5845	0.6774	0.8977	0.0604

TABLE VI: Ablation study of core components (IPR (Sec. III-C ) and DMS (Sec. III-D )) and module design choices.

Fig. 9: Ablation study on IPR (Sec. III-C ).

Fig. 10: Ablation study on DMS (Sec. III-D ).

Methods	Venues	Editing	Local Scores (Small)			Local Scores (Medium)			Local Scores (Large)
Methods	Venues	Scopes	LTC $\uparrow$	LTF $\uparrow$	IA $\uparrow$	LTC $\uparrow$	LTF $\uparrow$	IA $\uparrow$	LTC $\uparrow$	LTF $\uparrow$	IA $\uparrow$
ControlVideo [ 15 ]	ICLR’24	Global	0.9202	0.1610	0.3308	0.9552	0.1993	0.5258	0.9554	0.2008	0.5338
FLATTEN [ 16 ]	ICLR’24	Global	0.9128	0.1707	0.3228	0.9515	0.1890	0.2366	0.9521	0.1891	0.2285
RAVE [ 17 ]	CVPR’24	Global	0.9255	0.1710	0.3685	0.9552	0.1898	0.3580	0.9557	0.1898	0.3510
TokenFlow [ 19 ]	ICLR’24	Global	0.9209	0.1701	0.4015	0.9490	0.1883	0.3451	0.9490	0.1887	0.3405
FreSCo [ 18 ]	CVPR’24	Global	0.9021	0.1681	0.3686	0.9347	0.1869	0.2948	0.9348	0.1867	0.2825

TABLE VII: Quantitative comparison based on instance size. We only show Local Scores since these are the only scores that can be computed depending on the instance sizes.

Fig. 11: Qualitative comparison based on instance size. The yellow arrows highlight failure cases on small objects (e.g., “game console”), highlighting the difficulty of small object editing in diffusion-based video editing.

Methods	Venues	Editing	Global Scores			Local Scores			Leakage Scores
Methods	Venues	Scopes	GTC $\uparrow$	GTF $\uparrow$	FA $\uparrow$	LTC $\uparrow$	LTF $\uparrow$	IA $\uparrow$	CIA ( Ours ) $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Editing on One (1) Instance ( $\text{MIVE}_{E}$ ) - 200 Videos
ControlVideo [ 15 ]	ICLR’24	Global	0.9755	0.2717	0.6922	0.9611	0.2048	0.5351	-	0.4130	0.5090
FLATTEN [ 16 ]	ICLR’24	Global	0.9699	0.2538	0.3140	0.9586	0.1826	0.1971	-	0.7783	0.1567
RAVE [ 17 ]	CVPR’24	Global	0.9691	0.2774	0.4243	0.9599	0.1857	0.3374	-	0.7044	0.2716
TokenFlow [ 19 ]	ICLR’24	Global	0.9715	0.2621	0.4413	0.9564	0.1822	0.3006	-	0.7546	0.2148

TABLE VIII: Quantitative comparison for multi-instance video editing on various number of instances.

Fig. 12: Qualitative comparison based on the number of instances.

Methods	Reflective Surfaces			Tiny Objects		$\text{MIVE}_{X}$
Methods	FR $\downarrow$	IA $\uparrow$	CIA ( Ours ) $\uparrow$	FR $\downarrow$	IA $\uparrow$	FR $\downarrow$	IA $\uparrow$	CIA ( Ours ) $\uparrow$
ControlVideo [ 15 ]	0.5478	0.4522	0.3589	0.6692	0.3308	0.6398	0.3602	0.1698
GAV [ 20 ]	0.5989	0.4011	0.3978	0.6642	0.3358	0.7089	0.2911	0.2210
VideoGrain [ 24 ]	0.2978	0.7022	0.4056	0.5581	0.4419	0.5950	0.4050	0.2516
PRIMEdit (Ours)	0.3811	0.6199	0.5311	0.6130	0.3870	0.5791	0.4209	0.3413

TABLE IX: Quantitative evaluation on challenging subsets of the MIVE dataset: reflective surfaces, tiny objects (small instances), and extreme instance counts (

\text{MIVE}_{X}

>7

edited instances). FR denotes failure rate, defined as the percentage of instances with IA = 0 (i.e., the edited instance is more similar to its source caption than to its target caption).

Fig. 13: Editing results of Videograin [ 24 ] and PRIMEdit under imprecise instance mask conditions.

Methods	Mask	Global Scores			Local Scores			Leakage Scores			Mean %
Methods	Mask	GTC $\uparrow$	GTF $\uparrow$	FA $\uparrow$	LTC $\uparrow$	LTF $\uparrow$	IA $\uparrow$	CIA ( Ours ) $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	Change $\uparrow$
VideoGrain [ 24 ]	Clean	0.9600	0.2803	0.7430	0.9423	0.2026	0.4798	0.5868	0.8929	0.0488	-
PRIMEdit (Ours)	Clean	0.9677	0.2632	0.7707	0.9552	0.2048	0.5369	0.6705	0.9008	0.0586	-
VideoGrain [ 24 ]	Dilated	0.9604 (+0.05%)	0.2848 (+1.61%)	0.7732 (+4.07%)	0.9429 (+0.06%)	0.2046 (+0.95%)	0.4969 (+3.55%)	0.5912 (+0.74%)	0.8879 (-0.56%)	0.0565 (+15.87%)	+2.93
PRIMEdit (Ours)	Dilated	0.9691 (+0.14%)	0.2659 (+1.06%)	0.7912 (+2.65%)	0.9571 (+0.20%)	0.2064 (+0.79%)	0.5513 (+2.68%)	0.6703 (-0.03%)	0.8936 (-0.80%)	0.0698 (+19.23%)	+2.88
VideoGrain [ 24 ]	Eroded	0.9602 (+0.02%)	0.2688 (-4.08%)	0.5884 (-20.81%)	0.9424 (+0.01%)	0.1975 (-2.51%)	0.4037 (-15.86%)	0.5775 (-1.59%)	0.8968 (+0.43%)	0.0414 (-15.07%)	-6.61

TABLE X: Quantitative comparison between VideoGrain [ 24 ] and PRIMEdit under noisy instance masks. For each metric, the percentage change relative to the Clean setting is reported in parentheses alongside the raw score. Within each column, the numerically largest and second-largest percentage changes are highlighted in red and blue , respectively.

Fig. 14: Attention weight visualization.

Panel of Table 25 — TABLE XI: A user-study on human evaluation for attention leakage.

Methods	Time (s) $\downarrow$ / Memory (GB) $\downarrow$				IS	DS
Methods	I =1, F =15	I =1, F =32	I =3, F =15	I =3, F =32	IS	DS
ControlVideo [ 45 ]	106.80 / 7.97	359.73 / 11.35	106.96 / 7.85	360.11 / 11.21	50	50
FLATTEN [ 16 ]	162.86 / 11.07	566.66 / 29.44	162.91 / 11.13	564.98 / 29.53	100	50
RAVE [ 17 ]	206.59 / 12.23	409.62 / 12.83	208.04 / 12.22	409.49 / 12.83	50	50
TokenFlow [ 19 ]	238.91 / 11.45	471.70 / 14.89	239.60 / 11.44	473.94 / 14.90	500	50
FreSCo [ 18 ]	128.71 / 14.62	261.43 / 14.61	130.30 / 14.24	259.22 / 14.64	-	20

TABLE XII: Runtime (seconds) and GPU memory usage (GB) comparison for two categories: global video editing (top) and local video editing (bottom). We denote IS as the number of inversion steps, DS as the number of denoising steps, I as the number of instances, and F as the number of frames to edit.

Fig. 15: Limitations of our PRIMEdit and other SOTA multi-instance VE methods in editing scenes with reflective surfaces (top) and an excessive number of instances (bottom).

Video Editing

Video editing uses AI to alter, compose, relight, or reorganize moving footage while controlling how changes persist across frames.

See all 178 papers

Jun 20, 2026

IDAG-Edit: Multi-Object Video Editing via Instance-Decoupled Attention and Guidance

Diffusion-based video editing has made significant progress; however, achieving precise and temporally consistent object-level control, especially in multi-object scenarios, remains challenging due to attention leakage, identity drift, and unstable temporal dynamics. In this work, we propose IDAGEdit, a training-free framework for fine-grained multi-object video editing with strong temporal consistency. The framework adopts Layout-guided Attention Modulation to facilitate coherent multi-object editing, while Instance-level Masks are introduced to preserve individual object identity and enforce localized attention within each object region, thereby enabling fine-grained, object-level editing. Extensive qualitative and quantitative evaluations demonstrate that our method improves temporal stability and multi-object controllability over state-of-the-art video editing approaches.

Yuan-Zhih Lin, Huu-Thang Nguyen, Huu-Phu Do +2

Department of Computer Science, National Yang Ming Chiao Tung University, Taiwan

Diffusion Model Guidance Video Editing Diffusion-Based Video Editing

Ask PDF

Apr 6, 2026

MIRAGE: Benchmarking and Aligning Multi-Instance Image Editing

Instruction-guided image editing has seen remarkable progress with models like FLUX.2 and Qwen-Image-Edit, yet they still struggle with complex scenarios with multiple similar instances each requiring individual edits. We observe that state-of-the-art models suffer from severe over-editing and spatial misalignment when faced with multiple identical instances and composite instructions. To this end, we introduce a comprehensive benchmark specifically designed to evaluate fine-grained consistency in multi-instance and multi-instruction settings. To address the failures of existing methods observed in our benchmark, we propose Multi-Instance Regional Alignment via Guided Editing (MIRAGE), a training-free framework that enables precise, localized editing. By leveraging a vision-language model to parse complex instructions into regional subsets, MIRAGE employs a multi-branch parallel denoising strategy. This approach injects latent representations of target regions into the global representation space while maintaining background integrity through a reference trajectory. Extensive evaluations on MIRA-Bench and RefEdit-Bench demonstrate that our framework significantly outperforms existing methods in achieving precise instance-level modifications while preserving background consistency. Our benchmark and code are available at https://github.com/ZiqianLiu666/MIRAGE.

Ziqian Liu, Stephan Alaniz

LTCI, Télécom Paris, Institut Polytechnique de Paris, France

Image Editing Text-Guided Image Editing Image Editing Evaluation Diffusion Model Guidance

Ask PDF

Apr 18, 2026

LIVE: Leveraging Image Manipulation Priors for Instruction-based Video Editing

Video editing aims to modify input videos according to user intent. Recently, end-to-end training methods have garnered widespread attention, constructing paired video editing data through video generation or editing models. However, compared to image editing, the high annotation costs of video data severely constrain the scale, quality, and task diversity of video editing datasets when relying on video generative models or manual annotation. To bridge this gap, we propose LIVE, a joint training framework that leverages large-scale, high-quality image editing data alongside video datasets to bolster editing capabilities. To mitigate the domain discrepancy between static images and dynamic videos, we introduce a frame-wise token noise strategy, which treats the latents of specific frames as reasoning tokens, leveraging large pretrained video generative models to create plausible temporal transformations. Moreover, through cleaning public datasets and constructing an automated data pipeline, we adopt a two-stage training strategy to anneal video editing capabilities. Furthermore, we curate a comprehensive evaluation benchmark encompassing over 60 challenging tasks that are prevalent in image editing but scarce in existing video datasets. Extensive comparative and ablation experiments demonstrate that our method achieves state-of-the-art performance. The source code will be publicly available.

Weicheng Wang, Zhicheng Zhang, Zhongqi Zhang +6

Nankai University · Kuaishou Technology · Pengcheng Laboratory +1

Video Editing Benchmarks Video Diffusion Models Video Editing Instruction-Based Video Editing

Ask PDF

Abstract

Video Editing

Explore similar work

IDAG-Edit: Multi-Object Video Editing via Instance-Decoupled Attention and Guidance

MIRAGE: Benchmarking and Aligning Multi-Instance Image Editing

LIVE: Leveraging Image Manipulation Priors for Instruction-based Video Editing