PRIMEdit: Probability Redistribution for Instance-Aware Multi-Object Video Editing With a New Benchmark Dataset

Samuel Teodoro; Agus Gunawan; Soo Ye Kim; Jihyong Oh; Munchurl Kim

Fig. 1: Limitations of previous SOTA methods. (a) ControlVideo [ 15 ] relies on single global captions, and (b) GAV [ 20 ] depends on bounding box conditions that can sometimes overlap. Both are susceptible to unfaithful editing ( red arrow ) and attention leakage ( blue arrow ). Note that the colored words in Prompt correspond to the instances to be edited in Input .

Fig. 2: Given a video, instance masks, and target instance captions, our PRIMEdit enables faithful and disentangled edits in (a) single- and (b)-(c) multi-instance levels, as well as an applicability to more fine-grained (d) partial instance level without additional training. Unlike previous methods, our PRIMEdit does not rely on global edit captions, but leverages individual instance captions. Object masks are color-coded to match its corresponding edit caption.

TABLE I: Comparison of our PRIMEdit against previous multi-instance video editing state-of-the-art methods.

Fig. 3: The overall framework of our PRIMEdit. For a given

M

multi-instances, each instance has a caption

c_{i}

and instance masks

\boldsymbol{m_{i}}

for editing. Our Disentangled Multi-instance Sampling (DMS, Sec. III-D ) consists of series noise sampling (SNS, yellow box), latent fusion (green box) to fuse different instance latents, re-inversion (pink box) to harmonize the latents after fusion, and parallel noise sampling (PNS, blue box). In addition, our Instance-centric Probability Redistribution (IPR, Sec. III-C ) provides better spatial control.

Symbol / Acronym	Description
PRIMEdit	Proposed multi-instance video editing framework.
LDM	Latent Diffusion Model.
$\mathcal{E}$ / $\mathcal{D}$	VAE encoder / decoder of the LDM.
$\epsilon_{\theta}$	U-Net noise prediction network.
$f^{1:N}$	Input video frames ( $N$ total frames).
$M$	Number of instances in the video.

TABLE II: Key Notations and Acronyms in PRIMEdit.

Fig. 4: A comparative illustration of our IPR versus Vanilla Cross-Attention [ 5 ] and DenseDiffusion [ 27 ] (top). The attention manipulation details of our IPR is shown at the bottom.

Fig. 5: An illustration of the four-step video caption generation pipeline of our MIVE dataset. (1) Image Captioning - The caption generation process starts by prompting LLaVA [ 71 ] to generate caption for each video that includes all instances in the video. Since LLaVa can only accept images, we perform the prompting for each frame and select one representative caption that includes the most instances. (2) Text Summarization - We utilize Llama 3 [ 72 ] to summarize the initial caption generated by LLaVa. (3) Manual Fix and Annotation - We manually include all of the instances of interest that are not included in the caption and manually add tags to map the instance captions to corresponding segmentation masks. (4) Target Caption Generation - We utilize Llama 3 to generate target captions by retexturing or swapping instances similar to [ 11 ] for each instance.

Dataset Name	Number	Number of	Number of	Number of Object	Number of Instances	Instance	Instance	Range of Average Instance
Dataset Name	of Clips	Frames per Clip	Objects per Clip	Classes	per Object Class	Captions	Masks	Mask Size Per Video (%)
TGVE [ 73 ] & TGVE+ [ 55 ]	76	$32-128$	1-2	No Info	1-2	$\checkmark$	$\times$	No Masks
VideoGrain [ 24 ]	13	$16-32$	1-5	7	1-2	✓	✓	$0.01\sim 92.34$
MIVE Dataset ( Ours )	200	$12-46$	1-9	54	1-17	✓	✓	$0.02\sim 77.35$
$\text{MIVE}_{E}$	200	12-46	1	37	1	$\checkmark$	$\checkmark$	$0.32\sim 77.35$
$\text{MIVE}_{M}$	116	12-46	1-3	45	1-2	$\checkmark$	$\checkmark$	$0.02\sim 77.35$

TABLE III: Comparison of our MIVE dataset with existing text-guided video editing datasets (top), and the subset categorization of MIVE dataset according to difficulty levels (bottom).

\text{MIVE}_{E}

,

\text{MIVE}_{M}

,

\text{MIVE}_{H}

and

\text{MIVE}_{X}

indicate Easy Video , Medium Video , Hard Video and eXtreme Video subsets, respectively.

Fig. 6: An illustration of Cross-Instance Accuracy (CIA) Score computation. We calculate the Local Textual Faithfulness (LTF) between each cropped instance and all the instance captions. For each row, we assign 1 to the maximum LTF (shown in red ) and 0 to the rest. The CIA Score is calculated as the mean of the diagonal elements (shown in green ).

Metric	Full Name	What It Measures
GTC [ 24 , 17 , 16 , 18 ]	Global Temporal Consistency	Temporal consistency across consecutive frames (overall video smoothness).
GTF [ 24 , 17 , 16 , 19 ]	Global Textual Faithfulness	Alignment between edited frames and the global edit prompt.
FA [ 18 ]	Frame Accuracy	Percentage of frames more aligned with the global target prompt than the global source prompt.
LTC	Local Temporal Consistency	Temporal consistency of each cropped instance across consecutive frames.
LTF [ 28 , 62 ]	Local Textual Faithfulness	Alignment between each cropped instance and its corresponding target instance caption.
IA	Instance Accuracy	Percentage of instances more aligned with the target instance caption than the source instance caption.

TABLE IV: Evaluation metrics. Acronyms, full names, and concise descriptions of the metrics used in our experiments.

Fig. 7: Qualitative comparison for three videos (with increasing difficulty from left to right) in our MIVE dataset. (a) shows the color-coded masks overlaid on the input frames to match the corresponding instance captions. (b)-(d) use global target captions for editing. (e) uses global source caption, instance source captions, global target caption, and instance target captions along with bounding boxes (omitted in (a) for better visualization). (f) uses global source caption, instance source captions, global target caption, and instance target captions, along with instance masks. Our PRIMEdit in (g) uses instance captions and masks. Unfaithful editing examples are shown in red arrow and attention leakage are shown in green arrow .

Methods	Venue	Editing	Global Scores			Local Scores			Leakage Scores			User Study
Methods	Venue	Scopes	GTC $\uparrow$	GTF $\uparrow$	FA $\uparrow$	LTC $\uparrow$	LTF $\uparrow$	IA $\uparrow$	CIA ( Ours ) $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	TC $\uparrow$	TF $\uparrow$	Leakage $\uparrow$
ControlVideo [ 15 ]	ICLR’24	Global	0.9745	0.2738	0.8850	0.9548	0.1960	0.4944	0.4963	0.5327	0.4242	0.0167	0.0656	0.0208
FLATTEN [ 16 ]	ICLR’24	Global	0.9678	0.2389	0.2627	0.9507	0.1881	0.2463	0.5109	0.8220	0.1314	0.3260	0.0594	0.1646
RAVE [ 17 ]	CVPR’24	Global	0.9675	0.2728	0.5769	0.9551	0.1869	0.3504	0.4950	0.7699	0.2188	0.0667	0.0354	0.0458
TokenFlow [ 19 ]	ICLR’24	Global	0.9686	0.2571	0.5617	0.9477	0.1868	0.3499	0.5319	0.7991	0.1807	0.0125	0.0521	0.0531
FreSCo [ 18 ]	CVPR’24	Global	0.9543	0.2528	0.4202	0.9324	0.1860	0.2960	0.5172	0.7567	0.2427	0.0823	0.0073	0.0104

TABLE V: Quantitative results for multi-instance video editing. The best and second best scores are shown in red and blue , respectively.

Fig. 8: Our user study interface and questionnaire form. Participants are shown an input video with a source caption, an annotated video with a target caption, and 8 randomly ordered edited videos from PRIMEdit and 7 other SOTA VE methods. Each instance mask is color-coded to correspond with its target instance caption. Participants select the video with the Best Temporal Consistency , Best Textual Faithfulness , and Least Editing Leakage .

Methods		LTC $\uparrow$	LTF $\uparrow$	IA $\uparrow$	CIA $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Core Comp.	DMS only	0.9564	0.2018	0.4920	0.6497	0.9007	0.0600
	IPR + SNS only	0.9503	0.2047	0.5288	0.6743	0.9063	0.0500
	IPR + PNS only	0.9542	0.2044	0.5311	0.6718	0.8993	0.0585
	Ours, Full	0.9552	0.2048	0.5369	0.6705	0.9008	0.0586
IPR	No Modulation [ 5 ]	0.9564	0.2018	0.4920	0.6497	0.9007	0.0600
	Dense Diffusion [ 27 ]	0.9530	0.2078	0.5845	0.6774	0.8977	0.0604

TABLE VI: Ablation study of core components (IPR (Sec. III-C ) and DMS (Sec. III-D )) and module design choices.

Fig. 9: Ablation study on IPR (Sec. III-C ).

Fig. 10: Ablation study on DMS (Sec. III-D ).

Methods	Venues	Editing	Local Scores (Small)			Local Scores (Medium)			Local Scores (Large)
Methods	Venues	Scopes	LTC $\uparrow$	LTF $\uparrow$	IA $\uparrow$	LTC $\uparrow$	LTF $\uparrow$	IA $\uparrow$	LTC $\uparrow$	LTF $\uparrow$	IA $\uparrow$
ControlVideo [ 15 ]	ICLR’24	Global	0.9202	0.1610	0.3308	0.9552	0.1993	0.5258	0.9554	0.2008	0.5338
FLATTEN [ 16 ]	ICLR’24	Global	0.9128	0.1707	0.3228	0.9515	0.1890	0.2366	0.9521	0.1891	0.2285
RAVE [ 17 ]	CVPR’24	Global	0.9255	0.1710	0.3685	0.9552	0.1898	0.3580	0.9557	0.1898	0.3510
TokenFlow [ 19 ]	ICLR’24	Global	0.9209	0.1701	0.4015	0.9490	0.1883	0.3451	0.9490	0.1887	0.3405
FreSCo [ 18 ]	CVPR’24	Global	0.9021	0.1681	0.3686	0.9347	0.1869	0.2948	0.9348	0.1867	0.2825

TABLE VII: Quantitative comparison based on instance size. We only show Local Scores since these are the only scores that can be computed depending on the instance sizes.

Fig. 11: Qualitative comparison based on instance size. The yellow arrows highlight failure cases on small objects (e.g., “game console”), highlighting the difficulty of small object editing in diffusion-based video editing.

Methods	Venues	Editing	Global Scores			Local Scores			Leakage Scores
Methods	Venues	Scopes	GTC $\uparrow$	GTF $\uparrow$	FA $\uparrow$	LTC $\uparrow$	LTF $\uparrow$	IA $\uparrow$	CIA ( Ours ) $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Editing on One (1) Instance ( $\text{MIVE}_{E}$ ) - 200 Videos
ControlVideo [ 15 ]	ICLR’24	Global	0.9755	0.2717	0.6922	0.9611	0.2048	0.5351	-	0.4130	0.5090
FLATTEN [ 16 ]	ICLR’24	Global	0.9699	0.2538	0.3140	0.9586	0.1826	0.1971	-	0.7783	0.1567
RAVE [ 17 ]	CVPR’24	Global	0.9691	0.2774	0.4243	0.9599	0.1857	0.3374	-	0.7044	0.2716
TokenFlow [ 19 ]	ICLR’24	Global	0.9715	0.2621	0.4413	0.9564	0.1822	0.3006	-	0.7546	0.2148

TABLE VIII: Quantitative comparison for multi-instance video editing on various number of instances.

Fig. 12: Qualitative comparison based on the number of instances.

Methods	Reflective Surfaces			Tiny Objects		$\text{MIVE}_{X}$
Methods	FR $\downarrow$	IA $\uparrow$	CIA ( Ours ) $\uparrow$	FR $\downarrow$	IA $\uparrow$	FR $\downarrow$	IA $\uparrow$	CIA ( Ours ) $\uparrow$
ControlVideo [ 15 ]	0.5478	0.4522	0.3589	0.6692	0.3308	0.6398	0.3602	0.1698
GAV [ 20 ]	0.5989	0.4011	0.3978	0.6642	0.3358	0.7089	0.2911	0.2210
VideoGrain [ 24 ]	0.2978	0.7022	0.4056	0.5581	0.4419	0.5950	0.4050	0.2516
PRIMEdit (Ours)	0.3811	0.6199	0.5311	0.6130	0.3870	0.5791	0.4209	0.3413

TABLE IX: Quantitative evaluation on challenging subsets of the MIVE dataset: reflective surfaces, tiny objects (small instances), and extreme instance counts (

\text{MIVE}_{X}

,

>7

edited instances). FR denotes failure rate, defined as the percentage of instances with IA = 0 (i.e., the edited instance is more similar to its source caption than to its target caption).

Fig. 13: Editing results of Videograin [ 24 ] and PRIMEdit under imprecise instance mask conditions.

Methods	Mask	Global Scores			Local Scores			Leakage Scores			Mean %
Methods	Mask	GTC $\uparrow$	GTF $\uparrow$	FA $\uparrow$	LTC $\uparrow$	LTF $\uparrow$	IA $\uparrow$	CIA ( Ours ) $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	Change $\uparrow$
VideoGrain [ 24 ]	Clean	0.9600	0.2803	0.7430	0.9423	0.2026	0.4798	0.5868	0.8929	0.0488	-
PRIMEdit (Ours)	Clean	0.9677	0.2632	0.7707	0.9552	0.2048	0.5369	0.6705	0.9008	0.0586	-
VideoGrain [ 24 ]	Dilated	0.9604 (+0.05%)	0.2848 (+1.61%)	0.7732 (+4.07%)	0.9429 (+0.06%)	0.2046 (+0.95%)	0.4969 (+3.55%)	0.5912 (+0.74%)	0.8879 (-0.56%)	0.0565 (+15.87%)	+2.93
PRIMEdit (Ours)	Dilated	0.9691 (+0.14%)	0.2659 (+1.06%)	0.7912 (+2.65%)	0.9571 (+0.20%)	0.2064 (+0.79%)	0.5513 (+2.68%)	0.6703 (-0.03%)	0.8936 (-0.80%)	0.0698 (+19.23%)	+2.88
VideoGrain [ 24 ]	Eroded	0.9602 (+0.02%)	0.2688 (-4.08%)	0.5884 (-20.81%)	0.9424 (+0.01%)	0.1975 (-2.51%)	0.4037 (-15.86%)	0.5775 (-1.59%)	0.8968 (+0.43%)	0.0414 (-15.07%)	-6.61

TABLE X: Quantitative comparison between VideoGrain [ 24 ] and PRIMEdit under noisy instance masks. For each metric, the percentage change relative to the Clean setting is reported in parentheses alongside the raw score. Within each column, the numerically largest and second-largest percentage changes are highlighted in red and blue , respectively.

Fig. 14: Attention weight visualization.

TABLE XI: A user-study on human evaluation for attention leakage.

Methods	Time (s) $\downarrow$ / Memory (GB) $\downarrow$				IS	DS
Methods	I =1, F =15	I =1, F =32	I =3, F =15	I =3, F =32	IS	DS
ControlVideo [ 45 ]	106.80 / 7.97	359.73 / 11.35	106.96 / 7.85	360.11 / 11.21	50	50
FLATTEN [ 16 ]	162.86 / 11.07	566.66 / 29.44	162.91 / 11.13	564.98 / 29.53	100	50
RAVE [ 17 ]	206.59 / 12.23	409.62 / 12.83	208.04 / 12.22	409.49 / 12.83	50	50
TokenFlow [ 19 ]	238.91 / 11.45	471.70 / 14.89	239.60 / 11.44	473.94 / 14.90	500	50
FreSCo [ 18 ]	128.71 / 14.62	261.43 / 14.61	130.30 / 14.24	259.22 / 14.64	-	20

TABLE XII: Runtime (seconds) and GPU memory usage (GB) comparison for two categories: global video editing (top) and local video editing (bottom). We denote IS as the number of inversion steps, DS as the number of denoising steps, I as the number of instances, and F as the number of frames to edit.

Fig. 15: Limitations of our PRIMEdit and other SOTA multi-instance VE methods in editing scenes with reflective surfaces (top) and an excessive number of instances (bottom).

Figure 28

Figure 29

Figure 30

Figure 31

Figure 32

PRIMEdit: Probability Redistribution for Instance-Aware Multi-Object Video Editing With a New Benchmark Dataset

Abstract

Figures & tables

Video Editing

IDAG-Edit: Multi-Object Video Editing via Instance-Decoupled Attention and Guidance

MIRAGE: Benchmarking and Aligning Multi-Instance Image Editing

LIVE: Leveraging Image Manipulation Priors for Instruction-based Video Editing

Abstract

Video Editing

Explore similar work

IDAG-Edit: Multi-Object Video Editing via Instance-Decoupled Attention and Guidance

MIRAGE: Benchmarking and Aligning Multi-Instance Image Editing

LIVE: Leveraging Image Manipulation Priors for Instruction-based Video Editing