Diffusion Editing with Soft Mask: Pixel Level Redo of Image and Video with Adjustable Strength

Candi Zheng; Yuan Lan

cs.GRSep 30, 2026

Diffusion Editing with Soft Mask: Pixel Level Redo of Image and Video with Adjustable Strength

Authors: Candi Zheng, Yuan Lan

Organizations: Department of Mathematics, Hong Kong University of Science and Technology · Independent Researcher

Abstract

Diffusion models with prompt and reference image-guided editing have seen rapid progress, yet they remain too coarse for pixel-level control. One promising direction is to incorporate a soft mask that specifies spatially varying edit strengths but training such fine-grained control demands expensive pixel-wise annotations, while existing zero-shot methods often yield unsatisfactory results. We introduce SoftPaint, a new zero-shot sampling method that leverages soft masks to enable a continuous spectrum of edits, from fully preserving the original content to completely re-synthesizing the masked region. Going beyond zero-shot inpainting methods, we design a Langevin-iteration-based sampler that respects per-pixel soft mask strengths, which applies universally to image and video diffusion models, enabling tasks such as video editing. The method is gradient-free, memory-efficient, and achieves smooth, pixel-level edits across multiple image and video backbones.

Figures & tables

Panel of Figure 1 — Figure 1: Soft Image editing example . SoftPaint takes a reference image $\mathbf{x}_{0}$ (left) and a continuous-opacity editing mask (middle), where the mask opacity encodes redo strength : more opaque pixels correspond to stronger redo (weaker reference tether), while more transparent pixels keep the sample closer to $\mathbf{x}_{0}$ (stronger tether). This yields spatially graded edits within a single generation (right). Text prompt: “Chinese female with thick wavy black hair, white shirt, night city skyline behind, cinematic close-up.”

Figure 2: Fine-grained editing control: $r$ tunes the balance between preservation and redo.

Figure 3: Reference blending weight $\boldsymbol{\gamma}_{t}(\mathbf{r})$ at $\lambda=4$ . Curves span $r=0$ (top, exact reconstruction) to $r=1$ (bottom, free generation).

Figure 4: Image-editing controllability and quality on Qwen-Image (FlowEdit Benchmark). Left: soft-mask strength $r$ vs. CLIP-Text—SoftPaint provides the widest editing range. Right three panels: quality metrics against CLIP-Text as $r$ sweeps from $0.1$ to $1.0$ . SoftPaint Pareto-dominates DD; Naive Mixing has near-zero controllability. Source ( $r{=}0$ , gray square) and Qwen-Image-Edit (purple triangle) are reference points.

Method	NFE	PSNR ${}_{\text{edit}}\uparrow$	LPIPS ${}_{\text{edit}}\downarrow$	FID $\downarrow$	CLIP-IQA $\uparrow$
DD	20	20.3053	0.1343	102.8441	0.6088
DD	60	20.2097	0.1349	103.4106	0.6045
SoftPaint	60	21.0159	0.1293	100.9059	0.6226

Table 1: Spatially varying soft-mask editing on Qwen-Image (FlowEdit). Each image uses four vertical redo-strength zones,

r\in\{1,0.75,0.5,0.25\}

. All methods are evaluated at the medium CLIP-T anchor (

33.0\pm 0.2

); the

60

-NFE rows compare equal model-evaluation budgets.

Figure 7: Soft video editing: “Model Train” $\to$ “Wooden Train” at three soft-mask strengths $r$ on CogVideoX. Top row: original DAVIS-Edit frames ( $r{=}0$ , no edit). Subsequent rows: SoftPaint and DD outputs at $r{=}0.4$ , $0.8$ , $1.0$ .

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Backbone	Anchor	Method	$r$ used	LPIPS $\downarrow$	PSNR $\uparrow$	M-LPIPS $\downarrow$	VFID $\downarrow$	DOVER $\uparrow$
Wan2.2 V2V	Weak Edit (CLIP-T: 19.7 $\pm$ 0.5)	DD	0.1	0.053	27.34	0.022	0.109	54.46
	Weak Edit (CLIP-T: 19.7 $\pm$ 0.5)	SoftPaint	0.2	0.045	29.36	0.015	0.080	55.55
	Medium Edit (CLIP-T: 20.0 $\pm$ 0.5)	DD	0.2	0.062	25.61	0.029	0.147	54.39
	Medium Edit (CLIP-T: 20.0 $\pm$ 0.5)	SoftPaint	0.5	0.051	27.80	0.020	0.106	55.63
	Strong Edit (CLIP-T: 24.2 $\pm$ 0.5)	DD	0.8	0.104	20.07	0.065	0.275	55.06
	Strong Edit (CLIP-T: 24.2 $\pm$ 0.5)	SoftPaint	1.0	0.099	21.51	0.060	0.245	54.60

Appendix

Table 2: Video-editing on DAVIS-Edit at matched CLIP-T. Anchors are SoftPaint at

r\in\{0.2,0.5,1.0\}

(Wan2.2) and

r\in\{0.2,0.5,0.9\}

(CogVideoX). For DD we select the nearest measured

r

within

\pm 0.5

of each anchor.

Anchor	Method	NFE	LPIPS ${}_{\text{edit}}\downarrow$	PSNR ${}_{\text{edit}}\uparrow$	PSNR ${}_{\text{keep}}\uparrow$	CLIP-IQA $\uparrow$	FID $\downarrow$
Weak Edit (CLIP-T: 30.1 $\pm$ 0.2)	DD	20	0.082	25.34	65.54	0.626	56.3
	DD	60	0.088	25.19	65.50	0.620	54.33
	SoftPaint	60	0.075	26.36	66.31	0.626	52.7
Medium Edit (CLIP-T: 33.0 $\pm$ 0.2)	DD	20	0.131	21.31	65.34	0.619	102.2
	DD	60	0.134	20.80	65.25	0.613	106.4
	SoftPaint	60	0.121	22.05	66.29	0.627	99.9

Appendix

Table 3: Image editing at matched NFE on Qwen-Image (FlowEdit). SoftPaint uses

20

outer steps and two additional inner evaluations per step (

60

NFE); DD uses

20

60

outer steps. Rows within each block are selected at the same CLIP-T anchor. The DD

20

-NFE and SoftPaint

60

-NFE rows reproduce the corresponding values in Table 4 .

Anchor	Method	$r$ used	LPIPS ${}_{\text{edit}}\!\downarrow$	PSNR ${}_{\text{edit}}\!\uparrow$	PSNR ${}_{\text{keep}}\!\uparrow$	CLIP-IQA $\uparrow$	FID $\downarrow$
Reference Points
NA	Source	0	0.000	80.00	80.00	0.650	0.0
NA	QwenImage-Edit	NA	0.306	16.19	25.39	0.667	129.5
Soft-Mask Methods
Weak Edit (CLIP-T: 30.1 $\pm$ 0.2)	Naive Mixing	0.8	0.057	28.92	65.58	0.595	46.2
	DD	0.4	0.082	25.34	65.54	0.626	56.3

Appendix

Table 4: Image-editing on Qwen-Image at matched CLIP-T. Anchors are SoftPaint at

r\in\{0.2,0.6,1.0\}

. For DD and Naive Mixing we select the nearest measured

r

within

\pm 0.2

CLIP-T of each anchor. Source is listed for reference. Naive Mixing has no match at the medium anchor.

Figure 8: Ablation of $n$ and $s$ on soft video editing task(DAVIS-Edit) under backbone Wan2.2. Top: $n$ varied with $s{=}0.2$ fixed. Bottom: $s$ varied with $n{=}2$ fixed. All metrics averaged over $r\in\{0.2,0.4,0.6,0.8,1.0\}$ . $n{=}2$ , $s{=}0.2$ are the default operating point, also used for image editing without re-tuning.

Method	PSNR $\uparrow$	LPIPS $\downarrow$	CLIP-T $\uparrow$	VFID $\downarrow$
Naive mix.	32.46	0.0333	19.95 almost no edit	0.0668
Naive mix. + warm start	22.22	0.0786	24.10 $\uparrow$ edit	0.2410
SoftPaint	23.15	0.0757	24.32 $\uparrow$ edit	0.2305

Appendix

Table 5: Warm-start ablation for soft video editing at

r=0.8

on the CogVideoX backbone. Naive mixing has almost no editing ability at this mask strength. Adding the warm start restores editing, and SoftPaint further improves editability with the UR score and DDL Langevin inner loop while improving LPIPS and VFID relative to warm start alone. All rows use 50 denoising steps.

Mask	Inner steps $n$	Replace [ 18 ]	RePaint-style [ 19 ]	SoftPaint
Box	5	0.229 / 75.7	0.216 / 62.8	0.174 / 48.1
Box	10	0.229 / 75.7	0.215 / 61.0	0.165 / 43.5
Checkerboard	5	0.406 / 146.4	0.137 / 31.4	0.118 / 21.6
Checkerboard	10	0.406 / 146.4	0.135 / 32.8	0.110 / 19.9

Appendix

Table 6: Binary inpainting on ImageNet with ADM. LPIPS

\downarrow

/ FID

\downarrow

1{,}000

validation images at

256\times 256

. All methods use

20

outer Euler steps. RePaint-style and SoftPaint have matched inner-loop budgets for each

n

; Replace has no inner loop.

Figure 9: Effect of the exponent $\eta$ in $\Sigma(\mathbf{r};\eta)$ on edit strength, measured on the Flow Edit Benchmark with the Qwen-Image model. The two curves correspond to $\eta=2$ and $\eta=4$ , showing how the choice of exponent changes the effective redo strength for each mask value $r$ , and hence $\mathrm{clip}_{t}$ .

Figure 10: Soft-mask image editing: “Dalmatian” $\to$ “Cheetah”. Source: “A Dalmatian dog sitting on a white background…” Target: “A cheetah sitting on a white background…, looking to the camera.” Top: Naive Mixing. Middle: DD. Bottom: SoftPaint (ours). $r\in\{0.1,\ldots,1.0\}$ left to right; mask inset at top-right.

Figure 11: Soft-mask image editing: “Pink Flowers” $\to$ “Orange Flowers”. Source: “A vase filled with a beautiful bouquet of pink, red and white flowers.” Target: “…orange, yellow and white flowers.” Layout as in Fig. 10 .

Figure 12: Soft-mask video editing: “Dog” $\to$ “Wolf” at three soft-mask strengths $r$ on CogVideoX. Top row: original DAVIS-Edit frames ( $r{=}0$ , no edit). Subsequent rows: SoftPaint and DD outputs at $r{=}0.4$ , $0.8$ , $1.0$ .

Diffusion-Based Video Editing

See all 27 papers

Masked Generative Transformer Is What You Need for Image Editing
May 11, 2026Wei Chow, Linfeng Li, Xian Sun +14Transformer Text-Guided Image Editing
1ByteDance · 2National University of Singapore · 3Duke University +2
Refinement Is Inherently Editable: Training-Free Prompt-to-Prompt Image Editing with Generative Refinement Network
Sep 17, 2026Yulong Chen, Ziqian Zhang, Haoyu Zhang +4Text-Guided Image Editing Training-Free Image Editing
City University of Hong Kong (Dongguan), Guangdong, China · City University of Hong Kong, Hong Kong, China · Mohamed bin Zayed University of Artificial Intelligence, Masdar, Abu Dhabi
Prompt-Guided Image Editing with Masked Logit Nudging in Visual Autoregressive Models
Apr 16, 2026Amir El-Ghoussani, Marc Hölle, Gustavo Carneiro +1Text-Guided Image Editing Visual Autoregressive Models
Friedrich-Alexander-Universität Erlangen-Nürnberg, Germany · University of Surrey, United Kingdom

May 11, 2026J/K move · Enter open · S save

Masked Generative Transformer Is What You Need for Image Editing

Wei Chow, Linfeng Li, Xian Sun, Lingdong Kong+13

1ByteDance · 2National University of Singapore · 3Duke University+2

Transformer Text-Guided Image Editing Image Editing

Open paper AskPDF arXiv

Diffusion models dominate image editing, yet their global denoising mechanism entangles edited regions with surrounding context, causing modifications to propagate into areas that should remain intact. We propose a fundamentally different approach by leveraging Masked Generative Transformers (MGTs), whose localized token-prediction paradigm naturally confines changes to intended regions. We present EditMGT, an MGT-based editing framework that is the first of its kind. Our approach employs multi-layer attention consolidation to aggregate cross-attention maps into precise edit localization signals, and region-hold sampling to explicitly prevent token flipping in non-target areas. To support training, we construct CrispEdit-2M, a 2M-sample high-resolution (>1024) editing dataset spanning seven categories. With only 960M parameters, EditMGT achieves state-of-the-art image similarity on multiple benchmarks while delivering 6x faster editing, demonstrating that MGTs offer a compelling alternative to diffusion-based editing.

Abstract

Diffusion-Based Video Editing

Explore similar work

Masked Generative Transformer Is What You Need for Image Editing

Refinement Is Inherently Editable: Training-Free Prompt-to-Prompt Image Editing with Generative Refinement Network

Prompt-Guided Image Editing with Masked Logit Nudging in Visual Autoregressive Models