ReState: Noise-Level-Aware State Correction for Training-Free Object Removal in Few-Step Diffusion Models

Donghwan Lee; Sanghyun Jo; Sihyun Kim; Ziseok Lee; Kyungsu Kim

cs.CVOct 15, 2026

ReState: Noise-Level-Aware State Correction for Training-Free Object Removal in Few-Step Diffusion Models

Authors: Donghwan Lee, Sanghyun Jo, Sihyun Kim, Ziseok Lee, Kyungsu Kim

Organizations: Seoul National University · OGQ

Abstract

Training-free object removal commonly redirects self-attention away from the target region, but on few-step distilled diffusion models this often leaves object-shaped residuals. We identify a temporal mismatch: a structured prediction forms inside the mask at the first forward, and the short trajectory offers few later chances to revise it, while attention rerouting acts mainly in later steps. Across four distilled backbones, magnitude-matched single-forward interventions show that background-directed state correction affects the final result more than attention rerouting. We therefore propose ReState, which corrects the masked clean prediction toward background evidence retrieved from the input using a noise-level-aware weight that vanishes at the final forward. ReState retains object-excluding attention as a complementary constraint and adds no denoiser evaluations. Across two paired benchmarks and two few-step backbones, ReState achieves the best masked PSNR, SSIM, and LPIPS among training-free methods, improving PSNR by 3.17 dB on average over the strongest baseline. On OBER-Test, it also outperforms the 50-step attention-forcing reference on all three metrics at 6.7x lower latency. The same configuration transfers to two additional distilled backbones without retuning. Project page: https://tdrlee.github.io/ReState

Figures & tables

Figure 1: Motivation. With attention forcing applied, the multi-step model (SDXL) starts diffuse and forms the background gradually along the trajectory (top), while the few-step model (SDXL-Turbo) already fixes a structured image in the mask at the first forward, and the short trajectory does not absorb it. This committed structure is what attention forcing then fails to remove, leaving a residual (bottom); ReState instead corrects the prediction directly and removes it.

Property	DesignEdit AAAI’25	Attentive Eraser AAAI’25	AdaEraser ICML’26	PANDORA ICME’26	FuLLaMa WACV’26	ReState (Ours)
Writes an explicit background target into the predicted state	✗	✗	✗	✗	✗	✓
Adds no forwards or external models beyond the backbone	✓	✗	✗	✗	✗	✓
Operates at the native few-step schedule	✗	✗	✗	✗	✗	✓

Table 1: ReState is the only training-free removal method that writes an explicit background target into the predicted state at the native few-step cost. Cells classify each method as published (Appendix B.2 ); they state design properties, not measured performance, which Table 2 reports. The comparison covers training-free removal methods; training-based methods are evaluated in Appendix E.1 .

Figure 2: Overview of ReState. Features from the first generation forward retrieve background evidence $B$ from unmasked source regions without an additional denoiser evaluation. At each step, ReState moves the masked clean prediction $\hat{x}_{0,t}$ toward $B$ with a noise-level-aware weight $\omega_{t}$ . The weight vanishes at the final forward, while object-excluding attention remains active throughout sampling.

Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	Lat. (s) $\downarrow$	VRAM $\downarrow$
	OBER-Test			RORD
SD-Turbo (4 steps)	11.27	0.203	0.380	9.86	0.162	0.438	0.174	4.58
$+$ Attentive Eraser AAAI’25	7.28	0.063	0.516	7.22	0.060	0.555	0.200	4.58
$+$ DesignEdit AAAI’25	14.74	0.316	0.283	13.48	0.263	0.355	0.184	4.58
$+$ AdaEraser ICML’26	12.27	0.237	0.347	10.48	0.177	0.420	0.843	5.08
$+$ ReState (Ours)	18.19	0.485	0.186	16.59	0.440	0.280	0.305	4.58

Table 2: Object removal on OBER-Test and RORD: on the same frozen backbone and the same nominal 4-step schedule, ReState opens a large gap on every masked metric. Rows with

+

apply the method on the backbone of their block; bold is best per column. Latency is the median per image and VRAM the peak allocation in GB, both on OBER-Test (Appendix B ). Effective denoiser evaluations, which differ for batch-expanded baselines, are reported in Table 5 . Training-based comparisons are in Appendix E.1 .

Figure 3: Qualitative results on GT-paired benchmarks. Previous training-free methods often leave object-shaped residuals in the mask, whereas ReState more closely reconstructs the paired background.

	SDXL-Turbo		SD-Turbo		SDXL-Lightning		Hyper-SDXL
$\sigma$	$E_{\mathrm{AF}}$	$E_{\mathrm{direct}}$	$E_{\mathrm{AF}}$	$E_{\mathrm{direct}}$	$E_{\mathrm{AF}}$	$E_{\mathrm{direct}}$	$E_{\mathrm{AF}}$	$E_{\mathrm{direct}}$
14.6	$+.002$	$\mathbf{+.025}$	$-.001$	$\mathbf{+.029}$	$+.034$	$\mathbf{+.072}$	$+.030$	$\mathbf{+.052}$
4.1	$+.016$	$\mathbf{+.039}$	$+.018$	$\mathbf{+.045}$	$+.052$	$\mathbf{+.059}$	$+.049$	$\mathbf{+.062}$
1.6	$+.028$	$\mathbf{+.040}$	$+.026$	$\mathbf{+.036}$	$+.029$	$\mathbf{+.041}$	$+.027$	$\mathbf{+.042}$
0.7	$+.033$	$\mathbf{+.049}$	$+.021$	$\mathbf{+.040}$	$+.023$	$\mathbf{+.046}$	$+.020$	$\mathbf{+.046}$

Table 3: Direct correction acts from the first forward. Terminal effect

E

of a single matched intervention across noise levels and distilled backbones. Direct correction (

E_{\mathrm{direct}}

, bold) exceeds attention forcing (

E_{\mathrm{AF}}

) at every noise level and backbone.

Direct correction	AF (all forwards)	AF (last only)	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
✗	✗	✗	9.86	0.133	0.435
✗	✗	✓	12.15	0.182	0.370
✗	✓	✗	15.50	0.348	0.269
✓	✗	✗	17.62	0.477	0.237
✓	✗	✓	18.65	0.495	0.196
✓	✓	✗	18.75	0.498	0.194

Table 4: Component and timing ablation. Direct correction supplies most of the gain; attention forcing contributes mainly at the last forward.

Method	Forwards	Latency (s) $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
SDXL-Turbo	4	0.247	9.86	0.133	0.435
$+$ Attentive Eraser AAAI’25	8	0.338	7.09	0.046	0.514
$+$ DesignEdit AAAI’25	4	0.281	15.50	0.348	0.269
$+$ AdaEraser ICML’26	8	1.145	11.71	0.205	0.372
$+$ ReState (Ours)	4	0.373	18.75	0.498	0.194
SDXL $+$ DesignEdit AAAI’25	50	2.50	17.86	0.441	0.209

Table 5: Cost and comparison against the multi-step reference. Latency is median per image; forwards counts the effective model passes.

Evidence $B$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Global background mean	18.16	0.481	0.214
Retrieved (default)	18.75	0.498	0.194
Ground-truth background (oracle)	22.32	0.588	0.133

Table 6: Evidence quality, not the correction mechanism, is the remaining bottleneck. Each row supplies a different evidence

B

to the identical correction.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Training or adaptation data	OBER	RORD
PowerPaint ECCV’24	OpenImages v6 (seg. subset) + LAION-Aesthetics v2 5+	✗	✗
CLIPAway NeurIPS’24	frozen base; embedding projection trained on COCO	✗	✗
EntityErasure CVPR’25	Open Images V7, about 1.75M (pseudo-labeled)	✗	✗
SmartEraser CVPR’25	Syn4Removal, 1M (Open Images v7, SAM, COCONut)	✗	✗
EraseLoRA ECCV’26	none (dataset-free test-time adaptation)	✗	✗
OSOR (SDXL-Inp.) ECCV’26	CORNE, 280K pairs (curated from NHR-Edit)	✗	✗

Appendix

Table B1: Training data of the training-based removal methods and its overlap with the evaluation benchmarks. ✓ : stated to include the benchmark family; ✗ : no stated overlap.

Method	Nominal	Effective	Auxiliary pass
SDXL-Turbo	4	4	none
$+$ Attentive Eraser AAAI’25	4	8	SARG branch pair
$+$ DesignEdit AAAI’25	4	4	none
$+$ AdaEraser ICML’26	8	8	reference forward
$+$ ReState (Ours)	4	4	none

Appendix

Table B2: ReState adds no auxiliary pass and stays at the native forward count. Hook-measured on the denoiser under the native four-step schedule: nominal counts backbone calls and effective sums their batch sizes (Appendix B ), so the two doubling mechanisms differ; Attentive Eraser duplicates the batch, AdaEraser calls twice. ReState’s retrieval probe is shared with the first generation forward and adds none.

Figure C1: The first-forward asymmetry holds image by image on the distilled backbones and vanishes with the original weights. Each point is one image; the diamond marks the mean of Table C1 , and

d>0

is the fraction of images above the diagonal, where the direct correction wins. Axes are clipped at the

0.5

th and

99.5

th percentiles and scaled per panel, as the backbones operate at different resolutions.

Backbone	$E_{\mathrm{AF}}$	$E_{\mathrm{direct}}$	$d$	$p(E_{\mathrm{AF}}{=}0)$	$p(d{=}0)$
SDXL-Turbo	$+0.0020$	$+0.0251$	$+0.0231$	$0.27$	$1.0\times 10^{-4}$
SD-Turbo	$-0.0013$	$+0.0289$	$+0.0303$	$0.52$	$1.0\times 10^{-4}$
SDXL-Lightning	$+0.0339$	$+0.0723$	$+0.0384$	$1.0\times 10^{-4}$	$1.0\times 10^{-4}$
Hyper-SDXL	$+0.0302$	$+0.0516$	$+0.0214$	$1.0\times 10^{-4}$	$1.0\times 10^{-4}$
Original SDXL (4-step grid)	$+0.0087$	$+0.0088$	$+0.0001$	$1.0\times 10^{-4}$	$0.91$

Appendix

Table C1: At the first forward, attention forcing is not significantly different from zero on the Turbo family, while the direct correction and the gap

d=E_{\mathrm{direct}}-E_{\mathrm{AF}}

are not. Means restate the first row of Table 3 at full precision. The p-values are paired permutation tests (Appendix B.1 ), whose resolution floor is

1.0\times 10^{-4}

; the original-SDXL row is the weight control.

Forward index	Noise level $\sigma$	$E_{\mathrm{AF}}$	$E_{\mathrm{direct}}$
1	13.12	$+0.0015$	$+0.0012$
4	9.34	$+0.0024$	$+0.0020$
8	6.20	$+0.0030$	$+0.0033$
13	3.95	$+0.0036$	$+0.0040$
19	2.46	$+0.0028$	$+0.0036$
25	1.62	$+0.0027$	$+0.0033$

Appendix

Table C2: On the reference trajectory both single-forward interventions stay an order of magnitude below the few-step effects. Exact values at the eight probed noise levels; the corresponding curves are the right panel of Figure C2 .

Figure C2: On the reference trajectory both interventions are an order of magnitude weaker and the first-forward asymmetry vanishes. Terminal intervention effect

E

of a single intervened forward at each noise level; bands are

\pm 1

standard error over

n=163

images. The left panel plots the SDXL-Turbo column of Table 3 ; the right panel plots Table C2 , on a vertical scale an order of magnitude smaller.

Denoising evaluations	$E_{\mathrm{AF}}$	$p(E_{\mathrm{AF}}{=}0)$	$E_{\mathrm{direct}}$	$p(E_{\mathrm{direct}}{=}0)$
4	$+0.0020$	$0.27$	$+0.0251$	$1.0\times 10^{-4}$
8	$+0.0018$	$0.21$	$+0.0122$	$1.0\times 10^{-4}$
16	$+0.0018$	$0.09$	$+0.0059$	$1.0\times 10^{-4}$

Appendix

Table C3: Densifying the same Turbo weights shrinks the direct effect and does not unlock attention forcing. First-forward effects at 4, 8, and 16 denoising evaluations; the native row anchors to Table C1 .

Placement	$\sigma$ range	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
8 forwards, high-noise range	$13.1\rightarrow 6.2$	11.57	0.255	0.399
8 forwards, late range	$0.44\rightarrow 0.04$	12.47	0.284	0.353
Last 38 forwards	$3.95\rightarrow 0.04$	17.17	0.429	0.221
All 50 forwards	$13.1\rightarrow 0.04$	17.86	0.441	0.209

Appendix

Table C4: At a fixed count, late attention forcing beats high-noise placement, and only the full schedule reaches the reference. 50-step SDXL with the guidance of the reference configuration; the all-50 row is the 50-step attention-forcing reference of Table 5 .

Output	High-band ratio to ground truth
No intervention	8.34
Attention forcing only	1.92
Direct correction only	0.30
ReState	0.51

Appendix

Table C5: Attention forcing leaves object-shaped high-frequency structure; ReState eliminates it. Masked high-band energy as a per-image ratio to the identically processed ground truth, geometric mean over OBER-Test (SDXL-Turbo).

	Realized $\omega$			OBER-Test
$\lambda$	$\omega_{1}$	$\omega_{2}$	$\omega_{3}$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
1	0.943	0.722	0.325	18.61	0.487	0.194
2	0.971	0.839	0.490	18.70	0.492	0.193
4	0.985	0.912	0.658	18.74	0.496	0.194
8	0.993	0.954	0.794	18.75	0.498	0.194
16	0.996	0.977	0.885	18.73	0.498	0.195

Appendix

Table D1: Removal quality is insensitive to the correction strength

\lambda

. Realized weights

\omega_{t}=\lambda\sigma_{t-1}^{2}/(1+\lambda\sigma_{t-1}^{2})

at the first three forwards;

\omega_{4}=0

for all

\lambda

by the closed form. The default

\lambda=8

is highlighted.

Corrected forwards	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
None (attention forcing only)	15.50	0.348	0.269
$\{1\}$	16.85	0.411	0.237
$\{2\}$	18.18	0.469	0.203
$\{3\}$	18.47	0.482	0.200
$\{1,2\}$	18.35	0.477	0.198
$\{2,3\}$	18.74	0.496	0.194

Appendix

Table D2: The correction is cumulative and the final forward is left uncorrected. The direct correction is applied only at the listed forwards; object-excluding attention stays active at every forward in all rows. The last row keys

\omega

\sigma_{t}

, so the final forward retains a correction of

\omega_{4}\approx 0.79

whose output is final. Constant rows replace the closed-form weights by a single constant

c

while keeping

\omega_{4}=0

Figure D1: Removal quality is insensitive to the retained fraction

r

across three orders of magnitude. Masked metrics on OBER-Test with the bank widened up to the similarity-weighted average over the entire background (

r=1

); the stars mark the default

r=0.01

. Exact test statistics are given in the text.

Feature source	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Shared first forward (default)	18.75	0.498	0.194
No attention suppression	18.72	0.497	0.194
Near-clean input ( $\sigma=0.693$ )	17.44	0.474	0.214
Second forward ( $\sigma=4.08$ )	18.87	0.499	0.193
Encoder block (native $32^{2}$ )	17.96	0.475	0.216
Mid block (native $16^{2}$ )	18.75	0.499	0.194

Appendix

Table D4: The shared noisy probe is the operative choice: near-clean inputs contaminate the bank, while attention suppression and the feature layer admit slack. Each row rebuilds the background bank from a different feature source and regenerates with the identical four-forward pipeline; every variant harvests its features in a separate diagnostic pass, so the deployed method stays at the native count.

Training/Optimization-based methods
	OBER-Test			RORD
Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	Lat. (s)	VRAM
SDXL-Inpainting	11.65	0.365	0.391	10.57	0.348	0.403	3.75	9.0GB
FLUX.1-Fill	11.04	0.376	0.405	12.33	0.415	0.366	15.9	33.8GB
PowerPaint ECCV’24	14.89	0.463	0.255	14.91	0.438	0.282	3.86	5.1GB
CLIPAway NeurIPS’24	15.71	0.426	0.246	14.84	0.406	0.268	7.83	9.8GB

Appendix

Table E1: Without any training, ReState remains competitive with the training-based methods at the lowest latency. Latency and VRAM are measured as described in the text; the training-free rows repeat the SDXL-Turbo block of Table 2 .

	OBER-Test			RORD
Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
SDXL-Lightning (4 steps)	10.51	0.393	0.424	9.60	0.318	0.445
$+$ DesignEdit AAAI’25	15.95	0.517	0.296	15.08	0.475	0.325
$+$ ReState (Ours)	17.51	0.578	0.271	16.54	0.528	0.308
Hyper-SDXL (4 steps)	10.21	0.346	0.434	9.21	0.273	0.455
$+$ DesignEdit AAAI’25	15.20	0.487	0.315	14.14	0.434	0.358

Appendix

Table E2: The single frozen configuration transfers: ReState improves every masked metric on two further distilled backbones. Full removal at

1024^{2}

with the identical hyperparameters as Table 2 and no per-backbone tuning; rows with

+

apply the method on the backbone of their block. Bold is best per column within a block; test statistics are given in the text.

Configuration at 50 steps	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
$+$ DesignEdit AAAI’25 (reference)	17.86	0.441	0.209
$+$ Attentive Eraser AAAI’25	18.79	0.462	0.225
$+$ AdaEraser ICML’26	15.66	0.399	0.274
$+$ ReState, every forward corrected	17.57	0.479	0.231
$+$ ReState, corrections at $\sigma_{t-1}\geq 1.61$	18.83	0.485	0.181
ReState on SDXL-Turbo (4 steps; Table 2 )	18.75	0.498	0.194

Appendix

Table E3: Attention forcing recovers in its 50-step regime, and 4-step ReState matches the strongest 50-step configuration at

6.7\times

lower latency. OBER-Test under the 50-step reference configuration; the 4-step port numbers of the same methods are in Table 2 , and the correction-window sweep is given in the text.

		OBER-Test			RORD
Method	Configuration	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Attentive Eraser AAAI’25	SARG $9.0$ ( $1.0\times$ , official port)	7.09	0.046	0.514	7.35	0.065	0.538
	SARG $4.5$ ( $0.5\times$ )	7.61	0.054	0.509	7.54	0.056	0.537
	SARG $2.25$ ( $0.25\times$ )	12.92	0.264	0.377	12.56	0.277	0.416
	SARG $0.9$ ( $0.1\times$ , selected)	15.34	0.335	0.277	14.18	0.310	0.342
DesignEdit AAAI’25	all 4 forwards (official port, selected)	15.50	0.348	0.269	14.34	0.326	0.334

Appendix

Table E4: Limited re-tuning does not close the gap: the best four-step configuration of each baseline remains about

3

dB below ReState in masked PSNR. One pre-registered knob per method (SARG scale for Attentive Eraser; masked-forward window for DesignEdit), SDXL-Turbo, seed

0

, full benchmarks. Configuration selection is cross-benchmark by masked PSNR and both directions select the same configuration (marked).

Figure E1: Qualitative results on OBER-Test.

Object Removal

See all 27 papers

EraseLoRA: MLLM-Driven Foreground Exclusion and Background Subtype Aggregation for Dataset-Free Object Removal
Dec 25, 2025Sanghyun Jo, Donghwan Lee, Eunji Jung +2Object Removal Diffusion-Based Image Editing
OGQ, Seoul, Korea · Seoul National University, Seoul, Korea
OSOR: One-Step Diffusion Inpainting for Effect-Aware Object Removal
Jun 26, 2026Qinming Zhou, Chenxi Sun, Deyang Kong +6Image Inpainting Diffusion Model Acceleration
Shanghai Jiao Tong University · Tsinghua University · University of Electronic Science and Technology of China +3
ClickRemoval: An Interactive Open-Source Tool for Object Removal in Diffusion Models
May 14, 2026Ledun Zhang, Yatu Ji, Xufei Zhuang +1Attention Control in Diffusion Models Image Inpainting
Inner Mongolia University of Technology Hohhot, Inner Mongolia, China

Dec 25, 2025J/K move · Enter open · S save

EraseLoRA: MLLM-Driven Foreground Exclusion and Background Subtype Aggregation for Dataset-Free Object Removal

Sanghyun Jo, Donghwan Lee, Eunji Jung, Seong Je Oh+1

OGQ, Seoul, Korea · Seoul National University, Seoul, Korea

Object Removal Diffusion-Based Image Editing

Open paper AskPDF arXiv

Object removal must prevent the masked target from reappearing and reconstruct the occluded background with structural and contextual fidelity, rather than merely filling a hole plausibly. Recent dataset-free approaches manipulate the diffusion model's internal self-attention to prevent it from referencing the masked region, yet they fail in two critical ways: (i) they treat the masked region as the sole foreground, misinterpreting non-target objects as background and regenerating them, and (ii) they apply uniform attention constraints without distinguishing diverse background subtypes, leading to textural blurring and structural misalignment. Both failures stem from the absence of explicit background-aware reasoning. We propose EraseLoRA, a dataset-free framework that replaces attention surgery with background-aware reasoning and test-time adaptation. The first stage, Background-aware Foreground Exclusion (BFE), leverages a multimodal large-language model to separate target foreground, non-target foregrounds, and clean background from a single image-mask pair. The second stage, Background-aware Reconstruction with Subtype Aggregation (BRSA), performs test-time optimization that treats inferred background subtypes as complementary pieces, enforcing their consistent integration through reconstruction and alignment objectives without explicit attention intervention. As a model-agnostic plug-in applicable to diverse diffusion backbones, EraseLoRA reconstructs backgrounds at least 23% more faithful to the original scene than previous dataset-free methods while nearly halving unwanted foreground re-generation, and surpasses all dataset-driven approaches in both aspects despite requiring no training data. Code is available at https://shjo-april.github.io/EraseLoRA.

Abstract

Object Removal

Explore similar work

EraseLoRA: MLLM-Driven Foreground Exclusion and Background Subtype Aggregation for Dataset-Free Object Removal

OSOR: One-Step Diffusion Inpainting for Effect-Aware Object Removal

ClickRemoval: An Interactive Open-Source Tool for Object Removal in Diffusion Models