RefineAnything: Multimodal Region-Specific Refinement for Perfect Local Details

Dewei Zhou; You Li; Zongxin Yang; Yi Yang

cs.CVApr 8, 2026

RefineAnything: Multimodal Region-Specific Refinement for Perfect Local Details

Authors: Dewei Zhou, You Li, Zongxin Yang, Yi Yang

Organizations: RELER, CCAI, Zhejiang University · DBMI, HMS, Harvard University

Abstract

We introduce region-specific image refinement as a dedicated problem setting: given an input image and a user-specified region (e.g., a scribble mask or a bounding box), the goal is to restore fine-grained details while keeping all non-edited pixels strictly unchanged. Despite rapid progress in image generation, modern models still frequently suffer from local detail collapse (e.g., distorted text, logos, and thin structures). Existing instruction-driven editing models emphasize coarse-grained semantic edits and often either overlook subtle local defects or inadvertently change the background, especially when the region of interest occupies only a small portion of a fixed-resolution input. We present RefineAnything, a multimodal diffusion-based refinement model that supports both reference-based and reference-free refinement. Building on a counter-intuitive observation that crop-and-resize can substantially improve local reconstruction under a fixed VAE input resolution, we propose Focus-and-Refine, a region-focused refinement-and-paste-back strategy that improves refinement effectiveness and efficiency by reallocating the resolution budget to the target region, while a blended-mask paste-back guarantees strict background preservation. We further introduce a boundary-aware Boundary Consistency Loss to reduce seam artifacts and improve paste-back naturalness. To support this new setting, we construct Refine-30K (20K reference-based and 10K reference-free samples) and introduce RefineEval, a benchmark that evaluates both edited-region fidelity and background consistency. On RefineEval, RefineAnything achieves strong improvements over competitive baselines and near-perfect background preservation, establishing a practical solution for high-precision local refinement. Project Page: https://limuloo.github.io/RefineAnything/.

Figures & tables

Figure 2: Architecture of RefineAnything. Given an input image and an optional reference image, the user specifies an edit region via a scribble mask; the images, region cue, and text instruction are encoded by a frozen Qwen2.5-VL encoder into multimodal conditioning tokens. Conditioned on these tokens, a diffusion backbone built from MMDiT blocks (trainable, e.g., via LoRA [ 15 , 47 ] ) denoises a VAE latent from timestep $t$ to produce the locally refined result.

Figure 3: Motivation for Focus-and-Refine. We compare VAE reconstructing a local region (red box) from the full image versus first cropping the region and resizing it to original full image resolution before VAE encoding. Although the crop-and-resize step does not introduce new information, it substantially improves the reconstruction quality within the target region. This observation suggests that, under a fixed input resolution, directing the model to focus on the local area rather than the entire image leads to better detail recovery for region-specific refinement.

Figure 4: Overview of Focus-and-Refine Method.

Figure 5: Overview of Reference-Based Refine Data Construction Pipeline.

Figure 6: Qualitative Result on Reference-Based Refinement.

Method	MSE $\downarrow$	LP $\downarrow$	VGG $\downarrow$	DINO $\uparrow$	CLIP $\uparrow$	SSIM $\uparrow$	MSE ${}_{\text{bg}}$ $\downarrow$	LP ${}_{\text{bg}}$ $\downarrow$	SSIM ${}_{\text{bg}}$ $\uparrow$
Gemini2.5	0.049	0.250	0.592	0.717	0.817	0.423	0.201	0.103	0.7662
Gemini3	0.031	0.178	0.431	0.771	0.855	0.510	0.029	0.052	0.9061
GPT4o	0.083	0.370	0.918	0.620	0.801	0.302	0.815	0.309	0.6001
OmniGen2	0.155	0.602	1.691	0.384	0.717	0.219	2.094	0.624	0.4300
BAGEL	0.045	0.253	0.611	0.682	0.803	0.494	0.033	0.046	0.9360
Kontext	0.040	0.264	0.540	0.685	0.785	0.538	0.011	0.019	0.9660

Table 1: Evaluation on Reference-Based Image Refinement.

Figure 7: Qualitative Result on Reference-Free Refinement.

Method	VQ $\uparrow$	Nat. $\uparrow$	Aes. $\uparrow$	Det. $\uparrow$	Faith. $\uparrow$
OmniGen2	2.501	2.500	2.461	2.348	2.586
BAGEL	3.018	3.000	2.959	2.851	3.135
kontext	1.716	2.114	1.982	1.690	1.750
Qwen-Edit	3.081	3.110	3.105	2.975	3.214
Ours	3.806	3.868	3.876	3.720	3.644

Table 2: Evaluation on the Reference-Free Image Refinement.

Method	MSE $\downarrow$	LP $\downarrow$	VGG $\downarrow$	DINO $\uparrow$	CLIP $\uparrow$	SSIM $\uparrow$	MSE ${}_{\text{bg}}$ $\downarrow$	LP ${}_{\text{bg}}$ $\downarrow$	SSIM ${}_{\text{bg}}$ $\uparrow$
w/o focus	0.021	0.177	0.449	0.779	0.869	0.578	0.005	0.022	0.9601
w/o loss	0.023	0.191	0.482	0.736	0.858	0.563	0.000	0.000	0.9997
Ours	0.020	0.155	0.401	0.793	0.885	0.591	0.000	0.000	0.9997

Table 3: Ablation on Focus-and-Refine and Boundary Consistency Loss.

Figure 8: Ablation of the Focus-and-Refine strategy.

Figure 9: Ablation of the Boundary Consistency Loss.

Image Editing

Image editing changes selected visual content while preserving the parts of an image that should remain unchanged.

See all 315 papers

Refinement Is Inherently Editable: Training-Free Prompt-to-Prompt Image Editing with Generative Refinement Network
Sep 17, 2026Yulong Chen, Ziqian Zhang, Haoyu Zhang +4Text-Guided Image Editing Training-Free Image Editing
City University of Hong Kong (Dongguan), Guangdong, China · City University of Hong Kong, Hong Kong, China · Mohamed bin Zayed University of Artificial Intelligence, Masdar, Abu Dhabi
FineEdit: Fine-Grained Image Edit with Bounding Box Guidance
Apr 13, 2026Haohang Xu, Lin Liu, Zhibo Zhang +3Image Editing Diffusion-Based Image Editing
Huawei Inc.
HierEdit: Region-Aware Hierarchical Diffusion for Efficient High-Resolution Editing
May 17, 2026Yuyao Zhang, Alexander Huang-Menders, Yu-Wing TaiDiffusion Transformer Diffusion-Based Image Editing
Dartmouth College

Sep 17, 2026J/K move · Enter open · S save

Refinement Is Inherently Editable: Training-Free Prompt-to-Prompt Image Editing with Generative Refinement Network

Yulong Chen, Ziqian Zhang, Haoyu Zhang, Ao He+3

City University of Hong Kong (Dongguan), Guangdong, China · City University of Hong Kong, Hong Kong, China · Mohamed bin Zayed University of Artificial Intelligence, Masdar, Abu Dhabi

Text-Guided Image Editing Training-Free Image Editing Image Editing

Open paper AskPDF arXiv

Text-guided image editing must introduce the requested changes while preserving unrelated source content. In training-free editing, diffusion editors often use spatial controls whose inaccuracies can leave edits incomplete or alter unrelated regions. Causal autoregressive editors face a further constraint: their fixed decoding order limits revision of earlier decisions. As the first to explore training-free image editing with Generative Refinement Networks (GRN), we observe that its refinement process is inherently suitable for editing and offers a promising way to address these limitations. Motivated by this observation, we introduce RefineEdit, a training-free prompt-to-prompt image editing framework built on the GRN. Our key idea is to couple edit localization with content generation through the global refinement of binary image codes, allowing editing evidence to be revised as the image evolves. More specifically, RefineEdit combines bit routing with two stabilization mechanisms: adaptive spatial freezing and finite bit locking. Bit routing starts from an intermediate source state and uses signed probability differences between the two branches to identify editable positions and bits. It directs selected bits toward editing refinement while anchoring the rest to the evolving source trajectory. Adaptive spatial freezing limits unnecessary expansion of the editing region, while finite bit locking maintains recent bit activations to support continued editing. The overall framework requires no additional training, external masks, or attention control. Across nine editing categories of PIE-Bench, RefineEdit achieves the best background-preservation scores in PSNR, LPIPS, MSE, and SSIM, together with the highest whole-image and edited-region CLIP scores among the evaluated methods. Code is available at https://github.com/mura1n/RefineEdit.

Abstract

Image Editing

Explore similar work

Refinement Is Inherently Editable: Training-Free Prompt-to-Prompt Image Editing with Generative Refinement Network

FineEdit: Fine-Grained Image Edit with Bounding Box Guidance

HierEdit: Region-Aware Hierarchical Diffusion for Efficient High-Resolution Editing