RefineAnything: Multimodal Region-Specific Refinement for Perfect Local Details
Organizations: RELER, CCAI, Zhejiang University · DBMI, HMS, Harvard University
Abstract
We introduce region-specific image refinement as a dedicated problem setting: given an input image and a user-specified region (e.g., a scribble mask or a bounding box), the goal is to restore fine-grained details while keeping all non-edited pixels strictly unchanged. Despite rapid progress in image generation, modern models still frequently suffer from local detail collapse (e.g., distorted text, logos, and thin structures). Existing instruction-driven editing models emphasize coarse-grained semantic edits and often either overlook subtle local defects or inadvertently change the background, especially when the region of interest occupies only a small portion of a fixed-resolution input. We present RefineAnything, a multimodal diffusion-based refinement model that supports both reference-based and reference-free refinement. Building on a counter-intuitive observation that crop-and-resize can substantially improve local reconstruction under a fixed VAE input resolution, we propose Focus-and-Refine, a region-focused refinement-and-paste-back strategy that improves refinement effectiveness and efficiency by reallocating the resolution budget to the target region, while a blended-mask paste-back guarantees strict background preservation. We further introduce a boundary-aware Boundary Consistency Loss to reduce seam artifacts and improve paste-back naturalness. To support this new setting, we construct Refine-30K (20K reference-based and 10K reference-free samples) and introduce RefineEval, a benchmark that evaluates both edited-region fidelity and background consistency. On RefineEval, RefineAnything achieves strong improvements over competitive baselines and near-perfect background preservation, establishing a practical solution for high-precision local refinement. Project Page: https://limuloo.github.io/RefineAnything/.
Figures & tables
| Method | MSE | LP | VGG | DINO | CLIP | SSIM | MSE | LP | SSIM |
| Gemini2.5 | 0.049 | 0.250 | 0.592 | 0.717 | 0.817 | 0.423 | 0.201 | 0.103 | 0.7662 |
| Gemini3 | 0.031 | 0.178 | 0.431 | 0.771 | 0.855 | 0.510 | 0.029 | 0.052 | 0.9061 |
| GPT4o | 0.083 | 0.370 | 0.918 | 0.620 | 0.801 | 0.302 | 0.815 | 0.309 | 0.6001 |
| OmniGen2 | 0.155 | 0.602 | 1.691 | 0.384 | 0.717 | 0.219 | 2.094 | 0.624 | 0.4300 |
| BAGEL | 0.045 | 0.253 | 0.611 | 0.682 | 0.803 | 0.494 | 0.033 | 0.046 | 0.9360 |
| Kontext | 0.040 | 0.264 | 0.540 | 0.685 | 0.785 | 0.538 | 0.011 | 0.019 | 0.9660 |
| Method | VQ | Nat. | Aes. | Det. | Faith. |
|---|---|---|---|---|---|
| OmniGen2 | 2.501 | 2.500 | 2.461 | 2.348 | 2.586 |
| BAGEL | 3.018 | 3.000 | 2.959 | 2.851 | 3.135 |
| kontext | 1.716 | 2.114 | 1.982 | 1.690 | 1.750 |
| Qwen-Edit | 3.081 | 3.110 | 3.105 | 2.975 | 3.214 |
| Ours | 3.806 | 3.868 | 3.876 | 3.720 | 3.644 |
| Method | MSE | LP | VGG | DINO | CLIP | SSIM | MSE | LP | SSIM |
|---|---|---|---|---|---|---|---|---|---|
| w/o focus | 0.021 | 0.177 | 0.449 | 0.779 | 0.869 | 0.578 | 0.005 | 0.022 | 0.9601 |
| w/o loss | 0.023 | 0.191 | 0.482 | 0.736 | 0.858 | 0.563 | 0.000 | 0.000 | 0.9997 |
| Ours | 0.020 | 0.155 | 0.401 | 0.793 | 0.885 | 0.591 | 0.000 | 0.000 | 0.9997 |
Image Editing
Image editing changes selected visual content while preserving the parts of an image that should remain unchanged.
See all 315 papers