Organizations: Department of Electrical Engineering, Indian Institute of Science, Bangalore, India · Department of Electrical and Computer Engineering, Northwestern University, Evanston, Illinois, USA · Department of Electrical and Computer Engineering, University of Rochester, Rochester, NY, USA
The human brain effortlessly deciphers colors and textures from visual stimuli and reconstructs the world around us, yet preserving such details remains challenging in image denoising. While significant progress has been made in generic image denoising, the problem of texture denoising has remained relatively under-explored. Fine details, such as intricate patterns and repetitive structures, often suffer from over-smoothing even in deep learning based approaches for denoising. In this paper, we analyze the behavior of state-of-the-art (SOTA) denoisers on texture datasets and show that performance improvements are possible only by fine-tuning the denoisers on the texture datasets. Surprisingly, fine-tuning on texture datasets result in the models catastrophically forgetting the original dataset. These findings highlight the need for specialized treatment of textures in denoising. Since denoising quality directly impacts downstream tasks such as generative modelling, we argue that dedicated texture aware denoisers are essential for faithful texture reconstruction and synthesis.
Figures & tables
Fig. 1 : Comparison of a noisy and denoised texture using pretrained MIRNet [ 23 ] and its finetuned version (MIRNet † ). PSNR and SSIM values improve after finetuning.
Fig. 2 : Visual comparison of various denoising models on the NUUR-Texture500 [ 14 ] dataset. Top row : noisy and pre-trained model results, bottom row : reference and (Model † ) fine-tuned models. PSNR is shown for each result relative to the reference.
Fig. 3 : Visual comparison of denoising models on the SIDD [ 1 ] . Top row : noisy and pre-trained models, bottom row : reference and (Model † ) fine-tuned models. PSNR (dB) shown w.r.t. the reference.
Model
PSNR ↑
SSIM ↑
MS-SSIM ↑
CW-SSIM ↑
DISTS ↓
STSIM1 ↑
STSIM2 ↑
LPIPS ↓
Noisy
26.23 ± 0.36
0.860 ± 0.123
0.976 ± 0.024
0.962 ± 0.077
0.157 ± 0.076
0.989 ± 0.023
0.941 ± 0.052
0.1286 ± 0.1488
CAT [ 4 ]
27.73 ± 4.542
0.834 ± 0.149
0.956 ± 0.074
0.931 ± 0.127
0.165 ± 0.077
0.954 ± 0.092
0.926 ± 0.062
0.1904 ± 0.1778
CAT †
34.17 ± 3.389
0.959 ± 0.032
0.990 ± 0.009
0.973 ± 0.054
0.062 ± 0.034
0.997 ± 0.006
0.949 ± 0.050
0.0304 ± 0.0283
CycleISP [ 22 ]
25.52 ± 3.759
0.781 ± 0.192
0.933 ± 0.099
0.872 ± 0.212
0.239 ± 0.089
0.929 ± 0.136
0.911 ± 0.077
0.3047 ± 0.2425
CycleISP †
34.20 ± 3.346
0.959 ± 0.032
0.998 ± 0.005
0.973 ± 0.053
0.062 ± 0.034
0.998 ± 0.005
0.948 ± 0.050
0.0306 ± 0.0296
MIRNet [ 23 ]
25.23 ± 5.765
0.721 ± 0.204
0.917 ± 0.104
0.913 ± 0.144
0.232 ± 0.081
0.909 ± 0.127
0.909 ± 0.070
0.3425 ± 0.2464
TABLE I : Denoising performance (mean ± std) on NUUR-Texture500 dataset [ 14 ] using PSNR, SSIM, MS-SSIM, CW-SSIM, DISTS, STSIM1, STSIM2 and LPIPS . Dagger ( † ) indicates fine-tuned models.
Model
PSNR ↑
SSIM ↑
Noisy
27.86 ± 5.17
0.52 ± 0.23
MIRNet [ 23 ]
42.59 ± 4.08
0.96 ± 0.06
MIRNet †
32.68 ± 6.75
0.72 ± 0.26
MPRNet [ 24 ]
42.60 ± 4.10
0.96 ± 0.06
MPRNet †
32.48 ± 6.80
0.72 ± 0.26
CycleISP [ 22 ]
42.21 ± 4.12
0.95 ± 0.06
TABLE II : Original vs. fine-tuned ( † ) model performance on SIDD [ 1 ] .
We present a generative method for texture filtering, which exhibits surprisingly good performance and generalizability. Our core idea is to empower texture filtering by taking full advantage of the strong learned image prior of pre-trained generative models. To this end, we propose to fine-tune a pre-trained generative model via a two-stage strategy. Specifically, we first conduct supervised fine-tuning on a very small set of paired images, and then perform reinforcement fine-tuning on a large-scale unlabeled dataset under the guidance of a reward function that quantifies the quality of texture removal and structure preservation. Extensive experiments show that our method clearly outperforms previous methods, and is effective to deal with previously challenging cases. Our code is available at https://github.com/OnlyZZZZ/Generative_Texture_Filtering.
Rongjia Zheng, Shangwei Huang, Lei Zhu +2
Sun Yat-sen University, China · Hong Kong University of Science and Technology (Guangzhou), China
Text-to-image diffusion models generate images by gradually converting white Gaussian noise into a natural image. White Gaussian noise is well suited for producing diverse outputs from a single text prompt due to its absence of structure. However, this very property limits control over, and predictability of, specific visual attributes, as the noise is not human-interpretable. In this work, we investigate the characteristics of the input noise in diffusion models. We show that, although all frequencies in white Gaussian noise have comparable statistical energy, low-frequency components primarily determine the images global structure and color composition, while high-frequency components control finer details. Building on this observation, we demonstrate that simple manipulations of the low-frequency noise using low-frequency image priors can effectively condition the generation process to reconstruct these low-frequency visual cues. This allows us to define a simple, training-free method with minimal overhead that steers overall image structure and color, while letting high-frequency components freely emerge as fine details, enabling variability across generated outputs.
We propose a novel training-free method for inpainting with off-the-shelf text-to-image models. While guidance-based methods in theory allow generic models to be used for inverse problems such as inpainting -- in practice their effectiveness is limited, leading to the necessity of specialized inpainting-specific models. In this work, we argue the missing ingredient for training-free generic model usage is proper optimization of the initial noise sample. We optimize the initial noise to approximately reproduce the unmasked image, in as few as tens of optimization steps, then use it with a conventional training-free inpainting method. Critically, we propose two core ideas that make this possible: (i) we perform linear approximation that avoids the costly and often impractical unrolling required to relate the initial noise sample to model output -- which potentially is why this relationship was previously overlooked; and (ii) perform spectral preconditioning by optimizing the initial noise sample in the spectral domain with Adam, which stabilizes the optimization. We demonstrate our method on various inpainting tasks, outperforming the state of the art. Project website: https://ubc-vision.github.io/sonic/
Seungyeon Baek, Erqun Dong, Shadan Namazifard +2
University of British Columbia, Canada · Google DeepMind, USA