Residual Diffusion Implicit Models

João Guerreiro; Pedro Tomás; Helena Aidos; Jacinto C. Nascimento

cs.CVSep 26, 2026

Residual Diffusion Implicit Models

Authors: João Guerreiro, Pedro Tomás, Helena Aidos, Jacinto C. Nascimento

Organizations: INESC-ID, Instituto Superior Técnico, Universidade de Lisboa, Lisboa, Portugal · ISR-Lisboa, Instituto Superior Técnico, Universidade de Lisboa, Lisboa, Portugal · LASIGE, Faculdade de Ciências, Universidade de Lisboa, Lisboa, Portugal

Abstract

Diffusion models achieve state-of-the-art results across multiple tasks. However, in inverse problems, standard initialization from pure Gaussian noise misaligns the generative process with real-world degradations. More recent methods such as diffusion bridges impose strict endpoint constraints and often require long reverse processes that are prone to hallucinations. Alternative consistency models provide noise-invariant, one-step mappings but lack inherent variance modeling and can degrade under severe corruption. Hence, residual diffusion implicit models (RDIMs) are proposed, constituting a generalized framework that explicitly models the residuals between high-quality (HQ) and low-quality (LQ) images, aligning the forward process with the actual degradation. A non-Markovian implicit reverse sampler is derived, which can skip intermediate timesteps, enabling accurate few-step or even single-step reconstruction, while mitigating the hallucinations inherent to long diffusion chains. RDIM also introduces a controllable variance mechanism that interpolates between deterministic and stochastic sampling, balancing fidelity and diversity. Furthermore, it enables the straightforward use of perceptual losses, when needed. Experiments on denoising and super-resolution benchmarks demonstrate that RDIMs consistently outperforms the state of the art, including bridge and consistency models, in terms of PSNR, SSIM, and LPIPS, reducing hallucinations while requiring only a few sampling steps (often just one). The results position RDIMs as an efficient solution for a broad range of image restoration tasks.

Figures & tables

Figure 1: Overview of RDIM , a diffusion framework tailored for inverse problems, such as image reconstruction. The reverse process can accurately reconstruct back the data in $S\leq T$ steps.

Figure 2: Progression of weights $\beta_{t}$ and $\lambda_{t}$ across timesteps and impact of $p$ on the diffusion process.

Table 1: Comparative analysis of RDIM against relevant state-of-the-art techniques for (a) denoising and (b) SR. Green color highlights the best score overall and Blue color the second best.

Figure 4: RDIM - $10$ results in image inpainting, colorization, and deblurring on the FFHQ dataset. In (b), (c) and (d), the left side represents the input image and the right side the output.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: U-Net-based network. In the convolutional layers, the parameters $k$ , $st$ , and $pd$ represent the kernel size, stride, and padding, respectively. Additionally, $(*/s)$ denotes $(W/s,H/s)$ , where $s$ is a scale factor ( $s>1$ for SR tasks and $s=1$ otherwise).

Figure 6: Core blocks of the U-Net-based network. (a) Encoder Block, (b) Decoder Block, and (c) Bottleneck Block.

Figure 7: Building blocks. (a) Residual Block, (b) Self-Attention Block, (c) Upsample Block, and (d) Downsample Block.

Figure 8: Residual $\beta$ -schedule ( $p=5.0$ ) overlaid with red markers indicating the $\beta_{t}$ value at each sampling point. The schedule adopted selects timesteps uniformly spaced. In this illustration, the forward process involves $T=100$ timesteps and the number of sampling steps, where reverse transitions occur, is $S=10$ .

$p$	$S$	FMD-BPAE
$p$	$S$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
1.0	1	40.0836	0.9678	0.0205
	5	40.0772	0.9678	0.0205
	10	40.0565	0.9677	0.0205
5.0	1	40.1100	0.9681	0.0202
	5	40.0436	0.9679	0.0202

Appendix

Table 3: Impact of parameter

p

, which controls the steepness of the curve in the

\beta

-schedule. All RDIM configurations were trained using a forward process with

T=10

timesteps and variance hyperparameter

\gamma=9.0

. During inference, the reverse process variance hyperparameter is fixed to

\eta=1.0

. Orange color rows highlight ResShift scenarios, corresponding to particular cases where RDIM reduces to ResShift under the conditions

\eta=1.0

and

S=T

Figure 9: Impact of $\gamma$ on the diffusion process with $\beta$ -schedule parameter fixed to $p=5.0$ .

$T$	$S$	FMD-BPAE
		$\gamma=0.0$			$\gamma=3.0$			$\gamma=9.0$
		PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
10	1	38.3703	0.9575	0.0296	40.0998	0.9686	0.0196	40.1100	0.9681	0.0202
10	10	38.3487	0.9572	0.0299	39.3632	0.9644	0.0187	39.9524	0.9674	0.0202
50	1	38.4181	0.9578	0.0293	42.5354	0.9803	0.0093	43.1161	0.9821	0.0079
	10	38.4162	0.9578	0.0295	42.4566	0.9802	0.0079	43.2029	0.9824	0.0077

Appendix

Table 4: Denoising performance comparison of several RDIM configurations on BPAE confocal images from the benchmark dataset FMD. It exhibits the impact of the constant hyperparameter

\gamma

that controls the variance in the forward process and the impact of the number of diffusion timesteps during training (

T

) and inference (

S

). The constant hyperparameter that controls the variance in the reverse process is fixed to

\eta=1.0

. Orange color rows highlight ResShift scenarios, corresponding to cases where RDIM reduces to ResShift (

\eta=1.0

and

S=T

Figure 10: PSNR , SSIM , and LPIPS performance as a function of $S$ (number of sampling timesteps) on denoising of BPAE confocal images from the FMD dataset. RDIM is trained with $T=100$ and $\gamma=3.0$ . The constant hyperparameter that controls the variance in the reverse process is fixed to $\eta=1.0$ . The number of sampling timesteps evaluated includes a single-step prediction and then ranges from $10$ to $100$ in increments of $10$ .

$T$	$S$	$\eta$	FMD-BPAE
			$\gamma=0.0$			$\gamma=3.0$			$\gamma=9.0$
			PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
100	10	0.0	38.3752	0.9575	0.0295	44.1429	0.9855	0.0047	43.3732	0.9830	0.0073
		0.5	38.3752	0.9575	0.0295	44.1440	0.9855	0.0047	43.3738	0.9830	0.0073
		1.0	38.3752	0.9575	0.0295	44.1468	0.9855	0.0047	43.3743	0.9830	0.0073

Appendix

Table 5: Impact of the constant hyperparameter

\eta

, which controls the variance in the reverse process, on denoising BPAE confocal images from the benchmark dataset FMD.

Figure 11: Comparison of the perception-distortion trade-off between the proposed RDIM and the state-of-the-art on the DIV2K dataset for 4 $\times$ SR.

Method	FFHQ-Inpainting			FFHQ-Colorization			FFHQ-Deblurring
Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Corrupted	9.034	0.136	1.165	20.806 †	0.926 †	0.223 †	24.691	0.686	0.486
ResShift	31.721	0.922	0.022	25.320	0.948	0.103	28.331	0.812	0.089
RDIM- $1$	33.514	0.941	0.029	25.727	0.950	0.094	29.995	0.847	0.170
RDIM- $10$	32.330	0.931	0.025	25.565	0.949	0.099	28.870	0.826	0.111
† To compute colorization scores on corrupted images, the existing channel is replicated.

Appendix

Table 6: Performance on the FFHQ dataset for image inpainting, colorization, and deblurring.

Method	Poisson-corrupted DIV2K-Bicubic- $\times 4$
Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
ResShift	23.610 (-3.845)	0.521 (-0.259)	0.428 (+0.275)
GOUB-SDE	20.165 ( -6.725 )	0.335 ( -0.413 )	0.664 (+0.444)
UniDB-SDE	19.284 (-6.176)	0.314 (-0.372)	0.697 ( +0.518 )
CTMSR- $1$	22.987 (-4.100)	0.466 (-0.293)	0.495 (+0.365)
RDIM- $1$	26.260 ( -2.920 )	0.673 ( -0.151 )	0.519 (+0.262)

Appendix

Table 7: Evaluation on Poisson-corrupted DIV2K-Bicubic-

\times 4

images after training on the original dataset without adding Poisson noise. Values in parentheses indicate the performance drop relative to the evaluation on the original noise-free DIV2K-Bicubic-

\times 4

dataset (see Table 2 ). Red color indicates the worst performance drop overall and Green color the best.

Figure 12: Qualitative denoising analysis on cropped regions from the SIDD dataset. Since SIDD contains noisy images captured under challenging lighting conditions, brightness-adjusted crops of the marked regions are shown in the bottom row for enhanced visualization.

Figure 13: SR qualitative comparison on cropped regions from the DIV2K subset with unknown degradation.

Figure 14: SR qualitative comparison on cropped regions from the DIV2K subset with bicubic downsampled images.

Figure 15: Qualitative comparison of RDIM-PQ- $1$ on $\times 4$ SR . Cropped regions from the DIV2K subset, with bicubic downsampled images, suggest RDIM achieves greater structural and texture fidelity than bridge-based models.

Figure 16: Denoising results of RDIM - $10$ on images from the FMD and SIDD datasets. For improved visualization, only cropped regions are shown. RDIM is trained with $T=100$ and $\gamma=3.0$ . Inference is conducted with $S=10$ and $\eta=1.0$ .

Figure 17: $\times 2$ and $\times 4$ SR results of RDIM - $10$ on images from the DIV2K dataset under unknown degradations. RDIM is trained with $T=100$ and $\gamma=3.0$ . Inference is conducted with $S=10$ and $\eta=1.0$ . In (b) and (c), the left side represents the input image and the right side the output.

Figure 18: Image inpainting, colorization, and deblurring results of RDIM - $10$ on images from the FFHQ dataset. For inpainting, pixels in the original images are randomly masked and set to zero with probability $p_{\text{mask}}=0.5$ . For colorization, grayscale inputs are obtained by converting the original RGB images to luminance. For deblurring, synthetic blurred images are generated from ground truth images by applying a Gaussian blur with kernel size $15\times 15$ and standard deviation $\sigma=3.0$ . RDIM is trained with $T=50$ and $\gamma=3.0$ . Inference is conducted with $S=10$ and $\eta=1.0$ . In (b), (c) and (d), the left side represents the input image and the right side the output.

Diffusion Models for Image Restoration

Diffusion models for image restoration use generative priors to denoise, deblur, upscale, and reconstruct images from incomplete or corrupted measurements.

See all 181 papers

Dynamic Time Step Prediction in Inverse Heat Dissipation for Blur-Like Image Restoration Tasks
Oct 2, 2026Cap Dang Xuan Kiet, Tat-Jen ChamDiffusion Models for Image Restoration Diffusion Model Inversion
Nanyang Technological University Singapore
Energy-oriented Diffusion Bridge for Image Restoration with Foundational Diffusion Models
Apr 13, 2026Jinhui Hou, Zhiyu Zhu, Junhui HouDiffusion Models for Image Restoration Diffusion Models
Department of Computer Science, City University of Hong Kong · Department of Computer Science, City University of Hong Kong (Dongguan)
Bridging Restoration and Generation Manifolds in One-Step Diffusion for Real-World Super-Resolution
Apr 27, 2026Shyang-En Weng, Yi-Cheng Liao, Yu-Syuan Xu +2Diffusion Model Inversion Diffusion Models for Image Restoration
National Yang Ming Chiao Tung University, Hsinchu, Taiwan · MediaTek Inc., Taiwan

Oct 2, 2026J/K move · Enter open · S save

Dynamic Time Step Prediction in Inverse Heat Dissipation for Blur-Like Image Restoration Tasks

Cap Dang Xuan Kiet, Tat-Jen Cham

Nanyang Technological University Singapore

Diffusion Models for Image Restoration Diffusion Model Inversion Degradation-Aware Image Restoration Inverse Problems

Open paper AskPDF arXiv

When using diffusion models to target image restoration problems, diffusion inversion is typically employed to retain relevant image information from the degraded images. Instead of inverting back to the initial time step (i.e., T), many methods invert to a pre-determined intermediate time step, in order to better preserve information from degraded source images. However, a pre-determined time step for inversion is not ideal for reconstruction, as a severely degraded image requires an earlier starting time step than a mildly degraded one. In addition, DDIM-based models corrupt the original signal by adding Gaussian noise, which can be mismatched to the nature of blur-like degradations, such as blur, haze, and low-light. To address these problems, we propose two solutions: (1) we adopt an alternative diffusion process, called the Inverse Heat Dissipation Model, that diffuses the input image by gradually blurring a data point (2) we propose to implement a time predictor to estimate the starting time step for the inversion, with the model learning to adapt to the degradation severity. Extensive experiments on standard benchmarks show that our method achieves state-of-the-art performance in both quantitative and qualitative evaluations, with excellent generalization to many restoration tasks.

Abstract

Diffusion Models for Image Restoration

Explore similar work

Dynamic Time Step Prediction in Inverse Heat Dissipation for Blur-Like Image Restoration Tasks

Energy-oriented Diffusion Bridge for Image Restoration with Foundational Diffusion Models

Bridging Restoration and Generation Manifolds in One-Step Diffusion for Real-World Super-Resolution