Authors: João Guerreiro, Pedro Tomás, Helena Aidos, Jacinto C. Nascimento
Organizations: INESC-ID, Instituto Superior Técnico, Universidade de Lisboa, Lisboa, Portugal · ISR-Lisboa, Instituto Superior Técnico, Universidade de Lisboa, Lisboa, Portugal · LASIGE, Faculdade de Ciências, Universidade de Lisboa, Lisboa, Portugal
Diffusion models achieve state-of-the-art results across multiple tasks. However, in inverse problems, standard initialization from pure Gaussian noise misaligns the generative process with real-world degradations. More recent methods such as diffusion bridges impose strict endpoint constraints and often require long reverse processes that are prone to hallucinations. Alternative consistency models provide noise-invariant, one-step mappings but lack inherent variance modeling and can degrade under severe corruption. Hence, residual diffusion implicit models (RDIMs) are proposed, constituting a generalized framework that explicitly models the residuals between high-quality (HQ) and low-quality (LQ) images, aligning the forward process with the actual degradation. A non-Markovian implicit reverse sampler is derived, which can skip intermediate timesteps, enabling accurate few-step or even single-step reconstruction, while mitigating the hallucinations inherent to long diffusion chains. RDIM also introduces a controllable variance mechanism that interpolates between deterministic and stochastic sampling, balancing fidelity and diversity. Furthermore, it enables the straightforward use of perceptual losses, when needed. Experiments on denoising and super-resolution benchmarks demonstrate that RDIMs consistently outperforms the state of the art, including bridge and consistency models, in terms of PSNR, SSIM, and LPIPS, reducing hallucinations while requiring only a few sampling steps (often just one). The results position RDIMs as an efficient solution for a broad range of image restoration tasks.
Figures & tables
Figure 1: Overview of RDIM , a diffusion framework tailored for inverse problems, such as image reconstruction. The reverse process can accurately reconstruct back the data in S≤T steps.
Figure 2: Progression of weights βt and λt across timesteps and impact of p on the diffusion process.
Table 1: Comparative analysis of RDIM against relevant state-of-the-art techniques for (a) denoising and (b) SR. Green color highlights the best score overall and Blue color the second best.
Figure 4
Figure 4: RDIM - 10 results in image inpainting, colorization, and deblurring on the FFHQ dataset. In (b), (c) and (d), the left side represents the input image and the right side the output.
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: U-Net-based network. In the convolutional layers, the parameters k , st , and pd represent the kernel size, stride, and padding, respectively. Additionally, (∗/s) denotes (W/s,H/s) , where s is a scale factor ( s>1 for SR tasks and s=1 otherwise).
Figure 6: Core blocks of the U-Net-based network. (a) Encoder Block, (b) Decoder Block, and (c) Bottleneck Block.
Figure 7: Building blocks. (a) Residual Block, (b) Self-Attention Block, (c) Upsample Block, and (d) Downsample Block.
Figure 8: Residual β -schedule ( p=5.0 ) overlaid with red markers indicating the βt value at each sampling point. The schedule adopted selects timesteps uniformly spaced. In this illustration, the forward process involves T=100 timesteps and the number of sampling steps, where reverse transitions occur, is S=10 .
p
S
FMD-BPAE
PSNR ↑
SSIM ↑
LPIPS ↓
1.0
1
40.0836
0.9678
0.0205
5
40.0772
0.9678
0.0205
10
40.0565
0.9677
0.0205
5.0
1
40.1100
0.9681
0.0202
5
40.0436
0.9679
0.0202
Appendix
Table 3: Impact of parameter p , which controls the steepness of the curve in the β -schedule. All RDIM configurations were trained using a forward process with T=10 timesteps and variance hyperparameter γ=9.0 . During inference, the reverse process variance hyperparameter is fixed to η=1.0 . Orange color rows highlight ResShift scenarios, corresponding to particular cases where RDIM reduces to ResShift under the conditions η=1.0 and S=T .
Figure 9: Impact of γ on the diffusion process with β -schedule parameter fixed to p=5.0 .
T
S
FMD-BPAE
γ=0.0
γ=3.0
γ=9.0
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
10
1
38.3703
0.9575
0.0296
40.0998
0.9686
0.0196
40.1100
0.9681
0.0202
10
38.3487
0.9572
0.0299
39.3632
0.9644
0.0187
39.9524
0.9674
0.0202
50
1
38.4181
0.9578
0.0293
42.5354
0.9803
0.0093
43.1161
0.9821
0.0079
10
38.4162
0.9578
0.0295
42.4566
0.9802
0.0079
43.2029
0.9824
0.0077
Appendix
Table 4: Denoising performance comparison of several RDIM configurations on BPAE confocal images from the benchmark dataset FMD. It exhibits the impact of the constant hyperparameter γ that controls the variance in the forward process and the impact of the number of diffusion timesteps during training ( T ) and inference ( S ). The constant hyperparameter that controls the variance in the reverse process is fixed to η=1.0 . Orange color rows highlight ResShift scenarios, corresponding to cases where RDIM reduces to ResShift ( η=1.0 and S=T ).
Figure 10: PSNR , SSIM , and LPIPS performance as a function of S (number of sampling timesteps) on denoising of BPAE confocal images from the FMD dataset. RDIM is trained with T=100 and γ=3.0 . The constant hyperparameter that controls the variance in the reverse process is fixed to η=1.0 . The number of sampling timesteps evaluated includes a single-step prediction and then ranges from 10 to 100 in increments of 10 .
T
S
η
FMD-BPAE
γ=0.0
γ=3.0
γ=9.0
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
100
10
0.0
38.3752
0.9575
0.0295
44.1429
0.9855
0.0047
43.3732
0.9830
0.0073
0.5
38.3752
0.9575
0.0295
44.1440
0.9855
0.0047
43.3738
0.9830
0.0073
1.0
38.3752
0.9575
0.0295
44.1468
0.9855
0.0047
43.3743
0.9830
0.0073
Appendix
Table 5: Impact of the constant hyperparameter η , which controls the variance in the reverse process, on denoising BPAE confocal images from the benchmark dataset FMD.
Figure 11: Comparison of the perception-distortion trade-off between the proposed RDIM and the state-of-the-art on the DIV2K dataset for 4 × SR.
Method
FFHQ-Inpainting
FFHQ-Colorization
FFHQ-Deblurring
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
Corrupted
9.034
0.136
1.165
20.806 †
0.926 †
0.223 †
24.691
0.686
0.486
ResShift
31.721
0.922
0.022
25.320
0.948
0.103
28.331
0.812
0.089
RDIM- 1
33.514
0.941
0.029
25.727
0.950
0.094
29.995
0.847
0.170
RDIM- 10
32.330
0.931
0.025
25.565
0.949
0.099
28.870
0.826
0.111
† To compute colorization scores on corrupted images, the existing channel is replicated.
Appendix
Table 6: Performance on the FFHQ dataset for image inpainting, colorization, and deblurring.
Method
Poisson-corrupted DIV2K-Bicubic- ×4
PSNR ↑
SSIM ↑
LPIPS ↓
ResShift
23.610 (-3.845)
0.521 (-0.259)
0.428 (+0.275)
GOUB-SDE
20.165 ( -6.725 )
0.335 ( -0.413 )
0.664 (+0.444)
UniDB-SDE
19.284 (-6.176)
0.314 (-0.372)
0.697 ( +0.518 )
CTMSR- 1
22.987 (-4.100)
0.466 (-0.293)
0.495 (+0.365)
RDIM- 1
26.260 ( -2.920 )
0.673 ( -0.151 )
0.519 (+0.262)
Appendix
Table 7: Evaluation on Poisson-corrupted DIV2K-Bicubic- ×4 images after training on the original dataset without adding Poisson noise. Values in parentheses indicate the performance drop relative to the evaluation on the original noise-free DIV2K-Bicubic- ×4 dataset (see Table 2 ). Red color indicates the worst performance drop overall and Green color the best.
Figure 12: Qualitative denoising analysis on cropped regions from the SIDD dataset. Since SIDD contains noisy images captured under challenging lighting conditions, brightness-adjusted crops of the marked regions are shown in the bottom row for enhanced visualization.
Figure 13: SR qualitative comparison on cropped regions from the DIV2K subset with unknown degradation.
Figure 14: SR qualitative comparison on cropped regions from the DIV2K subset with bicubic downsampled images.
Figure 15: Qualitative comparison of RDIM-PQ- 1 on ×4 SR . Cropped regions from the DIV2K subset, with bicubic downsampled images, suggest RDIM achieves greater structural and texture fidelity than bridge-based models.
Figure 16: Denoising results of RDIM - 10 on images from the FMD and SIDD datasets. For improved visualization, only cropped regions are shown. RDIM is trained with T=100 and γ=3.0 . Inference is conducted with S=10 and η=1.0 .
Figure 17: ×2 and ×4 SR results of RDIM - 10 on images from the DIV2K dataset under unknown degradations. RDIM is trained with T=100 and γ=3.0 . Inference is conducted with S=10 and η=1.0 . In (b) and (c), the left side represents the input image and the right side the output.
Figure 18: Image inpainting, colorization, and deblurring results of RDIM - 10 on images from the FFHQ dataset. For inpainting, pixels in the original images are randomly masked and set to zero with probability pmask=0.5 . For colorization, grayscale inputs are obtained by converting the original RGB images to luminance. For deblurring, synthetic blurred images are generated from ground truth images by applying a Gaussian blur with kernel size 15×15 and standard deviation σ=3.0 . RDIM is trained with T=50 and γ=3.0 . Inference is conducted with S=10 and η=1.0 . In (b), (c) and (d), the left side represents the input image and the right side the output.
Diffusion models for image restoration use generative priors to denoise, deblur, upscale, and reconstruct images from incomplete or corrupted measurements.
When using diffusion models to target image restoration problems, diffusion inversion is typically employed to retain relevant image information from the degraded images. Instead of inverting back to the initial time step (i.e., T), many methods invert to a pre-determined intermediate time step, in order to better preserve information from degraded source images. However, a pre-determined time step for inversion is not ideal for reconstruction, as a severely degraded image requires an earlier starting time step than a mildly degraded one. In addition, DDIM-based models corrupt the original signal by adding Gaussian noise, which can be mismatched to the nature of blur-like degradations, such as blur, haze, and low-light. To address these problems, we propose two solutions: (1) we adopt an alternative diffusion process, called the Inverse Heat Dissipation Model, that diffuses the input image by gradually blurring a data point (2) we propose to implement a time predictor to estimate the starting time step for the inversion, with the model learning to adapt to the degradation severity. Extensive experiments on standard benchmarks show that our method achieves state-of-the-art performance in both quantitative and qualitative evaluations, with excellent generalization to many restoration tasks.
Diffusion bridge models have shown great promise in image restoration by explicitly connecting clean and degraded image distributions. However, they often rely on complex and high-cost trajectories, which limit both sampling efficiency and final restoration quality. To address this, we propose an Energy-oriented diffusion Bridge (E-Bridge) framework to approximate a set of low-cost manifold geodesic trajectories to boost the performance of the proposed method. We achieve this by designing a novel bridge process that evolves over a shorter time horizon and makes the reverse process start from an entropy-regularized point that mixes the degraded image and Gaussian noise, which theoretically reduces the required trajectory energy. To solve this process efficiently, we draw inspiration from consistency models to learn a single-step mapping function, optimized via a continuous-time consistency objective tailored for our trajectory, so as to analytically map any state on the trajectory to the target image. Notably, the trajectory length in our framework becomes a tunable task-adaptive knob, allowing the model to adaptively balance information preservation against generative power for tasks of varying degradation, such as denoising versus super-resolution. Extensive experiments demonstrate that our E-Bridge achieves state-of-the-art performance across various image restoration tasks while enabling high-quality recovery with a single or fewer sampling steps. Our project page is https://jinnh.github.io/E-Bridge/.
Jinhui Hou, Zhiyu Zhu, Junhui Hou
Department of Computer Science, City University of Hong Kong · Department of Computer Science, City University of Hong Kong (Dongguan)
Pretrained diffusion models have revolutionized real-world image super-resolution (Real-ISR) but suffer from computational bottlenecks due to iterative sampling. Recent single-step distillation accelerates inference but faces a stark perception-distortion trade-off due to rigid timestep initialization, distributional trajectory mismatches, and fragile stochastic modulation. To address this, we present Adaptive Inversion and Degradation-aware Sampling for Real-ISR (IDaS-SR), a one-step framework bridging the deterministic restoration and stochastic generation manifolds. At its core, the Manifold Inversion Noise Estimator (MINE) resolves these initialization and trajectory mismatches by predicting a severity-aware timestep and inversion noise, precisely anchoring low-quality latents onto the diffusion trajectory. Furthermore, to mitigate fragile stochastic modulation, we propose CHARIOT, a continuous generative steering mechanism. By rescheduling trajectories and interpolating noise, it enables explicit navigation of the perception-distortion boundary without compromising structural priors. Extensive experiments demonstrate that IDaS-SR outperforms state-of-the-art methods, seamlessly transitioning from a rigorous structural restorer to a sophisticated texture hallucinator in a single inference step.
Shyang-En Weng, Yi-Cheng Liao, Yu-Syuan Xu +2
National Yang Ming Chiao Tung University, Hsinchu, Taiwan · MediaTek Inc., Taiwan