Ultra-High-Definition Restoration Transformers with Correlation Matching Transformation
Organizations: School of Computing and Communications, Lancaster University
Abstract
We propose UHDformer++, a general Transformer-based framework to solve numerous Ultra-High-Definition (UHD) image restoration tasks. UHDformer++ operates across coordinated learning spaces: 1) a high-resolution space (HR) for multi-level feature extraction, 2) a low-resolution space (LR) for learning compact, representative features, 3) a super-resolution space (SR) for upsampling low-resolution features from SR, and 4) a low-high fusion and reconstruction space (LHFR) for final image restoration. Specifically, HR extracts multi-scale high-resolution features and fuses them with low-resolution cues to produce residual images, while LR distills complementary representations from HR to improve restoration quality. To supply LHFR with richer features, SR super-resolves LR outputs before fusion. We further introduce two modules to bridge the high- and low-resolution spaces. The Feature-Refined Correlation Matching Transformation (FR-CMT) module selects the top , from the fusion between max- and mean-pooled high-resolution features to replace less informative channels in the low-resolution Transformer. The Adaptive Channel Modulator (ACM) adaptively recalibrates multi-scale high-resolution features, ensuring that only task-relevant information propagates to LR. Extensive experiments demonstrate that UHDformer++ reduces model parameters by at least 86% compared with recent state-of-the-art methods while achieving substantial performance gains across UHD restoration tasks, including low-light image enhancement, dehazing, deblurring, deraining, and desnowing. Code will be released at https://github.com/supersupercong/uhdformerplus.
Figures & tables
| Dataset | Training samples | Testing samples | Resolution |
|---|---|---|---|
| UHD-LL | 2,000 | 150 | 3,840 2,160 |
| UHD-Haze | 2,290 | 231 | 3,840 2,160 |
| UHD-Blur | 1,964 | 300 | 3,840 2,160 |
| UHD-Rain | 3,000 | 200 | 3,840 2,160 |
| UHD-Snow | 3,000 | 200 | 3,840 2,160 |
| Method | Metrics | Computational Complexity | Full Size | |||||||
| Full Reference | No-Reference | |||||||||
| PSNR (dB) | SSIM | LPIPS | PI | NIQE | MIUSIQ | Params. (M) | FLOPs (G) | Time (s) | ||
| Training Set on LOL | ||||||||||
| SwinIR [ 15 ] | 17.900 | 0.7379 | 0.5217 | 6.5902 | 8.5836 | 28.3047 | 11.5 | 11992.17 | 1.41 | ✗ |
| Restormer [ 49 ] | 19.728 | 0.7703 | 0.4566 | 6.5099 | 8.3787 | 30.2802 | 26.1 | 2255.85 | 1.27 | ✗ |
| Uformer [ 16 ] | 18.168 | 0.7201 | 0.5593 | 7.1781 | 9.8494 | 29.5157 | 20.6 | 657.45 | 0.43 | ✗ |
| Method | Metrics | Computational Complexity | Full Size | |||||||
| Full Reference | No-Reference | |||||||||
| PSNR (dB) | SSIM | LPIPS | PI | NIQE | MIUSIQ | Params. (M) | FLOPs (G) | Time (s) | ||
| Training Set on SOTS-ITS | ||||||||||
| GridNet [ 66 ] | 14.783 | 0.8466 | 0.2518 | 4.4077 | 4.0874 | 35.6738 | 0.96 | 343.92 | 0.10 | ✔ |
| MSBDN [ 67 ] | 15.043 | 0.8570 | 0.2709 | 4.5582 | 4.4640 | 35.5302 | 31.4 | 664.87 | 0.09 | ✔ |
| Restormer [ 49 ] | 13.875 | 0.6405 | 0.4477 | 5.2136 | 6.1129 | 35.4127 | 26.1 | 2255.85 | 1.27 | ✗ |
| Method | Metrics | Computational Complexity | Full Size | |||||||
| Full Reference | No-Reference | |||||||||
| PSNR (dB) | SSIM | LPIPS | PI | NIQE | MIUSIQ | Params. (M) | FLOPs (G) | Time (s) | ||
| Training Set on GoPro | ||||||||||
| DMPHN [ 78 ] | 26.490 | 0.7985 | 0.2696 | 6.1592 | 6.0393 | 26.1362 | 21.7 | 1251.76 | 0.15 | ✔ |
| MIMO-Unet++ [ 48 ] | 24.290 | 0.7354 | 0.3723 | 6.0395 | 6.1247 | 25.3302 | 16.1 | 2470.52 | 0.31 | ✔ |
| MPRNet [ 14 ] | 24.571 | 0.7426 | 0.3597 | 6.9655 | 7.9493 | 27.6944 | 20.1 | 27317.71 | 1.13 | ✔ |
| Method | Metrics | Computational Complexity | Full Size | |||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Full Reference | No-Reference | |||||||||
| PSNR (dB) | SSIM | LPIPS | PI | NIQE | MIUSIQ | Params. (M) | FLOPs (G) | Time (s) | ||
| Restormer [ 49 ] | 19.408 | 0.7105 | 0.4775 | 8.8409 | 11.2731 | 24.9277 | 26.1 | 2255.85 | 1.27 | ✗ |
| Uformer [ 16 ] | 19.494 | 0.7163 | 0.4598 | 8.0579 | 9.9422 | 25.0728 | 20.6 | 657.45 | 0.43 | ✗ |
| SFNet [ 79 ] | 20.091 | 0.7092 | 0.4768 | 7.1914 | 8.0167 | 22.3553 | 13.23 | 1991.03 | 0.42 | ✗ |
| UHDformer [ 32 ] | 37.348 | 0.9748 | 0.0554 | 5.0464 | 4.9256 | 30.4390 | 0.3393 | 51.63 | 0.16 | ✔ |
| Method | Metrics | Computational Complexity | Full Size | |||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Full Reference | No-Reference | |||||||||
| PSNR (dB) | SSIM | LPIPS | PI | NIQE | MIUSIQ | Params. (M) | FLOPs (G) | Time (s) | ||
| Restormer [ 49 ] | 24.142 | 0.8691 | 0.3190 | 8.0490 | 9.4711 | 26.0491 | 26.1 | 2255.85 | 1.27 | ✗ |
| Uformer [ 16 ] | 23.717 | 0.8711 | 0.3095 | 7.5036 | 8.5964 | 26.1323 | 20.6 | 657.45 | 0.43 | ✗ |
| SFNet [ 79 ] | 23.638 | 0.8456 | 0.3528 | 7.1527 | 7.3665 | 22.5295 | 13.23 | 1991.03 | 0.42 | ✗ |
| UHDformer [ 32 ] | 36.614 | 0.9881 | 0.0245 | 5.0046 | 4.7680 | 31.1335 | 0.3393 | 51.63 | 0.16 | ✔ |
| ID | Experiment | Metrics | Computational Complexity | |||
|---|---|---|---|---|---|---|
| PSNR (dB) | SSIM | LPIPS | Params. (M) | FLOPs (G) | ||
| (a) | w/o LN | 26.812 | 0.9279 | 0.2369 | 0.3955 | 49.65 |
| (b) | LN in UHDformer [ 32 ] | 27.119 | 0.9277 | 0.2493 | 0.3962 | 49.65 |
| (c) | DyT in UHDformer++ | 27.256 | 0.9308 | 0.2304 | 0.3962 | 49.65 |
| (d) | GDFN in Restormer [ 49 ] | 26.499 | 0.9270 | 0.2366 | 0.3064 | 38.06 |
| (e) | CMTN in UHDformer [ 32 ] | 27.090 | 0.9290 | 0.2394 | 0.3379 | 48.70 |
| ID | Experiment | Metrics | Computational Complexity | |||
|---|---|---|---|---|---|---|
| PSNR (dB) | SSIM | LPIPS | Params. (M) | FLOPs (G) | ||
| (a) | w/o FR-CMT in CMTA & CMTN | 26.933 | 0.9279 | 0.2433 | 0.3060 | 27.94 |
| (b) | w/o FR-CMT in CMTA | 26.879 | 0.9278 | 0.2406 | 0.3511 | 38.8 |
| (c) | w/o FR-CMT in CMTFN | 26.650 | 0.9256 | 0.2347 | 0.3511 | 38.8 |
| (d) | w/o Using Max-Pooling in FR-CMT | 27.079 | 0.9270 | 0.2406 | 0.3962 | 49.45 |
| (e) | w/o Using Mean-Pooling in FR-CMT | 26.911 | 0.9281 | 0.2469 | 0.3962 | 49.45 |
| 1 | 2 | 4 ( Default ) | 8 | |
|---|---|---|---|---|
| PSNR (dB) | 26.7669 | 27.3854 | 27.2567 | 27.1235 |
| SSIM | 0.9271 | 0.9288 | 0.9308 | 0.9278 |
| LPIPS | 0.2376 | 0.2383 | 0.2304 | 0.2345 |
| Params. (M) | 0.3962 | 0.3962 | 0.3962 | 0.3962 |
| FLOPs (G) | 49.65 | 49.65 | 49.65 | 49.65 |
| ID | Experiment | Metrics | Computational Complexity | |||
|---|---|---|---|---|---|---|
| PSNR (dB) | SSIM | LPIPS | Params. (M) | FLOPs (G) | ||
| (a) | w/o Adaptive Channel Modulator (ACM) | 26.953 | 0.9289 | 0.2357 | 0.3953 | 48.65 |
| (b) | w/ Efficient Channel Attention (ECA) [ 82 ] | 26.750 | 0.9278 | 0.2407 | 0.3953 | 48.7 |
| (c) | w/ Squeeze-and-Excitation Attention (SE) [ 83 ] | 27.008 | 0.9286 | 0.2355 | 0.3969 | 48.7 |
| (d) | w/ Adaptive Channel Modulator (ACM) ( Ours ) | 27.256 | 0.9308 | 0.2304 | 0.3962 | 49.65 |
| ID | Experiment | Metrics | Computational Complexity | |||
|---|---|---|---|---|---|---|
| PSNR (dB) | SSIM | LPIPS | Params. (M) | FLOPs (G) | ||
| (a) | Individual Adaptive Channel Modulator | 26.981 | 0.9281 | 0.2404 | 0.4062 | 60.07 |
| (b) | Shared Adaptive Channel Modulator ( Ours ) | 27.256 | 0.9308 | 0.2304 | 0.3962 | 49.65 |
| ID | Experiment | Metrics | Computational Complexity | |||
|---|---|---|---|---|---|---|
| PSNR (dB) | SSIM | LPIPS | Params. (M) | FLOPs (G) | ||
| (a) | w/o GFR | 27.013 | 0.9270 | 0.2442 | 0.3705 | 49.23 |
| (b) | w/ Shared GFR ( Ours ) | 27.256 | 0.9308 | 0.2304 | 0.3962 | 49.65 |
| ID | Experiment | Metrics | Computational Complexity | |||
|---|---|---|---|---|---|---|
| PSNR (dB) | SSIM | LPIPS | Params. (M) | FLOPs (G) | ||
| (a) | w/o SR Space | 26.449 | 0.9274 | 0.2355 | 0.3929 | 46.21 |
| (b) | w/ SR Space ( Ours ) | 27.256 | 0.9308 | 0.2304 | 0.3962 | 49.65 |
| 4 | 8 | 16 ( Default ) | 24 | 32 | |
|---|---|---|---|---|---|
| PSNR (dB) | 22.789 | 24.773 | 27.256 | 26.690 | 27.158 |
| SSIM | 0.9086 | 0.9162 | 0.9308 | 0.9298 | 0.9312 |
| LPIPS | 0.306 | 0.2756 | 0.2304 | 0.2305 | 0.2250 |
| Params. (M) | 0.0399 | 0.1193 | 0.3962 | 0.8310M | 1.4237 |
| FLOPs (G) | 4.07 | 13.69 | 49.65 | 107.89 | 188.39 |
| 8 | 12 | 16 ( Default ) | 20 | 24 | |
|---|---|---|---|---|---|
| PSNR (dB) | 27.188 | 27.452 | 27.256 | 26.858 | 26.583 |
| SSIM | 0.9284 | 0.9299 | 0.9308 | 0.9274 | 0.9275 |
| LPIPS | 0.2438 | 0.2378 | 0.2304 | 0.2401 | 0.2350 |
| Params. (M) | 0.2913 | 0.3438 | 0.3962 | 0.4487 | 0.5011 |
| FLOPs (G) | 37.83 | 43.74 | 49.65 | 55.57 | 61.48 |
| 2 | 4 | 8 ( Default ) | 16 | 32 | |
|---|---|---|---|---|---|
| PSNR (dB) | 27.602 | 27.109 | 27.256 | 27.051 | 26.682 |
| SSIM | 0.9315 | 0.9295 | 0.9308 | 0.9280 | 0.9253 |
| LPIPS | 0.2335 | 0.2370 | 0.2304 | 0.2375 | 0.2426 |
| Params. (M) | 0.2426 | 0.2733 | 0.3962 | 0.8878 | 2.8538 |
| FLOPs (G) | 96.27 | 58.98 | 49.65 | 47.32 | 46.74 |
| 0 | 0.01 | 0.05 ( Default ) | 0.1 | 0.2 | 0.5 | |
|---|---|---|---|---|---|---|
| PSNR (dB) | 26.828 | 27.3703 | 27.256 | 26.634 | 27.215 | 27.005 |
| SSIM | 0.9118 | 0.9257 | 0.9308 | 0.9281 | 0.9291 | 0.9305 |
| LPIPS | 0.2545 | 0.2464 | 0.2304 | 0.2392 | 0.2360 | 0.2351 |
| 0 | 0.1 | 0.25 | 0.5 ( Default ) | 0.75 | 1 | |
| PSNR (dB) | 26.683 | 26.421 | 26.734 | 27.256 | 26.864 | 26.897 |
| SSIM | 0.9284 | 0.9283 | 0.9280 | 0.9308 | 0.9275 | 0.9290 |
| Method | PSNR (dB) | SSIM | Params. |
|---|---|---|---|
| Retinex-Net [ 62 ] | 16.77 | 0.54 | 0.45M |
| Zero-DCE [ 84 ] | 16.79 | 0.67 | 79.416K |
| AGLLNet [ 85 ] | 17.52 | 0.77 | 3.438K |
| Zhao et. al. [ 76 ] | 21.67 | 0.87 | 11.560M |
| RUAS [ 86 ] | 16.44 | 0.70 | 3K |
| SCI [ 87 ] | 14.78 | 0.62 | 0.258K |
| Method | PSNR (dB) | SSIM | Params. |
|---|---|---|---|
| SRN [ 11 ] | 31.751 | 0.9144 | 6.8M |
| DMPHN [ 78 ] | 31.734 | 0.9130 | 21.7M |
| MIMO-Unet+ [ 48 ] | 33.277 | 0.9348 | 16.1M |
| MPRNet [ 14 ] | 34.195 | 0.9454 | 20.1M |
| Restormer [ 49 ] | 34.483 | 0.9488 | 26.1M |
| Uformer [ 16 ] | 34.668 | 0.9502 | 20.6M |