GeoCR: Learning a Generalist Cloud Removal Prior from Heterogeneous Observations
Organizations: Korea Advanced Institute of Science and Technology (KAIST)
Abstract
Cloud removal methods are typically specialized to individual datasets and input configurations, limiting reuse across sensors, spectral bands, and observation settings. We introduce GeoCR, a generalist model that unifies RGB-only-based CR and multispectral-based CR from single- or multi-temporal cloudy observations, with optional SAR guidance, within a single network. To accommodate different spectral and sensing domains, compact input and output stems extend a pretrained RGB autoencoder while keeping its encoder and decoder trunks frozen. This shared latent interface enables a single flow transformer to jointly model clean RGB and non-RGB latents, conditioned on separate cloudy-observation streams and optional SAR tokens. Through joint pretraining on the training splits of ten datasets comprising 883,331 cloud-free target images, GeoCR learns a shared cloud removal prior across these heterogeneous configurations. The same pretrained checkpoint supports direct inference without dataset-specific fine-tuning and efficient adaptation through low-rank adaptation (LoRA). We evaluate GeoCR against general image restoration and cloud removal methods on test splits of the contributing datasets under full-band and RGB-only settings. GeoCR achieves the best FID and DISTS on full-band SEN12MS-CR and Sen2_MTC_New and RGB-only CUHK-CR2, outperforming existing models and demonstrating the effectiveness of a reusable generative model across diverse settings.
Figures & tables
| Source | Imagery source | Spectral bands | Delivered tile | GSD (m) | Cloudy cond. | SAR cond. | Clean targets |
| AllClear ( Zhou et al., 2024 ) | Sentinel-2 + Sentinel-1 | 13 (L1C, TOA) | 10 | 1–3 | ✓ | 662,022 | |
| SEN12MS-CR ( Ebel et al., 2020 ) | Sentinel-2 + Sentinel-1 | 13 (L1C, TOA) | 10 | 1 | ✓ | 107,143 | |
| WHUS2-CRv ( Li et al., 2022 ) | Sentinel-2 | 13 (BOA; B10: L1C, TOA) | 10 | 1 | ✗ | 18,816 | |
| Sen2_MTC_Old ( Sarukkai et al., 2020 ) | Sentinel-2 | 3 (RGB; +NIR for cloudy inputs) | 10 | 3 | ✗ | 88,874 | |
| Sen2_MTC_New ( Huang and Wu, 2022 ) | Sentinel-2 | 4 (RGB+B8, BOA) | 10 | 3 | ✗ | 2,380 | |
| T-CLOUD ( Ding et al., 2022 ) | Landsat-8 | 3 (RGB, 8-bit) | 30 | 1 | ✗ | 2,234 |
| Method | Venue | FID | DISTS | KID | DINO | LPIPS | SSIM | PSNR |
| General image translation/restoration methods | ||||||||
| pix2pix ( Isola et al., 2017 ) | CVPR’17 | 194.8 | 0.241 | 0.1201 | 0.538 | 0.319 | 0.452 | 19.46 |
| pix2pixHD ( Wang et al., 2018 ) | CVPR’18 | 257.1 | 0.262 | 0.2130 | 0.379 | 0.304 | 0.401 | 19.34 |
| BBDM ( Li et al., 2023 ) | CVPR’23 | 514.6 | 0.542 | 0.5827 | 0.131 | 0.737 | 0.326 | 17.68 |
| HI-Diff ( Chen et al., 2023 ) | NeurIPS’23 | 165.7 | 0.192 | 0.0769 | 0.664 | 0.253 | 0.638 | 23.54 |
| Cloud removal methods | ||||||||
| Recon. PSNR | RGB (w/o training) | Non-RGB (stem training) | SAR (stem training) | ||||||
| All | Clear | Cloudy | All | Clear | Cloudy | All | VV | VH | |
| Before | 45.78 | 46.77 | 45.11 | 18.84 | 20.60 | 18.20 | 18.40 | 18.57 | 18.24 |
| After | 45.78 | 46.77 | 45.11 | 40.08 | 41.41 | 38.34 | 29.14 | 28.54 | 30.21 |
| Gain | +0.00 | +0.00 | +0.00 | +21.24 | +20.81 | +20.14 | +10.74 | +9.97 | +11.98 |
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
| Section | Contents |
| Section A | Stem adaptation |
| Section B | Implementation details |
| Section C | Additional quantitative results |
| Section D | Additional qualitative results |
| Section E | Failure cases and limitations |
| Section F | Datasets and preprocessing |
| Route | Stem parameters | Trained | Reconstruction PSNR |
| 10-band TOA | 23,178 | Yes | AllClear 38.23; SEN12MS-CR 37.69 |
| 10-band BOA | 23,178 | Yes | WHUS2-CRv 37.89 |
| Single-band NIR | 2,433 | No | Sen2_MTC_New 42.06; CUHK-CR 40.41; Sen2_MTC_Old 49.51 |
| SAR VV/VH | 4,738 | Yes | AllClear 28.51; SEN12MS-CR 31.19 |
| Method | FID | DISTS | KID | DINO | LPIPS | SSIM | PSNR |
| (a) T-CLOUD: RGB display imagery | |||||||
| pix2pix ( Isola et al., 2017 ) | 107.6 | 0.254 | 0.0509 | 0.450 | 0.330 | 0.635 | 20.13 |
| pix2pixHD ( Wang et al., 2018 ) | 60.8 | 0.169 | 0.0173 | 0.616 | 0.177 | 0.741 | 24.75 |
| BBDM ( Li et al., 2023 ) | 186.6 | 0.422 | 0.1179 | 0.259 | 0.546 | 0.541 | 22.69 |
| HI-Diff ( Chen et al., 2023 ) | 40.2 | 0.124 | 0.0070 | 0.726 | 0.114 | 0.874 | 30.54 |
| UnCRtainTS ( Ebel et al., 2023 ) | 63.3 | 0.180 | 0.0188 | 0.640 | 0.178 | 0.809 | 26.28 |
| Source | Optical representation | SAR | Mix (%) | Training targets | |
| AllClear ( Zhou et al., 2024 ) | 13 bands, L1C TOA | VV/VH | 1–3 | 53.14 | 662,022 |
| SEN12MS-CR ( Ebel et al., 2020 ) | 13 bands, L1C TOA | VV/VH | 1 | 32.93 | 107,143 |
| WHUS2-CRv ( Li et al., 2022 ) | 13 bands, BOA; B10 TOA | – | 1 | 8.81 | 18,816 |
| Sen2_MTC_Old ( Sarukkai et al., 2020 ) | RGB; NIR in cloudy inputs only | – | 3 | 2.00 | 88,874 |
| Sen2_MTC_New ( Huang and Wu, 2022 ) | RGB+B8, BOA | – | 3 | 1.10 | 2,380 |
| T-CLOUD ( Ding et al., 2022 ) | RGB, 8-bit | – | 1 | 1.10 | 2,234 |
| Setting | PSNR/SSIM bands | |
| SEN12MS-CR | 13 bands | 7,899 |
| Sen2_MTC_New | RGB+B8 | 687 |
| WHUS2-CRv | 13 bands | 3,746 |
| T-CLOUD | RGB | 588 |
| CUHK-CR1 | RGB+NIR | 98 |
| CUHK-CR2 | RGB | 111 |