UENR-600K: A Large-Scale Physically Grounded Dataset for Nighttime Video Deraining
Organizations: Show Lab, National University of Singapore · National University of Singapore
Abstract
Nighttime video deraining is uniquely challenging because raindrops interact with artificial lighting. Unlike daytime white rain, nighttime rain takes on various colors and appears locally illuminated. Existing small-scale synthetic datasets rely on 2D rain overlays and fail to capture these physical properties, causing models to generalize poorly to real-world night rain. Meanwhile, capturing real paired nighttime videos remains impractical because rain effects cannot be isolated from other degradations like sensor noise. To bridge this gap, we introduce UENR-600K, a large-scale, physically grounded dataset containing 600,000 1080p frame pairs. We utilize Unreal Engine to simulate rain as 3D particles within virtual environments. This approach guarantees photorealism and physically real raindrops, capturing correct details like color refractions, scene occlusions, rain curtains. Leveraging this high-quality data, we establish a new state-of-the-art baseline by adapting the Wan 2.2 video generation model. Our baseline treat deraining as a video-to-video generation task, exploiting strong generative priors to almost entirely bridge the sim-to-real gap. Extensive benchmarking demonstrates that models trained on our dataset generalize significantly better to real-world videos. Project page: https://showlab.github.io/UENR-600K/.
Figures & tables
| Dataset | Year | Venue | Rain Type a | GT Type b | #Rainy Frames c | #GT Frames c | Resolution | Night d | Video e | Raining In-Scene f |
| Rain200L/H [ 32 ] | 2017 | CVPR | Synth. | Real | 4,000 | 4,000 | 435 366 | |||
| DDN-Data [ 9 ] | 2017 | CVPR | Synth. | Real | 13,000 | 13,000 | 489 428 | |||
| DID-Data [ 38 ] | 2018 | CVPR | Synth. | Real | 13,200 | 13,200 | 512 512 | |||
| NTURain g [ 6 ] | 2018 | CVPR | Synth. | Real | - | - | 640 480 | ✓ | ||
| RainSynAll25 [ 20 ] | 2018 | CVPR | Synth. | Real | - | - | - | ✓ | ||
| Rain800 [ 39 ] | 2017 | - | Synth. | Real | 800 | 800 | 518 419 |
| Input Size | SynNightRain [ 24 ] | Tie | Our Data | |
|---|---|---|---|---|
| ESTINet [ 40 ] | 224 224 | 16.4% | 6.5% | 77.2% |
| RDD-Net [ 28 ] | 128 128 | 32.3% | 1.6% | 66.1% |
| RLP [ 37 ] | 256 256 | 27.7% | 8.6% | 63.7% |
| Turtle [ 12 ] | 128 128 | 16.1% | 6.2% | 77.7% |
| UConNet [ 43 ] | 128 128 | 3.2% | 1.3% | 95.4% |
| WeatherDiff [ 23 ] | 64 64 | 91.7% | 3.0% | 5.4% |
| Architecture | Pretrained | SynNightRain [ 24 ] | Our Data | Average | |
|---|---|---|---|---|---|
| ESTINet [ 40 ] | CNN+RNN | 12.9% | 5.6% | 9.3% | |
| RDD-Net [ 28 ] | CNN | 10.5% | 0.0% | 5.2% | |
| RLP [ 37 ] | Transformer | 4.0% | 0.0% | 2.0% | |
| Turtle [ 12 ] | Transformer | 0.8% | 0.0% | 0.4% | |
| UConNet [ 43 ] | CNN | 0.0% | 0.0% | 0.0% | |
| WeatherDiff [ 23 ] | Diffusion | 1.6% | 0.0% | 0.8% |
| SynNightRain | Our Data | Average | ||||
|---|---|---|---|---|---|---|
| AFD | VLM | AFD | VLM | AFD | VLM | |
| ESTINet [ 40 ] | 6.40 | 3.22 | 5.70 | 3.32 | 6.05 | 3.27 |
| RDD-Net [ 28 ] | 7.04 | 3.37 | 9.93 | 2.37 | 8.49 | 2.87 |
| RLP [ 37 ] | 7.75 | 3.11 | 10.11 | 2.60 | 8.93 | 2.86 |
| Turtle [ 12 ] | 8.55 | 2.71 | 8.08 | 3.07 | 8.32 | 2.89 |
| UConNet [ 43 ] | 7.94 | 2.47 | 9.34 | 3.03 | 8.64 | 2.75 |
| Our Data | SynNightRain [ 24 ] | PSNR | |||||
|---|---|---|---|---|---|---|---|
| PSNR | SSIM | PSNR | SSIM | ||||
| ESTINet [ 40 ] | 24.88 | 0.845 | 26.89 | 0.867 | 2.01 | ||
| RDD-Net [ 28 ] | 24.34 | 0.835 | 25.03 | 0.851 | 0.69 | ||
| RLP [ 37 ] | 24.34 | 0.833 | 25.23 | 0.835 | 0.89 | ||
| Turtle [ 12 ] | 19.35 | 0.712 | 24.40 | 0.838 | 5.05 | ||
| UConNet [ 43 ] | 18.82 | 0.713 | 21.85 | 0.778 | 3.03 | ||
| UENR-600K (Ours) | SynNightRain [ 24 ] | Pexels [ 25 ] | |
| Rain type | Synth. | Synth. | Real |
| Total frames | 600,000 † | 13,515 | 11,160 |
| Train frames | 498,200 | 11,915 | – |
| Test frames | 1,800 (20 clips) | 1,600 (8 clips) | 11,160 (124 clips) |
| Ground truth | ✓ | ✓ | – |
| † 600,000 frames across two subsets (City Sample: 500,000; Kyoto: 100,000). | |||
| Architecture | Input | Patch | Batch | Iters | Optimizer | LR | Loss | |
| ESTINet [ 40 ] | ResNet18 + R-CLSTM + BiLSTM | Vid-5 | 224 | 5 | 79.7K | Adam | 1e-4 | MSE |
| RDD-Net [ 28 ] | RDD-Net | Vid-7 | 128 | 8 | 320K | Adam | 1e-4 | L1 |
| RLP [ 37 ] | Uformer-T + RLP/RPIM | Img | 256 | 20 | 74.7K | AdamW | 2e-4 | Charbonnier |
| Turtle [ 12 ] | Transformer | Vid-5 | 128 | 1 | 200K | Adam | 4e-4 | L1 |
| UConNet [ 43 ] | UConNet + AngleNet | Img | 128 | 16 | 375K | Adam | 1e-3 | MSE |
| WeatherDiff [ 23 ] | DDPM UNet (83M) | Img | 64 | 80 † | 697K | Adam | 2e-5 | -pred |
| Patch Size | Padded Size | Grid | Patches/Frame | |
|---|---|---|---|---|
| ESTINet [ 40 ] | 224 224 | 1344 896 | 6 4 | 24 |
| RDD-Net [ 28 ] | 128 128 | 1280 768 | 10 6 | 60 |
| RLP [ 37 ] | 256 256 | 1536 768 | 6 3 | 18 |
| Turtle [ 12 ] | 128 128 | 1280 768 | 10 6 | 60 |
| UConNet [ 43 ] | 128 128 | 1280 768 | 10 6 | 60 |
| WeatherDiff [ 23 ] | 64 64 ( =16) | Overlapping | 77 42 | 3,234 |
| Original Setting | Our Setting (UENR-600K) | |||||||
| Dataset | Size | Epochs | Iters | Size/Epoch | Batch | Epochs | Iters | |
| ESTINet [ 40 ] | NTU-derain [ 6 ] | 3,023 | 120 | 181K | 99,640 | 5 | 4 | 79.7K |
| RDD-Net [ 28 ] | RainMotion [ 28 ] | 5,000 | 500 | 2.5M | 71,171 | 8 | 36 | 320K |
| RLP [ 37 ] | GTAV-NightRain [ 36 ] | 5,000 | 250 | 312K | 498,201 | 20 | 3 | 74.7K |
| Turtle [ 12 ] | (iter-based) | – | 200K iter | 200K | 99,640 | 1 | (iter) | 200K |
| UConNet [ 43 ] | Rain100/800 [ 32 , 39 ] | 40,000 ∗ | 150 | 375K | 40,000 ∗ | 16 | 150 | 375K |
| Input (MTok) | Output (MTok) | Cost ($) | |
|---|---|---|---|
| Pairwise quality (Tab. 2 ) | 11.57 | 0.45 | 69.00 |
| Multi-way quality (Tab. 3 ) | 2.82 | 0.04 | 15.00 |
| Temporal consistency (Tab. 4 ) | 27.32 | 0.20 | 141.55 |
| Total | 41.71 | 0.69 | 225.55 |
| s/frame | Total Time | |
| ESTINet [ 40 ] | 4.34 | 130.2 min |
| RDD-Net [ 28 ] | 2.79 | 83.7 min |
| RLP [ 37 ] | 1.09 | 32.7 min |
| Turtle [ 12 ] | 6.18 | 185.4 min |
| UConNet [ 43 ] | 0.81 | 24.3 min |
| WeatherDiff [ 23 ] | 11.00 | 5.50 h |
Synthetic Data Augmentation
Synthetic data augmentation creates or modifies training examples to improve learning when real data are scarce, imbalanced, costly, or sensitive.
See all 453 papers