UENR-600K: A Large-Scale Physically Grounded Dataset for Nighttime Video Deraining

Pei Yang; Hai Ci; Beibei Lin; Yiren Song; Mike Zheng Shou

cs.CVApr 6, 2026

UENR-600K: A Large-Scale Physically Grounded Dataset for Nighttime Video Deraining

Authors: Pei Yang, Hai Ci, Beibei Lin, Yiren Song, Mike Zheng Shou

Organizations: Show Lab, National University of Singapore · National University of Singapore

Abstract

Nighttime video deraining is uniquely challenging because raindrops interact with artificial lighting. Unlike daytime white rain, nighttime rain takes on various colors and appears locally illuminated. Existing small-scale synthetic datasets rely on 2D rain overlays and fail to capture these physical properties, causing models to generalize poorly to real-world night rain. Meanwhile, capturing real paired nighttime videos remains impractical because rain effects cannot be isolated from other degradations like sensor noise. To bridge this gap, we introduce UENR-600K, a large-scale, physically grounded dataset containing 600,000 1080p frame pairs. We utilize Unreal Engine to simulate rain as 3D particles within virtual environments. This approach guarantees photorealism and physically real raindrops, capturing correct details like color refractions, scene occlusions, rain curtains. Leveraging this high-quality data, we establish a new state-of-the-art baseline by adapting the Wan 2.2 video generation model. Our baseline treat deraining as a video-to-video generation task, exploiting strong generative priors to almost entirely bridge the sim-to-real gap. Extensive benchmarking demonstrates that models trained on our dataset generalize significantly better to real-world videos. Project page: https://showlab.github.io/UENR-600K/.

Figures & tables

Figure 1: Paper Overview. Top: We use Unreal Engine 5 to simulate rain as 3D particles within virtual environments, producing 600,000 paired 1080p frames with physically grounded nighttime rain. Left: a rainy video frame; right: the paired ground truth. Bottom: We finetune a video Diffusion Transformer on our dataset for nighttime video deraining. Given a real nighttime rain video (left), our baseline removes rain and rain-induced fog near the streetlamp while preserving scene detail (right; see red crop).

Dataset	Year	Venue	Rain Type a	GT Type b	#Rainy Frames c	#GT Frames c	Resolution	Night d	Video e	Raining In-Scene f
Rain200L/H [ 32 ]	2017	CVPR	Synth.	Real	4,000	4,000	435 $\times$ 366
DDN-Data [ 9 ]	2017	CVPR	Synth.	Real	13,000	13,000	489 $\times$ 428
DID-Data [ 38 ]	2018	CVPR	Synth.	Real	13,200	13,200	512 $\times$ 512
NTURain g [ 6 ]	2018	CVPR	Synth.	Real	-	-	640 $\times$ 480		✓
RainSynAll25 [ 20 ]	2018	CVPR	Synth.	Real	-	-	-		✓
Rain800 [ 39 ]	2017	-	Synth.	Real	800	800	518 $\times$ 419

Table 1: Overview of deraining datasets. Among the datasets surveyed, our dataset is the only one providing video data of in-scene nighttime rain.

Figure 2: Properties of nighttime rain illustrated with frames from our dataset. Chromaticity: raindrops refract colored artificial light (blue, yellow, green, red) rather than appearing white. Localization: rain is visible near light sources but fades in unlit regions. Glimmer effect: raindrops produce sudden high-intensity flashes as they pass through focused light beams. Rain curtains: wind-driven sheets of rain form volumetric, shifting patterns.

Figure 3: Comparison of rain synthesis between SynNightRain [ 24 ] (top row) and our dataset (bottom row). Each pair shows a ground-truth frame alongside its rainy counterpart. SynNightRain overlays rain as a global white layer that uniformly covers the entire frame, without responding to scene geometry or lighting. Our dataset simulates rain within a virtual scene: raindrops are correctly occluded by scene objects, form volumetric rain curtains, and refract local artificial light to produce colorful, spatially varying streaks.

Figure 4: Our baseline architecture, adapted from the Wan 2.2 Video DiT. The rainy input is encoded into condition tokens (blue) and concatenated with generation tokens (red); the DiT denoises only the generation tokens while using the condition tokens as context. A unidirectional attention mask prevents condition tokens from attending to generation tokens, keeping the input uncorrupted. Only LoRA adapters on the QKV projections are trained; all other parameters stay frozen.

	Input Size	SynNightRain [ 24 ]	Tie	Our Data
ESTINet [ 40 ]	224 $\times$ 224	16.4%	6.5%	77.2%
RDD-Net [ 28 ]	128 $\times$ 128	32.3%	1.6%	66.1%
RLP [ 37 ]	256 $\times$ 256	27.7%	8.6%	63.7%
Turtle [ 12 ]	128 $\times$ 128	16.1%	6.2%	77.7%
UConNet [ 43 ]	128 $\times$ 128	3.2%	1.3%	95.4%
WeatherDiff [ 23 ]	64 $\times$ 64	91.7%	3.0%	5.4%

Table 2: Pairwise comparison of deraining results on real nighttime rain videos, where a VLM selects the better result based on rain removal and detail preservation. For each method, the version trained on our dataset competes against the version trained on SynNightRain [ 24 ] ; bold indicates the preferred training dataset. Input Size is the spatial resolution each method processes during training and inference. Training on our dataset yields preferred results for all methods with input sizes

\geq

128

\times

128, covering all architecture families. The two exceptions, WeatherDiff and NightRain, use 64

\times

64 inputs (less than 1% of a 1280

\times

720 frame), which limits their ability to distinguish localized rain from clean dark regions in our data.

	Architecture	SynNightRain [ 24 ]	Our Data	Average
ESTINet [ 40 ]	CNN+RNN	12.9%	5.6%	9.3%
RDD-Net [ 28 ]	CNN	10.5%	0.0%	5.2%
RLP [ 37 ]	Transformer	4.0%	0.0%	2.0%
Turtle [ 12 ]	Transformer	0.8%	0.0%	0.4%
UConNet [ 43 ]	CNN	0.0%	0.0%	0.0%
WeatherDiff [ 23 ]	Diffusion	1.6%	0.0%	0.8%

Table 3: Multi-way comparison of deraining results on real nighttime rain videos, where a VLM selects the best result among all eight methods trained on the same dataset. Pretrained : model was pretrained on large-scale video data before finetuning. Our baseline, the only pretrained method, dominates in both settings: 94.4% when trained on our dataset and 69.4% when trained on SynNightRain.

Figure 5: Qualitative comparison of all eight methods on four real nighttime rain scenes. All methods trained on our dataset. Red annotations highlight regions for comparison. Existing restoration methods (ESTINet through UConNet) reduce rain to varying degrees but leave visible streaks in heavy rain regions. The two 64 $\times$ 64 diffusion models (WeatherDiff, NightRain) introduce haze or darken the scene. Our baseline, leveraging its pretrained generative prior, removes rain almost completely without introducing artifacts, demonstrating that the sim-to-real gap is effectively bridged.

Figure 6: Our baseline finetuned on SynNightRain (middle) versus our dataset (right) across six real nighttime rain scenes from Pexels [ 25 ] . The SynNightRain-finetuned baseline struggles with chromatic rain near colored lighting (row 1), localized rain around streetlamps (row 4), and rain-induced haze (rows 5 and 6; see red crops). The version finetuned on our dataset handles all these conditions effectively.

	SynNightRain		Our Data		Average
	AFD $\downarrow$	VLM $\uparrow$	AFD $\downarrow$	VLM $\uparrow$	AFD $\downarrow$	VLM $\uparrow$
ESTINet [ 40 ]	6.40	3.22	5.70	3.32	6.05	3.27
RDD-Net [ 28 ]	7.04	3.37	9.93	2.37	8.49	2.87
RLP [ 37 ]	7.75	3.11	10.11	2.60	8.93	2.86
Turtle [ 12 ]	8.55	2.71	8.08	3.07	8.32	2.89
UConNet [ 43 ]	7.94	2.47	9.34	3.03	8.64	2.75

Table 4: Temporal consistency of deraining results on real nighttime rain videos, evaluated by two complementary metrics. AFD

\downarrow

: average inter-frame LPIPS (

\times 10^{2}

; lower is better), measuring perceptual difference between consecutive output frames. VLM

\uparrow

: VLM-rated temporal smoothness on a 1 to 5 scale (higher is better). Our baseline achieves the best temporal consistency under both metrics and both training datasets.

	Our Data		SynNightRain [ 24 ]		$\Delta$ PSNR
	PSNR $\uparrow$	SSIM $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	$\Delta$ PSNR
ESTINet [ 40 ]	24.88	0.845	26.89	0.867	$-$ 2.01
RDD-Net [ 28 ]	24.34	0.835	25.03	0.851	$-$ 0.69
RLP [ 37 ]	24.34	0.833	25.23	0.835	$-$ 0.89
Turtle [ 12 ]	19.35	0.712	24.40	0.838	$-$ 5.05
UConNet [ 43 ]	18.82	0.713	21.85	0.778	$-$ 3.03

Table 5: Same-dataset benchmarking: each method is trained and tested on the same dataset.

\Delta

PSNR = Our Data PSNR minus SynNightRain PSNR; positive values indicate that SynNightRain is easier to fit, while the negative value (bold) indicates the opposite. All seven existing methods achieve higher PSNR on SynNightRain, confirming its simpler degradation patterns. Our baseline is the only method that scores higher on our dataset, because its pretrained generative prior aligns better with rendering-based rain.

Figure 7: Cross-dataset evaluation of five representative methods. Each method is trained on one dataset and tested on both. Diagonal cells represent same-dataset performance, while off-diagonal cells show cross-dataset performance in PSNR. All methods experience a substantial performance drop when tested on a different dataset than their training source, highlighting the significant domain gap between our physically grounded data and the algorithmically synthesized SynNightRain.

† 600,000 frames across two subsets (City Sample: 500,000; Kyoto: 100,000).
	UENR-600K (Ours)	SynNightRain [ 24 ]	Pexels [ 25 ]
Rain type	Synth.	Synth.	Real
Total frames	600,000 †	13,515	11,160
Train frames	498,200	11,915	–
Test frames	1,800 (20 clips)	1,600 (8 clips)	11,160 (124 clips)
Ground truth	✓	✓	–

Table 6: Dataset splits used in our experiments. All frames are at 1280

\times

720. UENR-600K and SynNightRain provide paired rainy/clean frames for training and synthetic evaluation. The Pexels set has no ground truth and is used exclusively for VLM-based evaluation.

	Architecture	Input	Patch	Batch	Iters	Optimizer	LR	Loss
ESTINet [ 40 ]	ResNet18 + R-CLSTM + BiLSTM	Vid-5	224	5	79.7K	Adam	1e-4	MSE
RDD-Net [ 28 ]	RDD-Net	Vid-7	128	8	320K	Adam	1e-4	L1
RLP [ 37 ]	Uformer-T + RLP/RPIM	Img	256	20	74.7K	AdamW	2e-4	Charbonnier
Turtle [ 12 ]	Transformer	Vid-5	128	1	200K	Adam	4e-4	L1
UConNet [ 43 ]	UConNet + AngleNet	Img	128	16	375K	Adam	1e-3	MSE
WeatherDiff [ 23 ]	DDPM UNet (83M)	Img	64	80 †	697K	Adam	2e-5	$\epsilon$ -pred

Table 7: Training hyperparameters for each method. All existing methods are trained from scratch; our baseline finetunes a pretrained video generation model using LoRA. Input type indicates whether the method processes single images (Img) or video clips of

N

consecutive frames (Vid-

N

). Patch size is the spatial crop used during training. Total iterations (Iters) are matched to each method’s original setting (see Tab. 9 ).

	Patch Size	Padded Size	Grid	Patches/Frame
ESTINet [ 40 ]	224 $\times$ 224	1344 $\times$ 896	6 $\times$ 4	24
RDD-Net [ 28 ]	128 $\times$ 128	1280 $\times$ 768	10 $\times$ 6	60
RLP [ 37 ]	256 $\times$ 256	1536 $\times$ 768	6 $\times$ 3	18
Turtle [ 12 ]	128 $\times$ 128	1280 $\times$ 768	10 $\times$ 6	60
UConNet [ 43 ]	128 $\times$ 128	1280 $\times$ 768	10 $\times$ 6	60
WeatherDiff [ 23 ]	64 $\times$ 64 ( $r$ =16)	Overlapping	77 $\times$ 42	3,234

Table 8: Sliding window configuration for inference at 1280

\times

720 resolution. Since each method operates on fixed-size patches, the original frame must first be enlarged to fit an exact grid of patches. Padded Size is the resulting resolution after zero-padding the bottom and right edges of the 1280

\times

720 frame to the nearest multiple of the Patch Size ; for example, with 224

\times

224 patches, 1280

\times

720 is padded to 1344

\times

896 (= 6

\times

224 by 4

\times

224). The padded frame is then partitioned into a non-overlapping grid of tiles. WeatherDiff uses overlapping patches with stride

r

=16 instead of non-overlapping tiling, following its original inference procedure. Our baseline processes full-resolution frames without patch decomposition.

	Original Setting				Our Setting (UENR-600K)
	Dataset	Size	Epochs	Iters	Size/Epoch	Batch	Epochs	Iters
ESTINet [ 40 ]	NTU-derain [ 6 ]	3,023	120	$\sim$ 181K	99,640	5	4	$\sim$ 79.7K
RDD-Net [ 28 ]	RainMotion [ 28 ]	$\sim$ 5,000	500	$\sim$ 2.5M	71,171	8	36	$\sim$ 320K
RLP [ 37 ]	GTAV-NightRain [ 36 ]	5,000	250	$\sim$ 312K	498,201	20	3	$\sim$ 74.7K
Turtle [ 12 ]	(iter-based)	–	200K iter	200K	99,640	1	(iter)	200K
UConNet [ 43 ]	Rain100/800 [ 32 , 39 ]	40,000 ∗	150	$\sim$ 375K	40,000 ∗	16	150	$\sim$ 375K

Table 9: Iteration matching between original training settings and our setup. For each method, we compute the total gradient updates from the original paper (dataset size

\times

epochs / batch size) and reproduce a comparable number on our dataset by adjusting the number of epochs. Our baseline is finetuned for only 1,400 steps, consuming approximately one epoch of training data.

Figure 8: Example of a multi-way VLM evaluation item. The VLM receives the rainy input (top-left) and eight candidate derained images labeled A through H. Method-to-label assignments are randomized per evaluation item, so the VLM cannot learn positional patterns. For this example, the VLM selected Candidate C with the justification: “Candidate C produces the cleanest rain removal with natural color and detail preservation, free of the haze artifacts seen in H and the residual streaks in A and F.”

Figure 9: Example of a temporal consistency VLM evaluation item. Top: four consecutive rainy input frames providing scene context. Middle: four consecutive derained output frames to evaluate. Bottom: three inter-frame difference maps (amplified 10 $\times$ ), where bright regions indicate large pixel changes between adjacent output frames. In a temporally consistent result, only regions with real scene motion (e.g., swaying leaves) appear bright, while static regions (e.g., lamp post, sky) remain dark. Flickering artifacts would manifest as bright patches in static areas.

	Input (MTok)	Output (MTok)	Cost ($)
Pairwise quality (Tab. 2 )	11.57	0.45	69.00
Multi-way quality (Tab. 3 )	2.82	0.04	15.00
Temporal consistency (Tab. 4 )	27.32	0.20	141.55
Total	41.71	0.69	225.55

Table 10: API cost for reproducing all VLM evaluations using claude-opus-4-6 [ 1 ] . Each 1280

\times

720 image consumes 1,229 input tokens; the 5,208 evaluations use 41.7M input tokens and 0.7M output tokens in total. The temporal evaluation is the most expensive because each item requires 11 images (4 input frames, 4 output frames, and 3 difference maps).

	s/frame	Total Time
ESTINet [ 40 ]	4.34	130.2 min
RDD-Net [ 28 ]	2.79	83.7 min
RLP [ 37 ]	1.09	32.7 min
Turtle [ 12 ]	6.18	185.4 min
UConNet [ 43 ]	0.81	24.3 min
WeatherDiff [ 23 ]	11.00	5.50 h

Table 11: Inference time on the Ours test set (1,800 frames at 1280

\times

720) using a single GPU. Non-diffusion methods complete within minutes to hours, while diffusion-based methods (NightRain, WeatherDiff) require significantly longer due to iterative denoising over many patches. Our baseline processes full-resolution frames without patch decomposition.

Synthetic Data Augmentation

Synthetic data augmentation creates or modifies training examples to improve learning when real data are scarce, imbalanced, costly, or sensitive.

See all 454 papers

RainDancer: RGB-Event Video Deraining with Rain-Oriented Spiking Dynamics
Jul 15, 2026Kui Jiang, Runzhe Li, Zhaocheng Yu +3Event-Based Vision RGB-Event Sensor Fusion
School of Computer Science and Technology, Harbin Institute of Technology, Harbin, China · School of Computer Science and Technology, Harbin University of Science and Technology, Harbin, China
Cross-Scenario Deraining Adaptation with Unpaired Data: Superpixel Structural Priors and Multi-Stage Pseudo-Rain Synthesis
Mar 23, 2026Kangbo Zhao, Miaoxin Guan, Xiang Chen +2Image Deraining Synthetic-to-Real Domain Adaptation
School of Information Engineering, Guangdong University of Technology, Guangzhou, 510006, China · School of Computer Science and Engineering, Nanjing University of Science and Technology
UniRain: Unified Image Deraining with RAG-based Dataset Distillation and Multi-objective Reweighted Optimization
Mar 4, 2026Qianfeng Yang, Qiyuan Guan, Xiang Chen +3Image Deraining Image Enhancement
Dalian Polytechnic University · Nanjing University of Science and Technology

Jul 15, 2026J/K move · Enter open · S save

RainDancer: RGB-Event Video Deraining with Rain-Oriented Spiking Dynamics

Kui Jiang, Runzhe Li, Zhaocheng Yu, Guanglu Sun+2

School of Computer Science and Technology, Harbin Institute of Technology, Harbin, China · School of Computer Science and Technology, Harbin University of Science and Technology, Harbin, China

Event-Based Vision RGB-Event Sensor Fusion Multimodal Fusion

Open paper AskPDF arXiv

Video deraining aims to recover clean visual content from rainy videos for reliable perception under adverse weather. Existing methods mainly rely on RGB sequences and temporal redundancy, but RGB-only restoration remains ambiguous in dynamic rainy scenes, where rain streaks, textures, boundaries, motion, and occlusions may share similar visual patterns. Event cameras provide complementary motion-sensitive cues with high temporal resolution, but event streams also contain sensor noise and background-triggered responses, so direct RGB-Event fusion may introduce cross-modal interference. To address this issue, we propose RainDancer, a progressive RGB-Event video deraining framework based on a decompose-before-interact paradigm. The core idea is to separate rain and background components within each modality before cross-modal interaction. In the RGB branch, frame features are progressively decomposed into rain and background representations. In the event branch, a rain-oriented spiking neural network module captures sparse and bursty event dynamics associated with rain motion. Component-level fusion is then performed between semantically aligned representations for structure preservation and rain suppression. We further introduce event-domain supervision to regularize sparse event reconstruction, structural consistency, and gradient orientation. Experiments on synthetic and real RGB-Event video deraining datasets demonstrate superior quantitative performance, visual quality, and downstream perception robustness. Code is available at https://github.com/AE86-plus/RainDancer.

Abstract

Synthetic Data Augmentation

Explore similar work

RainDancer: RGB-Event Video Deraining with Rain-Oriented Spiking Dynamics

Cross-Scenario Deraining Adaptation with Unpaired Data: Superpixel Structural Priors and Multi-Stage Pseudo-Rain Synthesis

UniRain: Unified Image Deraining with RAG-based Dataset Distillation and Multi-objective Reweighted Optimization