S3VD: Semantic-Guidance Spatio-Temporal Scanning for Video Deraining

Kui Jiang; Yiang Chen; Yan Luo; Zhaocheng Yu; Junjun Jiang; Xianming Liu

cs.CVSep 18, 2026

S3VD: Semantic-Guidance Spatio-Temporal Scanning for Video Deraining

Authors: Kui Jiang, Yiang Chen, Yan Luo, Zhaocheng Yu, Junjun Jiang, Xianming Liu

Organizations: School of Computer Science and Technology, Harbin Institute of Technology, Harbin 150001, China. · Zhengzhou Advanced Research Institute, Harbin Institute of Technology, Zhengzhou 450001, China.

Abstract

Heavy rainfall severely degrades outdoor videos by corrupting high-frequency details and introducing motion blur, critically undermining the reliability of visual tasks. Recently, State Space Models (SSMs), particularly Mamba, have emerged as efficient alternatives for vision tasks with their linear complexity and ability to model long-range dependencies. However, when confronted with the poor visual representations in rainy videos, Mamba still faces difficulties in preserving the integrity of 2D spatial semantics and modeling 3D spatio-temporal correlations. To break these limitations, we introduce S3VD, a Semantic-Guidance Spatio-Temporal Scanning framework for video deraining, featuring two key innovations: Multi-Scale Semantic Fusion (MSSF) Module and Spatio-Temporal Scanning Fusion (STSF) Module. The former integrates temporal semantic priors from DINOv2 to guide precise feature representation and counteract the loss of local semantic context inherent to Mamba's 1D flatten operation, enhancing robustness against extreme degradation. The latter introduces a spatio-temporal scanning mechanism and devises a Decoupled-Gating Mamba (DG-Mamba) layer, which employs two independent gates to adaptively control preceding and subsequent contextual information within the input clip, optimizing intra-frame and inter-frame correlation modeling. Experiments on video deraining benchmarks demonstrate the superiority of S3VD, achieving state-of-the-art performance with an average 0.84 dB PSNR improvement over Mamba-based baselines.

Figures & tables

Figure 1: Motivation for S3VD. Without semantic guidance, existing Mamba architecture (independent SSM streams + rigid fusion) struggles with complex scenes, causing texture loss ( e.g. , white truck). S3VD integrates semantic priors and a DG-Mamba layer with decoupled gates for adaptive fusion, effectively removing rain while preserving background integrity.

Figure 2: FLOPs, Runtime and deraining performance comparison between our S3VD and other state-of-the-art algorithms. S3VD achieves the highest PSNR with the fastest runtime and highly competitive computational efficiency among state-of-the-art methods.

Figure 3: Overall architecture of our S3VD framework for video deraining. The proposed network leverages DINOv2 with our Multi-Scale Semantic Fusion (MSSF) Module to fuse robust semantic representations and pixel features. Features are then processed through Spatio-Temporal Scanning Fusion (STSF) Modules, each consisting of a Spatial Mamba Block (SMB) and Temporal Mamba Block (TMB) for comprehensive spatio-temporal modeling. Each Mamba block incorporates our novel Decoupled-Gating Mamba (DG-Mamba) layer that enables efficient long-range dependency modeling in both spatial and temporal dimensions. 3D convolution layers bridge between STSF modules at different scales to ensure effective multi-scale feature integration before final reconstruction through the decoder.

VRDS		RainVID&SS		RainSynAll100		LWDDS
Method	PSNR $\uparrow$ /SSIM $\uparrow$ /LPIPS $\downarrow$	Method	PSNR $\uparrow$ /SSIM $\uparrow$	Method	PSNR $\uparrow$ /SSIM $\uparrow$	Method	PSNR $\uparrow$ /SSIM $\uparrow$
CCN 28	23.75 / 0.8410 / 0.2091	FastDerain 15	17.88 / 0.4506	FCRVD 41	21.06 / 0.7405	CCN 28	27.53 / 0.9220
ESTINet 46	27.17 / 0.8436 / 0.2253	MSCSC 19	19.19 / 0.5250	RMFD 42	25.14 / 0.9172	VWR 37	30.72 / 0.9726
DRSformer 6	28.54 / 0.9075 / 0.1143	PReNet 30	24.90 / 0.7464	BasicVSR++ 3	27.67 / 0.9135	BasicVSR++ 3	32.37 / 0.9792
BasicVSR++ 3	29.75 / 0.9171 / 0.1023	S2VD 45	29.69 / 0.9136	NCFL 11	28.11 / 0.9235	ViMP-Net 38	34.22 / 0.9784
ViMP-Net 38	31.02 / 0.9283 / 0.0862	VDMamba 35	32.69 / 0.9233	SALN 44	29.78 / 0.9315	SALN 44	36.57 / 0.9802

Table 1: Quantitative comparison with state-of-the-art methods on four benchmarks. The best results are highlighted in bold .

Panel of Figure 4 — Figure 4: Visual comparison of derained results produced by S3VD and state-of-the-art methods on the VRDS 38 , RainVID&SS 34 , RainSynAll100 42 , and LWDDS 37 datasets. Please zoom in for a better illustration.

Panel of Figure 5 — Figure 5: Visual Comparison of derained results between our S3VD and other state-of-the-art methods on the real-world dataset from 34 . Please zoom in for a better illustration.

Panel of Figure 6 — Figure 6: Visual comparison demonstrating the robustness of DINOv2 features to rain. The figure contrasts the DINOv2 feature maps extracted from (a) three consecutive frames of ground truth with those from (b) the rainy counterpart. Despite the significant visual corruption in the rainy frames, their corresponding feature maps remain remarkably temporally consistent and structurally similar to the features from the clean sequence. This stability highlights DINOv2’s ability to provide reliable semantic context regardless of rain interference.

Method	MSCSC	RMFD	S2VD	RainMamba	S3VD
WE $\downarrow$	0.054	0.043	0.033	0.029	0.013

Table 2: Temporal-consistency comparison on real-world rainy videos.

Feature representation	Mean cosine similarity $\uparrow$
ConvNeXt features	$0.71\pm 0.08$
DINOv2 semantic features	$0.89\pm 0.04$

Table 3: Clean–rainy feature similarity on the VRDS dataset 38 .

Feature configuration	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Semantic features only	30.74	0.9291	0.0788
Pixel features only	32.35	0.9413	0.0642
Pixel features $+$ Semantic features	32.43	0.9427	0.0610

Table 4: Complementarity between pixel and semantic representations on the VRDS dataset 38 .

Method	BasicVSR++	ESTINet	DRSformer	RainMamba	VDMamba	Ours
Parameters(M) $\downarrow$	6.22	22.96	33.63	34.75	12.70	42.08
FLOPs(G) $\downarrow$	1616.44	681.83	1101.89	118.99	389.79	143.58
Time(s/frame) $\downarrow$	0.051	0.0341	0.0381	0.0112	0.0320	0.0089

Table 5: Model Complexity and Efficiency Comparisons.

Model	MSSF	STSF	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
D1			32.04	0.9366	0.0684
D2	✓		32.10	0.9403	0.0635
D3		✓	32.35	0.9413	0.0642
S3VD (Ours)	✓	✓	32.43	0.9427	0.0610

Table 6: Ablation of key network components on the VRDS dataset 38 .

$\lambda_{1}$	$\lambda_{2}$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
0	0	31.96	0.9388	0.0669
0.3	0	32.25	0.9416	0.0629
0	0.1	32.18	0.9409	0.0638
0.3	0.1	32.43	0.9427	0.0610
0.1	0.1	32.34	0.9420	0.0621
0.3	0.05	32.38	0.9423	0.0616

Table 7: Loss-function ablation and weight sensitivity on VRDS 38 .

Model	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	Param $\downarrow$	FLOPS $\downarrow$	Runtime $\downarrow$
DINOv2-small (Ours)	32.43	0.9427	0.0610	63.08	143.58	0.0089
DINOv2-small with register	32.39	0.9417	0.0622	63.08	143.58	0.0078
DINOv2-base	32.53	0.9431	0.0618	127.63	248.42	0.0094
DINOv2-large	32.58	0.9422	0.0618	345.40	602.17	0.0143
DINOv1-small	32.34	0.9410	0.0632	63.30	129.43	0.0076
CLIP-visual	32.42	0.9413	0.0627	127.76	193.21	0.0082

Table 8: Ablation of semantic extractors on the VRDS dataset 38 .

Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
single frame	32.22	0.9420	0.0622
cross-frame average	32.39	0.9425	0.0611
cross-frame concatenation (Ours)	32.43	0.9427	0.0610

Table 9: Results of different setting of query in the MSSF on the VRDS dataset 38 .

Fused levels	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
$L_{4}$	32.19	0.9408	0.0641
$L_{3}+L_{4}$	32.28	0.9415	0.0628
$L_{2}+L_{3}+L_{4}$	32.36	0.9422	0.0617
$L_{1}+L_{2}+L_{3}+L_{4}$	32.43	0.9427	0.0610

Table 10: Ablation of fused feature levels in MSSF on the VRDS dataset 38 .

Fusion operator	Params (M)	GFLOPs	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Concatenation $+$ $1\times 1$ Conv	40.31	132.75	32.17	0.9406	0.0638
Bidirectional Mamba fusion	42.36	146.91	32.31	0.9418	0.0622
Cross-attention (MSSF)	42.08	143.58	32.43	0.9427	0.0610

Table 11: Ablation off semantic–pixel fusion modules on the VRDS dataset 38 .

Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
T+S	32.05	0.9402	0.0638
S+S	32.13	0.9407	0.0625
T+T	32.31	0.9414	0.0634
S+T (Ours)	32.43	0.9427	0.0610

Table 12: Results of different combinations of spatio and temporal Mamba blocks on the VRDS dataset 38 .

Depth	Params $\downarrow$	GFLOPs $\downarrow$	Time $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
$(1,2,1)$	35.42	118.62	0.0074	32.18	0.9405	0.0642
$(2,2,2)$	39.76	135.47	0.0082	32.34	0.9420	0.0621
$\underline{(2,3,2)}$	42.08	143.58	0.0089	32.43	0.9427	0.0610
$(3,4,3)$	48.91	169.84	0.0106	32.45	0.9428	0.0609

Table 13: Ablation of stage-wise STSF depth on the VRDS dataset 38 .

Shared core	Direction gates	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
–	–	32.12	0.9410	0.0624
✓	–	32.24	0.9417	0.0619
–	✓	32.29	0.9421	0.0616
✓	✓	32.43	0.9427	0.0610

Table 14: Component-wise ablation of DG-Mamba on the VRDS dataset.

Video Restoration

Video restoration reconstructs clearer, more faithful, temporally consistent footage from blur, noise, compression, weather, and other degradations.

See all 59 papers

FluidRain: Incompressible Rain Flow as an Attention Bias for Loop-in-Loop Video Deraining
Sep 24, 2026Pu Wang, Yongcong Wang, Wenhao Li +6Attention Mechanisms Image Deraining
Shandong University · Central South University · Nanjing University of Science and Technology +2
RainDancer: RGB-Event Video Deraining with Rain-Oriented Spiking Dynamics
Jul 15, 2026Kui Jiang, Runzhe Li, Zhaocheng Yu +3Event-Based Vision RGB-Event Sensor Fusion
School of Computer Science and Technology, Harbin Institute of Technology, Harbin, China · School of Computer Science and Technology, Harbin University of Science and Technology, Harbin, China
MambaVF: State Space Model for Efficient Video Fusion
Feb 5, 2026Zixiang Zhao, Yukun Cui, Lilun Deng +4Multimodal Fusion
ETH Zürich · Xi’an Jiaotong University · Nanyang Technological University +1

Sep 24, 2026J/K move · Enter open · S save

FluidRain: Incompressible Rain Flow as an Attention Bias for Loop-in-Loop Video Deraining

Pu Wang, Yongcong Wang, Wenhao Li, Xiang Chen+5

Shandong University · Central South University · Nanjing University of Science and Technology+2

Attention Mechanisms Image Deraining

Open paper AskPDF arXiv

Existing video deraining methods typically exploit neighboring frames through either explicit alignment or implicit spatiotemporal aggregation. Explicit alignment relies on accurate motion estimation, which can become unreliable under dense rain, while implicit aggregation avoids alignment but lacks explicit guidance on the directional and temporally coherent structure of rain. This leaves a gap between reliable temporal aggregation and explicit modeling of rain motion. To address these limitations, we propose FluidRain, a lightweight video derainer that uses divergence-free rain flow to guide Loop-in-Loop attention across scales and neighboring frames. Motivated by fluid mechanics, we model rain motion as a divergence-free image-space flow and use it to organize multi-scale and temporal aggregation. Specifically, FluidRain first estimates a rain-flow field for each frame and projects it onto the divergence-free subspace. The resulting flow steers window attention along rain streaks, enabling neighboring frames to be aggregated without explicit alignment. Since rain-flow structure is preserved across scales and nearby frames, Loop-in-Loop reuses the same attention operator across both dimensions, resulting in a three-frame model with only 0.80M parameters. Experiments on four benchmarks show that FluidRain remains competitive with substantially larger restoration models. We further examine how temporal evidence scales with different input views. To evaluate whether the model remains reliable when rain motion changes across frames, we introduce RainSyn-Gust, which injects controlled changes in rain-streak direction into existing benchmarks. We also develop a physics-based no-reference metric that evaluates real-rain removal without requiring clean targets.

Abstract

Video Restoration

Explore similar work

FluidRain: Incompressible Rain Flow as an Attention Bias for Loop-in-Loop Video Deraining

RainDancer: RGB-Event Video Deraining with Rain-Oriented Spiking Dynamics

MambaVF: State Space Model for Efficient Video Fusion