Organizations: Department of Precision Instrument, Tsinghua University, Beijing, China. · The Hong Kong University of Science and Technology, HKSAR, China. · Primevision Technology, Shanghai, China. · Pen-Tung Sah Institute of Micro-Nano Science and Technology, Xiamen University, Xiamen, China.
Capturing scenes with extreme motion and high dynamic range is fundamentally constrained by trade-offs among temporal resolution, spatial detail, signal-to-noise ratio, and dynamic range under practical bandwidth, power, and quantum-efficiency limits. Conventional frame-based cameras therefore suffer from temporal undersampling, exposure-induced motion blur, and sensor saturation. The Complementary Vision Sensor (CVS) addresses these limitations by decomposing visual information across two complementary pathways, a spatially dense, temporally sparse, long-exposure RGB pathway and a temporally dense, spatially sparse, short-exposure patiotemporal difference pathway. Their complementary measurements provide appearance, structural, and motion cues across a wide dynamic range that enable reconstruction beyond conventional imaging. We present CoGenRec, a complementarity-driven generative reconstruction framework that unifies video deblurring, high-temporal-resolution frame interpolation, and HDR fusion. CoGenRec encodes complementary measurements into multi-scale spatiotemporal features using exposure-aware appearance modeling and bidirectional temporal propagation. These features constrain a video diffusion model and are injected into the VAE decoder to recover fine structures lost during latent-space compression, combining the scene fidelity of physical measurements with the appearance recovery capabilities of learned video priors. On SportsSloMo-CVS, CoGenRec improves PSNR over the strongest general-purpose baseline by 4.40 dB for deblurring, 3.89 dB for frame interpolation, and 5.04 dB for HDR fusion, respectively. It also achieves strong perceptual quality on 129 unseen scenes captured with real CVS. These results establish a unified approach to advance high-quality reconstruction under challenging motion and illumination conditions. Project page: https://cogenrec.github.io/.
Figures & tables
Fig. 1: Motivation for capturing extreme dynamic scenes with the Complementary Vision Sensor (CVS) and reconstructing them using our complementarity-driven generative reconstruction framework, CoGenRec. (a) Limitations of conventional dense-sampling image sensors and how CVS alleviates them. Under limited bandwidth and power, frame rate and dynamic range are mutually constrained, while signal-to-noise ratio (SNR) and motion blur involve an inherent trade-off. CVS alleviates these constraints through two synergistic visual pathways. (b) Regression-based reconstruction versus CoGenRec. In the ill-posed solution space induced by extreme scenes, regression models tend to average plausible solutions, producing over-smoothed results. CoGenRec instead exploits the spatial, temporal, and exposure complementarity of CVS to guide generative sampling and jointly reconstruct blur-free, high-frame-rate HDR-fused video.
Fig. 2: Overall architecture of CoGenRec. The Multi-Scale Spatio-Temporal Guidance Extractor first encodes the dual-pathway CVS measurements into dense multi-scale guidance volumes. These volumes condition the Complementarity-Guided Video Diffusion Backbone through a multi-scale ControlNet. Finally, the Detail-Preserving VAE Decoder fuses the uncompressed multi-scale guidance features into the VAE decoding hierarchy and produces sharp, high-frame-rate, and HDR-fused RGB frames.
Fig. 3: Qualitative comparisons on SportsSloMo-CVS under the unified training protocol. All baselines are retrained on the same large-scale mixed training dataset as our method. Each row shows one test scene. (a) Deblurring results for a blurry RGB endpoint frame. (b) Interpolation results at the temporal midpoint between two adjacent blurry RGB endpoint frames, RGB0 and RGB1. (c) HDR synthesis results at the temporal position corresponding to Mid-SD between two adjacent RGB frames; the highlights in RGB0 are saturated, whereas Mid-SD visualizes the complementary structural cues retained by the short-exposure differential pathway at that time. STGDNet is specific to motion deblurring and is therefore included only in (a).
Method
5
7
9
11
Average
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
Tianmouc-UNet [ 10 ]
27.50
0.8848
0.1648
27.70
0.8906
0.1652
27.46
0.8780
0.1844
27.11
0.8692
0.1952
27.44
0.8807
0.1774
IGFNet [ 11 ]
28.09
0.9008
0.1398
28.16
0.9038
0.1415
27.89
0.8910
0.1583
27.43
0.8839
0.1686
27.89
0.8949
0.1521
CBRDM [ 13 ]
22.57
0.7730
0.1487
22.93
0.8043
0.1503
22.43
0.7918
0.1708
22.13
0.7711
0.1836
22.52
0.7851
0.1633
STGDNet [ 14 ]
35.10
0.9598
0.0387
34.68
0.9569
0.0446
34.12
0.9517
0.0523
33.39
0.9463
0.0602
34.33
0.9537
0.0490
CoGenRec (Ours)
32.67
0.9452
0.0383
32.55
0.9441
0.0403
32.14
0.9380
0.0454
31.82
0.9347
0.0482
32.30
0.9405
0.0431
TABLE I: Quantitative motion-deblurring comparison on the SportsSloMo-CVS test set. The blurry RGB inputs integrate 5, 7, 9, and 11 consecutive sharp frames, respectively; Average denotes the arithmetic mean over the four blur levels. The best and second-best results in each column are highlighted in bold and underlined, respectively.
Method
Middle Frame
All Frames
TR ↓
PSNR ↑
SSIM ↑
LPIPS ↓
NIQE ↓
FID ↓
PSNR ↑
SSIM ↑
LPIPS ↓
NIQE ↓
FID ↓
Tianmouc-UNet [ 10 ]
27.17
0.8722
0.1835
6.56
47.86
27.32
0.8763
0.1793
6.52
33.51
0.019735
IGFNet [ 11 ]
27.67
0.8888
0.1557
6.53
38.72
27.77
0.8912
0.1541
6.53
27.05
0.018129
CBRDM [ 13 ]
21.21
0.7429
0.2082
5.15
62.02
21.77
0.7624
0.1851
5.16
35.43
0.029459
CoGenRec (Ours)
31.18
0.9309
0.0520
5.77
13.59
31.66
0.9355
0.0475
5.75
6.50
0.016047
TABLE II: Quantitative frame-interpolation comparison on the SportsSloMo-CVS test set. “Middle Frame” denotes the temporal midpoint of the reconstructed sequence, whereas “All Frames” denotes all seven reconstructed frames in each sample. PSNR, SSIM, LPIPS, and NIQE are averaged over the corresponding frames and samples, while FID [ 40 ] is computed over the corresponding pooled image set. TR is the motion-compensated temporal residual between adjacent predictions. The best and second-best results in each column are highlighted in bold and underlined, respectively.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
NIQE ↓
FID ↓
TR ↓
Tianmouc-UNet [ 10 ]
24.69
0.8416
0.2340
6.66
57.21
0.022850
IGFNet [ 11 ]
24.57
0.8511
0.2178
6.72
52.21
0.023154
CBRDM [ 13 ]
17.81
0.7069
0.2445
5.16
48.27
0.028478
CoGenRec (Ours)
29.72
0.9263
0.0647
5.69
8.70
0.016891
TABLE III: Quantitative comparison of HDR fusion on the SportsSloMo-CVS test set. PSNR, SSIM, LPIPS, and NIQE are first averaged over the seven reconstructed frames within each sample and then across all samples, while FID is computed over the pooled set of all reconstructed frames. The best and second-best results in each column are highlighted in bold and underlined, respectively.
Method
Training
NIQE ↓
MUSIQ ↑
MANIQA ↑
CLIP-IQA+ ↑
TOPIQ-NR ↑
TR ↓
(a) Single Frame at AOP=0
Tianmouc-UNet [ 10 ]
Official Checkpoint
19.01
20.03
0.1279
0.1759
0.1602
–
Retrained w/o Exp. Aug.
7.71
30.78
0.1737
0.2754
0.2248
–
Retrained w/ Exp. Aug.
7.89
30.79
0.1828
0.2786
0.2282
–
IGFNet [ 11 ]
Official Checkpoint
16.77
20.28
0.1295
0.1737
0.1619
–
Retrained w/o Exp. Aug.
8.00
32.55
0.1810
0.3031
0.2275
–
TABLE IV: No-reference image quality and video temporal stability evaluation on real CVS scenes. (a) Single-frame results at AOP=0. (b) Mean image quality scores over all reconstructed frames and motion-compensated temporal residuals (TR) over adjacent pairs. “–” denotes a metric inapplicable to single-frame methods. “Official” denotes the officially released checkpoint, whereas “Retrained” denotes a checkpoint retrained on our large-scale mixed dataset. The best and second-best results in each panel are highlighted in bold and underlined, respectively.
Fig. 4: Qualitative comparisons on real CVS captures. All baselines use their officially released checkpoints, whereas our method uses the checkpoint trained with exposure augmentation. Each row corresponds to one real scene. (a) RGB-frame deblurring. (b) Intermediate-frame interpolation between adjacent RGB endpoints. (c) HDR fusion at the intermediate time, where the long-exposure RGB provides color and shadow information and the short-exposure Mid-SD retains structural cues in saturated highlight regions.
Components
w/o Exposure Aug.
w/ Exposure Aug.
Guidance Extractor
Spatial Diff. Backbone
Video Temporal Layers
Detail-Preserving VAE Decoder
PSNR ↑
SSIM ↑
LPIPS ↓
NIQE ↓
FID ↓
TR ↓
PSNR ↑
SSIM ↑
LPIPS ↓
NIQE ↓
FID ↓
TR ↓
SportsSloMo-CVS
GT
–
–
–
∞
1.000
0.000
5.21
–
0.0131
∞
1.000
0.000
5.21
–
0.0131
✓
×
×
×
30.56
0.9197
0.0653
5.50
9.99
0.0186
28.50
0.9060
0.0936
5.48
18.41
0.0202
✓
✓
×
×
25.78
0.8211
0.1256
6.31
14.39
0.0330
23.53
0.7969
0.1645
6.10
17.40
0.0511
✓
✓
✓
×
26.98
0.8407
0.0753
4.78
7.56
0.0209
24.87
0.8244
0.0961
4.75
9.70
0.0211
TABLE V: Component ablation on the SportsSloMo-CVS and Ego4D-CVS datasets. All metrics are aggregated over seven reconstructed frames.
Fig. 5: Qualitative ablation in HDR scenes and under severe motion blur. RGB0 and RGB1 are the input frames. Extractor Only reconstructs images directly through the Multi-Scale Spatio-Temporal Guidance Extractor and its Stage-1 auxiliary decoder. Diffusion adds the Stable Diffusion spatial backbone and Video Temporal Layers, conditioned through the multi-scale ControlNet and decoded with the original VAE decoder. Diffusion + Decoder further adds the Detail-Preserving VAE Decoder. With the extractor alone, the building highlights look slightly red, while regions affected by severe motion blur exhibit faded and mixed colors. Diffusion produces more vivid and distinct colors; the detail-preserving decoder further refines local structures, including text strokes and edges, while retaining these color improvements.
Fig. 6: Reconstruction results from real CVS captures under different frame-rate modes. The first two rows show two high-speed trains passing each other, for which the spatio-temporal difference pathway operates at 1515 FPS and acquires 50 differential measurements between RGB0 and RGB1; reconstructions at selected time instants are shown. The last two rows show a punctured water balloon, for which the pathway operates at 3030 FPS and acquires 100 differential measurements between RGB0 and RGB1; the selected key frames capture the progressive retraction of the balloon membrane after rupture.
Motion blur arises when rapid scene changes occur during the exposure period, collapsing rich intra-exposure motion into a single RGB frame. Without explicit structural or temporal cues, RGB-only deblurring is highly ill-posed and often fails under extreme motion. Inspired by the human visual system, brain-inspired vision sensors introduce temporally dense information to alleviate this problem. However, event cameras still suffer from event rate saturation under rapid motion, while the event modality entangles edge features and motion cues, which limits their effectiveness. As a recent breakthrough, the complementary vision sensor (CVS), Tianmouc, captures synchronized RGB frames together with high-frame-rate, multi-bit spatial difference (SD, encoding structural edges) and temporal difference (TD, encoding motion cues) data within a single RGB exposure, offering a promising solution for RGB deblurring under extreme dynamic scenes. To fully leverage these complementary modalities, we propose Spatio-Temporal Difference Guided Deblur Net (STGDNet), which adopts a recurrent multi-branch architecture that iteratively encodes and fuses SD and TD sequences to restore structure and color details lost in blurry RGB inputs. Our method outperforms current RGB or event-based approaches in both synthetic CVS dataset and real-world evaluations. Moreover, STGDNet exhibits strong generalization capability across over 100 extreme real-world scenarios. Project page: https://tmcDeblur.github.io/
Yapeng Meng, Lin Yang, Yuguo Chen +6
Department of Precision Instrument, Tsinghua University, Beijing, China · Communication University of China, Beijing, China · Primevision Technology, Shanghai, China +1
Prevailing High Dynamic Range (HDR) video reconstruction methods are fundamentally trapped in a fragile alignment-and-fusion paradigm. While explicit spatial alignment can successfully recover fine details in controlled environments, it becomes a severe bottleneck in unconstrained dynamic scenes. By forcing rigid alignment across unpredictable motions and varying exposures, these methods inevitably translate registration errors into severe ghosting artifacts and temporal flickering. In this paper, we rethink this conventional prerequisite. Recognizing that explicit alignment is inherently vulnerable to real-world complexities, we propose LoCAtion, a Long-time Collaborative Attention framework that reformulates HDR video generation from a fragile spatial warping task into a robust, alignment-free collaborative feature routing problem. Guided by this new formulation, our architecture explicitly decouples the highly entangled reconstruction task. Rather than struggling to rigidly warp neighboring frames, we anchor the scene on a continuous medium-exposure backbone and utilize collaborative attention to dynamically harvest and inject reliable irradiance cues from unaligned exposures. Furthermore, we introduce a learned global sequence solver. By leveraging bidirectional context and long-range temporal modeling, it propagates corrective signals and structural features across the entire sequence, inherently enforcing whole-video coherence and eliminating jitter. Extensive experiments demonstrate that LoCAtion achieves state-of-the-art visual quality and temporal stability, offering a highly competitive balance between accuracy and computational efficiency.
Qianyu Zhang, Bolun Zheng, Lingyu Zhu +3
School of Automation, Hangzhou Dianzi University, Hangzhou 310018, China · Department of Computer Science, City University of Hong Kong
Real-world video deblurring remains challenging due to diverse motion patterns, complex degradations, and the scarcity of realistic training data, yet robust restoration is critical for downstream pipelines such as mobile imaging and 3D reconstruction. This work presents RealVDeblur, an efficient generative framework designed to improve in-the-wild robustness under diverse real capture conditions. First, a large-scale, physically grounded blur synthesis pipeline is constructed from scene-level 3D Gaussian Splatting (3DGS) assets and high-frame-rate videos, providing realistic training data covering both camera-induced and object-motion blur. Second, a video diffusion prior is leveraged for restoration; to better accommodate frame-dependent blur variations, temporal compression in the VAE is disabled and a frame-wise encoding scheme is adopted. For practical deployment on long videos, multi-step diffusion sampling is distilled into an efficient one-step generator, and a training-free Temporal Window Mask stabilizes inference beyond the training horizon with constant memory usage. Extensive experiments on diverse real-world benchmarks demonstrate strong perceptual quality, semantic fidelity, and temporal consistency on unseen videos, as well as improved robustness in downstream 3D reconstruction under severe motion blur. Project page: https://rbjin.github.io/RealVDeblur
Renbiao Jin, Mingxin Yang, Yutian Chen +8
1Shanghai Jiao Tong University · 2Shanghai AI Laboratory · 3CUHK MMLab +2