CoGenRec: A Unified Generative Framework for Complementary Vision Reconstruction

Yapeng Meng; Yuguo Chen; Xiangru Chen; Shaojun Xu; Xuezhen Wang; Lijian Wang; Zheyu Yang; Yihan Lin

cs.CVNov 5, 2026

CoGenRec: A Unified Generative Framework for Complementary Vision Reconstruction

Authors: Yapeng Meng, Yuguo Chen, Xiangru Chen, Shaojun Xu, Xuezhen Wang, Lijian Wang, Zheyu Yang, Yihan Lin,

Organizations: Department of Precision Instrument, Tsinghua University, Beijing, China. · The Hong Kong University of Science and Technology, HKSAR, China. · Primevision Technology, Shanghai, China. · Pen-Tung Sah Institute of Micro-Nano Science and Technology, Xiamen University, Xiamen, China.

Abstract

Capturing scenes with extreme motion and high dynamic range is fundamentally constrained by trade-offs among temporal resolution, spatial detail, signal-to-noise ratio, and dynamic range under practical bandwidth, power, and quantum-efficiency limits. Conventional frame-based cameras therefore suffer from temporal undersampling, exposure-induced motion blur, and sensor saturation. The Complementary Vision Sensor (CVS) addresses these limitations by decomposing visual information across two complementary pathways, a spatially dense, temporally sparse, long-exposure RGB pathway and a temporally dense, spatially sparse, short-exposure patiotemporal difference pathway. Their complementary measurements provide appearance, structural, and motion cues across a wide dynamic range that enable reconstruction beyond conventional imaging. We present CoGenRec, a complementarity-driven generative reconstruction framework that unifies video deblurring, high-temporal-resolution frame interpolation, and HDR fusion. CoGenRec encodes complementary measurements into multi-scale spatiotemporal features using exposure-aware appearance modeling and bidirectional temporal propagation. These features constrain a video diffusion model and are injected into the VAE decoder to recover fine structures lost during latent-space compression, combining the scene fidelity of physical measurements with the appearance recovery capabilities of learned video priors. On SportsSloMo-CVS, CoGenRec improves PSNR over the strongest general-purpose baseline by 4.40 dB for deblurring, 3.89 dB for frame interpolation, and 5.04 dB for HDR fusion, respectively. It also achieves strong perceptual quality on 129 unseen scenes captured with real CVS. These results establish a unified approach to advance high-quality reconstruction under challenging motion and illumination conditions. Project page: https://cogenrec.github.io/.

Figures & tables

Fig. 1: Motivation for capturing extreme dynamic scenes with the Complementary Vision Sensor (CVS) and reconstructing them using our complementarity-driven generative reconstruction framework, CoGenRec. (a) Limitations of conventional dense-sampling image sensors and how CVS alleviates them. Under limited bandwidth and power, frame rate and dynamic range are mutually constrained, while signal-to-noise ratio (SNR) and motion blur involve an inherent trade-off. CVS alleviates these constraints through two synergistic visual pathways. (b) Regression-based reconstruction versus CoGenRec. In the ill-posed solution space induced by extreme scenes, regression models tend to average plausible solutions, producing over-smoothed results. CoGenRec instead exploits the spatial, temporal, and exposure complementarity of CVS to guide generative sampling and jointly reconstruct blur-free, high-frame-rate HDR-fused video.

Fig. 2: Overall architecture of CoGenRec. The Multi-Scale Spatio-Temporal Guidance Extractor first encodes the dual-pathway CVS measurements into dense multi-scale guidance volumes. These volumes condition the Complementarity-Guided Video Diffusion Backbone through a multi-scale ControlNet. Finally, the Detail-Preserving VAE Decoder fuses the uncompressed multi-scale guidance features into the VAE decoding hierarchy and produces sharp, high-frame-rate, and HDR-fused RGB frames.

Fig. 3: Qualitative comparisons on SportsSloMo-CVS under the unified training protocol. All baselines are retrained on the same large-scale mixed training dataset as our method. Each row shows one test scene. (a) Deblurring results for a blurry RGB endpoint frame. (b) Interpolation results at the temporal midpoint between two adjacent blurry RGB endpoint frames, RGB0 and RGB1. (c) HDR synthesis results at the temporal position corresponding to Mid-SD between two adjacent RGB frames; the highlights in RGB0 are saturated, whereas Mid-SD visualizes the complementary structural cues retained by the short-exposure differential pathway at that time. STGDNet is specific to motion deblurring and is therefore included only in (a).

Method	5			7			9			11			Average
Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Tianmouc-UNet [ 10 ]	27.50	0.8848	0.1648	27.70	0.8906	0.1652	27.46	0.8780	0.1844	27.11	0.8692	0.1952	27.44	0.8807	0.1774
IGFNet [ 11 ]	28.09	0.9008	0.1398	28.16	0.9038	0.1415	27.89	0.8910	0.1583	27.43	0.8839	0.1686	27.89	0.8949	0.1521
CBRDM [ 13 ]	22.57	0.7730	0.1487	22.93	0.8043	0.1503	22.43	0.7918	0.1708	22.13	0.7711	0.1836	22.52	0.7851	0.1633
STGDNet [ 14 ]	35.10	0.9598	0.0387	34.68	0.9569	0.0446	34.12	0.9517	0.0523	33.39	0.9463	0.0602	34.33	0.9537	0.0490
CoGenRec (Ours)	32.67	0.9452	0.0383	32.55	0.9441	0.0403	32.14	0.9380	0.0454	31.82	0.9347	0.0482	32.30	0.9405	0.0431

TABLE I: Quantitative motion-deblurring comparison on the SportsSloMo-CVS test set. The blurry RGB inputs integrate 5, 7, 9, and 11 consecutive sharp frames, respectively; Average denotes the arithmetic mean over the four blur levels. The best and second-best results in each column are highlighted in bold and underlined, respectively.

Method	Middle Frame					All Frames					TR $\downarrow$
Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	NIQE $\downarrow$	FID $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	NIQE $\downarrow$	FID $\downarrow$	TR $\downarrow$
Tianmouc-UNet [ 10 ]	27.17	0.8722	0.1835	6.56	47.86	27.32	0.8763	0.1793	6.52	33.51	0.019735
IGFNet [ 11 ]	27.67	0.8888	0.1557	6.53	38.72	27.77	0.8912	0.1541	6.53	27.05	0.018129
CBRDM [ 13 ]	21.21	0.7429	0.2082	5.15	62.02	21.77	0.7624	0.1851	5.16	35.43	0.029459
CoGenRec (Ours)	31.18	0.9309	0.0520	5.77	13.59	31.66	0.9355	0.0475	5.75	6.50	0.016047

TABLE II: Quantitative frame-interpolation comparison on the SportsSloMo-CVS test set. “Middle Frame” denotes the temporal midpoint of the reconstructed sequence, whereas “All Frames” denotes all seven reconstructed frames in each sample. PSNR, SSIM, LPIPS, and NIQE are averaged over the corresponding frames and samples, while FID [ 40 ] is computed over the corresponding pooled image set. TR is the motion-compensated temporal residual between adjacent predictions. The best and second-best results in each column are highlighted in bold and underlined, respectively.

Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	NIQE $\downarrow$	FID $\downarrow$	TR $\downarrow$
Tianmouc-UNet [ 10 ]	24.69	0.8416	0.2340	6.66	57.21	0.022850
IGFNet [ 11 ]	24.57	0.8511	0.2178	6.72	52.21	0.023154
CBRDM [ 13 ]	17.81	0.7069	0.2445	5.16	48.27	0.028478
CoGenRec (Ours)	29.72	0.9263	0.0647	5.69	8.70	0.016891

TABLE III: Quantitative comparison of HDR fusion on the SportsSloMo-CVS test set. PSNR, SSIM, LPIPS, and NIQE are first averaged over the seven reconstructed frames within each sample and then across all samples, while FID is computed over the pooled set of all reconstructed frames. The best and second-best results in each column are highlighted in bold and underlined, respectively.

Method	Training	NIQE $\downarrow$	MUSIQ $\uparrow$	MANIQA $\uparrow$	CLIP-IQA+ $\uparrow$	TOPIQ-NR $\uparrow$	TR $\downarrow$
(a) Single Frame at AOP=0
Tianmouc-UNet [ 10 ]	Official Checkpoint	19.01	20.03	0.1279	0.1759	0.1602	–
	Retrained w/o Exp. Aug.	7.71	30.78	0.1737	0.2754	0.2248	–
	Retrained w/ Exp. Aug.	7.89	30.79	0.1828	0.2786	0.2282	–
IGFNet [ 11 ]	Official Checkpoint	16.77	20.28	0.1295	0.1737	0.1619	–
	Retrained w/o Exp. Aug.	8.00	32.55	0.1810	0.3031	0.2275	–

TABLE IV: No-reference image quality and video temporal stability evaluation on real CVS scenes. (a) Single-frame results at AOP=0. (b) Mean image quality scores over all reconstructed frames and motion-compensated temporal residuals (TR) over adjacent pairs. “–” denotes a metric inapplicable to single-frame methods. “Official” denotes the officially released checkpoint, whereas “Retrained” denotes a checkpoint retrained on our large-scale mixed dataset. The best and second-best results in each panel are highlighted in bold and underlined, respectively.

Fig. 4: Qualitative comparisons on real CVS captures. All baselines use their officially released checkpoints, whereas our method uses the checkpoint trained with exposure augmentation. Each row corresponds to one real scene. (a) RGB-frame deblurring. (b) Intermediate-frame interpolation between adjacent RGB endpoints. (c) HDR fusion at the intermediate time, where the long-exposure RGB provides color and shadow information and the short-exposure Mid-SD retains structural cues in saturated highlight regions.

Components				w/o Exposure Aug.						w/ Exposure Aug.
Guidance Extractor	Spatial Diff. Backbone	Video Temporal Layers	Detail-Preserving VAE Decoder	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	NIQE $\downarrow$	FID $\downarrow$	TR $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	NIQE $\downarrow$	FID $\downarrow$	TR $\downarrow$
SportsSloMo-CVS
GT	–	–	–	$\infty$	1.000	0.000	5.21	–	0.0131	$\infty$	1.000	0.000	5.21	–	0.0131
$\checkmark$	$\times$	$\times$	$\times$	30.56	0.9197	0.0653	5.50	9.99	0.0186	28.50	0.9060	0.0936	5.48	18.41	0.0202
$\checkmark$	$\checkmark$	$\times$	$\times$	25.78	0.8211	0.1256	6.31	14.39	0.0330	23.53	0.7969	0.1645	6.10	17.40	0.0511
$\checkmark$	$\checkmark$	$\checkmark$	$\times$	26.98	0.8407	0.0753	4.78	7.56	0.0209	24.87	0.8244	0.0961	4.75	9.70	0.0211

TABLE V: Component ablation on the SportsSloMo-CVS and Ego4D-CVS datasets. All metrics are aggregated over seven reconstructed frames.

Fig. 5: Qualitative ablation in HDR scenes and under severe motion blur. RGB0 and RGB1 are the input frames. Extractor Only reconstructs images directly through the Multi-Scale Spatio-Temporal Guidance Extractor and its Stage-1 auxiliary decoder. Diffusion adds the Stable Diffusion spatial backbone and Video Temporal Layers, conditioned through the multi-scale ControlNet and decoded with the original VAE decoder. Diffusion + Decoder further adds the Detail-Preserving VAE Decoder. With the extractor alone, the building highlights look slightly red, while regions affected by severe motion blur exhibit faded and mixed colors. Diffusion produces more vivid and distinct colors; the detail-preserving decoder further refines local structures, including text strokes and edges, while retaining these color improvements.

Fig. 6: Reconstruction results from real CVS captures under different frame-rate modes. The first two rows show two high-speed trains passing each other, for which the spatio-temporal difference pathway operates at 1515 FPS and acquires 50 differential measurements between RGB0 and RGB1; reconstructions at selected time instants are shown. The last two rows show a punctured water balloon, for which the pathway operates at 3030 FPS and acquires 100 differential measurements between RGB0 and RGB1; the selected key frames capture the progressive retraction of the balloon membrane after rupture.

HDR Video Reconstruction

See all 11 papers

Spatio-Temporal Difference Guided Motion Deblurring with the Complementary Vision Sensor
Apr 12, 2026Yapeng Meng, Lin Yang, Yuguo Chen +6Image Deblurring Multimodal Sensor Fusion
Department of Precision Instrument, Tsinghua University, Beijing, China · Communication University of China, Beijing, China · Primevision Technology, Shanghai, China +1
LoCAtion: Long-time Collaborative Attention Framework for High Dynamic Range Video Reconstruction
Mar 15, 2026Qianyu Zhang, Bolun Zheng, Lingyu Zhu +3HDR Video Reconstruction Temporal Consistency in Video Restoration
School of Automation, Hangzhou Dianzi University, Hangzhou 310018, China · Department of Computer Science, City University of Hong Kong
RealVDeblur: One-Step Diffusion for Generalizable Real-World Video Deblurring
Jul 22, 2026Renbiao Jin, Mingxin Yang, Yutian Chen +8Video Diffusion Models Temporal Consistency in Video Restoration
1Shanghai Jiao Tong University · 2Shanghai AI Laboratory · 3CUHK MMLab +2

Apr 12, 2026J/K move · Enter open · S save

Spatio-Temporal Difference Guided Motion Deblurring with the Complementary Vision Sensor

Yapeng Meng, Lin Yang, Yuguo Chen, Xiangru Chen+5

Department of Precision Instrument, Tsinghua University, Beijing, China · Communication University of China, Beijing, China · Primevision Technology, Shanghai, China+1

Image Deblurring Multimodal Sensor Fusion

Open paper AskPDF arXiv

Motion blur arises when rapid scene changes occur during the exposure period, collapsing rich intra-exposure motion into a single RGB frame. Without explicit structural or temporal cues, RGB-only deblurring is highly ill-posed and often fails under extreme motion. Inspired by the human visual system, brain-inspired vision sensors introduce temporally dense information to alleviate this problem. However, event cameras still suffer from event rate saturation under rapid motion, while the event modality entangles edge features and motion cues, which limits their effectiveness. As a recent breakthrough, the complementary vision sensor (CVS), Tianmouc, captures synchronized RGB frames together with high-frame-rate, multi-bit spatial difference (SD, encoding structural edges) and temporal difference (TD, encoding motion cues) data within a single RGB exposure, offering a promising solution for RGB deblurring under extreme dynamic scenes. To fully leverage these complementary modalities, we propose Spatio-Temporal Difference Guided Deblur Net (STGDNet), which adopts a recurrent multi-branch architecture that iteratively encodes and fuses SD and TD sequences to restore structure and color details lost in blurry RGB inputs. Our method outperforms current RGB or event-based approaches in both synthetic CVS dataset and real-world evaluations. Moreover, STGDNet exhibits strong generalization capability across over 100 extreme real-world scenarios. Project page: https://tmcDeblur.github.io/

Abstract

HDR Video Reconstruction

Explore similar work

Spatio-Temporal Difference Guided Motion Deblurring with the Complementary Vision Sensor

LoCAtion: Long-time Collaborative Attention Framework for High Dynamic Range Video Reconstruction

RealVDeblur: One-Step Diffusion for Generalizable Real-World Video Deblurring