Polarization Uncertainty-Guided Diffusion Model for Color Polarization Image Demosaicking
Authors: Chenggong Li, Yidong Luo, Junchao Zhang, Degui Yang
Organizations: School of Automation, Central South University, Changsha, China · Hunan Provincial Key Laboratory of Optic-Electronic Intelligent Measurement and Control, Changsha, China · Zhejiang University, Hangzhou, China · School of Engineering, Westlake University, Hangzhou, China
Color polarization demosaicking (CPDM) aims to reconstruct full-resolution polarization images of four directions from the color-polarization filter array (CPFA) raw image. Due to the challenge of predicting numerous missing pixels and the scarcity of high-quality training data, existing network-based methods, despite effectively recovering scene intensity information, still exhibit significant errors in reconstructing polarization characteristics (degree of polarization, DOP, and angle of polarization, AOP). To address this problem, we introduce the image diffusion prior from text-to-image (T2I) models to overcome the performance bottleneck of network-based methods, with the additional diffusion prior compensating for limited representational capacity caused by restricted data distribution. To effectively leverage the diffusion prior, we explicitly model the polarization uncertainty during reconstruction and use uncertainty to guide the diffusion model in recovering high error regions. Extensive experiments demonstrate that the proposed method accurately recovers scene polarization characteristics with both high fidelity and strong visual perception.
Figures & tables
Figure 1: The workflow of color polarization imaging. The second row presents the AOP results of different CPDM methods, where AOP reflects the accuracy of polarization property reconstruction. Our method exhibits the least noise interference and achieves the most visually faithful results.
Figure 2: The framework of the proposed PUGDiff. (a) The architecture of the base branch, the mosaic array is initialized as a low-quality color polarization image and fed into the base branch for processing, and the network is updated via LB ; (b) The architecture of the SD branch, results of the base branch is further refined by the SD branch. LoRA layers are injected into SD to adapt it to the CPDM task, while the cross-attention modules are pruned to improve efficiency, the LoRA parameters are updated via LSD ; (c) The fusion process of the two branches, which is also used during inference. The contribution of each branch is adaptively weighted by polarization uncertainty. In low-uncertainty regions, the base branch is favored for high fidelity; in high-uncertainty regions, the SD branch is used to enhance the polarization property.
Figure 3: The architecture of the uncertainty estimation network. The network’s backbone shares the same architecture as the base branch, but augmented with an additional estimation head to output polarization uncertainty. The entire network is updated via LPU , and during the training phase in Fig. 2 (c), only the polarization uncertainty is used. The right part of the figure shows the reconstructed DOP and the polarization uncertainty. High values in the heatmap indicate regions where DOP reconstruction is inaccurate.
Method
PSNRmean
PSNRS0
PSNRDOP
MAE
SSIMmean
SSIMS0
SSIMDOP
MQ’s
Polanalyser
38.2149
35.2560
32.5532
14.8220
0.9653
0.9541
0.8387
NLCSR
36.8877
33.5610
34.0964
11.1297
0.9523
0.9310
0.8557
CPDNet
39.4374
36.6062
32.6586
12.5138
0.9700
0.9631
0.8465
TCPDNet
42.5602
39.7694
36.0613
10.1765
0.9815
0.9765
0.8921
DCPM
42.0221
40.0263
36.1496
11.7376
0.9807
0.9759
0.8958
PIDSR
42.4990
40.6269
36.5413
9.6463
0.9822
0.9794
0.8950
Table 1: Quantitative comparison with state-of-the-art methods. The higher value means better effects. The bold black indicates the best result. The proposed method shows the overall optimal performance.
Figure 4: Visual comparisons for CPDM of different methods. Both on synthetic and real-captured images, our method achieves the most outstanding polarization reconstruction performance in terms of AOP and DOP.
Figure 5: Visualization of results from different branches. Our method fuses the two branches to achieve complementary strengths: using the SD branch in high-uncertainty regions to improve reconstruction, and the base branch in low-uncertainty regions to preserve fidelity.
Method
PSNRS0
PSNRDOP
PSNRAOP
Intensity
42.1747
40.2787
15.8751
Intensity ( S0 )
42.1165
40.3096
16.4514
Polarization
41.9823
40.6696
16.8207
Table 2: Ablation study on the forms of uncertainty. “Intensity”, “Intensity ( S0 )” and “Polarization” denote the uncertainty computed from the four direction polarization images, S0 , and DOP, respectively. The bold black indicates the best result.
Method
PSNRS0
PSNRDOP
PSNRAOP
All weight
–
–
–
Only U-Net
40.8976
40.2579
16.0951
rank 2
–
–
–
rank 4 (Ours)
41.9823
40.6696
16.8207
rank 8
42.0145
40.6652
16.5832
Table 3: Ablation study on the SD configuration. “All weight” denotes full-parameter fine-tuning, “Only U-Net” indicates that only the diffusion U-Net is updated, “rank” is a hyperparameter of LoRA and “–” indicates training failure. The bold black indicates the best result.
Figure 6: Visualization of different forms of uncertainty. Polarization uncertainty provides more direct information about polarization reconstruction, thus leading to the most significant improvement in DOP.
Figure 7: Visual comparisons for PRR of different methods. Our method achieves the clearest reflection removal results.
Division-of-focal-plane (DoFP) color polarization cameras enable snapshot acquisition of color polarization mosaic images, but the inherently sparse sampling pattern makes color polarization demosaicking severely ill-posed. Existing methods often fail to jointly exploit the correlations among polarization channels and the physical constraints inherent in polarization imaging, resulting in noticeable demosaicking artifacts. To address this issue, a quaternion-tensor-based color polarization demosaicking (CPDM) method incorporating Stokes-domain total variation (TV) regularization is proposed. Correlation analysis shows that the correlations among polarization channels are stronger than those among color channels. Accordingly, the color polarization images acquired at 0∘, 45∘, 90∘, and 135∘ are encoded into the four components of a third-order quaternion tensor, with the color channels organized along its third mode. A low-rank prior is then imposed on the quaternion tensor to exploit the global structural redundancy in the color polarization data. Moreover, spatial gradients are mapped to the Stokes domain through an orthogonal transformation to separate intensity, polarization and residual variations, with adaptive quaternion weights enabling component-specific regularization and preserving the energy consistency of the reconstructed Stokes vectors. An efficient optimization algorithm is derived for the resulting model. Extensive experiments demonstrate the superior demosaicking performance of the proposed method.
Yanqing Song, Jifei Miao, Chaoqian Li +3
School of Mathematics and Statistics, Yunnan University, Kunming, Yunnan, 650091, China · Department of Mathematics, Faculty of Science and Technology, University of Macau, Macau 999078, China · School of Computing and Artificial Intelligence Southwestern University of Finance and Economics Chengdu, Sichuan, China
Color-polarization imaging using a color-polarization filter array (CPFA) sensor captures both texture (color intensity) and physical (polarization) information of the scene in a single shot, enabling various applications in computer vision. However, the raw mosaic output from a CPFA sensor often suffers from severe noise and resolution loss, especially under low-light conditions. Existing methods generally focus on either denoising or demosaicking tasks, failing to capture the coupling between them and neglecting shared low-level features. In this paper, we propose a color-polarization denoising and demosaicking network (CPDDNet), which is a joint framework that performs noise removal and CPFA interpolation using a feature fusion module that retains the features from the CPFA raw data at both the denoising and the demosaicking stages. Experimental results demonstrate that CPDDNet significantly enhances image quality and polarization parameter accuracy, outperforming existing approaches on a real dataset.
Polarization imaging captures distinctive surface and geometric cues that benefit a wide range of vision tasks. However, real-world polarization acquisition is often affected by multiple coupled degradations, making image restoration essential for practical polarization vision. Existing methods are largely tailored to specific degradations and remain constrained by the limited scale and quality of polarization data. To address these limitations, we develop an all-in-one polarization restoration framework for diverse and composite degradations. We first study the impact of different polarization representations on restoration performance and identify the normalized Stokes representation as an effective choice for separating intensity and polarization information. Accordingly, we devise a dual-branch architecture that separates intensity and polarization modeling. To overcome the limitations of polarization-specific training, the intensity branch leverages pretrained general restoration priors and a mixture-of-experts extension for composite degradations, while its restoration knowledge is adaptively distilled into the symmetric polarization branch via a cross-domain feature transform. In addition, we establish a composite-degradation polarization benchmark to support all-in-one restoration research. Extensive experiments on public datasets and our proposed benchmark demonstrate the effectiveness of the proposed method.
Chenggong Li, Jinhao Liu, Caiyun Wu +3
School of Automation Central South University · School of Engineering Westlake University