Monocular surface normal estimators trained on large-scale RGB-normal data often perform poorly in the edge cases of reflective, textureless, and dark surfaces. Polarization encodes surface orientation independently of texture and albedo, offering a physics-based complement for these cases. Existing polarization methods, however, require multi-view capture or specialized training data, limiting generalization. We introduce Poppy, a training-free framework that refines normals from any frozen RGB backbone using single-shot polarization measurements at test time. Keeping backbone weights frozen, Poppy optimizes per-pixel offsets to the input RGB and output normal along with a learned reflectance decomposition. A differentiable rendering layer converts the refined normals into polarization predictions and penalizes mismatches with the observed signal. Across seven benchmarks and three backbone architectures (diffusion, flow, and feed-forward), Poppy reduces mean angular error by 23-26% on synthetic data and 6-16% on real data. These results show that guiding learned RGB-based normal estimators with polarization cues at test time refines normals on challenging surfaces without retraining.
Figures & tables
Figure 1: Test-time polarization guidance to enhance normal estimation. (a) Polarization-based feed-forward models have limited generalizability due to the scarcity of polarization–normal training pairs. (b) RGB-only monocular normal estimators produce oversmoothed or hallucinated details on challenging surfaces – normals of the textureless bunny object appear flatter than ground truth. (c) Poppy introduces polarization guidance into pretrained RGB-only models at test time – improving normal accuracy without retraining.
Figure 2: Radiance decomposition. The mixed radiance S0 is decomposed into diffuse radiance Ld and specular radiance Ls from our method. The learned Ls captures specular highlights and environment-dependent reflections (scaled 2 × for clarity), while Ld retains the object’s intrinsic diffuse shading and texture. From these radiance components and the predicted normals, the polarization maps (AoLP × DoLP) of the diffuse, specular, and combined components can be obtained.
Figure 3: Poppy pipeline. Given polarization measurements, we compute the observed Stokes map S , extract the RGB image x , and add learnable image offset Ox to x . A frozen backbone produces base normals n^base ; a learnable normal offset On yields the refined estimate n^t=n^base+On . Using Fresnel equations, the predicted Stokes S^ is computed from n^t and specular radiance Ls . We minimize the polarization consistency loss between S^ and S to update the image offset Ox , normal offset On , and specular map Ls over T steps, while keeping backbone weights fixed.
Figure 4: (a) Stokes reconstruction of hedgehog scene (from NeRSP) when: Ls=0 (diffuse only) in column 1; only Ls is learned from backbone predicted normals n^base in column 2; Ls,Ox, and On are jointly learned in column 3. (b) Jacobian magnitude maps for a selected input pixel, normalized by the 99th percentile for different backbones, showing how perturbations of a single input pixel (green dot) influence the output normal map at a global level, across spatial locations.
Real
Synthetic
Method
Mean
Median
RMSE
Acc11.25
Acc22.5
Acc30
Mean
Median
RMSE
Acc11.25
Acc22.5
Acc30
SfPUEL [ 39 ]
20.68
17.08
26.15
0.37
0.70
0.80
17.03
12.91
23.33
0.46
0.79
0.87
DSINE [ 6 ]
23.06
19.07
28.83
0.27
0.62
0.76
24.13
19.94
30.00
0.26
0.60
0.75
Lotus [ 27 ]
16.69
13.96
21.70
0.42
0.78
0.88
19.99
17.04
25.06
0.30
0.69
0.84
StableNormal [ 55 ]
17.80
14.77
23.29
0.41
0.75
0.86
18.45
14.65
24.55
0.39
0.75
0.86
Marigold [ 34 ]
18.18
15.25
23.30
0.36
0.74
0.86
20.99
17.40
26.43
0.30
0.68
0.82
Table 1: Aggregated normal estimation performance on real and synthetic benchmarks. Our polarization guidance (+ Poppy) consistently reduces MAE and RMSE across backbone RGB estimators while improving thresholded accuracy, confirming gains in both global surface orientation and fine-scale local geometric detail. +Poppy denotes joint optimization of the image offset and normal offset. Best result per backbone within each data split is highlighted in bold.
Figure 5: Ablation of guidance variants on real and synthetic datasets. (a) Mean angular error (MAE) across three backbones (Marigold, Lotus-v2, MoGe-2) with no guidance (None), image offset guidance (Image), and joint image and normal offsets guidance (Joint). Guidance improves normals, with joint guidance performing slightly better on synthetic datasets. (b) MAE over optimization steps on SfPUEL and NeRSP with MoGe-2 backbone. The red-dashed line at t=50 marks the activation of the normal offset. On synthetic data, joint guidance accelerates error reduction. On real data, sensor noise causes the normal offset to slightly increase MAE, though high-frequency detail is still recovered.
Figure 6: Comparison of image (global) and joint (global-then-local) guidance on synthetic data. We compare output normals under three guidance conditions across three backbones: no guidance (Backbone), image offsets (Image), and joint image and normal offsets (Joint). Joint guidance recovers fine geometric structures – shell ridges, mesh facets, and dragon scales – that image-offset guidance alone cannot resolve. Pixel-wise angular error maps (brighter is lower error) and mean angular error are also shown for each prediction. Best viewed at high magnification.
Figure 7: Qualitative comparison on real-world scenes. Columns show backbone predictions and results after Poppy guidance. Pixel-wise angular error maps accompany each prediction (brighter is lower error), with the mean angular error annotated. Scenes span three challenging categories: textureless and highly reflective objects, and low-SNR conditions (low illumination, low albedo, high noise). Poppy shows consistent improvement in normal quality across all categories and backbones.
Figure 8: Sensitivity to input noise. We add zero-mean Gaussian noise of increasing standard deviation σ to the NeRSP synthetic measurements. The plot reports mean angular error (MAE) for MoGe-2 with and without polarization guidance. Guidance improves accuracy at low noise levels, but the advantage diminishes with increasing σ .
Figure 9: Robustness to material reflectance properties. Using the NeISF dataset, we render scenes with purely specular, purely diffuse, and mixed diffuse and specular reflectance to evaluate polarization guidance under varying material properties. Specular reflectance produces the strongest polarization signal and yields the largest improvement, followed by diffuse reflectance. Mixed reflectance case yields smaller gains due to the challenge of accurately decomposing diffuse and specular components. It shows pixel-wise angular error maps (brighter is lower error) with the mean angular error annotated.
Figure 10: Mesh reconstruction with refined surface normals. We reconstruct meshes using VCR-GauS [ 9 ] from multi-view images and corresponding normals estimated with and without polarization guidance. Polarization-refined normals yield better mesh quality across the three backbones. Signed distance to the ground-truth mesh is shown for each reconstruction.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 11: Test-time image optimization behavior. The input image is iteratively refined using a learned offset. The image offset is multiplied by 50 for visualization. Although the perturbations appear as nearly imperceptible noise in the RGB image, it produces large changes in the predicted surface normals. The optimization progressively improves the normal prediction, reducing the mean angular error from 16.94∘ to 11.95∘ .
Figure 12: Optimization details for different backbone architectures in Poppy. (a) Marigold performs optimization at each denoising step using the normal preview obtained via Tweedie’s formula. (b) Lotus-v2 performs optimization during the latent inference stage before the detail sharpening step. (c) MoGe-2 is a feed-forward network, allowing gradients to directly propagate to the input image during optimization.
Dataset
Method
Mean
Median
RMSE
Acc 11.25
Acc 22.5
Acc 30
PISR
MoGe-2
12.82
9.82
17.92
0.58
0.87
0.94
MoGe-2 + Poppy
13.54
11.27
18.00
0.50
0.89
0.95
MoGe-2 + Poppy (FoV)
10.87
8.05
16.02
0.69
0.92
0.96
Appendix
Table 2: Quantitative comparison on the PISR dataset. We compare the backbone model (MoGe-2), MoGe-2 with our polarization guidance assuming orthogonal projection (+ Poppy), and our guidance with perspective projection when the field-of-view (FoV) is incorporated (+ Poppy (FoV)). In the FoV variant, the camera FoV is inferred by MoGe-2 and used during optimization. Lower values are better for Mean, Median, and RMSE, while higher values are better for Acc 11.25 , Acc 22.5 , and Acc 30 .
Real
Synthetic
Method
Mean
Median
RMSE
Acc11.25
Acc22.5
Acc30
Mean
Median
RMSE
Acc11.25
Acc22.5
Acc30
MoGe-2
10.84
9.87
12.85
0.60
0.94
0.98
10.93
8.84
13.98
0.62
0.91
0.95
MoGe-2 + Poppy ( η=1.5 )
8.74
7.61
10.91
0.75
0.97
0.99
9.19
6.84
12.50
0.75
0.94
0.96
MoGe-2 + Poppy ( η=3.2 )
8.75
7.62
10.92
0.75
0.97
0.99
9.19
6.84
12.50
0.75
0.94
0.96
Appendix
Table 3: Comparison of different refractive index η settings on the SfPUEL dataset, which contains both dielectric and metallic objects. Poppy assumes a refractive index η=1.5 , while the SfPUEL paper uses η=3.2 . To analyze sensitivity to the refractive index assumption, we additionally evaluate Poppy with η=3.2 . The best results are highlighted in bold.
Backbone
Poppy (GB)
Direct Finetuning (GB)
Marigold
35.12
47.9
Lotus-v2
66.64
> 96 (OOM)
MoGe-2
15.33
21.0
Appendix
Table 4: Memory usage comparison between Poppy and direct backbone finetuning. Peak memory usage of Poppy and direct finetuning of model weights are reported for each backbone. Direct finetuning requires gradients and optimizer states for all backbone parameters, making it infeasible for large models such as Lotus-v2 on a 96 GB GPU.
Backbone
Variant
Synthetic
Real
SfPUEL
NeRSP
NeISF
DeepPol
SfPUEL
NeRSP
PISR
Marigold
None
15.19
21.47
20.03
27.29
17.45
19.03
18.06
Finetune
13.11
17.15
12.38
21.38
11.02
17.23
17.44
Poppy
12.45
16.55
12.33
21.07
12.12
17.01
16.71
Lotus-v2
None
13.14
17.85
15.30
19.80
12.38
17.76
13.90
Poppy
10.22
13.82
9.92
15.08
9.64
15.48
12.82
Appendix
Table 5: Comparison of Poppy and finetuning. We compare the backbone models (None), direct backbone finetuning with the polarization loss (Finetune), and our polarization-guided optimization (Poppy). The best result for each backbone and dataset is highlighted in bold. Finetuning results for Lotus-v2 are not reported because the model could not be trained due to GPU memory constraints.
Figure 13: MAE convergence curves over optimization steps for Poppy and direct backbone weights finetuning (MoGe-2) across SfPUEL and NeRSP synthetic datasets. Poppy converges more smoothly, while finetuning exhibits unstable oscillations despite very small learning rates.
Figure 14: Qualitative comparison of the predicted diffuse ( Ld ) and specular ( Ls ) radiance components with ground truth (GT). The absolute error maps (shown as percentages) visualize per-pixel reconstruction error within the valid polarization mask, with overall PSNR and SSIM values reported as labels on each error map. The valid mask indicates pixels where polarization-based constraints are reliable and used during optimization. The results show that the predicted decomposition closely matches the ground truth across both diffuse and specular regions, with low absolute errors over most valid pixels.
Figure 15: Applications of radiance decomposition from Poppy for appearance editing. Given the estimated diffuse ( Ld ) and specular ( Ls ) components, different visual effects can be achieved by manipulating each component independently. Recoloring modifies the diffuse radiance Ld to change the surface color while preserving specular reflections, while the metallic appearance effect is produced by editing the specular radiance Ls . These examples demonstrate that the decomposition enables flexible and physically interpretable image editing.
Surface Normal Estimation recovers the orientation of visible 3D surfaces from images and other sensor data for shape understanding and reconstruction.
Monocular depth estimation is a fundamental yet challenging task in computer vision, especially under complex conditions such as textureless surfaces, transparency, and specular reflections. Recent diffusion-based approaches have significantly advanced performance by reformulating depth prediction as a denoising process in the latent space. However, existing methods rely solely on RGB inputs, which often lack sufficient cues in challenging regions. In this work, we present CDPR - Cross-modal Diffusion with Polarization for Reliable Monocular Depth Estimation - a novel diffusion-based framework that integrates physically grounded polarization priors to enhance estimation robustness. Specifically, we encode both RGB and polarization (AoLP/DoLP) images into a shared latent space via a pre-trained Variational Autoencoder (VAE), and dynamically fuse multi-modal information through a learnable confidence-aware gating mechanism. This fusion module adaptively suppresses noisy signals in polarization inputs while preserving informative cues, particularly around reflective or transparent surfaces, and provides the integrated latent representation for subsequent monocular depth estimation. Beyond depth estimation, we further verify that our framework can be easily generalized to surface normal prediction with minimal modification, showcasing its scalability to general polarization-guided dense prediction tasks. Experiments on both synthetic and real-world datasets validate that CDPR significantly outperforms RGB-only baselines in challenging regions while maintaining competitive performance in standard scenes.
We show that, with polarization cues, a lightweight model trained on a small dataset can outperform RGB-only vision foundation models (VFMs) in single-shot object-level surface normal estimation. Shape from polarization (SfP) has long been studied due to the strong physical relationship between polarization and surface geometry. Meanwhile, driven by scaling laws, RGB-only VFMs trained on large datasets have recently achieved impressive performance and surpassed existing SfP methods. This situation raises questions about the necessity of polarization cues, which require specialized hardware and have limited training data. We argue that the weaker performance of prior SfP methods does not come from the polarization modality itself, but from domain gaps. These domain gaps mainly arise from two sources. First, existing synthetic datasets use limited and unrealistic 3D objects, with simple geometry and random texture maps that do not match the underlying shapes. Second, real-world polarization signals are often affected by sensor noise, which is not well modeled during training. To address the first issue, we render a high-quality polarization dataset using 1,954 3D-scanned real-world objects. We further incorporate pretrained DINOv3 priors to improve generalization to unseen objects. To address the second issue, we introduce polarization sensor-aware data augmentation that better reflects real-world conditions. With only 40K training scenes, our method significantly outperforms both state-of-the-art SfP approaches and RGB-only VFMs. Extensive experiments show that polarization cues enable a 33x reduction in training data or an 8x reduction in model parameters, while still achieving better performance than RGB-only counterparts.
High-quality surface normal estimation is preferred for detailed surface shape recovery and image editing. Existing single image-based methods, though being a practical setup, often struggle to recover fine surface details and are sensitive to inherent shape-reflectance ambiguity. While photometric stereo achieves high-fidelity surface normal estimation from images under varying lights, its applicability is strictly limited by requiring a multi-illumination capture setup. To this end, we propose FlashNormal, a diffusion-based surface normal estimator from flash/no-flash image pairs. While retaining high practicability on modern smartphones, our proposal takes advantage of flash-induced shading variations, and leverages curvature-guided detail enhancement strategy, improving surface detail recovery and mitigating shape-reflectance ambiguity effectively. To evaluate our proposed method, we further present EvalFlash, the first real-world flash/no-flash evaluation dataset containing 20 objects aligned with ground-truth surface normals for quantitative benchmarking. Extensive experiments demonstrate the effectiveness of FlashNormal over state-of-the-art single image-based methods and show a significant out-performance over flash/no-flash-based normal estimation method on EvalFlash.
Ruiyang Chen, Feiran Li, Heng Guo +1
Pattern Recognition and Intelligent System Laboratory, School of Artificial Intelligence, Beijing University of Posts and Telecommunications (BUPT), Beijing 100080, China · independent researcher