Poppy: Polarization-based Plug-and-Play Guidance for Enhancing Monocular Normal Estimation

Irene Kim; Sai Tanmay Reddy Chakkera; Alexandros Graikos; Dimitris Samaras; Akshat Dave

cs.CVMar 29, 2026

Poppy: Polarization-based Plug-and-Play Guidance for Enhancing Monocular Normal Estimation

Authors: Irene Kim, Sai Tanmay Reddy Chakkera, Alexandros Graikos, Dimitris Samaras, Akshat Dave

Organizations: Stony Brook University

Abstract

Monocular surface normal estimators trained on large-scale RGB-normal data often perform poorly in the edge cases of reflective, textureless, and dark surfaces. Polarization encodes surface orientation independently of texture and albedo, offering a physics-based complement for these cases. Existing polarization methods, however, require multi-view capture or specialized training data, limiting generalization. We introduce Poppy, a training-free framework that refines normals from any frozen RGB backbone using single-shot polarization measurements at test time. Keeping backbone weights frozen, Poppy optimizes per-pixel offsets to the input RGB and output normal along with a learned reflectance decomposition. A differentiable rendering layer converts the refined normals into polarization predictions and penalizes mismatches with the observed signal. Across seven benchmarks and three backbone architectures (diffusion, flow, and feed-forward), Poppy reduces mean angular error by 23-26% on synthetic data and 6-16% on real data. These results show that guiding learned RGB-based normal estimators with polarization cues at test time refines normals on challenging surfaces without retraining.

Figures & tables

Figure 1: Test-time polarization guidance to enhance normal estimation. (a) Polarization-based feed-forward models have limited generalizability due to the scarcity of polarization–normal training pairs. (b) RGB-only monocular normal estimators produce oversmoothed or hallucinated details on challenging surfaces – normals of the textureless bunny object appear flatter than ground truth. (c) Poppy introduces polarization guidance into pretrained RGB-only models at test time – improving normal accuracy without retraining.

Figure 2: Radiance decomposition. The mixed radiance $S_{0}$ is decomposed into diffuse radiance $L_{d}$ and specular radiance $L_{s}$ from our method. The learned $L_{s}$ captures specular highlights and environment-dependent reflections (scaled 2 $\times$ for clarity), while $L_{d}$ retains the object’s intrinsic diffuse shading and texture. From these radiance components and the predicted normals, the polarization maps (AoLP $\times$ DoLP) of the diffuse, specular, and combined components can be obtained.

Figure 3: Poppy pipeline. Given polarization measurements, we compute the observed Stokes map $\mathbf{S}$ , extract the RGB image $x$ , and add learnable image offset $O_{x}$ to $x$ . A frozen backbone produces base normals $\hat{n}_{\text{base}}$ ; a learnable normal offset $O_{n}$ yields the refined estimate $\hat{n}_{t}=\hat{n}_{\text{base}}+O_{n}$ . Using Fresnel equations, the predicted Stokes $\hat{\mathbf{S}}$ is computed from $\hat{n}_{t}$ and specular radiance $L_{s}$ . We minimize the polarization consistency loss between $\hat{\mathbf{S}}$ and $\mathbf{S}$ to update the image offset $O_{x}$ , normal offset $O_{n}$ , and specular map $L_{s}$ over $T$ steps, while keeping backbone weights fixed.

Figure 4: (a) Stokes reconstruction of hedgehog scene (from NeRSP) when: $L_{s}=0$ (diffuse only) in column 1; only $L_{s}$ is learned from backbone predicted normals $\hat{n}_{\text{base}}$ in column 2; $L_{s},O_{x},$ and $O_{n}$ are jointly learned in column 3. (b) Jacobian magnitude maps for a selected input pixel, normalized by the 99th percentile for different backbones, showing how perturbations of a single input pixel (green dot) influence the output normal map at a global level, across spatial locations.

	Real						Synthetic
Method	Mean	Median	RMSE	Acc11.25	Acc22.5	Acc30	Mean	Median	RMSE	Acc11.25	Acc22.5	Acc30
SfPUEL [ 39 ]	20.68	17.08	26.15	0.37	0.70	0.80	17.03	12.91	23.33	0.46	0.79	0.87
DSINE [ 6 ]	23.06	19.07	28.83	0.27	0.62	0.76	24.13	19.94	30.00	0.26	0.60	0.75
Lotus [ 27 ]	16.69	13.96	21.70	0.42	0.78	0.88	19.99	17.04	25.06	0.30	0.69	0.84
StableNormal [ 55 ]	17.80	14.77	23.29	0.41	0.75	0.86	18.45	14.65	24.55	0.39	0.75	0.86
Marigold [ 34 ]	18.18	15.25	23.30	0.36	0.74	0.86	20.99	17.40	26.43	0.30	0.68	0.82

Table 1: Aggregated normal estimation performance on real and synthetic benchmarks. Our polarization guidance (+ Poppy) consistently reduces MAE and RMSE across backbone RGB estimators while improving thresholded accuracy, confirming gains in both global surface orientation and fine-scale local geometric detail. +Poppy denotes joint optimization of the image offset and normal offset. Best result per backbone within each data split is highlighted in bold.

Figure 5: Ablation of guidance variants on real and synthetic datasets. (a) Mean angular error (MAE) across three backbones (Marigold, Lotus-v2, MoGe-2) with no guidance (None), image offset guidance (Image), and joint image and normal offsets guidance (Joint). Guidance improves normals, with joint guidance performing slightly better on synthetic datasets. (b) MAE over optimization steps on SfPUEL and NeRSP with MoGe-2 backbone. The red-dashed line at $t{=}50$ marks the activation of the normal offset. On synthetic data, joint guidance accelerates error reduction. On real data, sensor noise causes the normal offset to slightly increase MAE, though high-frequency detail is still recovered.

Figure 6: Comparison of image (global) and joint (global-then-local) guidance on synthetic data. We compare output normals under three guidance conditions across three backbones: no guidance (Backbone), image offsets (Image), and joint image and normal offsets (Joint). Joint guidance recovers fine geometric structures – shell ridges, mesh facets, and dragon scales – that image-offset guidance alone cannot resolve. Pixel-wise angular error maps (brighter is lower error) and mean angular error are also shown for each prediction. Best viewed at high magnification.

Figure 7: Qualitative comparison on real-world scenes. Columns show backbone predictions and results after Poppy guidance. Pixel-wise angular error maps accompany each prediction (brighter is lower error), with the mean angular error annotated. Scenes span three challenging categories: textureless and highly reflective objects, and low-SNR conditions (low illumination, low albedo, high noise). Poppy shows consistent improvement in normal quality across all categories and backbones.

Figure 8: Sensitivity to input noise. We add zero-mean Gaussian noise of increasing standard deviation $\sigma$ to the NeRSP synthetic measurements. The plot reports mean angular error (MAE) for MoGe-2 with and without polarization guidance. Guidance improves accuracy at low noise levels, but the advantage diminishes with increasing $\sigma$ .

Figure 9: Robustness to material reflectance properties. Using the NeISF dataset, we render scenes with purely specular, purely diffuse, and mixed diffuse and specular reflectance to evaluate polarization guidance under varying material properties. Specular reflectance produces the strongest polarization signal and yields the largest improvement, followed by diffuse reflectance. Mixed reflectance case yields smaller gains due to the challenge of accurately decomposing diffuse and specular components. It shows pixel-wise angular error maps (brighter is lower error) with the mean angular error annotated.

Figure 10: Mesh reconstruction with refined surface normals. We reconstruct meshes using VCR-GauS [ 9 ] from multi-view images and corresponding normals estimated with and without polarization guidance. Polarization-refined normals yield better mesh quality across the three backbones. Signed distance to the ground-truth mesh is shown for each reconstruction.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 11: Test-time image optimization behavior. The input image is iteratively refined using a learned offset. The image offset is multiplied by 50 for visualization. Although the perturbations appear as nearly imperceptible noise in the RGB image, it produces large changes in the predicted surface normals. The optimization progressively improves the normal prediction, reducing the mean angular error from $16.94^{\circ}$ to $11.95^{\circ}$ .

Figure 12: Optimization details for different backbone architectures in Poppy. (a) Marigold performs optimization at each denoising step using the normal preview obtained via Tweedie’s formula. (b) Lotus-v2 performs optimization during the latent inference stage before the detail sharpening step. (c) MoGe-2 is a feed-forward network, allowing gradients to directly propagate to the input image during optimization.

Dataset	Method	Mean	Median	RMSE	Acc 11.25	Acc 22.5	Acc 30
PISR	MoGe-2	12.82	9.82	17.92	0.58	0.87	0.94
	MoGe-2 + Poppy	13.54	11.27	18.00	0.50	0.89	0.95
	MoGe-2 + Poppy (FoV)	10.87	8.05	16.02	0.69	0.92	0.96

Appendix

Table 2: Quantitative comparison on the PISR dataset. We compare the backbone model (MoGe-2), MoGe-2 with our polarization guidance assuming orthogonal projection (+ Poppy), and our guidance with perspective projection when the field-of-view (FoV) is incorporated (+ Poppy (FoV)). In the FoV variant, the camera FoV is inferred by MoGe-2 and used during optimization. Lower values are better for Mean, Median, and RMSE, while higher values are better for Acc 11.25 , Acc 22.5 , and Acc 30 .

	Real						Synthetic
Method	Mean	Median	RMSE	Acc11.25	Acc22.5	Acc30	Mean	Median	RMSE	Acc11.25	Acc22.5	Acc30
MoGe-2	10.84	9.87	12.85	0.60	0.94	0.98	10.93	8.84	13.98	0.62	0.91	0.95
MoGe-2 + Poppy ( $\eta=1.5$ )	8.74	7.61	10.91	0.75	0.97	0.99	9.19	6.84	12.50	0.75	0.94	0.96
MoGe-2 + Poppy ( $\eta=3.2$ )	8.75	7.62	10.92	0.75	0.97	0.99	9.19	6.84	12.50	0.75	0.94	0.96

Appendix

Table 3: Comparison of different refractive index

\eta

settings on the SfPUEL dataset, which contains both dielectric and metallic objects. Poppy assumes a refractive index

\eta=1.5

, while the SfPUEL paper uses

\eta=3.2

. To analyze sensitivity to the refractive index assumption, we additionally evaluate Poppy with

\eta=3.2

. The best results are highlighted in bold.

Backbone	Poppy (GB)	Direct Finetuning (GB)
Marigold	35.12	47.9
Lotus-v2	66.64	$>$ 96 (OOM)
MoGe-2	15.33	21.0

Appendix

Table 4: Memory usage comparison between Poppy and direct backbone finetuning. Peak memory usage of Poppy and direct finetuning of model weights are reported for each backbone. Direct finetuning requires gradients and optimizer states for all backbone parameters, making it infeasible for large models such as Lotus-v2 on a 96 GB GPU.

Backbone	Variant	Synthetic				Real
Backbone	Variant	SfPUEL	NeRSP	NeISF	DeepPol	SfPUEL	NeRSP	PISR
Marigold	None	15.19	21.47	20.03	27.29	17.45	19.03	18.06
	Finetune	13.11	17.15	12.38	21.38	11.02	17.23	17.44
	Poppy	12.45	16.55	12.33	21.07	12.12	17.01	16.71
Lotus-v2	None	13.14	17.85	15.30	19.80	12.38	17.76	13.90
Lotus-v2	Poppy	10.22	13.82	9.92	15.08	9.64	15.48	12.82

Appendix

Table 5: Comparison of Poppy and finetuning. We compare the backbone models (None), direct backbone finetuning with the polarization loss (Finetune), and our polarization-guided optimization (Poppy). The best result for each backbone and dataset is highlighted in bold. Finetuning results for Lotus-v2 are not reported because the model could not be trained due to GPU memory constraints.

Figure 13: MAE convergence curves over optimization steps for Poppy and direct backbone weights finetuning (MoGe-2) across SfPUEL and NeRSP synthetic datasets. Poppy converges more smoothly, while finetuning exhibits unstable oscillations despite very small learning rates.

Figure 14: Qualitative comparison of the predicted diffuse ( $L_{d}$ ) and specular ( $L_{s}$ ) radiance components with ground truth (GT). The absolute error maps (shown as percentages) visualize per-pixel reconstruction error within the valid polarization mask, with overall PSNR and SSIM values reported as labels on each error map. The valid mask indicates pixels where polarization-based constraints are reliable and used during optimization. The results show that the predicted decomposition closely matches the ground truth across both diffuse and specular regions, with low absolute errors over most valid pixels.

Figure 15: Applications of radiance decomposition from Poppy for appearance editing. Given the estimated diffuse ( $L_{d}$ ) and specular ( $L_{s}$ ) components, different visual effects can be achieved by manipulating each component independently. Recoloring modifies the diffuse radiance $L_{d}$ to change the surface color while preserving specular reflections, while the metallic appearance effect is produced by editing the specular radiance $L_{s}$ . These examples demonstrate that the decomposition enables flexible and physically interpretable image editing.

Surface Normal Estimation

Surface Normal Estimation recovers the orientation of visible 3D surfaces from images and other sensor data for shape understanding and reconstruction.

See all 25 papers

CDPR: Cross-modal Diffusion with Polarization for Reliable Monocular Depth Estimation
Apr 13, 2026Rongjia Yu, Tong Jia, Hao Wang +4Depth Estimation Monocular Depth Estimation
Revisiting Shape from Polarization in the Era of Vision Foundation Models
Mar 5, 2026Chenhao Li, Taishi Ono, Takeshi Uemori +1Surface Normal Estimation Polarimetric Imaging
Sony Semiconductor Solutions Corporation
FlashNormal: Detailed Surface Normal Estimation from Flash and No-Flash Images
Aug 26, 2026Ruiyang Chen, Feiran Li, Heng Guo +1Surface Normal Estimation
Pattern Recognition and Intelligent System Laboratory, School of Artificial Intelligence, Beijing University of Posts and Telecommunications (BUPT), Beijing 100080, China · independent researcher

Apr 13, 2026J/K move · Enter open · S save

CDPR: Cross-modal Diffusion with Polarization for Reliable Monocular Depth Estimation

Rongjia Yu, Tong Jia, Hao Wang, Xiaofang Li+3

Depth Estimation Monocular Depth Estimation Multimodal Diffusion Models Multimodal Sensor Fusion

Open paper AskPDF arXiv

Monocular depth estimation is a fundamental yet challenging task in computer vision, especially under complex conditions such as textureless surfaces, transparency, and specular reflections. Recent diffusion-based approaches have significantly advanced performance by reformulating depth prediction as a denoising process in the latent space. However, existing methods rely solely on RGB inputs, which often lack sufficient cues in challenging regions. In this work, we present CDPR - Cross-modal Diffusion with Polarization for Reliable Monocular Depth Estimation - a novel diffusion-based framework that integrates physically grounded polarization priors to enhance estimation robustness. Specifically, we encode both RGB and polarization (AoLP/DoLP) images into a shared latent space via a pre-trained Variational Autoencoder (VAE), and dynamically fuse multi-modal information through a learnable confidence-aware gating mechanism. This fusion module adaptively suppresses noisy signals in polarization inputs while preserving informative cues, particularly around reflective or transparent surfaces, and provides the integrated latent representation for subsequent monocular depth estimation. Beyond depth estimation, we further verify that our framework can be easily generalized to surface normal prediction with minimal modification, showcasing its scalability to general polarization-guided dense prediction tasks. Experiments on both synthetic and real-world datasets validate that CDPR significantly outperforms RGB-only baselines in challenging regions while maintaining competitive performance in standard scenes.

Abstract

Surface Normal Estimation

Explore similar work

CDPR: Cross-modal Diffusion with Polarization for Reliable Monocular Depth Estimation

Revisiting Shape from Polarization in the Era of Vision Foundation Models

FlashNormal: Detailed Surface Normal Estimation from Flash and No-Flash Images