PIE-PS: Photometric Stereo from Physical Irradiance Event Streams

Xiangze Meng; Guangyu Li; Jing Li; Di Mei; Songchen Ma; Mingkun Xu; Rui Ma

cs.CVOct 6, 2026

PIE-PS: Photometric Stereo from Physical Irradiance Event Streams

Authors: Xiangze Meng, Guangyu Li, Jing Li, Di Mei, Songchen Ma, Mingkun Xu, Rui Ma

Organizations: Jilin University, China · Guangdong Institute of Intelligence Science and Technology, China · Beijing Institute of Technology, China · The Hong Kong University of Science and Technology, China

Abstract

Event cameras record asynchronous log-image-irradiance changes with microsecond latency and high dynamic range. These properties are useful for photometric stereo under moving illumination, but raw events are sparse and depend on an unknown contrast threshold. We start from the event trigger model and derive a physical relation between adjacent events, light motion, and surface normals. This relation gives a direct physics-only solver, but the solver needs the threshold, enough events at each pixel, and independent per-pixel optimization. To address these limits, we introduce PIE-PS, a learning-based framework for dense surface normal reconstruction from raw event streams and known lighting. We form Physical Irradiance Events (PIEs) by pairing two adjacent events at the same pixel with their corresponding light directions. Each PIE provides a Physical Irradiance Event Feature (PIEF), defined as the signed event rate. PIEF does not require the unknown contrast threshold. To share spatial and temporal context across nearby PIEs, we introduce PIE-GNN, which treats each PIE as a graph node and encodes it with its light-pair geometry. Since the reliability of PIE observations can vary with local appearance, illumination geometry, and sensor noise, Reliability-Grading Attention (RGA) predicts reliability weights to down-weight unreliable PIEs. Pixel aggregation then produces dense normals. Experiments on synthetic and real data show that PIE-PS outperforms prior event-based photometric stereo methods and the direct solver baseline.

Figures & tables

Figure 1. Overview of PIE-PS. A prototype system captures raw event streams under moving light. Adjacent events at the same pixel are paired with their light directions to form Physical Irradiance Event (PIE) observations. Each PIE provides a threshold-free signed event-rate Physical Irradiance Event Feature (PIEF) and calibrated light-pair geometry. PIE-GNN encodes these PIE nodes, Reliability-Grading Attention (RGA) down-weights unreliable observations, and pixel aggregation produces dense surface normals. Overview of PIE-PS from raw event capture under moving light to PIE representation, graph-based RGA weighting, and dense normal reconstruction.

Figure 2. Overview of the PIE-PS framework. We first convert raw events under moving light into PIE observations. Each PIE provides a threshold-free signed event-rate feature and a corresponding light-pair geometry. The PIE-GNN encodes these observations, uses Reliability-Grading Attention (RGA) to weight them, aggregates pixel features, and predicts normals with a pixel aggregator.

Figure 3. The Reliability-Grading Attention module scores each event-pair observation. It detects spikes in the PIE sequence and predicts a confidence weight for each observation. The weighted observations are then aggregated into pixel features.

Figure 4. Prototype acquisition system used for real-world event-based photometric stereo. A prototype capture setup with an event camera, a macro lens, and a programmable ring light for moving illumination.

Figure 5. Qualitative comparison across synthetic and real scenes. Each row shows the object, the normal prediction and angular error map of each method, and the ground-truth normal. We include PIE-Sim, DiLiGenT-Sim, EventPS real scenes, and our captured scenes. Error maps use the same $0^{\circ}$ – $30^{\circ}$ color range, where darker colors indicate lower error. A qualitative comparison figure with eight rows. The rows cover PIE-Sim, DiLiGenT-Sim, EventPS real scenes, and captured scenes. Each row compares EventPS-FCN, EventPS-CNN, Direct Solver, and PIE-PS using normal maps and angular error maps.

Figure 6. RGA reliability visualization. Each object is shown at three time slices. For each time slice, we show the raw frame, the GT-identified unreliable PIE ratio, and the low-RGA PIE ratio. GT-identified unreliable PIEs are computed with ground-truth normals and known simulation thresholds only for analysis. The figure label GT-bad denotes these GT-identified unreliable PIEs. Low-RGA maps are produced by the learned RGA scorer. A visualization with one DiLiGenT object and two PIE-Sim objects. Each object has three time rows and columns for raw frame, GT-identified unreliable PIE, and low-RGA PIE.

Method	Ball	Buddha	Cat	Cow	Goblet	Harvest	Pot1	Pot2	Reading	Average
EventUPS	8.86	12.21	10.78	18.58	12.97	23.32	8.70	15.13	16.71	14.14
EventPS-OP	10.99	18.73	12.74	26.51	18.43	36.06	13.78	15.75	24.61	19.73
EventPS-FCN	7.49	18.13	11.42	20.61	18.07	26.05	12.83	16.59	15.16	16.26
EventPS-CNN	10.44	16.79	11.88	20.60	16.44	25.26	12.93	15.54	18.19	16.45
Direct Solver	15.36	15.09	13.75	14.55	15.62	20.07	13.59	14.67	17.72	15.60
PIE-PS	3.93	8.67	5.15	12.90	7.81	12.91	6.33	8.34	12.25	8.70

Table 1. Quantitative comparison on DiLiGenT-Sim. MAE is reported in degrees, and lower is better.

Method	Blobby			Sculpture			Avg.
Method	Circle	Hypo.	DiLiGenT	Circle	Hypo.	DiLiGenT	Avg.
EventPS-FCN	28.83	16.83	21.73	31.63	26.43	29.59	25.84
EventPS-CNN	43.50	42.11	26.08	48.31	46.85	28.08	39.15
Direct Solver	27.08	17.24	13.02	26.60	24.68	22.67	21.88
PIE-PS	10.68	6.41	4.81	17.17	12.09	12.77	10.66

Table 2. PIE-Sim results by object family and lighting trajectory. MAE is reported in degrees.

Method	EventPS Real			Captured			Avg.
Method	Real-1	Real-2	Real-3	Bunny	Rectangle	Flower	Avg.
EventPS-FCN	15.79	19.65	9.42	26.14	18.70	35.76	20.91
EventPS-CNN	17.65	20.16	14.85	48.78	48.11	43.14	32.12
Direct Solver	21.08	21.10	20.50	23.46	32.22	25.02	23.90
PIE-PS	6.51	6.09	5.92	12.69	11.52	19.57	10.38

Table 3. Real-world results on EventPS scenes and our captures. MAE is reported in degrees.

Metric	PIE-PS	w/o PIEF rate	w/o RGA
Avg. MAE	10.66	14.52	11.99

Table 4. Ablation results on PIE-Sim. MAE is reported in degrees.

Figure 7. RGA weight distribution on PIE-Sim. GT-identified unreliable PIEs receive lower weights on average, supporting RGA as a soft reliability-weighting module. A histogram comparing RGA weight distributions for reliable and unreliable PIEs on PIE-Sim. Reliable PIEs have a higher mean weight than unreliable PIEs.

Explore similar work

Aug 1, 2026cs.CV

E2Pano: Learning Event-to-Panorama Image Reconstruction

Event cameras offer microsecond-level temporal resolution and high dynamic range, potentially facilitating motion-blur-free panoramic imaging from fast rotational scanning. Nonetheless, existing optimization-based methods remain computationally demanding, while prior learning-based reconstruction methods are largely designed for perspective imagery and lack geometry-aware support for panoramic outputs. We present E2Pano, a geometry-guided event-to-panorama pipeline with an end-to-end learnable photometric reconstruction stage. Our framework preserves real spherical coordinates from geometric mapping throughout the pipeline, employs a lightweight enhancement module with frequency-domain supervision to bridge the event-image domain gap, and leverages a spherical Transformer with 3D positional embeddings for photometric reconstruction. Experiments on synthetic data and captured rotational scans show improved reconstruction quality and lower photometric reconstruction cost than optimization-based baselines, together with encouraging transfer to real captures under our acquisition protocol despite training purely on synthetic data. Additionally, we construct PanoScan, a dataset with 4,370 synthetic and 30 real-world panoramic scenes paired with event streams. Our dataset and code will be released.

Zhenyang Li, Zongqi He, Jia Pan +2

The University of Hong Kong Hong Kong, China

Omnidirectional Images Scene Reconstruction Panoptic Segmentation Sphere

Ask PDF

Jul 16, 2026eess.IV

ESAR: Event-Based Synthetic Aperture Reconstruction

Event cameras report asynchronous polarity events when changes in log--radiance exceed a fixed contrast threshold, producing signed temporal contrast measurements rather than conventional image frames. We formulate monocular event-based imaging as a synthetic-aperture inverse problem for a static ground-domain log--radiance field

θ\in \mathbb{R}^{N_g}

. Instead of reconstructing a latent pixel-time volume

v \in \mathbb{R}^{N_pN_t}

, we impose the geometric relation

v=Pθ

, where

P

maps the fixed scene into motion-dependent latent views. Aggregating events over finite time intervals gives the linearized model

APθ= b+η,

where

A

is a temporal differencing operator,

b

contains signed binned event counts, and

η

represents measurement and modeling errors. This decomposition exposes a synthetic-aperture structure: under near-nadir motion, successive projections are approximately shifted views of a common scene, while the composite operator

AP

remains ill-conditioned because it combines spatial averaging with temporal differencing. We therefore use regularized inversion to recover

θ

. Numerical experiments on simulated data and real near-nadir Falcon Neuro event data show that the proposed

θ

-based formulation recovers coherent large-scale spatial structure, relative to dynamic latent-image and learned event-reconstruction baselines, while suppressing fine-scale texture.

Harbir Antil, Daniel Blauvelt, David Sayre

Synthetic Aperture Radar Image Reconstruction Wireless Radiance Field Bayesian Inverse Problems

Ask PDF

Date pendingcs.CV

EVIS: Real-Time Event Camera Simulation with Multimodal Supervision in NVIDIA Isaac Sim

Event cameras are increasingly adopted in embodied perception for their microsecond temporal resolution, high dynamic range, and resilience to motion blur. However, training event-based models for robotics requires large-scale, action-conditioned data with dense physical annotations that are difficult to collect in the real world. We introduce EVIS, an open-source physics-grounded event simulator integrated into NVIDIA Isaac Sim that generates events from linear-HDR radiance from a closed-loop robot training episode. Rather than relying on learning-based video interpolation or expensive dense rendering, EVIS exploits renderer-provided motion vectors and depth maps through bi-directional warping and depth-based splatting. This enables high throughput, real-time, and high-fidelity event generation. We evaluate EVIS across runtime efficiency, sim-to-real transfer, and zero-shot model compatibility. A rotation-speed estimator trained solely on EVIS events achieves 2.75 rad/s MAE on real sensor data. Pretrained models for reconstruction, matching, and tracking perform competitively on EVIS events without any fine-tuning. EVIS sustains real-time generation across GPU-parallel environments on a single GPU. Code repository: https://github.com/spikelab-jhu/isaac-sim-event-camera-plugin.

Linli Shi, Ruijun Zhang, Ziyun Wang

Johns Hopkins University

Camera Robotic Perception Nvidia Isaac Sim

Ask PDF