PIE-PS: Photometric Stereo from Physical Irradiance Event Streams
Authors: Xiangze Meng, Guangyu Li, Jing Li, Di Mei, Songchen Ma, Mingkun Xu, Rui Ma
Organizations: Jilin University, China · Guangdong Institute of Intelligence Science and Technology, China · Beijing Institute of Technology, China · The Hong Kong University of Science and Technology, China
Event cameras record asynchronous log-image-irradiance changes with microsecond latency and high dynamic range. These properties are useful for photometric stereo under moving illumination, but raw events are sparse and depend on an unknown contrast threshold. We start from the event trigger model and derive a physical relation between adjacent events, light motion, and surface normals. This relation gives a direct physics-only solver, but the solver needs the threshold, enough events at each pixel, and independent per-pixel optimization. To address these limits, we introduce PIE-PS, a learning-based framework for dense surface normal reconstruction from raw event streams and known lighting. We form Physical Irradiance Events (PIEs) by pairing two adjacent events at the same pixel with their corresponding light directions. Each PIE provides a Physical Irradiance Event Feature (PIEF), defined as the signed event rate. PIEF does not require the unknown contrast threshold. To share spatial and temporal context across nearby PIEs, we introduce PIE-GNN, which treats each PIE as a graph node and encodes it with its light-pair geometry. Since the reliability of PIE observations can vary with local appearance, illumination geometry, and sensor noise, Reliability-Grading Attention (RGA) predicts reliability weights to down-weight unreliable PIEs. Pixel aggregation then produces dense normals. Experiments on synthetic and real data show that PIE-PS outperforms prior event-based photometric stereo methods and the direct solver baseline.
Figures & tables
Figure 1. Overview of PIE-PS. A prototype system captures raw event streams under moving light. Adjacent events at the same pixel are paired with their light directions to form Physical Irradiance Event (PIE) observations. Each PIE provides a threshold-free signed event-rate Physical Irradiance Event Feature (PIEF) and calibrated light-pair geometry. PIE-GNN encodes these PIE nodes, Reliability-Grading Attention (RGA) down-weights unreliable observations, and pixel aggregation produces dense surface normals. Overview of PIE-PS from raw event capture under moving light to PIE representation, graph-based RGA weighting, and dense normal reconstruction.
Figure 2. Overview of the PIE-PS framework. We first convert raw events under moving light into PIE observations. Each PIE provides a threshold-free signed event-rate feature and a corresponding light-pair geometry. The PIE-GNN encodes these observations, uses Reliability-Grading Attention (RGA) to weight them, aggregates pixel features, and predicts normals with a pixel aggregator.
Figure 3. The Reliability-Grading Attention module scores each event-pair observation. It detects spikes in the PIE sequence and predicts a confidence weight for each observation. The weighted observations are then aggregated into pixel features.
Figure 4. Prototype acquisition system used for real-world event-based photometric stereo. A prototype capture setup with an event camera, a macro lens, and a programmable ring light for moving illumination.
Figure 5. Qualitative comparison across synthetic and real scenes. Each row shows the object, the normal prediction and angular error map of each method, and the ground-truth normal. We include PIE-Sim, DiLiGenT-Sim, EventPS real scenes, and our captured scenes. Error maps use the same 0∘ – 30∘ color range, where darker colors indicate lower error. A qualitative comparison figure with eight rows. The rows cover PIE-Sim, DiLiGenT-Sim, EventPS real scenes, and captured scenes. Each row compares EventPS-FCN, EventPS-CNN, Direct Solver, and PIE-PS using normal maps and angular error maps.
Figure 6. RGA reliability visualization. Each object is shown at three time slices. For each time slice, we show the raw frame, the GT-identified unreliable PIE ratio, and the low-RGA PIE ratio. GT-identified unreliable PIEs are computed with ground-truth normals and known simulation thresholds only for analysis. The figure label GT-bad denotes these GT-identified unreliable PIEs. Low-RGA maps are produced by the learned RGA scorer. A visualization with one DiLiGenT object and two PIE-Sim objects. Each object has three time rows and columns for raw frame, GT-identified unreliable PIE, and low-RGA PIE.
Method
Ball
Buddha
Cat
Cow
Goblet
Harvest
Pot1
Pot2
Reading
Average
EventUPS
8.86
12.21
10.78
18.58
12.97
23.32
8.70
15.13
16.71
14.14
EventPS-OP
10.99
18.73
12.74
26.51
18.43
36.06
13.78
15.75
24.61
19.73
EventPS-FCN
7.49
18.13
11.42
20.61
18.07
26.05
12.83
16.59
15.16
16.26
EventPS-CNN
10.44
16.79
11.88
20.60
16.44
25.26
12.93
15.54
18.19
16.45
Direct Solver
15.36
15.09
13.75
14.55
15.62
20.07
13.59
14.67
17.72
15.60
PIE-PS
3.93
8.67
5.15
12.90
7.81
12.91
6.33
8.34
12.25
8.70
Table 1. Quantitative comparison on DiLiGenT-Sim. MAE is reported in degrees, and lower is better.
Method
Blobby
Sculpture
Avg.
Circle
Hypo.
DiLiGenT
Circle
Hypo.
DiLiGenT
EventPS-FCN
28.83
16.83
21.73
31.63
26.43
29.59
25.84
EventPS-CNN
43.50
42.11
26.08
48.31
46.85
28.08
39.15
Direct Solver
27.08
17.24
13.02
26.60
24.68
22.67
21.88
PIE-PS
10.68
6.41
4.81
17.17
12.09
12.77
10.66
Table 2. PIE-Sim results by object family and lighting trajectory. MAE is reported in degrees.
Method
EventPS Real
Captured
Avg.
Real-1
Real-2
Real-3
Bunny
Rectangle
Flower
EventPS-FCN
15.79
19.65
9.42
26.14
18.70
35.76
20.91
EventPS-CNN
17.65
20.16
14.85
48.78
48.11
43.14
32.12
Direct Solver
21.08
21.10
20.50
23.46
32.22
25.02
23.90
PIE-PS
6.51
6.09
5.92
12.69
11.52
19.57
10.38
Table 3. Real-world results on EventPS scenes and our captures. MAE is reported in degrees.
Metric
PIE-PS
w/o PIEF rate
w/o RGA
Avg. MAE
10.66
14.52
11.99
Table 4. Ablation results on PIE-Sim. MAE is reported in degrees.
Figure 7. RGA weight distribution on PIE-Sim. GT-identified unreliable PIEs receive lower weights on average, supporting RGA as a soft reliability-weighting module. A histogram comparing RGA weight distributions for reliable and unreliable PIEs on PIE-Sim. Reliable PIEs have a higher mean weight than unreliable PIEs.