Single-photon cameras based on single-photon avalanche diode (SPAD) technology are gaining popularity for 3D sensing, thanks to their extreme sensitivity and time resolution. There are two key challenges with single-photon cameras that limit their widespread use: (i) they suffer from non-linear distortions called ''pile-up'' when operated in high-photon-flux conditions, and (ii) they generate a large volume of raw photon data, creating a severe data bottleneck at each sensor pixel. In this work, we show that while compressive capture techniques successfully mitigate data transfer challenges, they exacerbate the effects of dead-time distortion because they fail to retain sufficient information about the photon detection history to allow post-processing pile-up correction via existing methods. We propose a new computational-imaging method that combines free-running capture with an analysis-by-synthesis software pipeline to mitigate pile-up distortions. Our results with hardware emulations and full-scene and single-pixel simulations show that our method can reliably capture scene distance and reflectance over a wide range of illumination conditions. Our work will enable high-resolution SPAD cameras that are severely bandwidth-constrained to operate in real-world high-flux scenarios.
Figures & tables
Fig. 1: Our method enables dense 3D imaging with dead-time-affected SPCs in memory-constrained settings and high-photon-flux conditions. Without compensation, uncorrected distance estimates (bottom left) suffer from dead-time distortion and lossy compression via a count-free histogrammer. In contrast, our approach extracts depth directly from these compressed, pile-up-distorted measurements, yielding dense 3D maps with orders-of-magnitude higher accuracy. (Bottom numbers report mean error [ME], standard deviation [SD], and 5% inliers).
Method
Pixel Resources
Robust to Pile-up
High-res. Hardware Compatible
High-flux SPL [ 11 ]
High
Yes
No
CSPH [ 4 , 5 ]
Low
No
Yes
Hist.-less [ 8 ]
Low
No
Yes
SplineSketch [ 7 ]
Low
No a
No
EDH [ 9 , 10 ]
Low
No
No
This Work
Low
Yes
Yes
TABLE I: Comparison of state-of-the-art. Although many existing techniques deal with the problems of SPC data compression and pile-up separately, there is no single technique that solves these problems jointly and is amenable to in-sensor implementation for dense pixel arrays. ( a The SplineSketch method handles signal peak pile-up, but only over a narrow range of signal flux levels using a look-up table).
Fig. 2: SPC image formation and memory-efficient histogramming techniques. (a) Each SPC pixel operates in tandem with a pulsed laser whose round-trip travel time gives an estimate of the scene distance. (b) The pixel captures the returning photon events as a stream of photon timestamps and attempts to locate the true peak location in the true underlying photon distribution Φ[t] . (c) High-resolution timestamp data consumes too much in-pixel memory. One way to compress it is to use an equi-width histogram with very few bins (8 bins in this case). These wide bins have limited distance resolution and introduce quantization artifacts. (d) An equi-depth histogram with 8 bins is a more efficient representation to track the laser-peak location as seen from a small cluster of bin boundaries around the true peak location. (Figure adapted from [ 10 ] .)
Fig. 3: Effect of dead-time distortion on EDH SPCs. In the absence of dead-time distortions, a compressive capture from a 32-bin EDH provides reliable estimates of the true scene distances. For a realistic dead-time of τd=75 ns, the scene distance estimates are severely corrupted by pile-up. Due to the non-linearity of pile-up distortions, statistical averaging by increasing the capture time does not help.
Fig. 4: Effect of dead-time on SPC measurements. (a) In case of no dead-time, the binner tracks the median of the photon arrivals, but in case of dead-time, the binner tracks just the detections and suffers from a negative bias due to dead-time distortion. (b) When dead-time = 0, the narrowest bin tracks the peak, but in the presence of dead-time, the narrowest bin no longer tracks the peak of the photon arrivals.
Fig. 5: Markov chain model for an equi-depth histogrammer. (a) The photon detection stream lives on a discrete state space with B possible locations; the binner control value lives on a state space of B+1 possible locations. The discrete photon detection times can be described using a Markov chain [ 11 ] . (b) A binner’s control value is incremented or decremented with some probability that is tied to the photon detection probabilities relative to the current control-value position. The binner’s control value, therefore, follows another Markov chain, which is driven by the Markov chain of photon detections. (c) An EDH consists of a bank of binners, each tracking a different quantile (shown by vertical dashed lines) of the perceived transient distribution, which may be different from the true transient distribution due to dead-time distortion. The binners converge to their respective stationary distributions that are centered around the actual quantiles.
Fig. 6: Three stages of the analysis-by-synthesis (AbS) pipeline. (1) Initialization stage uses SPC measurement Mq and background flux Φbkg estimated from passive capture. We also apply a spatial 3×3 median filter to reduce noise in the initial distance estimate. (2) The optimization stage minimizes a loss function to obtain model parameters that best explain the observed EDH data. (3) A postprocessing stage applies a 3x3 median filter on the distance estimate and effective signal estimate to get the final outputs.
Fig. 7: Single-pixel simulation results. The single-pixel Monte Carlo simulation results, over a varying combinations of Φsig and Φbkg , demonstrate that our 32-bin EDH with dead-time compensation (AbS) method achieves significantly lower distance RMSE (left) and higher 1% inliers as compared to other baselines. The MCHC [ 11 ] , Coates’s correction [ 29 ] , and 5% rule methods use 32-bin EWH measurements and are thus limited by the quantization effects. The histogramless method (HL) performs worse as the background flux increases due to nonlinearity in the measurements.
Fig. 8: Robustness of our AbS pipeline to calibration errors and model mismatch. The AbS pipeline estimates the scene distance and signal flux by assuming known calibrated values of Φbkg , laser pulse FWHM, and τd . These simulations demonstrate a gradual increase in distance RMSE (top row) and a slight decrease in 1% inliers (bottom row) with increasing calibration errors in the background estimate (left column), pulse FWHM (middle column), and τd (right column). The scene distance was fixed at the half the maximum distance range.f The smooth trends across five different illumination conditions demonstrate that our AbS estimator continues to work even in the presence of slight calibration mismatches.
Fig. 9: AbS robustness to non-ideal transients in real hardware data. Our AbS pipeline can handle non-ideal transients such as single-pixel transients from real hardware data. Observe that given the emulated EDH boundaries (vertical lines), there is good agreement between the actual perceived transient and the estimate in both (a) high flux (HF) conditions and (b) low flux (LF). In high-flux conditions, our method reliably predicts the within-peak pile-up and also the “peak shadow” (the dip in the transient’s shape) that immediately follows the peak location, and spans a variable number of time locations depending on the dead-time.
Dead Time
Distance Error (mm)
KL Div.
τd (ns)
32-bin EWH
32-bin EDH [Ours]
HF
LF
21
285
4
0.21
0.23
48
285
12
0.15
0.18
81
285
13
0.19
0.21
89
285
8
0.14
0.15
98
285
9
0.17
0.17
TABLE II: Quantitative results on real hardware. Here we show quantitative comparison of distance error vs. ground truth distance and KL divergence of the recovered transient from the true transient across different sensor dead-times ( τd ) using real hardware data from Rapp, et al. [ 11 ] . (HF=high flux, LF=low flux)
Fig. 10: Results on 3D scenes with multi-path interference from iToF2dToF dataset. We compare our EDH-based AbS method with standard baselines that use Equi Width Histograms (EWH) and a histogramless approach. Considering resource-constrained scenarios, we keep the number of EWH bins and EDH bins equal for comparison. Quantitative results: mean error (ME, in meters), standard deviation (SD, in meters), and 1% inlier rate (1%in)—are reported beneath each result. As shown, our proposed method significantly reduces distance artifacts and outperforms the baselines. Additionally, we show recovered transients for some selected pixels from each scene, demonstrating how, despite the presence of multi-path interference, our method is able to converge closer to the largest peak in most scenarios.
Supplementary Figure 1: Plots showing empirical histograms of final CV boundaries over multiple runs vs analytical CV stationary distributions. The x-axis in this plot denotes the discretized window locations (length 1000) and the y-axis denotes the probability that the CV is found at a given window location. There is good agreement between the analytical model (dotted lines) and the empirical Monte Carlo simulations of the binner CV (solid jagged lines). The vertical dashed lines denote the true quantile positions, which line up with the CV stationary distributions’ modes.
Supplementary Figure 2: Error plot showing the difference between the mode of the stationary distribution Dq and the qth quantile of the empirically simulated perceived transient, aggregated over different combinations of distance, signal strength, background strength, and dead-time. We show the spread of errors for each of the 31 quantiles that form a 32-bin EDH. Observe that the errors are quite small (hovering around the zero line), indicating good agreement between theory and empirical simulations.
Supplementary Figure 3: Example loss-function contour maps . We show three different loss functions (negative-log-likelihood, boundary MSE and quantile MSE) in a simulated example. NLL gives the best match between the final optimizer estimate and the true value but has a higher computational overhead. Boundary MSE provides reasonable estimates at slightly higher error. Quantile MSE provides reliable distance estimate but poor signal strength estimates. × denotes the starting guess in the AbS pipeline, ∘ denotes the final converged position, and + denotes the true signal and distance combination used in this specific example. The trajectory of estimates over each iteration of the optimizer is overlaid on the cost-function contours.
Supplementary Figure 4: Optimizer convergence of the AbS/EDH estimator: Each panel sweeps background flux Φbkg∈{0.2,0.5,1.0,2.0} ( SBR 10 – 1 ) for two target distances at one-fourth and one-half the maximum distance range, with a 3×3 spatial median filter applied to the distance initialization. (Left) Nelder-Mead iteration count distributions; (right) distance RMSE. The combined loss optimization (50-iteration RMSE followed by 500-iteration NLL) maintains sub- 30mm RMSE and a tight iteration spread across all conditions, whereas NLL-only degrades catastrophically at SBR≤2 for the shallower target (at one-fourth the maximum distance range), where dead-time distortion is most severe. All results use Φsig=2 , dead-time =75ns , Tmax=100ns , 200 independent pixels per condition.
Supplementary Figure 5: Depth RMSE vs. laser time period Tr for three scene distances (7.5, 15, 22.5 cm) with no dead-time distortion ( τd=0 ). The laser pulse width scales with Tr ( FWHM=0.01Tr ) to maintain constant pulse resolution across all conditions. All methods use 16 bins; ϕsig=ϕbkg=2.0 photons/cycle, 100 pixels per condition. EDH+AbS (ours) maintains near-constant RMSE as Tr grows, while EWH-based methods degrade proportionally as their bin width Tr/nEWH widens linearly with the laser time period. The result holds across all three scene distances.
Supplementary Figure 6: Results on 3D scenes with multi-path interference from iToF2dToF dataset We compare our AbS method with standard baselines using 16-bin Equal Width Histograms (EWH) and a histogramless approach. Quantitative results—Mean Error (ME, in meters), Standard Deviation (SD, in meters), and 1% inlier rate (1%in)—are reported beneath each result. As shown, our proposed method significantly reduces distance artifacts and outperforms the baselines. We also show results for signal flux estimation and distance estimated from 5% rule method and MCHC method when operated with high-resolution EWH. The CSPH and DeePEDH show consistent negative bias and lower 1% inliers. Additionally, we show recovered transients for some selected pixels from each scene, demonstrating how, despite the presence of multi-path interference, our method is able to converge closer to the largest peak in most scenarios.
Supplementary Figure 7: Full scene results for scenes from NYUv2 and Middlebury datasets with 0 dead-time. Observe that all methods provide reliable distance estimates, with the only source of noise being Poisson noise.
Supplementary Figure 8: Full scene results for scenes from NYUv2 and Middlebury datasets for dead-time = 75 ns. The longer dead-time and high flux conditions cause strong pile-up artifacts that cannot be addressed using the histogram-less method. Even an “oracle” EDH that has access to ground truth (GT) perceived transient suffers from within-peak pile-up distortions. Our AbS compensation provides reliable distance and signal strength estimates even in this high-flux pile-up regime.
Supplementary Figure 9: Robustness of our AbS pipeline to calibration errors and model mismatch. Our AbS pipeline estimates the scene distance and signal flux by assuming known calibrated values of Φbkg , laser pulse FWHM, and τd . The results demonstrate a gradual increase in Mean Error (first column), standard deviation (second column) in distance estimates and in signal estimates (column 3 and 4), with increasing calibration errors in the background estimate (top row), pulse FWHM (middle row), and τd (bottom row). The smooth trends across five different illumination conditions demonstrate that our AbS estimator does not break under mismatch in the fixed parameters and thus works in other than ideal calibration conditions.
Supplementary Figure 10: Hardware emulation results for real-world SPAD pixel dataset. We show results of transient reconstruction using signal and distance values estimated with our AbS pipeline on real-world photon data streams. Observe that there is good agreement between the actual perceived transient and the estimate over a wide range of dead-times, in both low flux (LF) and high flux (HF) conditions. In high-flux conditions, our method reliably predicts the within-peak pile-up and also the “peak shadow” i.e. the slight dip in the transient’s shape that immediately follows the peak location, and spans a variable number of time locations depending on the dead-time.
Optical flow remains challenging in high-speed and low-light scenes, where the limited frame rate and sensitivity of conventional cameras lead to motion blur and underexposure. Single-photon avalanche diode (SPAD) cameras offer single-photon sensitivity and extremely fine temporal sampling. However, individual slices in these high FPS binary photon streams are too sparse for dense correspondence. Temporal aggregation can provide the spatial cues required by optical flow, but accumulating photons at fixed coordinates blurs moving structures. Motion-aware aggregation can reduce this blur, yet it depends on the flow being estimated. To address this dependency, we propose QuantaFlow, the first method for dense optical flow directly from SPAD streams. Instead of constructing a fixed input representation, QuantaFlow embeds SPAD representation construction into iterative flow refinement. At each iteration, the current flow coarsely aligns the slices within the source and target sub-streams. A photon-flux transformation then constructs multi-scale representations containing intensity and structural cues, while adaptive multi-scale fusion balances photon noise and residual motion blur at each pixel. The fused representations drive a feature-warping flow update, and the refined flow guides representation construction in the next iteration. We further construct a synthetic dataset for SPAD optical-flow training and evaluation. Experiments on the synthetic dataset and real-world SPAD data demonstrate the effectiveness and generalization of QuantaFlow.
Single-photon avalanche diode (SPAD) cameras operate fundamentally differently from conventional cameras due to their photon-counting nature. Each frame produces a binary image: pixels report zero if no photons arrived during exposure, and one if one or more photons arrived. Reconstructing a scene or inferring its properties from a single binary frame is difficult because many different images could produce the same measurement; thus, the inverse problem is fundamentally one-to-many. As we gather more binary measurements, the inherent uncertainty associated with the inverse problem and any associated inference diminishes. With sufficient photon counts, photon noise becomes negligible relative to the signal mean, enabling near-deterministic scene recovery and inference. This work characterizes the transition from stochastic to near-deterministic scene understanding as photon budget increases, analyzing how the stochasticity in photon arrival affects downstream inference tasks. Technically, we develop a conditional generative framework based on a Hypergeometric frame-thinning process for accumulated binary SPAD measurements. Generative models capture the one-to-many nature of photon-starved inverse problems, enabling empirical characterization of how this ambiguity diminishes with increasing measurements and its impact on downstream tasks like character recognition, QR code decoding, and facial analysis.
Single-photon LiDAR (SPL) based on single-photon avalanche diode (SPAD) sensing enables time-resolved photon measurements with extreme sensitivity, offering unique potential for active 3D perception in photon-starved scenarios.However, real-world single photon perception remains fundamentally challenging due to unique measurement noise and complex multi-return transient phenomena, which jointly complicate geometric reconstruction and semantic scene understanding. Despite growing interest in SPAD-based sensing, existing studies are largely limited to simulated data or small-scale controlled captures. As a result, systematic evaluation of real-world single photon perception across depth estimation, multi-view reconstruction, and 3D semantic understanding remains underexplored. To bridge this gap, we introduce SP-TransientBench (STB), a real-captured multi-task benchmark for single photon perception. SP-TransientBenc comprises 10 diverse scenes and 10,297 views captured using a solid-state single-photon LiDAR at 256×192 resolution. Each view provides full time-of-flight histograms with multi-return behavior,standardized metadata, and calibrated camera poses for multi-view evaluation. We further provide 13-class 3D semantic annotations for selected scenes. By providing dedicated data splits and evaluation protocols for each task, STB enables consistent and reproducible benchmarking of real-world single photon perception across multiple 3D vision problems. The dataset and code will be released upon acceptance.
Hongzhou Dong, Zili Zhang, Ziting Wen +12
Shanghai University, Shanghai, China · Southern University of Science and Technology, Shenzhen, China · The University of Sydney, Sydney, Australia