One Photon, Many Worlds: Posteriors and Predictions with Single-Photon Cameras
Authors: Haejoon Lee, Mohit Gupta, Vijayakumar Bhagavatula, Aswin C. Sankaranarayanan
Organizations: Carnegie Mellon University · Carnegie Mellon University, Pittsburgh, PA, USA · University of Wisconsin-Madison · University of Wisconsin-Madison, Madison, WI, USA
Single-photon avalanche diode (SPAD) cameras operate fundamentally differently from conventional cameras due to their photon-counting nature. Each frame produces a binary image: pixels report zero if no photons arrived during exposure, and one if one or more photons arrived. Reconstructing a scene or inferring its properties from a single binary frame is difficult because many different images could produce the same measurement; thus, the inverse problem is fundamentally one-to-many. As we gather more binary measurements, the inherent uncertainty associated with the inverse problem and any associated inference diminishes. With sufficient photon counts, photon noise becomes negligible relative to the signal mean, enabling near-deterministic scene recovery and inference. This work characterizes the transition from stochastic to near-deterministic scene understanding as photon budget increases, analyzing how the stochasticity in photon arrival affects downstream inference tasks. Technically, we develop a conditional generative framework based on a Hypergeometric frame-thinning process for accumulated binary SPAD measurements. Generative models capture the one-to-many nature of photon-starved inverse problems, enabling empirical characterization of how this ambiguity diminishes with increasing measurements and its impact on downstream tasks like character recognition, QR code decoding, and facial analysis.
Figures & tables
Figure 1. SPAD cameras operated in binary-frame mode record at most one detection per pixel per exposure. The top four rows shows samples from the conditional posterior given a single (simulated) binary SPAD frame, shown in the first column where each pixel has measured, at most, one photon . In such photon starved regimes, the inverse problem of image reconstruction is one-to-many, leading to many worlds of possibilities in the output. This uncertainty reduces as we increase the number of binary SPAD frames, as seen in subsequent rows. The conditional samples were produced by training diffusion models for each of the four applications (digits, edges, QR codes, and faces), and empirically characterizes how uncertainty in photon arrivals map to uncertainty in reconstruction and inference. In the case of the faces data, notice how different features like sunglasses, gender, facial expression, and identity take different number of frames to resolve correctly.
Figure 2. Example MNIST images simulated with the SPAD sensor model. The maximum expected photon level per binary frame is set to τ=0.01 .
Figure 3. Deterministic restoration under photon starvation. For very small N , the same low-count SPAD observation is compatible with many high-count targets. A deterministic denoiser trained with a L2 reconstruction loss produces a single averaged output, which can be blurry and cannot represent the remaining conditional uncertainty.
Figure 4. The overall process of conditional generator for SPAD image restoration.
Figure 5. Overview of the proposed latent binomial model for a single step. The prior encoder takes in the source SPAD image xt+1 and the frame counts Nt and Nt+1 to predict the parameters of the Gaussian prior distribution of the latent variable z . The posterior encoder takes in the source SPAD image xt+1 , the increment image dt , and the frame counts Nt and Nt+1 to predict the parameters of the Gaussian posterior distribution. The decoder with a U-Net architecture takes in xt+1 , the frame counts Nt and Nt+1 , and z to predict the photon-detection probability map ρθ . During sampling, z is drawn from the prior, yt,i∼Binomial(Mt,ρθ,i) , and xt=xt+1+yt . Here z denotes the set of latent groups described in Section 4.3 .
xt
∼i∏Binomial(Nt,ρi),
xt+1
∼i∏Hypergeometric(Nt,xt,i,Nt+1)
ALGORITHM 1 Training the shared pairwise SPAD reverse model
z∼pψ(⋅∣xt+1,Nt,Nt+1)
ALGORITHM 2 Sampling by anchor-point Markov reversion
Figure 6. Conditional generation results for synthetic edge images. Each column fixes the same conditioning measurement xk at a given frame count Nk , followed by several high-count samples x1 generated conditioned on xk , their sample mean, and their sample normalized standard deviation.
Figure 7. Conditional generation results on a real SPAD capture of a printed edge target. For each column, the figure shows the generated samples, their sample mean, and their sample standard deviation. At the rightmost corner, we show the photo of the printed target.
Figure 8. Downstream edge-prediction results for generated samples as a function of the number of conditioning SPAD frames Nk . The red dotted line denotes performance on high-count target SPAD images with N1=10,000 frames.
Figure 9. Conditional generation results for MNIST. For each fixed conditioning measurement xk , the figure shows generated high-count samples, their sample mean, and their sample standard deviation.
Figure 10. Classification accuracy of generated MNIST samples as a function of the number of conditioning SPAD frames Nk .
Figure 11. Conditional generation results with real captures in OCR task. We exhibit the restored grid of digits with selected number of measurements.
Figure 12. Conditional QR-code generation results from the multistep model. Each column fixes the conditioning measurement xk at a given frame count Nk , followed by generated samples, their sample mean, and their sample standard deviation. Each standard deviation map is normalized to have maximum value of 1.
Figure 13. QR-code decoding accuracy of multistep generated samples compared with raw SPAD measurements. The generative model reaches stable decoding accuracy with far fewer conditioning frames.
Figure 14. Facial-geometry errors for generated FFHQ samples. We compare MediaPipe landmarks and head pose estimated from generated samples against those estimated from high-count target images with N1=100,000 frames.
Figure 15. MediaPipe landmark visualizations on generated FFHQ samples and raw SPAD measurements. The generated samples support landmark detection at much lower frame counts than the raw SPAD inputs.
Figure 16. Conditional generation results for real captured face images. For each fixed conditioning measurement xk , the figure shows generated high-count samples, their sample mean, and their sample standard deviation.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 17. QR-code decoding accuracy of generated samples as a function of the number of conditioning SPAD frames Nk .
Figure 18. Comparison of single-step and multistep generated QR-code samples from the same low-count conditioning measurements.
Capturing high-quality images from only a few detected photons is a fundamental challenge in computational imaging. Single-photon avalanche diode (SPAD) sensors promise high-quality imaging in regimes where conventional cameras fail, but raw quanta frames contain only sparse, noisy, binary photon detections. Recovering a coherent image from a burst of such frames requires handling alignment, denoising, and demosaicing (for color) under noise statistics far outside those assumed by standard restoration pipelines or modern generative models. We present an approach that adapts large text-to-image latent diffusion models to the photon-limited domain of quanta burst imaging. Our method leverages the structural and semantic priors of internet-scale diffusion models while introducing mechanisms to handle Bernoulli photon statistics. By integrating latent-space restoration with burst-level spatio-temporal reasoning, our approach produces reconstructions that are both photometrically faithful and perceptually pleasing, even under high-speed motion. We evaluate the method on synthetic benchmarks and new real-world datasets, including the first color SPAD burst dataset and a challenging Deforming (XD) video benchmark. Across all settings, the approach substantially improves perceptual quality over classical and modern learning-based baselines, demonstrating the promise of adapting large generative priors to extreme photon-limited sensing. Code at https://github.com/Aryan-Garg/gQIR.
Single-photon avalanche diodes (SPADs) record light as a discrete stream of individual detections. The signal is stochastic. Its statistical structure depends on the sensor's operation mode: binary detection in fixed bins, timestamped detection in fixed bins, or free-running timestamped detection. We derive the likelihood score function for each of these three passive modes. From this single object, stem both fundamental limits of recovery (Cramer-Rao bounds) and practical recovery algorithms based on diffusion posterior sampling. The paper further generalizes fundamental limits to Bayesian Cramer-Rao lower bounds. This generalization makes use of a learned approximation of the score function of signal priors. In prior art, analyses and diffusion-based reconstruction for SPAD data have treated individual modes in isolation. Our unified treatment shows a qualitative high-flux gap between modes: binary counts saturate exponentially, while timestamped modes degrade only linearly. We further extend diffusion posterior sampling, previously restricted to binary SPAD data, to a full timestamped case using the suitable score function. We demonstrate experimentally that matching the score to the operation mode is beneficial for high-fidelity reconstruction. By tying the recovery bounds and diffusion to the score function, this work aims to establish a common foundation for both asking what is recoverable in single-photon sensing, and building methods that approach the bound.
Lior Dvir, Nadav Torem, Mohit Gupta +1
Viterbi Faculty of Electrical and Computer Engineering, Technion-Israel Institute of Technology, Haifa 3200003, Israel · Department of Computer Sciences, University of Wisconsin-Madison, Madison, WI 53706, USA
Single-photon cameras based on single-photon avalanche diode (SPAD) technology are gaining popularity for 3D sensing, thanks to their extreme sensitivity and time resolution. There are two key challenges with single-photon cameras that limit their widespread use: (i) they suffer from non-linear distortions called ''pile-up'' when operated in high-photon-flux conditions, and (ii) they generate a large volume of raw photon data, creating a severe data bottleneck at each sensor pixel. In this work, we show that while compressive capture techniques successfully mitigate data transfer challenges, they exacerbate the effects of dead-time distortion because they fail to retain sufficient information about the photon detection history to allow post-processing pile-up correction via existing methods. We propose a new computational-imaging method that combines free-running capture with an analysis-by-synthesis software pipeline to mitigate pile-up distortions. Our results with hardware emulations and full-scene and single-pixel simulations show that our method can reliably capture scene distance and reflectance over a wide range of illumination conditions. Our work will enable high-resolution SPAD cameras that are severely bandwidth-constrained to operate in real-world high-flux scenarios.