cs.CVAug 27, 2026
SaveQuanta Perception as Probabilistic Events
Organizations: Department of Computer Sciences, University of Wisconsin–Madison, Wisconsin, United States.
Abstract
Autonomous systems rely on extracting information from light, yet remain brittle in extreme environments, from nighttime navigation to high-speed robotics. Conventional sensors aggregate photons over fixed exposures, imposing trade-offs between sensitivity, dynamic range, and temporal resolution that degrade perception when photons are scarce or dynamics are rapid. Quanta sensors detect individual photons, but their streams exceed real-time compute and latency budgets by orders of magnitude. Here we introduce , a computational primitive for real-time quanta perception from individual photon detections. By computing the posterior over the time since the last intensity change, we represent photon streams as recursive belief states. Rather than fixed-threshold event-camera triggers, this recursive Bayesian formulation yields three low-latency signals: motion-adaptive scene flux, high-fidelity activity maps, and entropy-based perceptual uncertainty. This representation enables perception in extreme conditions, including pose estimation of a running person at 0.05 lux---without retraining vision models. Our approach processes input streams exceeding 50{,}000 quanta frames per second on commodity GPU hardware---yielding kilohertz-scale outputs up to four orders of magnitude faster than state-of-the-art quanta reconstruction baselines, even for megapixel arrays. By replacing frame reconstruction with direct probabilistic inference over photon streams, this work bridges photon-counting quanta sensing with robotic vision.
Figures & tables
Figure 1: From photons to perception using probabilistic events. (A) Conventional image sensors function as photon buckets , integrating flux over coarse exposure windows (tens of milliseconds): a mechanism that fundamentally limits performance in low-light and high-dynamic-range scenarios. (B) Conversely, quanta image sensor pixels act as photon teaspoons , registering individual photon arrivals. While this detection capability allows quanta cameras to operate across extreme illumination and motion regimes, it comes at the cost of massive data volumes. Consequently, existing reconstruction techniques remain offline methods that impose prohibitive compute, readout, and latency demands, precluding real-time perception. (C) To bridge this gap, we introduce probabilistic event cameras, a computational sensing layer that operates directly on photon streams. Unlike standard event cameras that emit deterministic thresholded events, probabilistic events maintain a posterior distribution over change intervals, yielding representations that encode both signal and uncertainty—temporal stability, entropy shifts, and motion-aware flux aggregates. (D) This representation enables motion-aware photon aggregation and uncertainty estimation at low latency and constant memory, transforming quanta cameras into plug-and-play sensors for extreme computer vision. Using real acquisition from a 1-megapixel color quanta sensor , we demonstrate perception results for edge detection (Canny ( 16 ) ), optical flow estimation (RAFT small ( 17 ) ), monocular depth estimation (DepthAnything-v2-s ( 18 ) ), and instance segmentation (YOLO-v8 nano ( 19 ) ).
Figure 2: Probabilistic event cameras produce low-latency, uncertainty-aware representations. (A) We treat photon arrival as multiple hypothesis inference, evaluating run length: the time since the last abrupt change. (B) From the run-length posterior, we derive entropy shifts and per-pixel temporal stability maps. The latter drives a motion-aware flux aggregate that balances the noise–blur trade-off. Tracking a specific pixel (cyan cross) shows the posterior peaking at high stability before a ping pong ball crosses (t=17 ms), spreading across recent changes during occlusion (t=27 ms), and regaining a sharp peak afterwards (t=37 ms). (C) Entropy fluctuations (green: increase, purple: decrease) reveal subtle ego-motion; short integration preserves high-frequency details (snapping tape); and our model robustly tracks objects despite extreme photon sparsity (low-light juggling). We blur faces for privacy. (D) Sustaining an input throughput of 51 kqFPS and an output throughput of 1650 FPS at 256×512, our approach operates orders of magnitude faster than reconstruction and denoising baselines, enabling real-time downstream perception ( Fig. 3 ).
Figure 3: Low-latency quanta perception in the wild. Motion-aware flux aggregates produced by probabilistic events serve as plug-and-play inputs to standard vision pipelines, enabling robust perception in regimes where conventional cameras are constrained by the noise–blur trade-off. We evaluate tasks spanning classical vision and modern deep networks without retraining or task-specific tuning. (A–D) High-speed regimes: edge detection at 4,000 FPS (Canny ( 16 ) ); tracking the swinging spheres of a Newton’s cradle at 550 FPS (MOSSE ( 23 ) ); QR-code decoding at 165 FPS; and optical flow on a rapidly spinning prize wheel at 140 FPS (RAFT ( 17 ) ). (E–F) Low-light regimes: nighttime vehicle detection at 400 FPS (RT-DETR ( 24 ) ); and human pose estimation of a running person in a dim hall at 1,000 FPS (YOLOv8 ( 19 ) ). FPS numbers indicate end-to-end throughput , including the time needed to compute the probabilistic event representation and run subsequent inference. Each row shows a representative single quanta frame , the extent of motion under a conventional 25 ms integration, the probabilistic representation (color indicates the inferred temporal stability / adaptive integration timescale), and the resulting perception output . Throughputs measured while processing quanta camera streams (edge detection and human pose estimation utilize the faster probabilistic event variant, capable of output throughputs up to 9,000 FPS).
Figure 4: Comparison with offline quanta-reconstruction on low-light scenes. (A) We consider low-light scenes in which the detector (SwissSPAD2 sensor ( 57 ) ) registers, on average, less than photons per pixel (PPP)—corresponding to less than lux illumination. Summing photon detections incurs the noise–blur trade-off at the frame level; even when this trade-off is balanced carefully, we find that virtual exposures (summed frames) do not facilitate computer-vision tasks. (B) An offline quanta reconstruction technique (Quanta Burst Photography ( 9 ) ) reconstructs high quality images by performing non-causal align-and-merge operations on the raw binary sequence, leading to good task performance, but with very high latencies. (C) In comparison, the probabilistic event camera balances the noise–blur trade-off at the pixel level, running significantly faster (about 10,000 faster); insets visualize the temporal stability maps. Annotated frame rates indicate the output representation rate supplied to downstream perception.
Figure 5: Probabilistic events compared to specialized sensing modalities. We benchmark our computational approach applied to a quanta sensor against a conventional DSLR (Canon 77D, 30 FPS), high-speed camera (Photron Infinicam, 200 FPS), low-light camera (Bosch DINION), and event camera (Prophesee EVK4). We tuned conventional sensor and downstream inference parameters for near-optimal performance. Each row shows conventional camera capture (showing failure), a single quanta frame, and our probabilistic event representation; insets show temporal stability maps. (A) City Nighttime: The DSLR capture is insufficient for object detection (RT-DETR ( 24 ) ). Our method enables car detection at 100 Hz. (B) High-Speed QR Decoding in Low Light: The Infinicam fails to capture usable data at 200 FPS. Its compression quantizes the low signal-to-noise ratio (SNR) signal to zero, producing dark frames and decoding failure. In contrast, our method decodes the QR code in 6 ms. (C) Low-Light Depth: The security camera suffers motion blur from ego-motion. Probabilistic events retain structural edges (motion-aware aggregate in inset), enabling depth estimation at 100 Hz (DepthAnythingV2-s ( 18 ) ). (D) Segment Anything: Event cameras lack intensity in static regions, causing segmentation failure (SAM ( 29 ) ) despite E2VID reconstruction (inset). Our approach captures full context, enabling accurate segmentation. Faces blurred for privacy.
Figure 6: Limitations in signal-starved regimes. (A) We evaluate performance in an extreme low-light scenario (0.02 photons per pixel) featuring rapid motion: a picture of a Mandrill mounted on a rotating CPU fan (sensor acquisition from ( 37 ) ). (B) We benchmark against Bit2Bit (top) ( 37 ) to define a performance ceiling. This method employs inference-time training to perform self-supervised denoising on the test data itself; while this recovers sharp details, the optimization is computationally exhaustive. We report its batch throughput as FPS, corresponding to roughly 300 s per output frame after optimization; because the optimization is performed over the sequence, the latency before outputs are available is a few hours. (C) In contrast, our probabilistic events operate in real-time (0.62 ms), representing a speedup of five orders of magnitude. Here, we visualize the resulting temporal stability map, entropy shifts, and flux aggregate. The trade-off for this ultra-low latency is the inability to employ iterative optimization or non-causal alignment. Consequently, our method struggles to distinguish rapid changes from shot noise, resulting in motion blur. Nevertheless, the motion-aware aggregation preserves discernible structure; notably, Segment Anything 2 (SAM 2, [ 32 ] ) successfully tracks the object despite the aggregate being an imperfect reconstruction.
Extended Data Figure 1: Comparing probabilistic events’ motion-aware aggregation to quanta-image restoration. We compare against burst restoration using QUIVER ( 11 ) , video denoising using FastDVDnet ( 58 ) , and online synthesis using Streaming EMA ( 21 ) . We show three imaging scenarios: (A) camera motion in ambient lighting, (B) fast object motion in indoor lighting, and (C) moderate object motion in low light. Faces have been blurred to protect privacy. We also show a single binary frame of the quanta sensor acquisition and the extent of motion in 30 ms. Our approach, despite being computationally much lighter, provides output fidelity comparable to, and in certain cases, better than, the considered baselines. Insets show the temporal stability maps computed by probabilistic events: dynamic regions are assigned shorter windows to minimize motion blur. In challenging conditions involving fast motion or low light, probabilistic events better preserve high-frequency details (e.g., numbers and markings on the measure tape) that are oversmoothed by learned baselines. (D) Our approach (red bars) is the only method capable of exceeding 1,000 FPS, operating – faster than the considered baselines; we illustrate this using a stacked bar chart. All runtime measurements assume desktop-grade computing resources: an Intel Xeon Silver CPU fitted with an NVIDIA 4090 GPU accelerator.
| Method | Output FPS | PPP= | PPP= | ||
| PSNR (dB) | SSIM | PSNR (dB) | SSIM | ||
| Baselines | |||||
| Virtual exposure | 10000 | 11.0 | 0.068 | 19.3 | 0.212 |
| Exposure stack ( 21 ) | 80 | 28.8 | 0.809 | 30.6 | 0.869 |
| FastDVDNet ( 58 ) | 50 | 24.1 | 0.528 | 33.5 | 0.896 |
| NAFNet ( 59 ) | 20 | 22.2 | 0.448 | 27.0 | 0.774 |
Extended Data Table 1: Image quality comparisons on the i2k high-speed dataset ( 11 ) .
Extended Data Figure 2: Qualitative image-quality comparisons on the i2k high-speed dataset. We show two representative examples from the i2k benchmark ( 11 ) , at moderate light ( PPP; A) and low light ( PPP; B), complementing the aggregate PSNR, SSIM, and throughput results in Extended Data Tab. 1 . Each example compares the clean ground truth with exposure stacks ( 21 ) , FastDVDNet ( 58 ) , NAFNet ( 59 ) , QUIVER ( 11 ) , gQIR ( 38 ) , Quanta Burst Photography ( 9 ) , and the proposed probabilistic-event representation. The image- and video-denoising baselines, NAFNet and FastDVDNet, rely on limited temporal context and degrade markedly at very low photon counts, whereas quanta-specific restoration methods exploit the binary photon stream more effectively. Among these methods, QBP produces visually strong reconstructions, while QUIVER benefits from end-to-end training and achieves competitive image-quality metrics. Probabilistic events provide comparable PSNR and SSIM to state-of-the-art quanta restoration methods at moderate light while operating up to four orders of magnitude faster in this benchmark. At low light, reconstruction-first methods recover higher-SNR images, but probabilistic events preserve sufficient scene structure for downstream perceptual inference at kilohertz output rates, as illustrated in Fig. 3 and quantified for depth estimation and optical flow in Extended Data Tab. 2 . Values below each image report PSNR and SSIM relative to the ground truth.
| Avg. PPP | Motion | Method | Image quality | Monocular depth (DepthAnything-v2) | Optical flow (RAFT) | |||
| PSNR | SSIM | Relative error | Threshold accuracy | Endpoint error | Bad-pixel rate | |||
| 0.025 20 fewer photons | 1 | Reference intensity | 1.000 | 0.067 | 0.940 | 1.08 | 0.053 | |
| Virtual exposure | 9.01 | 0.273 | 0.184 | 0.704 | 21.15 | 0.516 | ||
| QBP | 25.93 | 0.945 | 0.086 | 0.916 | 0.97 | 0.069 | ||
| QUIVER | 25.38 | 0.937 | 0.159 | 0.763 | 5.50 | 0.202 | ||
| Probabilistic events | 29.04 | 0.970 | 0.081 | 0.917 | 1.37 | 0.122 | ||
Extended Data Table 2: Evaluating task performance on the VisionSim dataset.
Extended Data Figure 3: Controlled depth and flow evaluation on the VisionSim dataset. Representative examples from a controlled task-performance study at – fewer photons and faster motion. Columns compare fixed-window virtual exposures, reconstruction-first perception using quanta-burst and QUIVER reconstructions, and probabilistic events. Rows in panels A and B show the input intensity representation, predicted monocular depth, and predicted optical flow. Short virtual exposures preserve motion but are photon-starved; long exposures reduce noise but introduce motion blur. Reconstruction-first methods are strong align-and-merge baselines, although residual artifacts can propagate to downstream estimates. Probabilistic events use causal, motion-adaptive aggregation rather than full reconstruction, yet match reconstruction-first depth and flow at fewer photons. At fewer photons, depth remains robust, while optical flow is more sensitive to residual noise and extreme motion.
Extended Data Figure 4: Balancing noise and blur with probabilistic events. Standard fixed integration windows ( eq. 2 ) impose a rigid global trade-off: a single raw quanta frame (A) or short integration (B, 32 frames) results in significant noise, whereas extended integration (C, 8000 frames) induces motion blur. In contrast, probabilistic events utilize adaptive exponential smoothing driven by estimated temporal stability. Depicted are the temporal stability maps (D), shifts in distribution entropy (E, green and purple indicating increase and decrease, respectively), and the motion-aware flux aggregate (F). While this Bernoulli formulation mediates the noise–blur trade-off more effectively than fixed windows, it neglects spatial information; consequently, it remains less responsive to abrupt changes, as evidenced by the blur in the insets.
Extended Data Figure 5: Incorporating spatial information in estimating temporal stability maps. (A) On a sequence simulated from the i2k high-speed video dataset, the purely pixel-wise binomial instantiation treats locations independently, resulting in laggard responses (increased motion blur). (B) Minimum filtering reduces blur by enforcing smaller temporal stability values if any pixel in a local patch detects change, but does so at the cost of excess noise, as it remains agnostic to scene content. (C–E) In contrast, by explicitly modeling linear spatial features alongside binomial counts, we selectively increase pixel responsivity in textured regions while maintaining robust temporal denoising elsewhere. Features attuned to scale and orientation (e.g., Log-Gabor, E) offer superior selectivity, while simpler features (e.g., gradients, C) provide a computationally lighter alternative. For each instantiation, we show the motion-aware flux aggregate and the corresponding temporal stability map.
Extended Data Figure 6: Benchmarking input and output throughput on a desktop-grade GPU (RTX 4090) and an embedded edge platform (Jetson Orin Nano). Input throughput denotes the binary quanta-frame rate processed by the method, while output throughput denotes the rate at which probabilistic-event representations or reconstructed frames are emitted. (A) On the RTX 4090, the Gradient ( fast variant) and Log-Gabor ( robust variant) configurations support a megapixel quanta camera at 41,000 and 6,700 qFPS, respectively, corresponding to 1,286 and 209 FPS output throughput. (B) On the Jetson Orin Nano, the Binomial and Gradient variants sustain and qFPS at 1 MPixel ( and FPS output throughput), confirming viability on power-constrained edge platforms. (C) On the RTX 4090, reconstruction-first quanta baselines such as QBP ( 9 ) , QUIVER ( 11 ) , and gQIR ( 38 ) operate at sub-Hz to low-Hz output rates. OOM denotes out of (GPU) memory here. Probabilistic events run up to four orders of magnitude faster across the tested resolutions and variants.
Extended Data Figure 7: Denoising using spatially-varying Wiener filtering. We use the temporal stability maps to denoise based on the amount of integration: lesser spatial denoising in regions with strong temporal integration. Going from insets on the left to the right, the amount of motion decreases, the exposure times are larger, and we perform lesser spatial denoising.
Extended Data Figure 8: Rate-distortion comparison between event-based and video codec compression. We compare the compression efficiency of transmitting the motion-aware flux aggregate (after the proposed spatial denoising), using a sparse event-based scheme versus a GPU-accelerated H.264 video codec. The ground truth scene is shown alongside examples of both encoding methods at high compression ratios. The event-based encoding (threshold 0.06) achieves a 50× compression ratio at 21 dB PSNR but suffers from distinct laggard artifacts, visible as ghosting on the signboard during camera panning. In contrast, the H.264 encoding (constant rate factor, CRF, 32) achieves a significantly higher 2660 compression ratio at 27 dB PSNR. While the video codec exhibits compression artifacts at this extreme CRF setting, such as the loss of high-frequency details leading to a slightly washed-out appearance, it preserves structural integrity far better than the event-based encoding.
Extended Data Figure 9: Probabilistic event representations in color. Color imaging with quanta sensors requires addressing the coupled challenges of demosaicking and denoising. Specifically, we must aggregate sufficient photons to facilitate reliable debayering without incurring motion blur. We demonstrate this noise–blur trade-off on 1-megapixel color quanta acquisition across three scenes: rapid camera swiveling, high-speed falling objects, and nighttime driving. (A–B) We show virtual exposures corresponding to a short exposure time (suffering from noise, A) and a long exposure time (suffering from motion blur, B). (C) We also display our probabilistic event representation, visualizing per-pixel temporal stability and entropy changes, where purple and green indicate per-pixel decreases and increases in entropy, respectively. (D) By aggregating photons till the per-pixel noise–blur limit, our demosaicked motion-aware aggregate—which includes a Wiener denoising step ( Extended Data Fig. 7 )—yields the high-fidelity color imaging.
Extended Data Figure 10: Evaluating temporal stability estimation. We compare our probabilistic events (Log-Gabor) against the theoretical limit of a conventional sensor (Oracle Global Shutter) and ground truth. Histograms show error in stops; positive values indicate over-exposure (motion blur, marked in red), while negative values indicate under-exposure (noise, marked in blue). (A) In scenes with uniform ego-motion, optimal integration windows are spatially correlated. The oracle global shutter approximates the ground truth reasonably well (1.01 stops blur RMSE), though our method tightens the distribution (0.58 stops blur RMSE). (B) In scenes with distinct velocities ( e.g. , shuffling cards), the optimal distribution is not unimodal. The global shutter is forced into a compromise, yielding high error (1.68 stops blur). Our method adapts locally, assigning short integration windows to the falling cards and long integration windows to the background, reducing blur RMSE to 0.26 stops. (C) Aggregate error across the dataset. While minimum filtering (blue dashed line) eliminates blur, it causes excessive noise (skewing left). Our Log-Gabor approach balances this trade-off, outperforming the oracle global shutter in both blur and noise metrics.
Extended Data Figure 11: Temporal stability estimation as a function of light level and summed-frame count. We visualize the effect of summed-frame count and photon rate on probabilistic-event inference for a single example sequence, complementing the aggregate error distributions in Extended Data Fig. 12 . Columns vary the number of binary quanta frames summed before inference, , while rows vary the incident photon rate, from the original light level to fewer photons and fewer photons, corresponding to average photon detection rates of PPP and PPP, respectively. At low , estimates remain temporally responsive but are evidence-limited, especially at reduced light levels. Increasing improves local evidence and stabilizes the estimate, but overly long temporal aggregation eventually introduces blur in regions with scene or camera motion. This example illustrates the same noise–blur trade-off observed across the dataset (shown in Extended Data Fig. 12 ): lower light requires more temporal aggregation, but the optimal summed-frame count is bounded by the temporal stability of the scene.
Extended Data Figure 12: Temporal stability estimation across light levels and summed-frame counts. We evaluate aggregate error distributions while varying the number of binary quanta frames summed before probabilistic-event inference, (see eqs. 2 and 13 ), and the incident photon rate. Columns sweep from 16 to 160, while rows show the original simulated light level, fewer photons, and fewer photons. Histograms report error in stops; positive values indicate over-exposure and motion blur (red), while negative values indicate under-exposure and noise (blue). Values below each histogram report blur RMSE and noise RMSE, respectively. As light decreases, larger summed-frame counts reduce noise but eventually increase blur, exposing the operating trade-off between photon aggregation and temporal responsiveness.
Extended Data Figure 13: Motion-aware aggregation maximizes Signal-to-Noise Ratio (SNR) relative to global shutter baselines. (A) Analytical upper bound on image SNR as a function of the expected photon count per frame, , and the integration window, (Eq. 22 ). (B) Performance comparison on the VisionSIM dataset [ 25 ] . We plot the SNR of the optimal per-pixel integration window, , against the optimal global shutter baseline, . The global exposure is constrained by the maximum scene velocity to prevent blur, with a minimum duration of s. Points falling below the gray line indicate pixels with dynamics too fast for this minimum exposure, resulting in inevitable motion blur. (C) Histogram of the relative increase in integration time, , demonstrating that static background regions integrate light significantly longer than the global bottleneck permits. (D) The corresponding distribution of SNR improvement (in dB), quantifying the benefit of motion-aware aggregation.