Machine vision with small numbers of detected photons per inference
Authors: Shi-Yuan Ma, Jérémie Laydevant, Mandar M. Sohoni, Logan G. Wright, Tianyu Wang, Peter L. McMahon
Organizations: School of Applied and Engineering Physics, Cornell University, Ithaca, NY 14853, USA · Present address: Research Laboratory of Electronics, Massachusetts Institute of Technology, Cambridge, MA 02139, USA. · NTT Physics and Informatics Laboratories, NTT Research, Inc., Sunnyvale, CA 94085, USA · Present address: Department of Applied Physics, Yale University, New Haven, CT 06511, USA · Present address: Department of Electrical and Computer Engineering, Boston University, Boston, MA 02215, USA.
Machine vision, including object recognition and image reconstruction, is a central technology in many consumer devices and scientific instruments. The design of machine-vision systems has been revolutionized by the adoption of end-to-end optimization, in which the optical front end and the post-processing back end are jointly optimized. However, while machine vision currently works extremely well in moderate-light or bright-light situations -- where a camera may detect thousands of photons per pixel and billions of photons per frame -- it is far more challenging in very low-light situations. We introduce photon-aware neuromorphic sensing (PANS), an approach for end-to-end optimization in highly photon-starved scenarios. The training incorporates knowledge of the low photon budget and the stochastic nature of light detection when the average number of photons per pixel is near or less than 1. We report a proof-of-principle experimental demonstration in which we performed low-light image classification using PANS, achieving 73% (82%) accuracy on FashionMNIST with an average of only 4.9 (17) detected photons in total per inference, and 86% (97%) on MNIST with 8.6 (29) detected photons -- orders of magnitude more photon-efficient than conventional approaches. We also report simulation studies showing how PANS could be applied to other classification, event-detection, and image-reconstruction tasks. By taking into account the statistics of measurement results for non-classical states or alternative sensing hardware, PANS could in principle be adapted to enable high-accuracy results in quantum and other photon-starved setups.
Figures & tables
Figure 1: Detection bottleneck in optical sensing and photon-aware neuromorphic sensing (PANS) under limited photon counts . A , Conceptual optical sensing pipeline. An object (illustrated by a cat) interacts with a probe signal (e.g., light) shaped by a configurable optical front end (green), which may operate in active (controlled illumination) or passive (incoming-signal modulation) modes. The detector then converts the incident optical signal into digital data via single-photon detection (SPD). When photon budgets are highly limited, this conversion presents a detection bottleneck with significant information loss (lower schematic), which cannot be recovered by subsequent digital processing. A digital back end (e.g., a post-processing neural network) extracts task-relevant information from the detected data. B , Direct imaging vs. PANS under limited photons. Top: in a conventional direct-imaging pipeline, photon-limited image frames (shown as repeated realizations across independent trials 1, 2, 3) exhibit strong shot noise, making downstream inference challenging (e.g., cat vs. dog). Bottom: PANS introduces a parameterized optical front end that transforms the optical field before detection, producing photon-efficient feature measurements; the front end and digital back end are jointly optimized end-to-end through the stochastic detection bottleneck . C , Photon detection bottleneck. With mean photon energy λ , SPD produces a discrete stochastic digital readout a . PANS faithfully models this stochastic forward propagation and applies gradient estimation to enable estimated backpropagation (backprop) through the detection bottleneck, allowing end-to-end optimization under photon-budget constraints.
Figure 2: Active photon-aware neuromorphic sensing (PANS) demonstrated on FashionMNIST object classification. A , Direct imaging (conventional approach). Uniform illumination probes the object, and single-photon detectors directly capture an image frame with dobj pixels. Nillu and Ndet denote average total illumination and detection photon budgets, respectively. The example object image (a sneaker) is taken from the FashionMNIST dataset. B , Image frames degrade with decreasing Ndet (denoted above each column) for a pullover (top) and a shirt (bottom). Frames become increasingly noisy as the photon budget decreases. C , Quantifying information loss at the detection bottleneck. As Ndet decreases (left to right), three metrics decline: mutual information with labels (top), Fisher discriminant ratio (FDR; middle), and the test accuracy using a convolutional neural network (bottom). D , Active PANS (our approach). df illumination patterns are projected onto the object, producing a df -dimensional feature vector through single-photon detection (see Fig. 3 and Appendix 12 for details of the experimental protocol). E , Experimental results on FashionMNIST. Top: confusion matrices at two different photon budgets. Bottom: Test accuracy vs. Ndet (left) and Nillu (right). Nillu is the total illumination incident on the object (uniform for direct imaging; sum of pattern intensities for structured illumination; Appendix 5). Red markers: active PANS experiment (mean ± std over 30 trials per image) for df=3,4,6,10,16,24,32 ; light red shade: corresponding simulation (mean ± 3 std). Blue curve: direct imaging baseline (from C ). Green curve: conventional E2E without photon-aware modeling (non-PA E2E; Appendix 8C). F , 2D t-SNE [ 98 ] visualization comparing feature distributions. Active PANS (red boxes) versus direct imaging (blue boxes) at different Ndet values, with test accuracies shown.
Figure 3: Stochastic single-shot inference and experimental validation of active PANS. A , Stochastic single-shot inference under extreme photon constraints. df learned illumination patterns {wi}i=1df are sequentially projected onto an object with transmission x , each producing a binary single-photon detection (SPD) readout. Because detection is highly stochastic at these photon levels, nT independent trials on the same object yield different feature vectors (bottom), each processed by the digital back end. Despite this trial-to-trial variability, the system consistently identifies the correct class across trials. The aggregate output distribution over nT inferences (right) reflects the classification confidence (Appendix 13). B–C , Experimental classification accuracy (red, mean ± std over nT=30 trials) versus number of illumination patterns df for FashionMNIST ( B ) and MNIST ( C ) under single-shot operation. The FashionMNIST data correspond to the red markers in Fig. 2 E, here plotted against df . Simulation results (light red band, mean ±3 std) show close agreement with experiment. Annotations indicate the average total detected photon budget per inference Ndet at selected df (Appendix 13).
Figure 4: Proposed real-time image sensing with active photon-aware neuromorphic sensing (PANS) in simulation. A , Conceptual wavelength-multiplexed implementation for flow-cytometric cell sorting. Multiple static illumination patterns at distinct optical wavelengths (illustrated as w1,w2,w3 with different colors) are applied simultaneously; wavelength demultiplexing routes each channel to a dedicated photon counter, producing activations (a1,a2,a3) in parallel for real-time digital processing. B , Simulated test accuracy for cell-organelle classification versus total detected photons Ndet under different detector dark-count rates (DCRs), compared with an ideal direct-imaging baseline. C , Example real-time sequence. Left: representative frames as a cell traverses the illumination field (top to bottom). Right: corresponding model outputs over time across classes (Mem.: membrane; Nuc.: nucleolus; Mit.: mitochondria; Null: no cell present). D , Barcode identification task. The illumination field spans a 10-bar window (red box); the goal is to decide whether the target subsequence “1010” appears at any position. E , Simulated test accuracy for barcode identification versus Ndet under multiple DCR values. Direct imaging accounts only for ideal shot noise (no dark counts or additional detector noise), highlighting the robustness of active PANS under realistic counting noise.
Figure 5: Diagram and applications of passive photon-aware neuromorphic sensing (PANS) in simulation. A , Passive PANS vs. direct imaging for sensing images transmitted through a scattering multimode fiber (MMF). Input images propagate through the MMF, emerging as speckle patterns that scramble spatial information. In passive PANS, speckles pass through a passive optical encoder before detection; in direct imaging, speckle frames are captured directly at the image plane with Ndet photons per frame. Both schemes use post-processing neural networks for classification or reconstruction. B , Classification accuracy on MNIST speckle images versus Ndet , with dark count rates (DCRs) of 1%, 5%, and 10% for passive PANS. Direct imaging (blue curve) simulated using only ideal shot noise. Inset: direct imaging accuracy at higher Ndet . Red markers show passive PANS with df=4,5,6,8,10,16,24,32 . C , Average structural similarity index (SSIM) of reconstructed images from scattered speckles, evaluated with different DCRs. Passive PANS data points: df=4,6,8,10,32,48,64 . D , Example images showing original MNIST digits (top), corresponding speckle patterns (middle), and reconstructed images (bottom) from passive PANS ( df=64 , DCR = 1%). E , Transient event detection: fleeting objects in noisy backgrounds are identified in a monitored scene (left). Right: test accuracy vs. Ndet . Passive PANS: df=2,4,5,6,8,10 . F , Tissue blood flow detection via speckle contrast imaging. G , Compact nebula classification. H , Optical fiber end-face contamination inspection. Insets show direct imaging performance at higher photon counts.
Appendix figures & tables36 assets
Supplementary material from the paper’s appendix.
Appendix
Figure Fig. A1: Taxonomy of optical sensing approaches. A flowchart illustrating the hierarchy of algorithmic approaches in optical sensing. Starting from the top, if no pre-detection operation is applied, the system performs direct imaging (blue). If an optical encoding operation is applied, the approach branches based on whether task-specific training is used: fixed transformations (white) use predetermined configurations such as downsampling or compressive sensing patterns, while end-to-end optimization learns task-specific optical parameters jointly with the digital back-end. End-to-end methods further divide based on whether photon-aware modeling is employed: conventional end-to-end methods (green) do not explicitly model stochastic photon detection, whereas photon-aware neuromorphic sensing (PANS, red) faithfully represents the probabilistic, discrete nature of the detection process during optimization. The color coding agrees with Fig. 2 in the main text. Shaded background regions group approaches by the level of sophistication in the optical front-end design.
Approach
Optical operation
Task-optimized
Photon-aware
Direct imaging
×
—
—
Fixed optical transformation
✓
×
—
Conventional (non-PA) E2E
✓
✓
×
Photon-aware neuromorphic sensing (PANS)
✓
✓
✓
Appendix
Table Table A1: Taxonomy of sensing approaches based on optical front-end design choices.
Figure Fig. A2: Diagram of the optical front end operation in active sensing. The light source provide a spatially distributed illumination pattern and project to the object, the object then attentuate the illuminatino light, equivalent to element-wise multiplication. then the optical energy was focused on to a detector to be measured. the output is equivalent to a dot product process.
Approach
Photon level ( η )
K -shot
df
Direct imaging
Applicable
≡η
N/A
Conventional optical encoding
Applicable
≡η
Applicable
PANS
Limited
Practical
Optimized
Appendix
Table Table A2: Photon budget scaling methods and their applicability across sensing approaches.
Parameter
Value
Optimizer
Adam
Learning rate (optical front-end, conv layers)
10−3
Learning rate (FC layers)
10−4
Batch size
128
Maximum epochs
500
Training shots per activation ( K )
1
Appendix
Table Table A3: Baseline training configuration for active PANS on FashionMNIST.
Figure Fig. A3: Training dynamics under different illumination regularization α for df=10 . Each panel shows training accuracy (blue), test accuracy (orange), and total illumination photon budget Nillu (green dashed, right axis) versus training epoch. The df and α values are indicated above each panel. All models use the training configuration in Table A3 with only α varied.
Figure Fig. A4: Training dynamics under different illumination regularization α for df=16 . Each panel shows training accuracy (blue), test accuracy (orange), and total illumination photon budget Nillu (green dashed, right axis) versus training epoch. The df and α values are indicated above each panel. All models use the training configuration in Table A3 with only α varied.
Figure Fig. A5: Training dynamics under different illumination regularization α for df=24 . Each panel shows training accuracy (blue), test accuracy (orange), and total illumination photon budget Nillu (green dashed, right axis) versus training epoch. The df and α values are indicated above each panel. All models use the training configuration in Table A3 with only α varied.
Figure Fig. A6: Training dynamics under different illumination regularization α for df=32 . Each panel shows training accuracy (blue), test accuracy (orange), and total illumination photon budget Nillu (green dashed, right axis) versus training epoch. The df and α values are indicated above each panel. All models use the training configuration in Table A3 with only α varied.
Figure Fig. A7: Evolution of learned illumination patterns during training. Each panel displays the df=10 patterns from a model trained with α=10−4 , shown at four stages with training progressing from a to d . For visualization, patterns are normalized such that the mean pixel value across all patterns equals 1 (corresponding to a total of 784 per pattern); the number above each pattern shows its individual mean. The color bar indicates pixel value range. Early in training ( a ), patterns are spatially diffuse with moderate intensity variation. Through training ( b – d ), patterns become increasingly sparse and localized, concentrating intensity into small regions while the maximum value expands from ∼ 18 to ∼ 100 times the mean (color bar), reflecting efficient photon allocation to discriminative spatial locations.
K=1
K=2
K=3
K=5
df=3
54.46±1.31
55.20±1.67
55.88±1.35
56.68±1.44
df=4
65.11±2.07
66.26±1.90
67.20±1.55
68.38±1.59
df=6
68.37±2.06
70.15±1.84
71.33±1.50
71.29±1.32
df=10
74.31±1.84
77.55±1.32
77.53±1.32
78.48±1.16
df=16
78.30±1.96
81.42±1.48
81.85±1.36
82.34±1.32
df=24
80.35±1.78
81.98±1.39
82.22±1.26
82.35±1.16
Appendix
Table Table A4: Test accuracy (%) on FashionMNIST for different feature dimensions df and shot counts K . Each entry shows mean ± standard deviation over 100 trials with 200 test images.
Figure Fig. A8: Multi-shot evaluation validates df scaling over K scaling. Test accuracy versus total number of detections ( df×K ). Different curves correspond to models trained with different feature dimensions df∈{3,4,6,10,16,24,32} , as indicated in the legend. Points along each curve correspond to increasing K (1, 2, 3, 5) from left to right. At the same total detection budget, larger df with smaller K consistently achieves higher accuracy than smaller df with larger K . Dot points show mean values computed over 100 repetitions; error bars indicate standard deviation.
Figure Fig. A9: Effect of photon level scaling η . Test accuracy versus photon scaling parameter η for trained PANS models with different feature dimensions df∈{3,4,6,10,16,24,32} , as indicated in the legend. The horizontal axis uses logarithmic scale spanning η∈[10−2,102] . All models achieve peak accuracy near η=1 (the trained operating point) and degrade for both lower and higher η values. Dot points show mean values computed over 100 repetitions; error bars indicate standard deviation.
Figure Fig. A10: Robustness to dark counts. Test accuracy versus dark count rate (expected dark counts per detector per measurement) for different feature dimensions df∈{3,4,6,10,16,24,32} at η=1 , as indicated in the legend. The horizontal axis uses logarithmic scale. Accuracy degrades gradually up to DCR ∼0.1 and more rapidly at higher DCR values. Dot points show mean values computed over 100 repetitions; error bars indicate standard deviation.
Figure Fig. A11: Shot-noise-limited frames at different photon budgets. Each row shows the same test image captured at decreasing average Ndet (left to right: 106 , 103 , 102 , 10). For Ndet≤103 , three independent samples are shown to illustrate stochastic variation. Class-discriminative features are progressively destroyed by shot noise. At low photon counts, even human observers cannot reliably identify the object category.
Figure Fig. A12: t-SNE visualization of information loss. Each panel shows the procedure and resulting embedding at a different Ndet : each test image (10,000 total) is sampled 3 times at the target photon budget, and all 30,000 resulting vectors are projected via t-SNE. Points are colored by class label. a, Ndet=106 : well-separated clusters indicate preserved class structure. b, Ndet=102 : clusters overlap as shot noise corrupts discriminative features. c, Ndet=10 : class structure is entirely lost; the embedding collapses into an undifferentiated distribution.
Figure Fig. A13: Training compressive sensing classifiers. Training loss (left) and accuracy (right) versus epoch for random binary patterns at different df values. Models are trained on clean (high photon count) measurements.
Figure Fig. A14: Compressive sensing performance. Test accuracy (%) as a function of number of patterns ( df , rows) and photon budget ( Ndet , columns). Performance collapses in the few-photon regime regardless of df .
Figure Fig. A15: Conventional end-to-end optimization results. Test accuracy versus Ndet for different df (light to dark curves). The thick curve shows the envelope—maximum accuracy at each Ndet across all df —corresponding to the green curve in main text Fig. 2E.
K=1
K=2
K=3
K=5
df=4
60.28±2.08
64.15±1.92
65.38±1.49
67.03±1.60
df=9
81.23±2.03
85.59±1.72
86.93±1.27
88.49±1.21
df=16
89.96±1.42
92.32±1.37
93.11±1.11
93.42±0.89
df=25
94.39±1.35
96.35±0.97
96.88±0.69
97.20±0.66
df=36
97.28±0.98
98.20±0.77
98.47±0.68
98.63±0.56
df=64
98.46±0.75
99.20±0.54
99.40±0.45
99.52±0.38
Appendix
Table Table A5: Test accuracy (%) on MNIST for different feature dimensions df and shot counts K . Each entry shows mean ± standard deviation over 100 trials.
Figure Fig. A16: Cell-organelle classification dataset. Images randomly sampled from the test set, representing category: membrane, nucleolus, and mitochondria. Category labels are shown above each image.
Figure Fig. A17: Barcode “1010” identification dataset. Example 10-element barcode segments randomly sampled from the test set. Positive examples (checkmark) contain the “1010” pattern at some position (highlighted in green box). Negative examples (cross) do not contain the target pattern. The pattern can appear at any location within the segment, requiring position-invariant feature detection.
Figure Fig. A18: Image sensing through multimode fiber. Top: schematic showing original image → MMF propagation → speckle pattern → PANS encoder → single-photon detection (SPD) → reconstruction. Bottom: example results showing original MNIST digits (gray shade), corresponding speckle patterns after MMF propagation (yellow shade), and reconstructed images from passive PANS at different configurations ( df=64 with DCR 1%, df=64 with DCR 5%, and df=32 with DCR 1%). Three independent samples per digit demonstrate consistency and variation across stochastic detection events. Spatial structure can be recovered even from apparently random speckle at few-photon levels.
Figure Fig. A19: Transient event detection dataset. Example frames showing background-only (gray border) and transient-present (green border) cases. The mean pixel intensity is annotated above each frame. The similar mean values across both classes demonstrate that simple intensity thresholding cannot distinguish transient events, inferring that detection requires learning the spatial signature of the transient against background fluctuations.
Figure Fig. A20: Tissue speckle imaging dataset. Images randomly sampled from the test set, showing static tissue (gray border) with high speckle contrast versus perfused tissue (green border) with reduced speckle contrast due to blood flow. The mean pixel intensity is annotated above each frame. Classification performance relies on speckle texture analysis—the contrast difference between sharp (static) and blurred (perfused) patterns—rather than intensity measurements.
Figure Fig. A21: Nebula classification dataset. Images randomly sampled from the test set, showing planetary nebulae (green border), characterized by roughly circular or ring-like morphology from expanding stellar ejecta, versus emission nebulae (gray border), showing more irregular structure from illuminated interstellar medium. The mean pixel intensity is annotated above each frame. The overlapping intensity distributions demonstrate that classification requires morphological analysis, not simple photometry.
Figure Fig. A22: Fiber end-face inspection dataset. Images randomly sampled from the test set, showing clean fiber ends (gray border) with uniform transmission versus contaminated fiber ends (green border) with debris, scratches, or surface damage causing localized intensity variations. The mean pixel intensity is annotated above each frame. Detecting contamination requires identifying localized anomalies against the fiber’s baseline profile, not intensity thresholding.
Figure Fig. A23: Experimental apparatus for active PANS. a , Top-down photograph of the optical system with the three main components highlighted: OLED display for programmable illumination patterns (green), SLM for object emulation (white), and qCMOS camera for single-photon detection (yellow). Arrows indicate the light path from illumination source through the object plane to the detector. b , Side-view photograph of the setup. c , Schematic rendering illustrating the sensing pipeline: learned illumination patterns displayed on the OLED probe the object on the SLM, and the transmitted light is measured by the photon detector to produce classification results.
Figure Fig. A24: Experimentally collected single-photon detection data for FashionMNIST. a , Data for df=32 : each 8×4 grid displays the binary detection outcomes from one single-shot measurement, with white indicating photon detected and black indicating no detection. Columns correspond to different test images; rows correspond to repeated trials of the same image. b , Data for df=16 : each 4×4 grid represents one measurement. Right panels show t-SNE visualizations of the feature distributions across all test images, with colors indicating class labels.
Figure Fig. A25: Single-shot inference pipeline with experimental data. a , Diagram of the inference process (compare to Fig. 2D in the main text): df=16 trained illumination patterns probe the input image (a “Shirt” from FashionMNIST), single-photon detection produces a 4×4 binary feature frame, and the digital back end outputs a 10-dimensional vector for classification. b , Six trials randomly sampled from the 30 repeated measurements of the same input image. Left column: experimentally collected feature frames showing trial-to-trial variability. Right column: corresponding output vectors from the digital classifier, with the predicted class indicated. Five of six trials correctly predict “Shirt”; one trial (Trial 5) incorrectly predicts “Coat”.
Figure Fig. A26: Distribution of classifier outputs across 30 trials for representative test images. Each row corresponds to one input image from FashionMNIST (with corresponding category labeled on the top). For each image, the output vectors from all 30 experimentally implemented single-shot inferences ( df=16 ) are overlaid with transparency to show the density of outputs across trials.
Figure Fig. A27: Experimental confusion matrices for FashionMNIST at K=1 . a – f show results for df=4,6,10,16,24,32 , with the average detected photon count and overall accuracy indicated below each matrix. Rows indicate true labels; columns indicate predicted labels.
Figure Fig. A28: Experimental confusion matrices for MNIST at K=1 . a – f show results for df=4,9,16,25,36,64 , with the average detected photon count and overall accuracy indicated below each matrix.
K=1
K=2
K=3
K=5
df=3
52.35±1.43
54.53±2.41
56.30±1.35
56.17±1.14
df=4
62.58±2.03
65.20±1.84
66.95±1.42
69.08±1.10
df=6
65.77±2.35
69.87±1.86
71.95±1.77
72.50±1.08
df=10
73.20±2.22
76.63±1.32
77.80±1.82
78.92±1.30
df=16
77.20±2.20
80.47±1.72
82.05±1.75
83.00±1.19
df=24
79.37±2.06
81.53±1.02
83.15±1.42
83.67±1.03
Appendix
Table Table A6: Experimental test accuracy (%) on FashionMNIST for different df and K . Each entry shows mean ± standard deviation across 200 test images.
K=1
K=2
K=3
K=5
df=4
58.33±2.29
61.83±2.53
64.85±1.64
68.67±1.14
df=9
76.48±2.30
82.57±2.12
85.40±0.83
87.75±0.95
df=16
85.70±2.06
90.87±1.37
91.85±1.57
93.08±0.98
df=25
91.87±1.70
95.50±1.10
96.60±0.86
96.58±1.54
df=36
95.12±1.41
97.73±0.96
98.05±0.69
98.67±0.37
df=64
97.23±1.00
99.00±0.66
99.20±0.68
99.75±0.25
Appendix
Table Table A7: Experimental test accuracy (%) on MNIST for different df and K . Each entry shows mean ± standard deviation across 200 test images.
Figure Fig. A29: Experimental classification accuracy versus number of illumination patterns df for different shot counts K . a , FashionMNIST and b , MNIST. Markers with error bars show experimental results (mean ± standard deviation); curves connect points for visual clarity. The four curves correspond to K=1,2,3,5 accumulated single-shot measurements per pattern before classification. Red annotations indicate the average total detected photon budget Ndet per inference at selected df values for K=1 .
Illumination using correlated photon sources has been established as an approach to allowing high-fidelity images to be reconstructed from noisy camera frames by taking advantage of the knowledge that signal photons are spatially correlated whereas detector clicks due to noise are uncorrelated. However, in computer-vision tasks, the goal is often not ultimately to reconstruct an image, but to make inferences about a scene -- such as what object is present. Here we show how correlated-photon illumination can be used to gain an advantage in a hybrid optical-electronic computer-vision pipeline for object recognition. We demonstrate correlation-aware training (CAT): end-to-end optimization of a trainable correlated-photon illumination source and a Transformer backend in a way that the Transformer can learn to benefit from the correlations, using a small number (<= 100) of shots. We show a classification accuracy enhancement of up to 15 percentage points over conventional, uncorrelated-illumination-based computer vision in ultra-low-light and noisy imaging conditions, as well as an improvement over using untrained correlated-photon illumination. Our work illustrates how specializing to a computer-vision task -- object recognition -- and training the pattern of photon correlations in conjunction with a digital backend allows us to push the limits of accuracy in highly photon-budget-constrained scenarios beyond existing methods focused on image reconstruction.
Mandar M. Sohoni, Jérémie Laydevant, Mathieu Ouellet +7
School of Applied and Engineering Physics, Cornell University, Ithaca, NY 14853, USA · NTT Physics and Informatics Laboratories, NTT Research Inc., Sunnyvale, CA 94085, USA · Department of Electrical and Computer Engineering, Boston University, Boston, MA 02215, USA +2
Optical readout in low-light imaging is fundamentally limited by measurement noise, including photon shot noise, detector noise, and quantization error. In this regime, downstream inference depends not only on the optical front end, but also on how noisy high-dimensional sensor measurements are represented before classification or decision-making. Here we show that eigentasks provide a measurement-adapted representation for optical sensor outputs by ordering readout features according to their resolvability under noise. Using experimental data from a lens-based optical imaging system and a reanalysis of published data from a single-photon-detection neural network, we find that eigentask representations frequently outperform standard baselines including principal component analysis and filtering-based compression. The advantage is most pronounced in photon-limited, few-shot, and higher-difficulty classification regimes. In few-shot MPEG-7 classification, for example, the advantage over other methods reaches about 10 percentage points as the number of classes increases. In these settings, eigentasks yield more informative low-dimensional features and improve sample-efficient downstream learning. These results identify measurement-adapted representation as a promising strategy for optical inference when photon budget, acquisition time, and task complexity are constrained.
Tianyang Chen, Mandar M. Sohoni, Saeed A. Khan +5
Department of Electrical and Computer Engineering, Princeton University, Princeton, NJ 08544, USA · School of Applied and Engineering Physics, Cornell University, Ithaca, NY 14853, USA · USRA Research Institute for Advanced Computer Science, Mountain View, CA 94035, USA +1
Autonomous systems rely on extracting information from light, yet remain brittle in extreme environments, from nighttime navigation to high-speed robotics. Conventional sensors aggregate photons over fixed exposures, imposing trade-offs between sensitivity, dynamic range, and temporal resolution that degrade perception when photons are scarce or dynamics are rapid. Quanta sensors detect individual photons, but their streams exceed real-time compute and latency budgets by orders of magnitude. Here we introduce probabilistic events, a computational primitive for real-time quanta perception from individual photon detections. By computing the posterior over the time since the last intensity change, we represent photon streams as recursive belief states. Rather than fixed-threshold event-camera triggers, this recursive Bayesian formulation yields three low-latency signals: motion-adaptive scene flux, high-fidelity activity maps, and entropy-based perceptual uncertainty. This representation enables perception in extreme conditions, including pose estimation of a running person at ∼0.05 lux---without retraining vision models. Our approach processes input streams exceeding 50{,}000 quanta frames per second on commodity GPU hardware---yielding kilohertz-scale outputs up to four orders of magnitude faster than state-of-the-art quanta reconstruction baselines, even for megapixel arrays. By replacing frame reconstruction with direct probabilistic inference over photon streams, this work bridges photon-counting quanta sensing with robotic vision.
Varun Sundar, Pavan Thodima, Sacha Jungerman +1
Department of Computer Sciences, University of Wisconsin–Madison, Wisconsin, United States.