physics.opticsMar 25, 2026
SaveMachine vision with small numbers of detected photons per inference
Organizations: School of Applied and Engineering Physics, Cornell University, Ithaca, NY 14853, USA · Present address: Research Laboratory of Electronics, Massachusetts Institute of Technology, Cambridge, MA 02139, USA. · NTT Physics and Informatics Laboratories, NTT Research, Inc., Sunnyvale, CA 94085, USA · Present address: Department of Applied Physics, Yale University, New Haven, CT 06511, USA · Present address: Department of Electrical and Computer Engineering, Boston University, Boston, MA 02215, USA.
Abstract
Machine vision, including object recognition and image reconstruction, is a central technology in many consumer devices and scientific instruments. The design of machine-vision systems has been revolutionized by the adoption of end-to-end optimization, in which the optical front end and the post-processing back end are jointly optimized. However, while machine vision currently works extremely well in moderate-light or bright-light situations -- where a camera may detect thousands of photons per pixel and billions of photons per frame -- it is far more challenging in very low-light situations. We introduce photon-aware neuromorphic sensing (PANS), an approach for end-to-end optimization in highly photon-starved scenarios. The training incorporates knowledge of the low photon budget and the stochastic nature of light detection when the average number of photons per pixel is near or less than 1. We report a proof-of-principle experimental demonstration in which we performed low-light image classification using PANS, achieving 73% (82%) accuracy on FashionMNIST with an average of only 4.9 (17) detected photons in total per inference, and 86% (97%) on MNIST with 8.6 (29) detected photons -- orders of magnitude more photon-efficient than conventional approaches. We also report simulation studies showing how PANS could be applied to other classification, event-detection, and image-reconstruction tasks. By taking into account the statistics of measurement results for non-classical states or alternative sensing hardware, PANS could in principle be adapted to enable high-accuracy results in quantum and other photon-starved setups.
Figures & tables
Figure 1: Detection bottleneck in optical sensing and photon-aware neuromorphic sensing (PANS) under limited photon counts . A , Conceptual optical sensing pipeline. An object (illustrated by a cat) interacts with a probe signal (e.g., light) shaped by a configurable optical front end (green), which may operate in active (controlled illumination) or passive (incoming-signal modulation) modes. The detector then converts the incident optical signal into digital data via single-photon detection (SPD). When photon budgets are highly limited, this conversion presents a detection bottleneck with significant information loss (lower schematic), which cannot be recovered by subsequent digital processing. A digital back end (e.g., a post-processing neural network) extracts task-relevant information from the detected data. B , Direct imaging vs. PANS under limited photons. Top: in a conventional direct-imaging pipeline, photon-limited image frames (shown as repeated realizations across independent trials 1, 2, 3) exhibit strong shot noise, making downstream inference challenging (e.g., cat vs. dog). Bottom: PANS introduces a parameterized optical front end that transforms the optical field before detection, producing photon-efficient feature measurements; the front end and digital back end are jointly optimized end-to-end through the stochastic detection bottleneck . C , Photon detection bottleneck. With mean photon energy , SPD produces a discrete stochastic digital readout . PANS faithfully models this stochastic forward propagation and applies gradient estimation to enable estimated backpropagation (backprop) through the detection bottleneck, allowing end-to-end optimization under photon-budget constraints.
Figure 2: Active photon-aware neuromorphic sensing (PANS) demonstrated on FashionMNIST object classification. A , Direct imaging (conventional approach). Uniform illumination probes the object, and single-photon detectors directly capture an image frame with pixels. and denote average total illumination and detection photon budgets, respectively. The example object image (a sneaker) is taken from the FashionMNIST dataset. B , Image frames degrade with decreasing (denoted above each column) for a pullover (top) and a shirt (bottom). Frames become increasingly noisy as the photon budget decreases. C , Quantifying information loss at the detection bottleneck. As decreases (left to right), three metrics decline: mutual information with labels (top), Fisher discriminant ratio (FDR; middle), and the test accuracy using a convolutional neural network (bottom). D , Active PANS (our approach). illumination patterns are projected onto the object, producing a -dimensional feature vector through single-photon detection (see Fig. 3 and Appendix 12 for details of the experimental protocol). E , Experimental results on FashionMNIST. Top: confusion matrices at two different photon budgets. Bottom: Test accuracy vs. (left) and (right). is the total illumination incident on the object (uniform for direct imaging; sum of pattern intensities for structured illumination; Appendix 5). Red markers: active PANS experiment (mean std over 30 trials per image) for ; light red shade: corresponding simulation (mean 3 std). Blue curve: direct imaging baseline (from C ). Green curve: conventional E2E without photon-aware modeling (non-PA E2E; Appendix 8C). F , 2D t-SNE [ 98 ] visualization comparing feature distributions. Active PANS (red boxes) versus direct imaging (blue boxes) at different values, with test accuracies shown.
Figure 3: Stochastic single-shot inference and experimental validation of active PANS. A , Stochastic single-shot inference under extreme photon constraints. learned illumination patterns are sequentially projected onto an object with transmission , each producing a binary single-photon detection (SPD) readout. Because detection is highly stochastic at these photon levels, independent trials on the same object yield different feature vectors (bottom), each processed by the digital back end. Despite this trial-to-trial variability, the system consistently identifies the correct class across trials. The aggregate output distribution over inferences (right) reflects the classification confidence (Appendix 13). B–C , Experimental classification accuracy (red, mean std over trials) versus number of illumination patterns for FashionMNIST ( B ) and MNIST ( C ) under single-shot operation. The FashionMNIST data correspond to the red markers in Fig. 2 E, here plotted against . Simulation results (light red band, mean std) show close agreement with experiment. Annotations indicate the average total detected photon budget per inference at selected (Appendix 13).
Figure 4: Proposed real-time image sensing with active photon-aware neuromorphic sensing (PANS) in simulation. A , Conceptual wavelength-multiplexed implementation for flow-cytometric cell sorting. Multiple static illumination patterns at distinct optical wavelengths (illustrated as with different colors) are applied simultaneously; wavelength demultiplexing routes each channel to a dedicated photon counter, producing activations in parallel for real-time digital processing. B , Simulated test accuracy for cell-organelle classification versus total detected photons under different detector dark-count rates (DCRs), compared with an ideal direct-imaging baseline. C , Example real-time sequence. Left: representative frames as a cell traverses the illumination field (top to bottom). Right: corresponding model outputs over time across classes (Mem.: membrane; Nuc.: nucleolus; Mit.: mitochondria; Null: no cell present). D , Barcode identification task. The illumination field spans a 10-bar window (red box); the goal is to decide whether the target subsequence “1010” appears at any position. E , Simulated test accuracy for barcode identification versus under multiple DCR values. Direct imaging accounts only for ideal shot noise (no dark counts or additional detector noise), highlighting the robustness of active PANS under realistic counting noise.
Figure 5: Diagram and applications of passive photon-aware neuromorphic sensing (PANS) in simulation. A , Passive PANS vs. direct imaging for sensing images transmitted through a scattering multimode fiber (MMF). Input images propagate through the MMF, emerging as speckle patterns that scramble spatial information. In passive PANS, speckles pass through a passive optical encoder before detection; in direct imaging, speckle frames are captured directly at the image plane with photons per frame. Both schemes use post-processing neural networks for classification or reconstruction. B , Classification accuracy on MNIST speckle images versus , with dark count rates (DCRs) of 1%, 5%, and 10% for passive PANS. Direct imaging (blue curve) simulated using only ideal shot noise. Inset: direct imaging accuracy at higher . Red markers show passive PANS with . C , Average structural similarity index (SSIM) of reconstructed images from scattered speckles, evaluated with different DCRs. Passive PANS data points: . D , Example images showing original MNIST digits (top), corresponding speckle patterns (middle), and reconstructed images (bottom) from passive PANS ( , DCR = 1%). E , Transient event detection: fleeting objects in noisy backgrounds are identified in a monitored scene (left). Right: test accuracy vs. . Passive PANS: . F , Tissue blood flow detection via speckle contrast imaging. G , Compact nebula classification. H , Optical fiber end-face contamination inspection. Insets show direct imaging performance at higher photon counts.
Appendix figures & tables36 assets
Supplementary material from the paper’s appendix.
Appendix
Figure Fig. A1: Taxonomy of optical sensing approaches. A flowchart illustrating the hierarchy of algorithmic approaches in optical sensing. Starting from the top, if no pre-detection operation is applied, the system performs direct imaging (blue). If an optical encoding operation is applied, the approach branches based on whether task-specific training is used: fixed transformations (white) use predetermined configurations such as downsampling or compressive sensing patterns, while end-to-end optimization learns task-specific optical parameters jointly with the digital back-end. End-to-end methods further divide based on whether photon-aware modeling is employed: conventional end-to-end methods (green) do not explicitly model stochastic photon detection, whereas photon-aware neuromorphic sensing (PANS, red) faithfully represents the probabilistic, discrete nature of the detection process during optimization. The color coding agrees with Fig. 2 in the main text. Shaded background regions group approaches by the level of sophistication in the optical front-end design.
| Approach | Optical operation | Task-optimized | Photon-aware |
| Direct imaging | × | — | — |
| Fixed optical transformation | ✓ | × | — |
| Conventional (non-PA) E2E | ✓ | ✓ | × |
| Photon-aware neuromorphic sensing (PANS) | ✓ | ✓ | ✓ |
Table Table A1: Taxonomy of sensing approaches based on optical front-end design choices.
Figure Fig. A2: Diagram of the optical front end operation in active sensing. The light source provide a spatially distributed illumination pattern and project to the object, the object then attentuate the illuminatino light, equivalent to element-wise multiplication. then the optical energy was focused on to a detector to be measured. the output is equivalent to a dot product process.
| Approach | Photon level ( ) | -shot | |
| Direct imaging | Applicable | N/A | |
| Conventional optical encoding | Applicable | Applicable | |
| PANS | Limited | Practical | Optimized |
Table Table A2: Photon budget scaling methods and their applicability across sensing approaches.
| Parameter | Value |
| Optimizer | Adam |
| Learning rate (optical front-end, conv layers) | |
| Learning rate (FC layers) | |
| Batch size | 128 |
| Maximum epochs | 500 |
| Training shots per activation ( ) | 1 |
Table Table A3: Baseline training configuration for active PANS on FashionMNIST.
Figure Fig. A3: Training dynamics under different illumination regularization for . Each panel shows training accuracy (blue), test accuracy (orange), and total illumination photon budget (green dashed, right axis) versus training epoch. The and values are indicated above each panel. All models use the training configuration in Table A3 with only varied.
Figure Fig. A4: Training dynamics under different illumination regularization for . Each panel shows training accuracy (blue), test accuracy (orange), and total illumination photon budget (green dashed, right axis) versus training epoch. The and values are indicated above each panel. All models use the training configuration in Table A3 with only varied.
Figure Fig. A5: Training dynamics under different illumination regularization for . Each panel shows training accuracy (blue), test accuracy (orange), and total illumination photon budget (green dashed, right axis) versus training epoch. The and values are indicated above each panel. All models use the training configuration in Table A3 with only varied.
Figure Fig. A6: Training dynamics under different illumination regularization for . Each panel shows training accuracy (blue), test accuracy (orange), and total illumination photon budget (green dashed, right axis) versus training epoch. The and values are indicated above each panel. All models use the training configuration in Table A3 with only varied.
Figure Fig. A7: Evolution of learned illumination patterns during training. Each panel displays the patterns from a model trained with , shown at four stages with training progressing from a to d . For visualization, patterns are normalized such that the mean pixel value across all patterns equals 1 (corresponding to a total of 784 per pattern); the number above each pattern shows its individual mean. The color bar indicates pixel value range. Early in training ( a ), patterns are spatially diffuse with moderate intensity variation. Through training ( b – d ), patterns become increasingly sparse and localized, concentrating intensity into small regions while the maximum value expands from 18 to 100 times the mean (color bar), reflecting efficient photon allocation to discriminative spatial locations.
Table Table A4: Test accuracy (%) on FashionMNIST for different feature dimensions and shot counts . Each entry shows mean standard deviation over 100 trials with 200 test images.
Figure Fig. A8: Multi-shot evaluation validates scaling over scaling. Test accuracy versus total number of detections ( ). Different curves correspond to models trained with different feature dimensions , as indicated in the legend. Points along each curve correspond to increasing (1, 2, 3, 5) from left to right. At the same total detection budget, larger with smaller consistently achieves higher accuracy than smaller with larger . Dot points show mean values computed over 100 repetitions; error bars indicate standard deviation.
Figure Fig. A9: Effect of photon level scaling . Test accuracy versus photon scaling parameter for trained PANS models with different feature dimensions , as indicated in the legend. The horizontal axis uses logarithmic scale spanning . All models achieve peak accuracy near (the trained operating point) and degrade for both lower and higher values. Dot points show mean values computed over 100 repetitions; error bars indicate standard deviation.
Figure Fig. A10: Robustness to dark counts. Test accuracy versus dark count rate (expected dark counts per detector per measurement) for different feature dimensions at , as indicated in the legend. The horizontal axis uses logarithmic scale. Accuracy degrades gradually up to DCR and more rapidly at higher DCR values. Dot points show mean values computed over 100 repetitions; error bars indicate standard deviation.
Figure Fig. A11: Shot-noise-limited frames at different photon budgets. Each row shows the same test image captured at decreasing average (left to right: , , , 10). For , three independent samples are shown to illustrate stochastic variation. Class-discriminative features are progressively destroyed by shot noise. At low photon counts, even human observers cannot reliably identify the object category.
Figure Fig. A12: t-SNE visualization of information loss. Each panel shows the procedure and resulting embedding at a different : each test image (10,000 total) is sampled 3 times at the target photon budget, and all 30,000 resulting vectors are projected via t-SNE. Points are colored by class label. a, : well-separated clusters indicate preserved class structure. b, : clusters overlap as shot noise corrupts discriminative features. c, : class structure is entirely lost; the embedding collapses into an undifferentiated distribution.
Figure Fig. A13: Training compressive sensing classifiers. Training loss (left) and accuracy (right) versus epoch for random binary patterns at different values. Models are trained on clean (high photon count) measurements.
Figure Fig. A14: Compressive sensing performance. Test accuracy (%) as a function of number of patterns ( , rows) and photon budget ( , columns). Performance collapses in the few-photon regime regardless of .
Figure Fig. A15: Conventional end-to-end optimization results. Test accuracy versus for different (light to dark curves). The thick curve shows the envelope—maximum accuracy at each across all —corresponding to the green curve in main text Fig. 2E.
Table Table A5: Test accuracy (%) on MNIST for different feature dimensions and shot counts . Each entry shows mean standard deviation over 100 trials.
Figure Fig. A16: Cell-organelle classification dataset. Images randomly sampled from the test set, representing category: membrane, nucleolus, and mitochondria. Category labels are shown above each image.
Figure Fig. A17: Barcode “1010” identification dataset. Example 10-element barcode segments randomly sampled from the test set. Positive examples (checkmark) contain the “1010” pattern at some position (highlighted in green box). Negative examples (cross) do not contain the target pattern. The pattern can appear at any location within the segment, requiring position-invariant feature detection.
Figure Fig. A18: Image sensing through multimode fiber. Top: schematic showing original image MMF propagation speckle pattern PANS encoder single-photon detection (SPD) reconstruction. Bottom: example results showing original MNIST digits (gray shade), corresponding speckle patterns after MMF propagation (yellow shade), and reconstructed images from passive PANS at different configurations ( with DCR 1%, with DCR 5%, and with DCR 1%). Three independent samples per digit demonstrate consistency and variation across stochastic detection events. Spatial structure can be recovered even from apparently random speckle at few-photon levels.
Figure Fig. A19: Transient event detection dataset. Example frames showing background-only (gray border) and transient-present (green border) cases. The mean pixel intensity is annotated above each frame. The similar mean values across both classes demonstrate that simple intensity thresholding cannot distinguish transient events, inferring that detection requires learning the spatial signature of the transient against background fluctuations.
Figure Fig. A20: Tissue speckle imaging dataset. Images randomly sampled from the test set, showing static tissue (gray border) with high speckle contrast versus perfused tissue (green border) with reduced speckle contrast due to blood flow. The mean pixel intensity is annotated above each frame. Classification performance relies on speckle texture analysis—the contrast difference between sharp (static) and blurred (perfused) patterns—rather than intensity measurements.
Figure Fig. A21: Nebula classification dataset. Images randomly sampled from the test set, showing planetary nebulae (green border), characterized by roughly circular or ring-like morphology from expanding stellar ejecta, versus emission nebulae (gray border), showing more irregular structure from illuminated interstellar medium. The mean pixel intensity is annotated above each frame. The overlapping intensity distributions demonstrate that classification requires morphological analysis, not simple photometry.
Figure Fig. A22: Fiber end-face inspection dataset. Images randomly sampled from the test set, showing clean fiber ends (gray border) with uniform transmission versus contaminated fiber ends (green border) with debris, scratches, or surface damage causing localized intensity variations. The mean pixel intensity is annotated above each frame. Detecting contamination requires identifying localized anomalies against the fiber’s baseline profile, not intensity thresholding.
Figure Fig. A23: Experimental apparatus for active PANS. a , Top-down photograph of the optical system with the three main components highlighted: OLED display for programmable illumination patterns (green), SLM for object emulation (white), and qCMOS camera for single-photon detection (yellow). Arrows indicate the light path from illumination source through the object plane to the detector. b , Side-view photograph of the setup. c , Schematic rendering illustrating the sensing pipeline: learned illumination patterns displayed on the OLED probe the object on the SLM, and the transmitted light is measured by the photon detector to produce classification results.
Figure Fig. A24: Experimentally collected single-photon detection data for FashionMNIST. a , Data for : each grid displays the binary detection outcomes from one single-shot measurement, with white indicating photon detected and black indicating no detection. Columns correspond to different test images; rows correspond to repeated trials of the same image. b , Data for : each grid represents one measurement. Right panels show t-SNE visualizations of the feature distributions across all test images, with colors indicating class labels.
Figure Fig. A25: Single-shot inference pipeline with experimental data. a , Diagram of the inference process (compare to Fig. 2D in the main text): trained illumination patterns probe the input image (a “Shirt” from FashionMNIST), single-photon detection produces a binary feature frame, and the digital back end outputs a 10-dimensional vector for classification. b , Six trials randomly sampled from the 30 repeated measurements of the same input image. Left column: experimentally collected feature frames showing trial-to-trial variability. Right column: corresponding output vectors from the digital classifier, with the predicted class indicated. Five of six trials correctly predict “Shirt”; one trial (Trial 5) incorrectly predicts “Coat”.
Figure Fig. A26: Distribution of classifier outputs across 30 trials for representative test images. Each row corresponds to one input image from FashionMNIST (with corresponding category labeled on the top). For each image, the output vectors from all 30 experimentally implemented single-shot inferences ( ) are overlaid with transparency to show the density of outputs across trials.
Figure Fig. A27: Experimental confusion matrices for FashionMNIST at . a – f show results for , with the average detected photon count and overall accuracy indicated below each matrix. Rows indicate true labels; columns indicate predicted labels.
Figure Fig. A28: Experimental confusion matrices for MNIST at . a – f show results for , with the average detected photon count and overall accuracy indicated below each matrix.
Table Table A6: Experimental test accuracy (%) on FashionMNIST for different and . Each entry shows mean standard deviation across 200 test images.
Table Table A7: Experimental test accuracy (%) on MNIST for different and . Each entry shows mean standard deviation across 200 test images.
Figure Fig. A29: Experimental classification accuracy versus number of illumination patterns for different shot counts . a , FashionMNIST and b , MNIST. Markers with error bars show experimental results (mean standard deviation); curves connect points for visual clarity. The four curves correspond to accumulated single-shot measurements per pattern before classification. Red annotations indicate the average total detected photon budget per inference at selected values for .