Ultra-low-light computer vision using trained photon correlations
Authors: Mandar M. Sohoni, Jérémie Laydevant, Mathieu Ouellet, Shi-Yuan Ma, Ryotatsu Yanagimoto, Benjamin A. Ash, Tatsuhiro Onodera, Tianyu Wang, +2 more
Organizations: School of Applied and Engineering Physics, Cornell University, Ithaca, NY 14853, USA · NTT Physics and Informatics Laboratories, NTT Research Inc., Sunnyvale, CA 94085, USA · Department of Electrical and Computer Engineering, Boston University, Boston, MA 02215, USA · Department of Applied Physics, Yale University, New Haven, CT 06511, USA · Kavli Institute at Cornell for Nanoscale Science, Cornell University, Ithaca, NY 14853, USA
Illumination using correlated photon sources has been established as an approach to allowing high-fidelity images to be reconstructed from noisy camera frames by taking advantage of the knowledge that signal photons are spatially correlated whereas detector clicks due to noise are uncorrelated. However, in computer-vision tasks, the goal is often not ultimately to reconstruct an image, but to make inferences about a scene -- such as what object is present. Here we show how correlated-photon illumination can be used to gain an advantage in a hybrid optical-electronic computer-vision pipeline for object recognition. We demonstrate correlation-aware training (CAT): end-to-end optimization of a trainable correlated-photon illumination source and a Transformer backend in a way that the Transformer can learn to benefit from the correlations, using a small number (<= 100) of shots. We show a classification accuracy enhancement of up to 15 percentage points over conventional, uncorrelated-illumination-based computer vision in ultra-low-light and noisy imaging conditions, as well as an improvement over using untrained correlated-photon illumination. Our work illustrates how specializing to a computer-vision task -- object recognition -- and training the pattern of photon correlations in conjunction with a digital backend allows us to push the limits of accuracy in highly photon-budget-constrained scenarios beyond existing methods focused on image reconstruction.
Figures & tables
Figure 1: Ultra-low-light computer vision with uncorrelated and correlated illumination. a, A comparison of conventional and our proposed computer vision pipelines. (i) , A conventional computer vision pipeline. A light source emits uncorrelated photons towards an object and the photons that are not blocked by the object are recorded on a single-photon-sensitive camera. The shots (frames) from the camera are sent to a Set Transformer backend to classify which object was being imaged. (ii) , A computer vision pipeline with a correlated-photon illumination source. Instead of uncorrelated photons as in (i), the light source emits spatially correlated photons. (iii) , Our proposed computer vision pipeline with a trainable-correlated-photon source. The light source emits trainable correlated photons that, when trained, allow the object to classified with fewer shots and/or photons. Appendix A introduces an example of a classification task where photon correlations are useful. b, Correlation-aware training (CAT). A schematic of the end-to-end optimization loop used to train photon correlations in a computer vision pipeline. A Set Transformer uses shots from the camera to classify the object being image, and backpropagates the error through a digital model of the correlated light source to update the photon correlations and the weights of the Set Transformer.
Figure 2: Characterization of the trainable-correlated-photon source. a, A schematic of the experimental setup to create programmable correlations in SPDC photons. An SLM shapes the angular spectrum of a 355nm laser which then produces signal and idler SPDC photons at 710nm in a β -BBO crystal. Momentum conservation ensures that the SPDC photons are correlated. By changing the angular spectrum of the pump, one can shape the correlations of these photons. The inset figure originating from the β -BBO crystal is a 2-D representation of the phasematching function. b, Trainable photon correlations. The columns show the phase pattern uploaded to the SLM to tune the angular spectrum of the pump, followed by the experimentally measured mean fields and correlations of the down-converted light averaged over 40 , 000 shots. qx and qy denote the transverse wave vector components of the SPDC biphotons (camera pixels since the camera was placed at the fourier plane of the β -BBO crystal). A diagonal mask was applied to the correlation images to filter out the variance. Appendix B.5 contains more information about how the phasematching function was fit, how the correlations are computed and the agreement with simulations. We’d like to note here that the covariance matrices were constructed only to demonstrate control of the photon correlations.
Figure 3: Photon-budget-efficient computer vision enabled by trained photon correlations. a, A schematic of the end-to-end optimization framework used to train this system. The objects to be classified were a subset of the MPEG-7 dataset (see Appendix D.1 ). The cross-entropy loss computed using experimentally measured shots was used to compute gradients of the SLM parameters using a digital model of the illumination source. b, Classification accuracy of trained correlated illumination (SPDC light generated by angular spectrum shaping of the pump beam; dark blue), untrained correlated illumination (SPDC light generated by a plane wave pump beam; light blue), and untrained uncorrelated illumination (coherent light; light orange) as a function of the illumination power when 10 shots were used for inference. The advantage in classification accuracy is most apparent at lower powers and just enough shots for the Transformer to be able to extract correlation information. Appendix D.1 discusses what objects were used in the dataset. Appendix D.4 has a detailed description of the classification accuracy curves across a various range of illumination powers and shots. c, The trained SLM pattern, illumination pattern (averaged over 2000 shots) and example of a series of individual shots after end-to-end optimization was performed for ∼162 total clicks per shot (with no object present). d, Classification accuracy of trained correlated illumination, untrained correlated illumination, and untrained uncorrelated illumination as a function of the illumination power when 10 shots are used for inference. As the number of shots increases, the benefit of trained correlations becomes apparent, with an enhancement of ∼ 10% over untrained correlations with just 10 shots.
Figure 4: Simulation of cell organelle classification using trained uncorrelated and correlated light. a, An overview of the cell organelle dataset that consists of 5 classes. In simulation, we trained three kinds of illumination to classify cell organelles: (1) an ideal correlated bi-photon source (assuming engineered phase matching and pump shaping, purple), (2) our experimental correlated bi-photon source (blue), and (3) uncorrelated illumination (a laser and an SLM, dark orange). We also compared it to the conventional computer vision pipeline with untrained, uncorrelated illumination (light orange) b, Classification accuracy of the different illuminations as a function of the number of illumination photons per shot with 50 shots used per inference. c, The trained illumination patterns for the three different illumination sources mention in panel a at different photon-budgets. d, Classification accuracy of the different illuminations as a function of the number of shots used during inference at 60 illumination photons per shot. e, The difference between the classification accuracies for trained correlated and trained uncorrelated illumination as a function of the number of illumination photons per shot and the number of shots used for classification. The dotted line (hand-drawn) divides the regions in parameter space where correlated illumination has a higher classification accuracy than uncorrelated illumination. f, The difference between the classification accuracies for trained correlated (engineered phasematching) and trained uncorrelated illumination as a function of the number of illumination photons per shot and the number of shots used for classification. All the simulations were trained using 100 shots at different photon fluxes and used a background-noise probability distribution derived from the EMCCD camera used in experiments. More details can be found in Appendix D.5 .
Appendix figures & tables47 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A1 : Toy example demonstrating the benefit of correlations. a, A coherent state input to a two-mode photonic circuit with a trainable beamsplitter. b, A two-mode-squeezed vacuum input to a two-mode photonic circuit. c, The different classes to be differentiated between: (1) no object present, (2) the object blocks all photons in mode 1, (3) the object blocks all photons in mode 0, (4) the object blocks all photons in mode 0 and mode 1.
Table A1 : Probabilities of measuring ∣00⟩ from different illuminations and object classes
Table A2 : Probabilities of measuring ∣10⟩ from different illuminations and object classes
Table A3 : Probabilities of measuring ∣01⟩ from different illuminations and object classes
Table A4 : Probabilities of measuring ∣11⟩ from different illuminations and object classes
Figure A2 : Single-shot error probabilities for correlated and uncorrelated illumination. The single-shot error probability for both cases when 0.1 illumination photons were used for classification. For the coherent case, the beamsplitter was optimized for each value of ϵ using a grid search.
Figure B1 : Schematic of the experimental setup. The uncorrelated illumination was a coherent source at 710nm produced using a laser diode. The correlated illumination source was generated using spontaneous parametric down conversion (SPDC) of 355nm pump photons.
Figure B2 : Picture of the experimental setup (part 1). A picture of the experimental setup used to shape the angular spectrum of the 355nm pump and the beam path of the 710nm laser used for the conventional computer vision scheme.
Figure B3 : Picture of the experimental setup (part 2). A picture of the experimental setup inside the light-tight enclosure from supplementary figure B2 .
Figure B4 : Phasematching function terms. a, β as a function of θ . b, γ as a function of θ . All the curves were generated using the pump wavelength ( 355nm ) and the sellmeier equations.
Figure B5 : Detailed schematic of the experimental setup.
Figure B6 : Phasematching function.
Figure B7 : A digital model of the correlated light source.
Figure B8 : Histogram of pixel values for a single pixel across 5000 background frames. a, The histogram (represented as a probability density) of pixel values for a randomly chosen pixel across 5000 frames. The red line depicts the gaussian fit and the green line depicts the exponential fit for the gain. b, A log-scale representation of the histogram plotted in a for better visualization of the exponential fit. The black dotted line depicts the mean value of the pixel across the 5000 frames, and the gray line depicts the value of μ+2σ from the gaussian fit.
Figure B9 : Conversion of raw camera frame pixel values to estimated photon counts. a, A raw camera frame recorded at an EM gain of 1000 with an exposure time of 1ms . b, The processed raw frame turned into estimated photon counts using equation 43 .
Figure C1 : Training pipeline. We chain our differentiable sampler with a digital model (Transformer), that allows us to trained the entire model end-to-end with backpropagation.
Figure C2 : CAT increases detectable correlations. Simulation results of how CAT can increase the number of correlated photons transmitted through different objects.
Figure D1 : Fabircated MPEG7 objects. A subset of the MPEG7 dataset chosen for experiments. The objects were etched into a chromium film on top of a fused silica substrate.
Figure D2 : Schematic for correlated-photon computer vision. A 4−f system images the objects onto an EMCCD camera. The objects are placed at the fourier plane of the β -BBO crystal.
Figure D3 : Schematic for uncorrelated-photon computer vision. A 4−f system images the objects onto an EMCCD camera. The illumination is a laser diode source connected to an single-mode fiber whose spot size is adjusted by a telescope.
Figure D4 : EMCCD images of the MPEG7 objects. Images of the 25 MPEG7 objects for both the untrained correlated and uncorrelated illuminations for the first corresponding data point in figure 3 .
Figure D5 : EMCCD images of the MPEG7 objects. Images of the 25 MPEG7 objects for both the untrained correlated and uncorrelated illuminations for the second corresponding data point in figure 3 .
Figure D6 : EMCCD images of the MPEG7 objects. Images of the 25 MPEG7 objects for both the untrained correlated and uncorrelated illuminations for the third corresponding data point in figure 3 .
Figure D7 : EMCCD images of the MPEG7 objects. Images of the 25 MPEG7 objects for both the untrained correlated and uncorrelated illuminations for the fourth corresponding data point in figure 3 .
Figure D8 : EMCCD images of the MPEG7 objects. Images of the 25 MPEG7 objects for both the untrained correlated and uncorrelated illuminations for the fifth corresponding data point in figure 3 .
Figure D9 : EMCCD images of the MPEG7 objects. Images of the 25 MPEG7 objects for both the untrained correlated and uncorrelated illuminations for the sixth corresponding data point in figure 3 .
Figure D10 : EMCCD images of the MPEG7 objects. Images of the 25 MPEG7 objects for both the untrained correlated and uncorrelated illuminations for the seventh corresponding data point in figure 3 .
Figure D11 : EMCCD images of the MPEG7 objects. Images of the 25 MPEG7 objects for both the untrained correlated and uncorrelated illuminations for the eighth corresponding data point in figure 3 .
Figure D12 : Examples of single frames of the MPEG7 objects. Examples of single frames from each class that were fed as part of the input to the set Transformer. The data is from the first corresponding point for the untrained correlated and uncorrelated illumination points in figure 3 .
Figure D13 : Examples of single frames of the MPEG7 objects. Examples of single frames from each class that were fed as part of the input to the set Transformer. The data is from the second corresponding point for the untrained correlated and uncorrelated illumination points in figure 3 .
Figure D14 : Examples of single frames of the MPEG7 objects. Examples of single frames from each class that were fed as part of the input to the set Transformer. The data is from the third corresponding point for the untrained correlated and uncorrelated illumination points in figure 3 .
Figure D15 : Examples of single frames of the MPEG7 objects. Examples of single frames from each class that were fed as part of the input to the set Transformer. The data is from the fourth corresponding point for the untrained correlated and uncorrelated illumination points in figure 3 .
Figure D16 : Examples of single frames of the MPEG7 objects. Examples of single frames from each class that were fed as part of the input to the set Transformer. The data is from the fifth corresponding point for the untrained correlated and uncorrelated illumination points in figure 3 .
Figure D17 : Examples of single frames of the MPEG7 objects. Examples of single frames from each class that were fed as part of the input to the set Transformer. The data is from the sixth corresponding point for the untrained correlated and uncorrelated illumination points in figure 3 .
Figure D18 : Examples of single frames of the MPEG7 objects. Examples of single frames from each class that were fed as part of the input to the set Transformer. The data is from the seventh corresponding point for the untrained correlated and uncorrelated illumination points in figure 3 .
Figure D19 : Examples of single frames of the MPEG7 objects. Examples of single frames from each class that were fed as part of the input to the set Transformer. The data is from the eighth corresponding point for the untrained correlated and uncorrelated illumination points in figure 3 .
Figure D20 : Classification accuracy curve for the MPEG7 experiment. Classification accuracy as a function of illumination power when 2 shots were used for inference
Figure D21 : Classification accuracy curve for the MPEG7 experiment. Classification accuracy as a function of illumination power when 6 shots were used for inference
Figure D22 : Classification accuracy curve for the MPEG7 experiment. Classification accuracy as a function of illumination power when 20 shots were used for inference
Figure D23 : Classification accuracy curve for the MPEG7 experiment. Classification accuracy as a function of illumination power when 50 shots were used for inference
Figure D24 : Classification accuracy curve for the MPEG7 experiment. Classification accuracy as a function of illumination power when 100 shots were used for inference
Figure D25 : Classification accuracy curves for the MPEG7 experiment. Extended data from Figure 3 .
Figure D26 : Classification accuracy curves for the MPEG7 experiment. Classification accuracy as a function of the number of shots used for inference when ∼100 photons were used for illumination. Extended data from Figure 3 .
Figure D27 : Mean-field illumination patterns for the MPEG7 experiment. Trained illumination for ∼162 and ∼220 average clicks per shot from Figure 3 .
Figure D28 : Simulations of cell organelle classification with different illuminations. Classification accuracy surfaces for the illuminations as a function of the number of illumination photons per shot and the number of shots used for classification. The dotted lines depict the cross sections of these curves that were shown in Figure 4 .
Figure D29 : Simulation: Effect of loss on correlated illumination.
Figure E1 : Comparison between different experimental demonstrations of object/image classification in ultra-low-light conditions.
Machine vision, including object recognition and image reconstruction, is a central technology in many consumer devices and scientific instruments. The design of machine-vision systems has been revolutionized by the adoption of end-to-end optimization, in which the optical front end and the post-processing back end are jointly optimized. However, while machine vision currently works extremely well in moderate-light or bright-light situations -- where a camera may detect thousands of photons per pixel and billions of photons per frame -- it is far more challenging in very low-light situations. We introduce photon-aware neuromorphic sensing (PANS), an approach for end-to-end optimization in highly photon-starved scenarios. The training incorporates knowledge of the low photon budget and the stochastic nature of light detection when the average number of photons per pixel is near or less than 1. We report a proof-of-principle experimental demonstration in which we performed low-light image classification using PANS, achieving 73% (82%) accuracy on FashionMNIST with an average of only 4.9 (17) detected photons in total per inference, and 86% (97%) on MNIST with 8.6 (29) detected photons -- orders of magnitude more photon-efficient than conventional approaches. We also report simulation studies showing how PANS could be applied to other classification, event-detection, and image-reconstruction tasks. By taking into account the statistics of measurement results for non-classical states or alternative sensing hardware, PANS could in principle be adapted to enable high-accuracy results in quantum and other photon-starved setups.
Shi-Yuan Ma, Jérémie Laydevant, Mandar M. Sohoni +3
School of Applied and Engineering Physics, Cornell University, Ithaca, NY 14853, USA · Present address: Research Laboratory of Electronics, Massachusetts Institute of Technology, Cambridge, MA 02139, USA. · NTT Physics and Informatics Laboratories, NTT Research, Inc., Sunnyvale, CA 94085, USA +2
Optical readout in low-light imaging is fundamentally limited by measurement noise, including photon shot noise, detector noise, and quantization error. In this regime, downstream inference depends not only on the optical front end, but also on how noisy high-dimensional sensor measurements are represented before classification or decision-making. Here we show that eigentasks provide a measurement-adapted representation for optical sensor outputs by ordering readout features according to their resolvability under noise. Using experimental data from a lens-based optical imaging system and a reanalysis of published data from a single-photon-detection neural network, we find that eigentask representations frequently outperform standard baselines including principal component analysis and filtering-based compression. The advantage is most pronounced in photon-limited, few-shot, and higher-difficulty classification regimes. In few-shot MPEG-7 classification, for example, the advantage over other methods reaches about 10 percentage points as the number of classes increases. In these settings, eigentasks yield more informative low-dimensional features and improve sample-efficient downstream learning. These results identify measurement-adapted representation as a promising strategy for optical inference when photon budget, acquisition time, and task complexity are constrained.
Tianyang Chen, Mandar M. Sohoni, Saeed A. Khan +5
Department of Electrical and Computer Engineering, Princeton University, Princeton, NJ 08544, USA · School of Applied and Engineering Physics, Cornell University, Ithaca, NY 14853, USA · USRA Research Institute for Advanced Computer Science, Mountain View, CA 94035, USA +1
Vision transformers typically treat every image token as equally important, yet for most tasks in computer vision only a fraction are needed. Adaptive computation methods accelerate inference by choosing which tokens to process, but existing methods struggle at extreme sparsity and require heuristics that may not generalize like token diversity and attention scores. We address these limitations with LookThere, achieving a new pareto frontier in performance-compute trade-offs through an end-to-end reinforcement learning framework that jointly trains a shallow input selector and a deep representation extractor. The selector learns where to look and the extractor learns what to see, together saving computation by selecting only what is worth processing for a given task without relying on auxiliary signals. We show that LookThere only selects the task-specific input, excelling at sparse recognition in high-resolution settings (traffic signs, billiards), and maintaining accuracy with as little as 0.2% of the input. It generalizes across tasks and models, including global recognition (ImageNet classification), local recognition (ADE20K segmentation), zero-shot classification (by distillation), and regression (counting). Across all settings, LookThere surpasses state-of-the-art selection to provide a general and scalable framework for specialized and efficient adaptive computation.
Sreehari Rammohan, Yousef Yassin, Anthony Fuller +3
Columbia University · Carleton University · Vector Institute +1