cs.CVApr 13, 2026
SaveUltra-low-light computer vision using trained photon correlations
Organizations: School of Applied and Engineering Physics, Cornell University, Ithaca, NY 14853, USA · NTT Physics and Informatics Laboratories, NTT Research Inc., Sunnyvale, CA 94085, USA · Department of Electrical and Computer Engineering, Boston University, Boston, MA 02215, USA · Department of Applied Physics, Yale University, New Haven, CT 06511, USA · Kavli Institute at Cornell for Nanoscale Science, Cornell University, Ithaca, NY 14853, USA
Abstract
Illumination using correlated photon sources has been established as an approach to allowing high-fidelity images to be reconstructed from noisy camera frames by taking advantage of the knowledge that signal photons are spatially correlated whereas detector clicks due to noise are uncorrelated. However, in computer-vision tasks, the goal is often not ultimately to reconstruct an image, but to make inferences about a scene -- such as what object is present. Here we show how correlated-photon illumination can be used to gain an advantage in a hybrid optical-electronic computer-vision pipeline for object recognition. We demonstrate correlation-aware training (CAT): end-to-end optimization of a trainable correlated-photon illumination source and a Transformer backend in a way that the Transformer can learn to benefit from the correlations, using a small number (<= 100) of shots. We show a classification accuracy enhancement of up to 15 percentage points over conventional, uncorrelated-illumination-based computer vision in ultra-low-light and noisy imaging conditions, as well as an improvement over using untrained correlated-photon illumination. Our work illustrates how specializing to a computer-vision task -- object recognition -- and training the pattern of photon correlations in conjunction with a digital backend allows us to push the limits of accuracy in highly photon-budget-constrained scenarios beyond existing methods focused on image reconstruction.
Figures & tables
Figure 1: Ultra-low-light computer vision with uncorrelated and correlated illumination. a, A comparison of conventional and our proposed computer vision pipelines. (i) , A conventional computer vision pipeline. A light source emits uncorrelated photons towards an object and the photons that are not blocked by the object are recorded on a single-photon-sensitive camera. The shots (frames) from the camera are sent to a Set Transformer backend to classify which object was being imaged. (ii) , A computer vision pipeline with a correlated-photon illumination source. Instead of uncorrelated photons as in (i), the light source emits spatially correlated photons. (iii) , Our proposed computer vision pipeline with a trainable-correlated-photon source. The light source emits trainable correlated photons that, when trained, allow the object to classified with fewer shots and/or photons. Appendix A introduces an example of a classification task where photon correlations are useful. b, Correlation-aware training (CAT). A schematic of the end-to-end optimization loop used to train photon correlations in a computer vision pipeline. A Set Transformer uses shots from the camera to classify the object being image, and backpropagates the error through a digital model of the correlated light source to update the photon correlations and the weights of the Set Transformer.
Figure 2: Characterization of the trainable-correlated-photon source. a, A schematic of the experimental setup to create programmable correlations in SPDC photons. An SLM shapes the angular spectrum of a laser which then produces signal and idler SPDC photons at in a -BBO crystal. Momentum conservation ensures that the SPDC photons are correlated. By changing the angular spectrum of the pump, one can shape the correlations of these photons. The inset figure originating from the -BBO crystal is a 2-D representation of the phasematching function. b, Trainable photon correlations. The columns show the phase pattern uploaded to the SLM to tune the angular spectrum of the pump, followed by the experimentally measured mean fields and correlations of the down-converted light averaged over , shots. and denote the transverse wave vector components of the SPDC biphotons (camera pixels since the camera was placed at the fourier plane of the -BBO crystal). A diagonal mask was applied to the correlation images to filter out the variance. Appendix B.5 contains more information about how the phasematching function was fit, how the correlations are computed and the agreement with simulations. We’d like to note here that the covariance matrices were constructed only to demonstrate control of the photon correlations.
Figure 3: Photon-budget-efficient computer vision enabled by trained photon correlations. a, A schematic of the end-to-end optimization framework used to train this system. The objects to be classified were a subset of the MPEG-7 dataset (see Appendix D.1 ). The cross-entropy loss computed using experimentally measured shots was used to compute gradients of the SLM parameters using a digital model of the illumination source. b, Classification accuracy of trained correlated illumination (SPDC light generated by angular spectrum shaping of the pump beam; dark blue), untrained correlated illumination (SPDC light generated by a plane wave pump beam; light blue), and untrained uncorrelated illumination (coherent light; light orange) as a function of the illumination power when 10 shots were used for inference. The advantage in classification accuracy is most apparent at lower powers and just enough shots for the Transformer to be able to extract correlation information. Appendix D.1 discusses what objects were used in the dataset. Appendix D.4 has a detailed description of the classification accuracy curves across a various range of illumination powers and shots. c, The trained SLM pattern, illumination pattern (averaged over 2000 shots) and example of a series of individual shots after end-to-end optimization was performed for total clicks per shot (with no object present). d, Classification accuracy of trained correlated illumination, untrained correlated illumination, and untrained uncorrelated illumination as a function of the illumination power when 10 shots are used for inference. As the number of shots increases, the benefit of trained correlations becomes apparent, with an enhancement of 10% over untrained correlations with just 10 shots.
Figure 4: Simulation of cell organelle classification using trained uncorrelated and correlated light. a, An overview of the cell organelle dataset that consists of 5 classes. In simulation, we trained three kinds of illumination to classify cell organelles: (1) an ideal correlated bi-photon source (assuming engineered phase matching and pump shaping, purple), (2) our experimental correlated bi-photon source (blue), and (3) uncorrelated illumination (a laser and an SLM, dark orange). We also compared it to the conventional computer vision pipeline with untrained, uncorrelated illumination (light orange) b, Classification accuracy of the different illuminations as a function of the number of illumination photons per shot with 50 shots used per inference. c, The trained illumination patterns for the three different illumination sources mention in panel a at different photon-budgets. d, Classification accuracy of the different illuminations as a function of the number of shots used during inference at 60 illumination photons per shot. e, The difference between the classification accuracies for trained correlated and trained uncorrelated illumination as a function of the number of illumination photons per shot and the number of shots used for classification. The dotted line (hand-drawn) divides the regions in parameter space where correlated illumination has a higher classification accuracy than uncorrelated illumination. f, The difference between the classification accuracies for trained correlated (engineered phasematching) and trained uncorrelated illumination as a function of the number of illumination photons per shot and the number of shots used for classification. All the simulations were trained using 100 shots at different photon fluxes and used a background-noise probability distribution derived from the EMCCD camera used in experiments. More details can be found in Appendix D.5 .
Appendix figures & tables47 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A1 : Toy example demonstrating the benefit of correlations. a, A coherent state input to a two-mode photonic circuit with a trainable beamsplitter. b, A two-mode-squeezed vacuum input to a two-mode photonic circuit. c, The different classes to be differentiated between: (1) no object present, (2) the object blocks all photons in mode 1, (3) the object blocks all photons in mode 0, (4) the object blocks all photons in mode 0 and mode 1.
Table A1 : Probabilities of measuring from different illuminations and object classes
Table A2 : Probabilities of measuring from different illuminations and object classes
Table A3 : Probabilities of measuring from different illuminations and object classes
Table A4 : Probabilities of measuring from different illuminations and object classes
Figure A2 : Single-shot error probabilities for correlated and uncorrelated illumination. The single-shot error probability for both cases when illumination photons were used for classification. For the coherent case, the beamsplitter was optimized for each value of using a grid search.
Figure B1 : Schematic of the experimental setup. The uncorrelated illumination was a coherent source at produced using a laser diode. The correlated illumination source was generated using spontaneous parametric down conversion (SPDC) of pump photons.
Figure B2 : Picture of the experimental setup (part 1). A picture of the experimental setup used to shape the angular spectrum of the pump and the beam path of the laser used for the conventional computer vision scheme.
Figure B3 : Picture of the experimental setup (part 2). A picture of the experimental setup inside the light-tight enclosure from supplementary figure B2 .
Figure B4 : Phasematching function terms. a, as a function of . b, as a function of . All the curves were generated using the pump wavelength ( ) and the sellmeier equations.
Figure B5 : Detailed schematic of the experimental setup.
Figure B6 : Phasematching function.
Figure B7 : A digital model of the correlated light source.
Figure B8 : Histogram of pixel values for a single pixel across 5000 background frames. a, The histogram (represented as a probability density) of pixel values for a randomly chosen pixel across 5000 frames. The red line depicts the gaussian fit and the green line depicts the exponential fit for the gain. b, A log-scale representation of the histogram plotted in a for better visualization of the exponential fit. The black dotted line depicts the mean value of the pixel across the 5000 frames, and the gray line depicts the value of from the gaussian fit.
Figure B9 : Conversion of raw camera frame pixel values to estimated photon counts. a, A raw camera frame recorded at an EM gain of 1000 with an exposure time of . b, The processed raw frame turned into estimated photon counts using equation 43 .
Figure C1 : Training pipeline. We chain our differentiable sampler with a digital model (Transformer), that allows us to trained the entire model end-to-end with backpropagation.
Figure C2 : CAT increases detectable correlations. Simulation results of how CAT can increase the number of correlated photons transmitted through different objects.
Figure D1 : Fabircated MPEG7 objects. A subset of the MPEG7 dataset chosen for experiments. The objects were etched into a chromium film on top of a fused silica substrate.
Figure D2 : Schematic for correlated-photon computer vision. A system images the objects onto an EMCCD camera. The objects are placed at the fourier plane of the -BBO crystal.
Figure D3 : Schematic for uncorrelated-photon computer vision. A system images the objects onto an EMCCD camera. The illumination is a laser diode source connected to an single-mode fiber whose spot size is adjusted by a telescope.
Figure D4 : EMCCD images of the MPEG7 objects. Images of the 25 MPEG7 objects for both the untrained correlated and uncorrelated illuminations for the first corresponding data point in figure 3 .
Figure D5 : EMCCD images of the MPEG7 objects. Images of the 25 MPEG7 objects for both the untrained correlated and uncorrelated illuminations for the second corresponding data point in figure 3 .
Figure D6 : EMCCD images of the MPEG7 objects. Images of the 25 MPEG7 objects for both the untrained correlated and uncorrelated illuminations for the third corresponding data point in figure 3 .
Figure D7 : EMCCD images of the MPEG7 objects. Images of the 25 MPEG7 objects for both the untrained correlated and uncorrelated illuminations for the fourth corresponding data point in figure 3 .
Figure D8 : EMCCD images of the MPEG7 objects. Images of the 25 MPEG7 objects for both the untrained correlated and uncorrelated illuminations for the fifth corresponding data point in figure 3 .
Figure D9 : EMCCD images of the MPEG7 objects. Images of the 25 MPEG7 objects for both the untrained correlated and uncorrelated illuminations for the sixth corresponding data point in figure 3 .
Figure D10 : EMCCD images of the MPEG7 objects. Images of the 25 MPEG7 objects for both the untrained correlated and uncorrelated illuminations for the seventh corresponding data point in figure 3 .
Figure D11 : EMCCD images of the MPEG7 objects. Images of the 25 MPEG7 objects for both the untrained correlated and uncorrelated illuminations for the eighth corresponding data point in figure 3 .
Figure D12 : Examples of single frames of the MPEG7 objects. Examples of single frames from each class that were fed as part of the input to the set Transformer. The data is from the first corresponding point for the untrained correlated and uncorrelated illumination points in figure 3 .
Figure D13 : Examples of single frames of the MPEG7 objects. Examples of single frames from each class that were fed as part of the input to the set Transformer. The data is from the second corresponding point for the untrained correlated and uncorrelated illumination points in figure 3 .
Figure D14 : Examples of single frames of the MPEG7 objects. Examples of single frames from each class that were fed as part of the input to the set Transformer. The data is from the third corresponding point for the untrained correlated and uncorrelated illumination points in figure 3 .
Figure D15 : Examples of single frames of the MPEG7 objects. Examples of single frames from each class that were fed as part of the input to the set Transformer. The data is from the fourth corresponding point for the untrained correlated and uncorrelated illumination points in figure 3 .
Figure D16 : Examples of single frames of the MPEG7 objects. Examples of single frames from each class that were fed as part of the input to the set Transformer. The data is from the fifth corresponding point for the untrained correlated and uncorrelated illumination points in figure 3 .
Figure D17 : Examples of single frames of the MPEG7 objects. Examples of single frames from each class that were fed as part of the input to the set Transformer. The data is from the sixth corresponding point for the untrained correlated and uncorrelated illumination points in figure 3 .
Figure D18 : Examples of single frames of the MPEG7 objects. Examples of single frames from each class that were fed as part of the input to the set Transformer. The data is from the seventh corresponding point for the untrained correlated and uncorrelated illumination points in figure 3 .
Figure D19 : Examples of single frames of the MPEG7 objects. Examples of single frames from each class that were fed as part of the input to the set Transformer. The data is from the eighth corresponding point for the untrained correlated and uncorrelated illumination points in figure 3 .
Figure D20 : Classification accuracy curve for the MPEG7 experiment. Classification accuracy as a function of illumination power when 2 shots were used for inference
Figure D21 : Classification accuracy curve for the MPEG7 experiment. Classification accuracy as a function of illumination power when 6 shots were used for inference
Figure D22 : Classification accuracy curve for the MPEG7 experiment. Classification accuracy as a function of illumination power when 20 shots were used for inference
Figure D23 : Classification accuracy curve for the MPEG7 experiment. Classification accuracy as a function of illumination power when 50 shots were used for inference
Figure D24 : Classification accuracy curve for the MPEG7 experiment. Classification accuracy as a function of illumination power when 100 shots were used for inference
Figure D25 : Classification accuracy curves for the MPEG7 experiment. Extended data from Figure 3 .
Figure D26 : Classification accuracy curves for the MPEG7 experiment. Classification accuracy as a function of the number of shots used for inference when photons were used for illumination. Extended data from Figure 3 .
Figure D27 : Mean-field illumination patterns for the MPEG7 experiment. Trained illumination for and average clicks per shot from Figure 3 .
Figure D28 : Simulations of cell organelle classification with different illuminations. Classification accuracy surfaces for the illuminations as a function of the number of illumination photons per shot and the number of shots used for classification. The dotted lines depict the cross sections of these curves that were shown in Figure 4 .
Figure D29 : Simulation: Effect of loss on correlated illumination.
Figure E1 : Comparison between different experimental demonstrations of object/image classification in ultra-low-light conditions.