Training single-electron and single-photon stochastic physical neural networks
Authors: Tong Dou, Shiro Kumara, Josh Burns, Ethan Sigler, Parth Girdhar, David Petty, Gerard Milburn, Jo Plested, Matt Woolley
Organizations: School of Engineering and Technology, UNSW Canberra, Campbell, ACT 2600, Australia · Sussex Centre for Quantum Technologies, University of Sussex, Brighton BN1 9RH, United Kingdom · Department of Engineering Science, University of Oxford, Oxford OX1 3PJ, United Kingdom · National Quantum Computing Centre, Rutherford Appleton Laboratory, Didcot OX11 0QX, United Kingdom · School of Systems and Computing, UNSW Canberra, Campbell, ACT 2600, Australia
The computational demands of deep learning motivate the investigation of alternative approaches to computation. One alternative is physical neural networks~(PNNs), in which learning and inference are performed directly via physical processes. Stochastic PNNs arise when the underlying neurons are realized by the dynamics of a stochastic activation switch. Here we propose novel electronic and photonic stochastic neurons. The electronic realization is implemented by single-electron tunneling through a quantum dot. The photonic realization is implemented via a single-photon source driving one of two modes coupled via a controllable beam-splitter-like interaction. In the electronic case, the charge state of the quantum dot forms the basis for the stochastic neuron, whereas in the photonic case the occupation of the undriven mode serves as the basis for the stochastic neuron. Training of stochastic PNNs is performed with models of stochastic neurons, as well as with coherently-driven, single-photon detector stochastic neurons previously introduced. Several training strategies for MNIST handwritten digit classification have been investigated using single-hidden-layer stochastic PNNs, including varying the number of trials in each layer to control forward pass stochasticity and employing either true probability or empirical outputs in the backward pass to evaluate their influence on gradient estimation. We show that when empirical outputs are used in the backward pass, the network achieves more than 97% test accuracy with few trials per layer. Despite the simplicity of the model architecture, high test accuracy is maintained in the presence of a high degree of noise and model uncertainty. The results demonstrate the potential of embracing stochastic PNNs for deep learning.
Figures & tables
Figure 1: Schematic of an SPD stochastic neuron adapted from [ 17 ] . The mean photon number λ(z) incident on the detector in a given measurement interval is a function of the pre-activation z . The output of the SPD neuron is modeled as the stochastic output from the single-photon detector, given by h=fSPD(z) , where h=0 corresponds to no-clock and h=1 corresponds to a click.
Figure 2: Schematic of a SET stochastic neuron realized in a semiconductor quantum dot system. The pre-activation z modulates the dot energy level ε(z) via the gate electrode. Electron tunneling between the dot and leads (source and drain) is governed by the tunneling rates ΓS/D± , where the total electron tunneling rates in and out of the dot are defined as Γin=ΓS++ΓD− and Γout=ΓD++ΓS− , respectively. The resulting dot occupation can be modeled as the output of the SET stochastic neuron, which is monitored by the current IQPC of a nearby quantum point contact (QPC), provided by h=fSET(z) , where h=0 denotes the unoccupied state ( N−1 electrons) and h=1 denotes the occupied state ( N electrons) of the quantum dot.
Figure 3: Schematic of a TSP stochastic neuron realized in an optomechanical system. The neuron is driven by an incident single-photon pulse ξ(t) with a temporal decay rate ζ . The pre-activation z modulates the interaction between the cavity mode a and the mechanical mode b via the time-dependent coherent control pulse α(t) . The system evolution is subject to the cavity decay rate κ and mechanical dissipation γ . Driven by the control pulse, the single-photon state is stochastically transferred to the mechanical mode, whose occupation at any time t is at most unity, providing a stochastic binary neuron output. The neuron activation can be obtained by measuring the mechanical occupation number, provided by h=fTSP(z) , where h=0 denotes the unoccupied state and h=1 denotes the occupied state of mode b .
Figure 4: Benchmarking stochastic physics-aware training on SPD, SET, and TSP stochastic neurons. The TP approach is applied to both the hidden and output layers. Inset: Activation probabilities pPSN(z) as functions of the pre-activation z for corresponding PSNs. The curves are plotted over different z -ranges: [−3,3] for SPD, [−5,5] for SET, [−50,50] for TSP, and for TSP neuron, parameters are set to t=0.21 , γ=0.02 , κ=30.0 , ζ=10.7 . All three PSNs support effective training even with only 1 trial in the hidden layer, and test accuracy improves as the number of trials increases.
Figure 5: Performance of the EG estimator under finite sampling constraints compared to the TP baseline. The EG estimator applied to the hidden layer only and to both layers enables effective training under finite-sampling constraints ( K≤10 ). Test accuracy quickly approaches TP upper bound as the trial count increases. K=1 is omitted in all plots for the EG estimators in the hidden layer dues to vanishing empirical gradients.
Figure 6: Performance of ST estimators under finite sampling. Applying the identity ST estimator to the hidden layer limits the learning capacity, with test accuracy saturating below roughly 93% across all evaluated output configurations. In contrast, applying the EG estimator in the hidden layer while retaining a ST estimator in the output layer yields highly competitive performance ( >98% ). The configuration applying the TP approach in both hidden and output layers is a baseline. Note that for K=1 , the EG estimator at the output layer is enabled by introducing a smoothing parameter ϵ (detailed in Sec. III.4 ).
Figure 7: Output-layer performance under infinite and few trial regimes. Finite output-layer sampling remains effective under strict trial-budget constraints ( K≤10 ). Test accuracy approaches the infinite-trial limit as the trial count increases.
Figure 8: Comparison of test accuracy using softmax and unnormalized linear activations in the output layer for models with (a) a single hidden layer and (b) two hidden layers. While the linear and MSE formulation underperforms in shallow networks, increasing the network depth to 2 hidden layers substantially improves its accuracy, making it comparable to the 1-layer softmax and CE configuration.
Science Division, New York University Abu Dhabi, UAE · eBrain Lab, Division of Engineering, New York University Abu Dhabi, PO Box 129188, Abu Dhabi, UAE · Center for Quantum and Topological Systems, NYUAD Research Institute, New York University Abu Dhabi, UAE +1
Department of Materials Science and Engineering, Delft University of Technology, Mekelweg 2, Delft, 2628CD, The Netherlands · Department of Precision and Microsystems Engineering, Delft University of Technology, Mekelweg 2, Delft, 2628CD, The Netherlands · Department of Intelligent Systems, Delft University of Technology, Van Mourik Broekmanweg 6, Delft 2628XE, The Netherlands