Photonic convolutional neural network with pre-trained in situ training

Saurabh Ranjan; Sonika Thakral; Amit Sehgal

cs.ETApr 2, 2026

Photonic convolutional neural network with pre-trained in situ training

Authors: Saurabh Ranjan, Sonika Thakral, Amit Sehgal

Organizations: Shaheed Sukhdev College of Business Studies, University of Delhi, New Delhi, Delhi 110085 · Hansraj College, University of Delhi, New Delhi, Delhi 110007

Abstract

Convolutional neural networks (CNNs) have transformed image processing, but the energy consumption and inference latency of electronic based implementations remain fundamental bottlenecks. These limitations have motivated the search for alternative hardware architectures beyond Complementary metal-oxide-semiconductor (CMOS) chips. Optical systems can perform linear matrix operations at the speed of light with extremely low energy dissipation, making them attractive for CNN acceleration. However, building a fully coherent photonic CNN that performs both linear and nonlinear operations and training it efficiently remains an open challenge. Here we present a fully photonic convolutional neural network (PCNN) that executes image classification in the optical domain, including convolution, max-pooling, nonlinear activation, and fully connected layers. The network achieves 94.49 percent accuracy on the MNIST dataset distributed across Mach Zehnder Interferometer (MZI) meshes, weighted Multimode Interferometer (MMI) trees, and a microring resonator based nonlinearity. A mathematically exact differentiable digital twin, enables backpropagation for ex situ pre training, reaches 97.45 percent digital accuracy. Trained phases are transferred one-to-one to the photonic hardware and refined via a gradient free algorithm that estimates the full gradient with only two forward passes. The architecture exhibits inherent robustness to non idealities, under the compound effect of propagation loss, MZI insertion loss, fabrication disorder, and thermal crosstalk. A bottom-up power analysis yields 10.83 W static chip consumption and 843 ns inference latency, translating to 220 to 330 times greater energy efficiency than state of the art electronic GPUs for single-image inference.

Figures & tables

Figure 1: Block diagram of the PCNN and its operation in MNIST data classification. A $28\times 28$ input image of digit “1” is amplitude-modulated at 1 GHz and propagated through the photonic pipeline. Conv1 extracts 4 feature channels ( $26\times 26$ ) via MZI mesh processing; Pool1 reduces spatial dimensions to $13\times 13$ using all-optical MMI+GST comparators; Conv2 deepens to 8 channels ( $11\times 11$ ) via depthwise separable MZI trees (2A) and pointwise CMXUs (2B); Pool2 applies WDM-multiplexed max-pooling ( $5\times 5$ ); FC1 + NOFU produce 32 nonlinearly activated features via microring resonators; FC2 maps to 10 output probabilities, correctly predicting digit 1 with high confidence.

Figure 2: Conv1 patch extraction flow, matrix fidelity, and physical layout. (a) Conv1 patch extraction and kernel multiplication flow: a $3\times 3$ sliding window traverses the $28\times 28$ input image in raster order at 1 GHz, yielding 676 overlapping patches. Each patch is multiplied element-wise with Kernel 0 to produce Feature Map 0. (b) $10\times 10$ CMXU matrix fidelity distribution over 500 Haar-random unitary matrices under direct programming (orange, $\langle F\rangle=0.899$ ) and SPSA-corrected programming (blue, $\langle F\rangle=0.920$ ). Both distributions assume $\sigma_{\theta}=0.04$ rad phase noise and nearest-neighbour crosstalk $\xi=0.05$ . (c) Layout schematic of the $10\times 10$ Coherent Matrix Multiplication Unit (CMXU), consisting of 45 MZIs configured in a symmetric Clements mesh topology split across 10 layers.

Figure 3: All-optical Pool1 max-pooling hardware. (a) Schematic of the Optical Max Unit (OMU): two data signals $a$ and $b$ (at $\lambda_{0}$ ) are compared in the power domain. Their control copies at $\lambda_{1}$ are subtracted via M1 and the difference is injected into a bistable Graphene-on-Silicon microring (Disk #1) biased by a reference signal $r_{0}$ . The comparator output $C$ is inverted by an all-optical NOT gate (Disk #2) to yield $\overline{C}$ ; these complementary signals gate GST-clad attenuators ATN1 and ATN2, selecting the larger field, while add-drop filters AD1 and AD2 remove the $\lambda_{1}$ control tone to deliver the winner exclusively at $\lambda_{0}$ . (b) Lorentzian through-port transmission of the GST-clad microring attenuator for the amorphous (blocking, $L_{\text{block}}=20.0$ dB) and crystalline (passing, $L_{\text{pass}}=0.3$ dB) phase states, yielding a $19.7$ dB extinction ratio. (c) Simulated optical intensity $|E|^{2}$ in the WDM add-drop microring filter: at $\lambda_{1}=1568.7$ nm (left) the field couples into the ring and exits the drop port with $25$ dB isolation; at $\lambda_{0}=1553.6$ nm (right) the bus waveguide is off-resonance and the field passes with only $0.3$ dB loss.

Figure 4: Architectural dataflow and pipeline of Convolutional Layer 2. The $13\times 13\times 4$ feature maps from Pool1 are processed via depthwise-separable convolution. First, in Step 2A (Depthwise Convolution), four parallel $3\times 3$ kernels (each implemented with a 9-tap delay line and a $9\rightarrow 1$ MZI combiner tree, for a total of 32 MZIs) perform spatial filtering on each channel independently, yielding a $11\times 11\times 4$ intermediate volume. Second, in Step 2B (Pointwise Convolution), a single $8\times 8$ Clements mesh (64 parameters) mixes the 4-channel intermediate fields across channels, yielding the final $11\times 11\times 8$ feature maps for input to Pool2.

Figure 5: WDM Pooling Layer 2 and Nonlinear Optical Function Unit (NOFU). (a) WDM Pool 2 architecture: eight C-band wavelengths $\lambda_{1}$ – $\lambda_{8}$ are combined via an $8\times 1$ AWG multiplexer into a single shared delay network (4 spiral delay lines). A $1\times 8$ AWG demultiplexer bank separates the aligned outputs to individual lanes; eight OMU-based max-pool units compare values using 1310 nm O-band control signals (crosstalk-free), and WDM bandpass filters remove the control path, delivering the $5\times 5\times 8$ feature volume. (b) Optoelectronic feedback schematic of one NOFU unit: a tunable directional coupler (tap fraction $\beta$ ) routes a fraction of the input field to a photodiode ( $R_{\text{pd}}$ ), whose photocurrent is injected under bias $V_{B}$ into the carrier-injection section of a high-Q microring (detuning $\Delta\lambda$ ), nonlinearly modifying the through-port output $E_{\text{out}}$ . (c) Simulated power transmission $T$ vs. input power $P_{\text{in}}$ ( $\beta=0.35$ , $V_{B}=0.8$ V, three detunings). All curves begin near $T\approx 0.63$ ; the on-resonance and $+0.03$ nm curves sweep through a deep minimum as carrier injection shifts the resonance through the operating wavelength. (d) Corresponding nonlinear phase shift $\Delta\Phi_{\text{NL}}$ , accumulating a negative shift reaching ${\sim}{-1.5}$ rad at the resonance crossing.

Figure 6: PCNN training, fine-tuning, and hardware evaluation. ( a ) Digital twin pre-training: training loss (blue, left axis) and train/test accuracy (orange/green, right axis) over 100 epochs on the param Rudra A100 cluster; test accuracy peaks at $\mathbf{97.45\%}$ at epoch 96 with no overfitting. ( b ) In situ SPSA fine-tuning: cross-entropy loss (left) and validation accuracy (right) vs. SPSA step; the best checkpoint at step 300 yields $\mathbf{94.49\%}$ on the full MNIST test set. ( c ) Hardware confusion matrix ( $10{,}000$ images) after fine-tuning; the strong diagonal confirms uniformly high per-class accuracy across all ten digit classes.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 1: MZI transfer function. Simulated optical transmission through the bar (blue) and cross (red) ports as a function of the internal phase shift $\theta_{1}$ .

Figure 2: $9\rightarrow 1$ MZI binary combiner tree for Conv2 Step 2A (depthwise convolution). Nine $3\times 3$ patch inputs ( $x_{0}$ – $x_{8}$ ) are coherently combined into a single output through four tree levels using 8 Mach–Zehnder interferometers. Terminated ( $\times$ ) stubs denote discarded output ports whose power is absorbed; all useful optical power is routed to the single output port. Four identical trees operate in parallel (one per input channel) for a total depthwise count of 32 MZIs and 64 phase parameters.

Time	Event	Registers filled	Buffer state
$t=330$ ns	Pool2 output starts	R[0], R[25],…,R[175]	8 / 200
$t=331$ ns	2nd sample	R[1], R[26],…,R[176]	16 / 200
$t=332$ ns	3rd sample	R[2], R[27],…,R[177]	24 / 200
$\vdots$	$\vdots$	$\vdots$	$\vdots$
$t=354$ ns	25th sample	R[24], R[49],…,R[199]	200 / 200
$t=355$ ns	HOLD phase begins	all 200 stable	ready

Appendix

Table 1: FC1 buffer filling timeline. Pool 2 first output arrives at

t\approx 330

ns from the start of frame processing.

Component	Conventional	WDM-based
Delay networks	72 mm 2 ( $N\times 9$ )	9 mm 2 ( $1\times 9$ )
AWG combiner / demux	—	5 mm 2
Max-pool units	0.56 mm 2	0.56 mm 2
Routing / misc.	3 mm 2	3 mm 2
Total	$\sim$ 81 mm 2	$\sim$ 19 mm 2

Appendix

Table 2: Pool2 component-level area at

N=8

: conventional vs. WDM.

Channels ( $N$ )	Conventional	WDM-based	Savings	Reduction
4	$\sim$ 40 mm 2	$\sim$ 32 mm 2	8 mm 2	20%
8	$\sim$ 81 mm 2	$\sim$ 19 mm 2	62 mm 2	76%
16	$\sim$ 162 mm 2	$\sim$ 26 mm 2	136 mm 2	84%
32	$\sim$ 324 mm 2	$\sim$ 40 mm 2	284 mm 2	88%

Appendix

Table 3: Total area scaling with channel count

N

. Conventional scales as

\mathcal{O}(N)

; WDM scales as

\mathcal{O}(1)

(only AWG/pool units grow).

Figure 3: Effect of mini-batch size on PCNN pre-training ( a ) Training loss vs. epoch: batch 128 converges to 0.21 while batch 512 stagnates above 0.44. ( b ) Test accuracy vs. epoch: at epoch 5, batch 128 already achieves 86.5% while batch 512 reaches only 39.3%. Final best accuracy is 96.50% (batch 128) vs. 91.48% (batch 512). Batch 32 was terminated early due to GPU under-utilisation on the A100 hardware.

Parameter	Value	Reference
On-chip photonic components
Phase shifter $P_{\pi}$ (TiN thermo-optic)	10 mW	[ 1 ]
Phase shifter $P_{\mathrm{avg}}$ (mean bias $\pi/2$ )	5 mW	derived
Thermo-optic heaters ( $2\times N_{\mathrm{MZI}}$ )	1,562	verified
Phase-shifter subtotal	7.81 W	derived
NOFU microring (carrier injection)	0.5 mW each	[ 10 ]

Appendix

Table 4: Parameters used for energy efficiency calculation. On-chip photonic values are verified against the hardware implementation.

Phase shifter	Power	Energy/op	Ratio
Thermal (this work)	10.83 W	36.5 pJ/OP	1 $\times$
Undercut thermal [ 8 ]	5.36 W	18.1 pJ/OP	0.49 $\times$
MEMS [ 9 , 11 ]	3.02 W	10.2 pJ/OP	0.28 $\times$

Appendix

Table 5: Energy comparison under alternative phase-shifter technologies.

Explore similar work

Photonic AI: A Hybrid Diffractive Holographic Neural System for Passive Optical Real-Time Image Classification
Apr 14, 2026Prakul Sunil HiremathPhotonic Computing Image Classification
Department of Computer Science and Engineering, Visvesvaraya Technological University (VTU), Belagavi, India · Aliens on Earth (AoE) Autonomous Research Group, Belagavi, India
Small-scale photonic Kolmogorov-Arnold networks using standard telecom nonlinear modules
Apr 9, 2026Luca Nogueira Calçado, Sergei K. Turitsyn, Egor ManuylovichPhotonic Computing Kolmogorov-Arnold Networks
Aston Institute of Photonic Technologies (AiPT), Aston University, Birmingham, B4 7ET, United Kingdom.
Constrained Co-Design for Photonic Bayesian Neural Networks
Aug 3, 2026Hendrik Borras, Xiao Wang, Bernhard Klein +4Bayesian Neural Networks Uncertainty Quantification
Hardware and Artificial Intelligence Lab, Institute of Computer Engineering, Heidelberg University, Germany · Neuromorphic Quantumphotonics Kirchhoff-Institute for Physics, Heidelberg University, Germany

Apr 14, 2026J/K move · Enter open · S save

Photonic AI: A Hybrid Diffractive Holographic Neural System for Passive Optical Real-Time Image Classification

Prakul Sunil Hiremath

Department of Computer Science and Engineering, Visvesvaraya Technological University (VTU), Belagavi, India · Aliens on Earth (AoE) Autonomous Research Group, Belagavi, India

Photonic Computing Image Classification

Open paper AskPDF arXiv

Edge intelligence is constrained by the energy and latency costs of shuttling data through electronic memory hierarchies. Optical systems offer a fundamentally different computational regime: once an input wavefront is launched into a structured medium, propagation, diffraction, and interference jointly enact a linear transformation whose cost is determined by wave physics rather than by clocked arithmetic. This paper develops a rigorous systems-level treatment of that regime and introduces a hybrid diffractive holographic architecture for image classification. The proposed model couples a Diffractive Optical Neural Network (DONN) with a Holographic Interference-Based Learning (HIBL) operator a formal map from digitally optimized phase distributions to physically realizable, fabrication-compatible interference patterns embeddable in passive optical elements. We express the full inference pipeline as a composition of encoding, phase modulation, free-space propagation, and intensity measurement operators, making explicit which quantities are learned, which are fixed by design, and where nonlinearity enters through photodetection. This operator-theoretic view resolves a persistent gap in the optical-ML literature between learning a transformation and physically realizing it. In physics-informed simulation on MNIST, a three-layer system with approximately 25,000 phase elements achieves 91.2% test accuracy with propagation-limited nanosecond-scale latency. The primary contribution is not a performance claim but a precise computational framework: learned representations can be physically embedded into structured optical media so that inference is executed by wavefront transformation through a passive, fabricated object rather than by sequential electronic multiply accumulate operations.

Abstract

Explore similar work

Photonic AI: A Hybrid Diffractive Holographic Neural System for Passive Optical Real-Time Image Classification

Small-scale photonic Kolmogorov-Arnold networks using standard telecom nonlinear modules

Constrained Co-Design for Photonic Bayesian Neural Networks