In Situ Training of Optical Neural Networks by Coding Agents
Authors: Ilker Oguz, Colin Christian, Christophe Moser, Demetri Psaltis, Gordon Wetzstein
Organizations: Department of Electrical Engineering, Stanford University, Stanford, CA, USA · School of Engineering, Institute of Electrical and Micro Engineering, Ecole polytechnique fédérale de Lausanne, Lausanne, Switzerland
Optical neural networks promise fast, energy-efficient computing, but every device departs from its design and must be trained with the hardware. In situ training relies on an expert-built calibrated digital model of each device or on gradient-free trial and error. Coding agents, large language models that write and run programs, have shown an emergent ability to understand and edit the weights of digital neural networks. Here we introduce optics as code, a method in which a coding agent trains an optical neural network from measurements alone, through a harness that returns camera images of its probes and requires a hypothesis per request. We discovered that the agent forms a physical understanding of the optics and reparametrizes the network with interpretable optical elements tuned on the hardware. It outperforms gradient-free methods in classification and image encoding, approaching backpropagation through a calibrated model at a fraction of the cost. Needing neither per-device calibration nor gradients through the hardware, and improving with more capable models, optics as code offers a practical route to large, scalable optical neural networks and a general tool for calibrating and optimizing optical experiments.
Figures & tables
Figure 1: A coding agent trains an optical neural network in situ, from measurements alone. a , Conventional deployment of optical neural networks. Phase masks trained in silico on an ideal model (angular-spectrum method, ASM) lose accuracy on the optical experiment, where gradients are not directly accessible. b , The two earlier routes to in situ training, and optics as code, in which a large language model writes code whose output, the phase masks θ and the probes, is displayed on the experiment. f^ , calibrated model. L , loss. r , reward. c , Photograph and schematic of the experiment. SLM, spatial light modulator. d , The coding agent, a large language model, reads camera images, scores and its notes, and writes hypotheses, code, phase masks and notes of its findings. It reaches the experiment only through the harness, the software that executes its requests and returns the measurements. The test set is sealed. e , Best validation accuracy versus measurements on the experiment for the agent with the notes of an earlier session (circles, validations) and without, and for proximal policy optimization (PPO) and policy gradient (PG). Diamond, backpropagation through a calibrated model, whose calibration alone used 67,000 camera images (arrow, beyond the axis). Dotted, start. Right, φ2 (top) and camera data (bottom) at numbered reasoning steps. 1, probes verifying the earlier notes. 2, the class displaces the focused spots (difference of two classes). 3, one spot per block of phase mask φ2 . 4, the agent’s first own design from these spots.
Figure 2: The agent trains an optical classifier on the experiment to the accuracy of backpropagation. a , Schematic of the experiment, and the two phase masks at the start, at the agent’s first own design and at the end of the best agent run, given the notes of an earlier session, with the measurements spent. b , Test accuracy on the 3,147 test digits at the end of the budget, for the best run of each method. Filled circle, agent with the notes of an earlier session. Open circle, agent without notes. Squares, PPO and policy gradient. Dashed, phase masks trained by backpropagation through a calibrated model of the device and scored on the experiment (96.3%). Dotted, start. c , Camera images averaged over the practice digits of each class, at the three stages, with the three detectors. d , Fraction of the detected energy on each detector, per digit class. e , Confusion matrix, in per cent of the digits of each class. Start and final phase masks in d and e , 3,147 test digits. First design, 128 practice digits.
Figure 3: The agent trains an optical encoder that learns a compact representation of images. The optics compress each Fashion-MNIST image into 16 detector signals, which play the role of the latent space of an autoencoder, and a digital decoder reconstructs the image from them. a , Phase masks φ1 and φ2 with random values, after an agent run without notes and after the agent run that received the notes and code of an earlier session, with the mean test peak signal-to-noise ratio (PSNR). b , Ground-truth test images (top) and their reconstructions by the digital decoder for the phase masks of a (arrows), with the PSNR of each reconstruction. The five images are spread evenly over the 50 test images that the agent run with notes improved most. c , Progress during training of the two agent runs of a and of PPO, the gradient-free baseline without a language model. Lines, gain in validation PSNR over the random start. Markers, single validations. The run with notes starts high because the agent rebuilt the earlier design before its first measurement. d , Gain in test PSNR over the random start at the end of the budget, one run per row, for three language models (Claude Fable 5.2, Opus 5.5 and Sonnet 5) and for PPO. Filled circles, runs with the notes and code of an earlier session. Open circles, runs with general lessons only. Square, PPO. Dashed, phase masks trained by backpropagation through a calibrated model.
Figure 4: The agent understands the experiment through its own measurements and designs the optics from this understanding. a , Probes chosen and written by the agent on the input and the two phase masks (top), and their camera images (bottom). Each probe occupies one plane, with the others flat. A disk locates each plane on the camera, and a phase step tests interference across each phase mask. b , Displacement of the intensity centroid on the camera for the same tilt (20 phase cycles) on each plane. Dots, x and y tilts. c , Fringe contrast for the same phase step on each phase mask. Dotted, flat. d , Both phase masks at the start and after the two routes of the runs, fine-tuning or a new design. The two new designs come from separate runs. The Fourier router (run from the task text alone) splits φ1 into four lens sections. The spot router (run with earlier notes) focuses 49 blocks of φ2 onto the detectors. Numbers, test accuracy. e , Test accuracy on the calibrated model with reduced information for the agent, and with earlier notes (filled circle). Square, PPO. Added rules did not help (Methods). f , Reasoning traces of the run from the task text alone, in seven steps from its first probe to the tuned Fourier router. Each step gives the hypothesis, the probe (blue) or design (red) that tested it, the finding, what the agent learned and the validation accuracy that followed. Top, measurements used (Methods).
Optical neural networks (ONNs) are attracting significant attention as alternative platforms for machine learning (ML). In an ONN, an optical module often takes over part of the computation inside a digital ML pipeline. How much of the task does the optical module itself perform, and how much does it contribute to any claimed system-level advantages, such as throughput or energy? This paper offers a simple, function-oriented evaluation methodology. Whatever its internal physics, the optical module appears to the surrounding digital pipeline as a mapping between real-valued tensors. We therefore evaluate it in three steps: (i) define its boundary in the pipeline, as real-valued tensors in and out; (ii) replace it with standard digital modules to identify those that match the optical system's performance on the same task and metric; and (iii) state system-level claims, such as speed or energy, per inference against the performance-matched digital module. We summarize steps (i) and (ii) in an Expressivity Card and demonstrate its use on published ONN systems. This helps isolate the optical module's contribution and provides a concrete digital reference for assessing its claimed system-level advantages, without the need to understand the physics. While we focus on optical systems as the most prominent example, the same approach applies to any physical analog computing platform used in ML, wherever a physical module takes over part of a digital pipeline.
Shi-Yuan Ma, Xingjian Bai, Ryan Hamerly +1
Massachusetts Institute of Technology, Cambridge, MA, USA. · NTT Research Inc., Sunnyvale, CA, USA.
Distributed learning is widely used for training large models on large datasets by distributing parts of the model or dataset across multiple devices and aggregating the computed results for subsequent computations or parameter updates. Existing communication algorithms for distributed learning such as ring all-reduce result in heavy communication overhead between servers. Since communication in large-scale systems uses optical fibers, we propose an Optical In-Network-Computing (OptINC) architecture to offload the computation in servers onto the optical interconnects. To execute gradient averaging and quantization in the optical domain, we incorporate optical devices such as Mach-Zehnder-Interferometers (MZIs) into the interconnects. Such a de facto optical neural network (ONN) can effectively reduce the communication overhead in existing distributed training solutions. To reduce dataset complexity for training this neural network, a preprocessing algorithm implemented in the optical domain is also proposed. Hardware cost is lowered by approximating the weight matrices of the optical neural network with unitary and diagonal matrices, while the accuracy is maintained by a proposed hardware-aware training algorithm. The proposed solution was evaluated on real distributed learning tasks, including ResNet50 on CIFAR-100, and a LLaMA-based network on Wikipedia-1B. In both cases, the proposed framework can achieve comparable training accuracy to the ring all-reduce baseline, while eliminating communication overhead.
Sijie Fei, Grace Li Zhang, Bing Li +1
Technical University of Munich, Munich, Germany · Technical University of Darmstadt, Darmstadt, Germany · Technical University of Ilmenau, Ilmenau, Germany
Equilibrium Propagation offers a compelling alternative to traditional machine learning for training energy-based networks. Here we demonstrate a hybrid optical-digital implementation of EP using a Spatial Photonic Ising Machine (SPIM). The SPIM exploits the gauge transformation method to optically encode both continuous neuron states and rank-1 binary trainable patterns as phase modulations via a spatial light modulator, with inference realized using a finite difference scheme. The experimental system is evaluated on the Wine classification dataset. The potential of this approach, including the use of continuous couplings and structured coupling matrices, is evaluated numerically on the more complex MNIST dataset. Our work provides a concrete pathway toward energy-efficient physical implementations of Equilibrium Propagation.