physics.opticsNov 18, 2026

In Situ Training of Optical Neural Networks by Coding Agents

Authors: Ilker Oguz, Colin Christian, Christophe Moser, Demetri Psaltis, Gordon Wetzstein

Organizations: Department of Electrical Engineering, Stanford University, Stanford, CA, USA · School of Engineering, Institute of Electrical and Micro Engineering, Ecole polytechnique fédérale de Lausanne, Lausanne, Switzerland

Abstract

Optical neural networks promise fast, energy-efficient computing, but every device departs from its design and must be trained with the hardware. In situ training relies on an expert-built calibrated digital model of each device or on gradient-free trial and error. Coding agents, large language models that write and run programs, have shown an emergent ability to understand and edit the weights of digital neural networks. Here we introduce optics as code, a method in which a coding agent trains an optical neural network from measurements alone, through a harness that returns camera images of its probes and requires a hypothesis per request. We discovered that the agent forms a physical understanding of the optics and reparametrizes the network with interpretable optical elements tuned on the hardware. It outperforms gradient-free methods in classification and image encoding, approaching backpropagation through a calibrated model at a fraction of the cost. Needing neither per-device calibration nor gradients through the hardware, and improving with more capable models, optics as code offers a practical route to large, scalable optical neural networks and a general tool for calibrating and optimizing optical experiments.

Figures & tables

Explore similar work

CardsList
  1. Optical Neural Networks Should Be Evaluated by Function, Not Only Physics

    Nov 17, 2026Shi-Yuan Ma, Xingjian Bai, Ryan Hamerly +1Physical Neural NetworksAnalog Neural Networks

  2. OptINC: Optical In-Network-Computing for Scalable Distributed Learning

    Mar 30, 2026Sijie Fei, Grace Li Zhang, Bing Li +1Gradient QuantizationCommunication-Efficient Distributed Training

  3. Optical Implementation of Equilibrium Propagation Using Spatial Photonic Ising Machines

    Jun 11, 2026Dimitri Vanden Abeele, Daniele Veraldi, Davide Pierangeli +2Equilibrium PropagationEnergy-Efficient ML