Physical policy gradient theorem for in situ stochastic-adjoint training

William Tuxbury; Zin Lin

physics.opticsSep 5, 2026

Physical policy gradient theorem for in situ stochastic-adjoint training

Authors: William Tuxbury, Zin Lin

Organizations: Bradley Department of Electrical and Computer Engineering, Virginia Tech, Blacksburg, Virginia 24061, USA

Abstract

In situ adjoint training extracts parameter gradients directly from measurement, but has so far been limited to reciprocal or restricted systems. Here, we introduce the physical counterpart of the policy gradient theorem: a stochastic-adjoint gradient estimator that lifts these constraints by trading reciprocity for nondegenerate diffusion. As validation, we train a nonlinear resonator network, whose own dynamics supply the policy, against antagonistic temporal modulations with gradients from measured stochastic trajectories alone, without finite differences or a separate adjoint experiment.

Figures & tables

Figure 1: Physical Policy Training (a) Expected return and (b) trajectories per iteration versus training iteration for antagonistic modulation amplitudes $\sigma_{\kappa}$ indicated by the common legend in (a) . Inset: continuations of the $\sigma_{\kappa}=0$ optimization from a common breakpoint (vertical dashed line) at escalated, constant, and de-escalated trajectory budgets, overlaid with $25$ -iteration rolling averages. The green ( $\mathcal{E}=25{,}000$ ) and black ( $\mathcal{E}=20{,}000$ ) curves share a similar rate of progress, while the red ( $\mathcal{E}=1{,}000$ ) curve drifts off the gradient flow. Hence, the trajectory budget was over-provisioned, but without escalation the achievable optimum is limited by the fixed noise floor of the gradient estimate (see End Matter). (c) Resonator frequency, (d) internal coupling, and (e) Kerr coefficient parameters versus training iteration for $\sigma_{\kappa}=0.2$ . Curve opacities are set by the time-integrated field exposure of the respective elements, and since high-exposure sites are near $\chi_{n}=0$ , nonlinearity is largely decorative in the optimization; find opacity mapping details in the text. (f) Network diagram of the trained optimum for $\sigma_{\kappa}=0.2$ forming an effective waveguide (green bonds) with detuned flanks. The input (output) port is indicated with a green triangle (blue star), resonator frequencies are represented by their disk radii, coupling magnitude by bond thickness and color, and Kerr coefficient by disk color (color bars are beneath the diagram with parameter mapping details in the text).

Figure 2: Estimator Properties Convergence of single components of the gradient estimate associated with resonator frequency for (a) $\sigma_{\kappa}=0$ and (b) $\sigma_{\kappa}=0.2$ versus the number of trajectories accumulated in the ensemble average. Respective iteration and site index were selected to favor the resolvability of systematic bias, avoiding pathologies (e.g., lead-coupled sites or detuned flanks). The horizontal dashed lines indicate the finite-difference values (using $\mathcal{E}=1{,}000$ with common noise realizations across the differenced legs) and the vertical dashed lines indicate the maximum trajectory budget in optimization (see Fig. 1 ). (c) Gradient estimate variance versus instrumental noise amplitude with (solid) and without (dashed) baselines computed at a representative mid-training iteration ( $800$ ) using $\mathcal{E}=1{,}000$ , shown for $\sigma_{\kappa}=0,0.05,$ and $0.2$ . Apart from the baselined $\sigma_{\kappa}=0$ case, whose variance is independent of small $\sigma_{W}$ before the score goes numerically singular, all follow the $\sim\sigma_{W}^{-2}$ dotted trend line. (d) Bias in the expected return due to $\sigma_{W}\neq 0$ using $\mathcal{E}=10{,}000$ , taken as a paired difference against $\sigma_{W}=0$ under common noise realizations, for each $\sigma_{\kappa}$ in the legend. The $\sim\sigma_{W}^{2}$ dotted trend line shows that excessive instrumental noise biases the measured return (see End Matter). Inset: Standard deviation of the return with dotted trend line $\sim\sigma_{W}$ . The visible crossovers are controlled by whether $\sigma_{W}$ or $\sigma_{\kappa}$ dominate return variance. The vertical dashed lines in (c) and (d) indicate the $\sigma_{W}$ value used in optimization (see Fig. 1 ), so a contribution $\Delta J\approx 0.01$ to the expected return is attributable to the instrumental noise.

Policy Gradient Methods

Policy gradient methods optimize stochastic policies by estimating how parameter changes affect expected reward.

See all 158 papers

Unifying Physical Backpropagation
Aug 12, 2026Cyrill Bösch, Yigithan Gediz, Clara C. Wanjura +1On-Device Training Backpropagation
Department of Computer Science, Princeton University, Princeton, NJ 08540, USA · Department of Electrical and Computer Engineering, Princeton University, Princeton, NJ 08540, USA
Automatic Differentiation from Scratch: How PyTorch Computes Gradients in Physics-Informed Neural Networks
Jun 13, 2026Abdeladhim TahimiAutomatic Differentiation Backpropagation
CECA, Universidade Federal de Alagoas (UFAL), Rio Largo, Alagoas, 57100-000, Brazil
SNAPO: Smooth Neural Adjoint Policy Optimization for Optimal Control via Differentiable Simulation
May 7, 2026Dmitri Goloubentsev, Natalija KarpichinaPolicy Optimization Differentiable Programming
MatLogica, U.K.

Aug 12, 2026J/K move · Enter open · S save

Unifying Physical Backpropagation

Cyrill Bösch, Yigithan Gediz, Clara C. Wanjura, Hakan E. Türeci

Department of Computer Science, Princeton University, Princeton, NJ 08540, USA · Department of Electrical and Computer Engineering, Princeton University, Princeton, NJ 08540, USA

On-Device Training Backpropagation

Open paper AskPDF arXiv

Physical computing systems exploit device dynamics for computation, but their gradient-based optimization is challenging: backpropagation through a digital twin suffers from a model-reality gap. On-device gradient computation could resolve this issue, and a handful of theoretical and experimental studies have proposed ways to achieve it. Yet a unifying theory identifying when a physical system can compute the gradient of its own performance has been missing. Here we develop such a unification based on the adjoint method: we identify sufficient conditions under which the adjoint field required for formally exact gradients can be generated on the same hardware that performs the computation. Linear and nonlinear systems obey fundamentally different conditions: for linear systems, damping or gain is admissible provided reciprocity is preserved. For nonlinear trajectory systems, the sufficient conditions are reciprocity of the linearized system and the existence of a time-reversal mirror. Algorithmically, the nonlinear case requires infinitesimal nudging, whereas linear systems admit a finite-amplitude experiment. We recover (quantum) Equilibrium Propagation, Hamiltonian echo backpropagation, fully forward mode training and in situ gradient methods in integrated-photonic and free-space-optical systems. Finally, we show that reciprocity is a special case of more general intertwining conditions. For linear systems, these permit exact on-device gradients in a class of non-Hermitian, non-reciprocal systems. For nonlinear trajectories, they combine with generalized time-reversal mirrors to cover, e.g., PT-symmetric equations. The framework also includes time-dependent parameters and Onsager-reciprocal dynamics, providing a unified basis for formally exact physical learning.

Abstract

Policy Gradient Methods

Explore similar work

Unifying Physical Backpropagation

Automatic Differentiation from Scratch: How PyTorch Computes Gradients in Physics-Informed Neural Networks

SNAPO: Smooth Neural Adjoint Policy Optimization for Optimal Control via Differentiable Simulation