An Educated Guess: Deriving Statistically Aligned Gradient Estimates for Zeroth-Order Optimisation
Organizations: Donders Institute for Brain, Cognition, and Behaviour, Radboud University, Nijmegen, the Netherlands
Abstract
As novel hardware architectures are explored for the purposes of machine learning, including neuromorphic chips, asynchronous systems, and ASICs, effective learning algorithms are highly desirable. While backpropagation is the standard for differentiation on standard hardware, its requirement for global synchrony and exact differentiability limits the exploration of exotic algorithms on non-traditional substrates and devices. Zeroth-order (ZO) optimisation methods, such as SPSA and weight perturbation, offer a compelling alternative by estimating gradients through inference-only passes; however, these methods have historically suffered due to the `curse of dimensionality', where random perturbations become increasingly misaligned with the true gradient in high-dimensional spaces. In this work, we move beyond blind random noise by deriving zeroth-order search directions from activation geometry available during the forward pass. This leads to a family of Spectral Zeroth-Order methods, SZO-, whose directions solve local response-based gradient-guessing objectives under explicit stochastic assumptions. These methods explore feature-supported directions that are more likely to produce informative loss responses. Using multilayer and convolutional networks, we analyse gradient alignment, bias-variance structure, and training stability of these methods relative to standard perturbation baselines and backpropagation.
Figures & tables
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
| Correction | What it does | Origin |
|---|---|---|
| Iterative layer-wise perturbation | Sequentially perturbs a single layer at a time across a network, removing inter-layer crosstalk terms for an extra forward-pass cost. When all layers are instead perturbed simultaneously, the cross-layer covariance vanishes in expectation for all methods, but loss changes driven by other layers’ noise inflate each layer’s within-layer variance (Appendix F ). | Dalm et al. [2026] |
| Activation Awareness | When perturbing nodes, or considering the effect of weight perturbation at the node level, one can additionally take into account the locally-available derivative of the node’s activation function. This ultimately can allow ensuring that the updates applied to weights are guided by how susceptible the output nodes are to the weight change (Appendix G ). | Singhal et al. [2023] |
| Whitening / preconditioning | Absorbs the rotation induced by using diagonal Adam-style second moments, Kronecker-factored Shampoo/SOAP, or Newton–Schulz orthogonalisation as in Muon. For SZO- this turns the intended spectral bias into an explicit preconditioned step; the same correction applies to NTWP, whose activity-derived eligibility shares SZO- geometry. | Jordan et al. [2024] , Lang et al. [2026] ; see also Kingma and Ba [2015] , Gupta et al. [2018] , Vyas et al. [2025] |
| Method | Effective covariance | Whitening correction | Practical form |
|---|---|---|---|
| WP/SPSA | in population; realised at finite | optional correction by or optimiser state | Adam, Muon |
| NP | activity-noise covariance | whiten active activity locus before mapping to weights | Adam, Muon |
| SZO- | input factor | right multiply by , or keep as preconditioner | Shampoo/SOAP |
| NTWP | activity-induced factor | right whitening of activity eligibility; optional two-sided correction | preconditioned |
| Method | Expected update geometry | Dominant variance scale | Main tradeoff |
|---|---|---|---|
| WP/SPSA | when | Unbiased target, high dimension | |
| NP | when | Lower dimension, may be unrealisable | |
| SZO- | nominal | WP endpoint | |
| SZO- | Broad spectral bias, lower effective dimension | ||
| SZO- | Lowest coverage, strongest variance reduction | ||
| NTWP | up to normalisation (per-batch credit); (per-sample) | Realisable node-like credit |
| Fashion-MNIST | CIFAR-10 | |||||||
|---|---|---|---|---|---|---|---|---|
| Method | MACs | ms/step | time | peak MiB | MACs | ms/step | time | peak MiB |
| BP | 1.0 | 2.1 | 1.0 | 77 | 1.0 | 7.1 | 1.0 | 573 |
| NP | 2.0 | 6.2 | 2.9 | 69 | 2.0 | 19.2 | 2.7 | 563 |
| WP (SZO-0) | 2.0 | 6.7 | 3.1 | 70 | 1.7 | 19.8 | 2.8 | 343 |
| NTWP | 2.5 | 6.7 | 3.1 | 69 | 2.0 | 20.3 | 2.9 | 563 |
| SZO-1 | 2.0 | 6.8 | 3.2 | 69 | 2.5 | 27.7 | 3.9 | 388 |
| Layer | Type | Configuration | Output | Weights |
| Fashion-MNIST MLP (input ) | ||||
| 1 | Linear | |||
| 2 | Linear (output) | |||
| CIFAR-10 CNN (input ) | ||||
| 1 | Conv | |||
| 2 | Conv + max-pool | |||
| Hyperparameter | Symbol | Value |
| Architecture and data | ||
| Architecture Fashion-MNIST | MLP, | |
| Architecture CIFAR-10 | CNN, | |
| Activation function | Leaky ReLU | |
| Preprocessing | Fashion-MNIST: normalise; CIFAR-10: normalise, random crop (padding 4), horizontal flip | |
| Loss | Cross-entropy with label smoothing | |
| Fashion-MNIST | CIFAR-10 | |||||
| Method | SGD | Adam | SOAP | SGD | Adam | SOAP |
| BP | ||||||
| NP | ||||||
| WP (SZO-0) | ||||||
| NTWP | ||||||
| SZO-1 | ||||||
Zeroth-Order Optimization
Zeroth-order optimization uses function values or comparisons, rather than gradients, to optimize objectives when differentiation is unavailable or costly.
See all 99 papers