cs.LGNov 5, 2026

An Educated Guess: Deriving Statistically Aligned Gradient Estimates for Zeroth-Order Optimisation

Authors: Justus Huebotter, Serge Thill, Marcel van Gerven, Nasir Ahmad

Organizations: Donders Institute for Brain, Cognition, and Behaviour, Radboud University, Nijmegen, the Netherlands

Abstract

As novel hardware architectures are explored for the purposes of machine learning, including neuromorphic chips, asynchronous systems, and ASICs, effective learning algorithms are highly desirable. While backpropagation is the standard for differentiation on standard hardware, its requirement for global synchrony and exact differentiability limits the exploration of exotic algorithms on non-traditional substrates and devices. Zeroth-order (ZO) optimisation methods, such as SPSA and weight perturbation, offer a compelling alternative by estimating gradients through inference-only passes; however, these methods have historically suffered due to the `curse of dimensionality', where random perturbations become increasingly misaligned with the true gradient in high-dimensional spaces. In this work, we move beyond blind random noise by deriving zeroth-order search directions from activation geometry available during the forward pass. This leads to a family of Spectral Zeroth-Order methods, SZO-κκ, whose directions solve local response-based gradient-guessing objectives under explicit stochastic assumptions. These methods explore feature-supported directions that are more likely to produce informative loss responses. Using multilayer and convolutional networks, we analyse gradient alignment, bias-variance structure, and training stability of these methods relative to standard perturbation baselines and backpropagation.

Figures & tables

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Model Evolution Under Zeroth-Order Optimization: A Neural Tangent Kernel Perspective

    Mar 22, 2026Chen Zhang, Yuxin Cheng, Chenchen Ding +5Zeroth-Order OptimizationNeural Network Training Dynamics

  2. Position: Zeroth-Order Optimization in Deep Learning Is Underexplored, Not Underpowered

    May 15, 2026Sijia Liu, Yicheng Lang, Soumyadeep Pal +6Deep Learning OptimizationZeroth-Order Optimization