As novel hardware architectures are explored for the purposes of machine learning, including neuromorphic chips, asynchronous systems, and ASICs, effective learning algorithms are highly desirable. While backpropagation is the standard for differentiation on standard hardware, its requirement for global synchrony and exact differentiability limits the exploration of exotic algorithms on non-traditional substrates and devices. Zeroth-order (ZO) optimisation methods, such as SPSA and weight perturbation, offer a compelling alternative by estimating gradients through inference-only passes; however, these methods have historically suffered due to the `curse of dimensionality', where random perturbations become increasingly misaligned with the true gradient in high-dimensional spaces. In this work, we move beyond blind random noise by deriving zeroth-order search directions from activation geometry available during the forward pass. This leads to a family of Spectral Zeroth-Order methods, SZO-κ, whose directions solve local response-based gradient-guessing objectives under explicit stochastic assumptions. These methods explore feature-supported directions that are more likely to produce informative loss responses. Using multilayer and convolutional networks, we analyse gradient alignment, bias-variance structure, and training stability of these methods relative to standard perturbation baselines and backpropagation.
Figures & tables
Figure 1: The angles between estimated gradients computed by NP, WP, SZO- κ and NTWP against the true backpropagation gradient for randomly initialised layers in MLP and Convolutional Networks. For each network, the left panel uses per-batch loss-difference credit and the right panel per-sample credit; points show the mean angle over 20 randomly seeded initialisations (batch of 128 ) against the number of probes n , with error bars of ±1 standard deviation. Left pair, the gradient alignment angle is shown for the first layer of our Fashion-MNIST MLP network. Under per-sample credit these angles clearly show the equivalence of NP and SZO- κ on a linear layer, as theorised in this work. Right pair, the gradient alignment angle is shown for the last 3×3 convolutional layer ( 64→64 channels) of our CIFAR-10 CNN model. On convolutional layers NP suffers significantly from its lack of shared perturbation across kernel positions. Note, these angles were measured by perturbing only the layer in question and leaving all other network layers unperturbed.
Figure 2: The performance of the SZO-family of methods as well as NP, WP, and NTWP when training an MLP (Fashion-MNIST) and a Convolutional Network (CIFAR-10). (a,c) Test accuracy over training with SGD and n=1 (mean over 10 seeds, ±1 standard deviation); on the MLP, NP, SZO- 1 and SZO- ∞ coincide and are drawn interleaved; BP (not shown) reaches 89.9% and 83.6% . (b,d) Validation accuracy after 250 epochs against the number of probes per step n (mean over 3 seeds, 95% confidence intervals; Appendix M ). The learning rate of each method was selected via a grid search on validation loss. Final test accuracies for each method, as well as backpropagation, are reported in Appendix L .
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Correction
What it does
Origin
Iterative layer-wise perturbation
Sequentially perturbs a single layer at a time across a network, removing inter-layer crosstalk terms for an extra L× forward-pass cost. When all layers are instead perturbed simultaneously, the cross-layer covariance vanishes in expectation for all methods, but loss changes driven by other layers’ noise inflate each layer’s within-layer variance (Appendix F ).
Dalm et al. [2026]
Activation Awareness
When perturbing nodes, or considering the effect of weight perturbation at the node level, one can additionally take into account the locally-available derivative of the node’s activation function. This ultimately can allow ensuring that the updates applied to weights are guided by how susceptible the output nodes are to the weight change (Appendix G ).
Singhal et al. [2023]
Whitening / preconditioning
Absorbs the rotation E[g^n]≈MoutgMin induced by Mn=I using diagonal Adam-style second moments, Kronecker-factored Shampoo/SOAP, or Newton–Schulz orthogonalisation as in Muon. For SZO- κ this turns the intended spectral bias into an explicit preconditioned step; the same correction applies to NTWP, whose activity-derived eligibility shares SZO- 1 geometry.
Jordan et al. [2024] , Lang et al. [2026] ; see also Kingma and Ba [2015] , Gupta et al. [2018] , Vyas et al. [2025]
Appendix
Table A.1: Optional corrections that can be layered on top of zeroth-order and perturbation-based gradient estimators. Each targets a distinct source of error in g^n=gMn+O(ϵ2)+ξ, .
Method
Effective covariance
Whitening correction
Practical form
WP/SPSA
I in population; realised M−I at finite n
optional correction by M−1 or optimiser state
Adam, Muon
NP
activity-noise covariance Mϕ
whiten active activity locus before mapping to weights
Adam, Muon
SZO- κ
input factor Cκ/Tr(Cκ)
right multiply by (Cκ+λI)−1 , or keep as preconditioner
Shampoo/SOAP
NTWP
activity-induced factor C
right whitening of activity eligibility; optional two-sided correction
preconditioned
Appendix
Table A.2: Whitening corrections.
Figure 3: Final test accuracy with and without activation awareness. Open circles, activation-unaware; filled diamonds, activation-aware; SGD with n=1 , each setting at its own selected learning rate. Points show the mean over the last 10 of 250 epochs and 10 seeds, with 95% confidence intervals.
Method
Expected update geometry
Dominant variance scale
Main tradeoff
WP/SPSA
g when Mn→Im
O(m/n)∥g∥2
Unbiased target, high dimension
NP
∇hLJh,θ when Mϕ→Ik
O(k/n)∥∇hL∥2
Lower dimension, may be unrealisable
SZO- 0
G
O(kq/n) nominal
WP endpoint
SZO- 1
GC/Tr(C)
∝Tr(GCG⊤)/(nTr(C))
Broad spectral bias, lower effective dimension
SZO- ∞
(Gu1)u1⊤
O(k/n)∥Gu1∥2
Lowest coverage, strongest variance reduction
NTWP
GC up to normalisation (per-batch credit); G (per-sample)
O(keff/n)
Realisable node-like credit
Appendix
Table A.3: Bias-variance structure of the perturbation estimators. Here m is parameter dimension, k is activity dimension, q is layer-input dimension, C=AA⊤ , u1 is the leading eigenvector of C , and n is the number of probes.
Figure 5: Layer-wise gradient alignment for the Fashion-MNIST MLP at initialisation. Each row is one weight layer; left panels use per-batch loss-difference credit and right panels per-sample credit. Points show the mean angle between the estimated and the backpropagation gradient over 20 seeds (initialisation and batch of 128 ); error bars are ±1 standard deviation. Only the layer shown is perturbed. On a linear layer per-sample SZO- κ is the same estimator for every κ>0 , so per-sample SZO- 1 and SZO- ∞ show identical values; per-sample WP is omitted.
Figure 6: Layer-wise gradient alignment for the CIFAR-10 CNN at initialisation. Each row is one weight layer; left panels use per-batch loss-difference credit and right panels per-sample credit. Points show the mean angle between the estimated and the backpropagation gradient over 20 seeds (initialisation and batch of 128 ); error bars are ±1 standard deviation. Only the layer shown is perturbed. On the linear readout per-sample SZO- κ is the same estimator for every κ>0 , so per-sample SZO- 1 and SZO- ∞ show identical values; per-sample WP is omitted.
Figure 7: Gradient alignment as a function of the spectral exponent κ at various probe budgets. Rows show the first layer of the Fashion-MNIST MLP and the last 3×3 convolutional layer of the CIFAR-10 CNN; left panels use per-batch loss-difference credit and right panels per-sample credit. Points show the mean angle between the estimated and the backpropagation gradient over 20 seeds (initialisation and batch of 128 ); error bars are ±1 standard deviation. κ=0 (WP), κ=1 and κ=∞ are highlighted in their method colours; the κ axis is categorical.
Figure 8: Training the CIFAR-10 CNN with per-sample SZO- κ for different spectral exponents κ . Left, validation accuracy over training (mean over 3 seeds, smoothed over 5 epochs, shaded bands ±1 standard deviation); right, its mean over the last 10 of 250 epochs with 95% confidence intervals. SGD with n=1 at the learning rate of SZO- 1 ( 3×10−3 ) for every κ ; κ=1 is SZO- 1 and shown in its method colour, and κ=0 denotes the limit κ→0+ (see text).
Figure 9: Final test accuracy per method and optimiser. SGD (circles, 10 seeds), Adam (triangles) and SOAP (squares, 5 seeds each), n=1 , each method at its own selected learning rate. Points show the mean over the last 10 of 250 epochs with 95% confidence intervals. Learning rates were selected on a validation split (20% of the training set; Table A.7 ); the final runs train on the full training set and are evaluated on the official test split. BP (not shown) reaches 89.9 , 90.2 and 90.4% on Fashion-MNIST and 83.6 , 84.2 and 83.7% on CIFAR-10 with SGD, Adam and SOAP.
Figure 11: Validation accuracy against the number of probes per step n . Top, after 250 epochs, as in Figure 2 (b,d); bottom, the same with activation awareness. SGD at each method’s learning rate for n=1 in that setting; points show the mean over 3 seeds with 95% confidence intervals.
Figure 12: Probes or examples at a fixed compute per step, measured. Change of validation accuracy (points) of the CIFAR-10 CNN after 1250 SGD steps with n probes on a batch of B examples, where every step costs the same B(n+1)=256 forward passes: the n perturbed passes and the clean pass of each example. Every run starts from the same weights: a random initialisation (left), or that initialisation after 3 epochs of BP (right, 40.9% validation accuracy). Each method uses the learning rate selected per (n,B) from the pretrained start (lowest final validation loss, 3 seeds); points show the mean ± one standard deviation over 3 further seeds. For reference, BP ( B=128 ) changes the validation accuracy over the same 1250 steps by +35.8±0.8 (left) and +9.8±1.2 points (right).
Figure 14: Validation accuracy dynamics under iterative (one layer at a time) versus simultaneous (all layers perturbed at once) probing modes. The learning rate of each method was selected separately for each mode by the same grid search; all other hyperparameters are matched between the two configurations. SGD with n=1 ; lines show the mean over 3 seeds, smoothed over 5 epochs, and shaded bands ±1 standard deviation.
Fashion-MNIST
CIFAR-10
Method
MACs
ms/step
time
peak MiB
MACs
ms/step
time
peak MiB
BP
1.0 ×
2.1
1.0 ×
77
1.0 ×
7.1
1.0 ×
573
NP
2.0 ×
6.2
2.9 ×
69
2.0 ×
19.2
2.7 ×
563
WP (SZO-0)
2.0 ×
6.7
3.1 ×
70
1.7 ×
19.8
2.8 ×
343
NTWP
2.5 ×
6.7
3.1 ×
69
2.0 ×
20.3
2.9 ×
563
SZO-1
2.0 ×
6.8
3.2 ×
69
2.5 ×
27.7
3.9 ×
388
Appendix
Table A.4: Compute and memory per training step. Batch size 128 , n=1 , FP32, iterative probing, final configuration; MACs: multiply-accumulates per training step counted on our implementation, relative to BP on the same task (Computational complexity, below); time: measured time per step relative to BP, on one dedicated NVIDIA A100 (40 GB), all methods in one session, median over 3 repeated runs of the mean over epochs 2–3 (repeats differ by at most 3% ), without activation awareness, which adds 3 to 13% (Appendix G ); peak: peak allocated GPU memory. MAC counts leave out element-wise operations, memory traffic and kernel-launch overhead, which dominate the step time of networks this small, so the measured time ratios are larger. Our implementations are research code and not optimised for speed; faster implementations of every method are likely possible.
Fashion-MNIST: normalise; CIFAR-10: normalise, random crop (padding 4), horizontal flip
Loss
Cross-entropy with label smoothing 0.1
Appendix
Table A.6: Overview of hyperparameters used for the Fashion-MNIST MLP and CIFAR-10 CNN experiments. Shared settings are listed once; dataset- or architecture-specific settings are indicated where they differ.
Fashion-MNIST
CIFAR-10
Method
SGD
Adam
SOAP
SGD
Adam
SOAP
BP
10−2
3×10−5
10−5
3×10−2
3×10−4
10−4
NP
10−3
3×10−5
10−4
3×10−4
10−4
10−4
WP (SZO-0)
10−4
10−4
10−4
3×10−4
3×10−4
3×10−4
NTWP
10−3
3×10−5
10−4
10−3
10−4
10−4
SZO-1
10−3
3×10−5
10−4
3×10−3
3×10−4
3×10−4
Appendix
Table A.7: Selected learning rates. For each method, optimiser and dataset, without and with (bottom block) activation awareness: the learning rate with the lowest mean validation loss over the last 10 epochs (validation split, 3 seeds), among grid learning rates at which every seed completed.
Figure 15: Validation accuracy across the learning-rate grid , activation-unaware (top two rows) and activation-aware (bottom two rows, Appendix G ). Mean over the last 10 of 250 epochs and 3 seeds, n=1 , for SGD, Adam and SOAP; stars mark the selected learning rate (Table A.7 ). Learning rates at which at least one seed diverged are omitted.
Zeroth-order optimization uses function values or comparisons, rather than gradients, to optimize objectives when differentiation is unavailable or costly.