Exploiting heterogeneous delays for efficient computation in low-bit neural networks
Authors: Pengfei Sun, Jascha Achterberg, Zhe Su, Dan F. M. Goodman, Danyal Akarca
Organizations: Department of Electrical and Electronic Engineering, Imperial College London. · Centre for Neural Circuits and Behaviour, University of Oxford. · Institute of Neuroinformatics, University of Zurich and ETH Zurich. · Imperial-X, Imperial College London. · MRC Cognition and Brain Sciences Unit, University of Cambridge.
Neural networks rely on learning synaptic weights. However, this overlooks other learnable neural parameters. One such parameter is delay: neurons communicate asynchronously with heterogeneous transmission delays. We propose that, rather than a cost to minimize, this heterogeneity supports computation when task-relevant information unfolds over time. We test this hypothesis by jointly learning weights and delays at different precisions in feedforward spiking neural networks. Delay heterogeneity supports competitive performance on neuromorphic benchmarks while compressing weights over an order of magnitude more aggressively than typically studied weight-only networks. Performance depends on task-matched delay distributions, with temporally complex tasks selectively relying on longer delays. These benefits persist under explicit delay constraints, while neural time constants only partially substitute for long delays. Our results connect the temporal structure of data with efficient model design, highlighting temporal heterogeneity as a computational resource with implications for embodied intelligence and neuromorphic hardware.
Figures & tables
Figure 1: Training delays provides parameter-efficient performance improvements according to task temporal complexity. a . Each neuron maintains one learned delay parameter shared across all outgoing connections ( O(n) scaling), while synaptic weights require individual parameters for each connection ( O(n2) scaling). Circles represent neurons, lines represent connections. b . Classification accuracy learning curve across four neuromorphic datasets comparing networks with weights only (dashed line) versus weights and axonal delays (solid line). Performance deviation represents standard error across n=10 independent runs. Delays consistently improve performance. c . Per-parameter performance analysis shows the efficiency advantage of delays relative to weights. The y-axis shows accuracy per parameter. Delays (solid) achieve significantly greater performance gains per parameter than equivalent weight additions (dashed) across all tasks. Circle size indicates neuron count ( n=16×2k , up to 2048 ).
Figure 2: Weight-delay trade-offs enable extreme model compression through complementary parameter roles a . Bit-budget landscape showing classification accuracy as a function of weight bits (y-axis) and delay bits (x-axis) for SHD, b . SSC, and c . NTIDIGITS. For each dataset, the left sub-panel reports the hidden-layer memory footprint, with delay memory shown as an offset corresponding to 0–7 delay bits, which contributes only a small fraction of the total. The right sub-panel shows accuracy across quantisation combinations (yellow indicates higher performance); dashed contours denote iso-performance boundaries. Black circles highlight two operating points: Solution I, an accuracy-matched configuration competitive with strong baselines at markedly lower bit budgets, and Solution II, which attains peak accuracy. d . Parameter distribution histograms comparing high-efficiency (Solution I, top panels) versus high-precision (Solution II, bottom panels) approaches for SHD, e . SSC, and f . NTIDIGITS dataset. Left panels show synaptic weight distributions and right panels show axonal delay distributions (time steps) across both first and second layers. Across datasets and layers, learning reshapes the delay distributions under different weight quantisation levels. g . Memory-performance Pareto frontier. Each point denotes a quantisation configuration (x-axis: memory footprint, log scale; y-axis: classification accuracy). Introducing delay precision shifts the frontier outward, enabling accuracy-matched solutions at 20 × , 20 × , and 19 × lower parameter-memory budgets for SHD (left), SSC (middle), and NTIDIGITS (right), respectively. Black dots indicate individual configurations, and the red dot marks the common solution (full-precision weights without delays).
Figure 3: Performance is selectively sensitive to long delays and can be partially traded off with time constants and regularisation. a . Weight connection-pruning ablation on SHD, SSC, NTIDIGITS and DVS Gesture. Test accuracy is plotted against the fraction of delays removed either Short → Long (dark solid line) or Long → Short (light dashed line). Performance deviation represents standard error across n = 5 independent training runs. Pruning many neurons with short delays has a relatively little effect, whereas removing a small set of the longest delays causes a sharp drop in accuracy for the well-controlled datasets (SHD and NTIDIGITS). DVS Gesture shows minimal sensitivity for long delays, consistent with the lower temporal complexity of the task. b . Time constant trade-off. With plastic delays, increasing the layer-shared time constant ( τ ) systematically shifts the learned delay distributions towards shorter values and reduces dispersion. A non-trivial long tail persists, especially in deeper layers, indicating that longer τ only partially substitutes for long-range delays. Accuracy decreases at excessively large τ . c . Relationship between delay and L2 regularisation (strength λ ). As λ increases, the mean delay magnitude and long-delay fraction decrease while test accuracy remains high. The vertical dashed line marks 89% accuracy (higher to the left). Earlier-layer delays contract first, whereas long delays in later layers are preferentially preserved. The horizontal dashed lines correspond to the zero-delay network.
Figure 4: Delay learning, quantization, regularization, and time constants jointly optimize resource efficiency on SHD dataset. (Top) Test accuracy (%) with error bars representing 3 σ ; delay-enabled configurations achieve similar accuracies. (Middle) Memory footprint (Mbits) demonstrates parameter-efficient gains from delay heterogeneity, enabling compact models without sacrificing accuracy. (Bottom) Learned axonal delay distributions per layer under distinct configurations. Starting from weight-only feedforward baselines, direct weight quantization reduces memory but degrades accuracy. Adding learned delays with regularization and longer time constants ( τ ) restores high accuracy while achieving substantially lower memory footprint. The optimal solution combines 1.58-bit weight quantization, 5-step delay quantization, delay regularization, and τ=2 , achieving 90% accuracy with only 0.174 Mbits and sparse long-delay distributions.
Symbol
Description
uil(t)
Membrane potential of neuron i in layer l at time t
sil(t)
Spike output of neuron
Wijl−1
Synaptic weight
θu
Membrane potential threshold
τs , τr
Synaptic and refractory time constants
τscale,τθ
scaling factor and sharpness around θu
Table 1: Notation summary. Key variables used in the spiking neuron model, quantization, and loss functions.
Dataset
τs
τr
τscale,τθ
θu
Δd
δd
θd
SHD
1
2
(0.1, 1)
10
[1,70]
0/learnable
+∞
SSC
1
1
(0.1, 1)
10
[1,70]
0/learnable
+∞
NTIDIGITS
1
1
(0.1, 1)
10
[1,70]
0/learnable
+∞
DVS Gesture
5
5
(0.1, 1)
10
[1,70]
0/learnable
+∞
Table 2: Hyperparameter settings across datasets.
Dataset
SHD
SSC
NTIDIGITS
DVS Gesture
Input size
700
700
64
128 × 128 × 2
Number of hidden layers
2
2
2
3
Hidden spiking units
128
128
256
-
Output spiking neurons
20
35
11
11
Table 3: Network configurations across datasets. Input size refers to cochleagram channels (SHD/SSC), auditory features (NTIDIGITS), or event-camera resolution (DVS Gesture). For DVS Gesture, we use two spiking convolutional layers (16 and 32 channels) followed by one spiking fully connected layer (512 neurons).
The ability to represent temporal information is often hypothesized to underlie the efficiency and learning capabilities of biological neural networks. This motivates the development of spiking neural networks. However, it remains unclear how temporal information is actually represented in neural networks and how different forms of temporal representation affect performance and computational efficiency. We introduce the notion of the degree of temporal representation of a neural network, defined by whether its constituent functions (activation, learning, encoding, and decoding) explicitly manipulate time as a computational variable. This perspective yields a continuum ranging from non-temporal representations, where time does not explicitly appear as a computational variable in the network functions, to fully temporal representations, where computations explicitly operate on temporal variables. Using a fixed deep architecture, we compare different encoding, activation mechanisms and decoding with increasing temporal expressiveness: non-temporal weighted-sum activations used in usual artificial neural networks, non-temporal binary spiking activations used in spiking neural networks, and spike-time-based activations used in delay neural networks. Experiments on event-based datasets show that increasing the degree of temporal representation improves both performance and computational efficiency, allowing state-of-the-art performances on the Google Speech Commands dataset. We further analyze how temporal representations induce distinct forward and backward computational trajectories, leading to gradients that depend explicitly on the history of event times.
Exploiting temporal information in datasets with high temporal dependencies (such as spoken keyword recognition) by spiking models is still an open problem. Promising frameworks include Time-To-First coding (TTFS) and the inclusion of synaptic delays into learning. In TTFS, neurons are forced to spike at most once, forcing the network to found its learning on the spiking times of neurons rather than their spiking rate. Some models use the TTFS framework to directly compute the spiking time of Integrate-and-Fire (IF) or Leaky Integrate-and-Fire (LIF) neurons. However, most models built in the TTFS framework have not been tested against datasets containing temporal information. Little is known about the latency of these models on temporal datasets, where the latency is the time taken by the network to correctly classify a sample. In this work, we study the behaviour of Delay Neural Networks (DeNN) models, i.e. models that directly compute the spiking time of neurons by bypassing the membrane potential and thresholding mechanisms of spiking neurons. We show how to analytically control their latency, and empirically study the latency-accuracy tradeoff in a dynamic temporal dataset.
Spiking Neural Networks (SNNs) are widely regarded as an energy-efficient paradigm for modeling and processing temporal and event-driven information. Incorporating delays in SNNs has been proven to be an effective mechanism for improving spike alignment in event-driven tasks. However, existing delay learning approaches predominantly assign static delays to individual synapses, resulting in a large number of delay parameters and limited adaptability to input-dependent activity dynamics. To this end, we propose a Congestion-Aware Dynamic Axonal Delay (CADAD) mechanism, which decomposes the delay into a channel-wise static base delay for temporal structuring and a global, activity-conditioned shift that dynamically regulates the state update rate under varying spike intensities. The delay parameters are learned using differentiable linear interpolation and discretized at inference time, preserving the benefits of dynamic delay modulation while incurring only minimal additional cost. Experiments on speech benchmarks, including the Spiking Heidelberg Dataset, Spiking Speech Commands, and Google Speech Commands, demonstrate that introducing congestion-aware delays into synaptic signal transmission effectively improves accuracy on temporal tasks, notably achieving 93.75% accuracy on SHD, 80.69% accuracy on SSC, and 95.58% on GSC-35, while reducing the parameter count by approximately 50% compared to state-of-the-art delay-based methods with the same architecture.
Dewei Bai, Hongxiang Peng, Yunyun Zeng +2
University of Electronic Science and Technology of China