Exploiting heterogeneous delays for efficient computation in low-bit neural networks

Pengfei Sun; Jascha Achterberg; Zhe Su; Dan F. M. Goodman; Danyal Akarca

cs.NEOct 31, 2025

Exploiting heterogeneous delays for efficient computation in low-bit neural networks

Authors: Pengfei Sun, Jascha Achterberg, Zhe Su, Dan F. M. Goodman, Danyal Akarca

Organizations: Department of Electrical and Electronic Engineering, Imperial College London. · Centre for Neural Circuits and Behaviour, University of Oxford. · Institute of Neuroinformatics, University of Zurich and ETH Zurich. · Imperial-X, Imperial College London. ·

Abstract

Neural networks rely on learning synaptic weights. However, this overlooks other learnable neural parameters. One such parameter is delay: neurons communicate asynchronously with heterogeneous transmission delays. We propose that, rather than a cost to minimize, this heterogeneity supports computation when task-relevant information unfolds over time. We test this hypothesis by jointly learning weights and delays at different precisions in feedforward spiking neural networks. Delay heterogeneity supports competitive performance on neuromorphic benchmarks while compressing weights over an order of magnitude more aggressively than typically studied weight-only networks. Performance depends on task-matched delay distributions, with temporally complex tasks selectively relying on longer delays. These benefits persist under explicit delay constraints, while neural time constants only partially substitute for long delays. Our results connect the temporal structure of data with efficient model design, highlighting temporal heterogeneity as a computational resource with implications for embodied intelligence and neuromorphic hardware.

Figures & tables

Figure 1: Training delays provides parameter-efficient performance improvements according to task temporal complexity. a . Each neuron maintains one learned delay parameter shared across all outgoing connections ( $O(n)$ scaling), while synaptic weights require individual parameters for each connection ( $O(n^{2})$ scaling). Circles represent neurons, lines represent connections. b . Classification accuracy learning curve across four neuromorphic datasets comparing networks with weights only (dashed line) versus weights and axonal delays (solid line). Performance deviation represents standard error across $n=10$ independent runs. Delays consistently improve performance. c . Per-parameter performance analysis shows the efficiency advantage of delays relative to weights. The y-axis shows accuracy per parameter. Delays (solid) achieve significantly greater performance gains per parameter than equivalent weight additions (dashed) across all tasks. Circle size indicates neuron count ( $n=16\times 2^{k}$ , up to $2048$ ).

Figure 2: Weight-delay trade-offs enable extreme model compression through complementary parameter roles a . Bit-budget landscape showing classification accuracy as a function of weight bits (y-axis) and delay bits (x-axis) for SHD, b . SSC, and c . NTIDIGITS. For each dataset, the left sub-panel reports the hidden-layer memory footprint, with delay memory shown as an offset corresponding to 0–7 delay bits, which contributes only a small fraction of the total. The right sub-panel shows accuracy across quantisation combinations (yellow indicates higher performance); dashed contours denote iso-performance boundaries. Black circles highlight two operating points: Solution I, an accuracy-matched configuration competitive with strong baselines at markedly lower bit budgets, and Solution II, which attains peak accuracy. d . Parameter distribution histograms comparing high-efficiency (Solution I, top panels) versus high-precision (Solution II, bottom panels) approaches for SHD, e . SSC, and f . NTIDIGITS dataset. Left panels show synaptic weight distributions and right panels show axonal delay distributions (time steps) across both first and second layers. Across datasets and layers, learning reshapes the delay distributions under different weight quantisation levels. g . Memory-performance Pareto frontier. Each point denotes a quantisation configuration (x-axis: memory footprint, log scale; y-axis: classification accuracy). Introducing delay precision shifts the frontier outward, enabling accuracy-matched solutions at 20 $\times$ , 20 $\times$ , and 19 $\times$ lower parameter-memory budgets for SHD (left), SSC (middle), and NTIDIGITS (right), respectively. Black dots indicate individual configurations, and the red dot marks the common solution (full-precision weights without delays).

Figure 3: Performance is selectively sensitive to long delays and can be partially traded off with time constants and regularisation. a . Weight connection-pruning ablation on SHD, SSC, NTIDIGITS and DVS Gesture. Test accuracy is plotted against the fraction of delays removed either Short $\rightarrow$ Long (dark solid line) or Long $\rightarrow$ Short (light dashed line). Performance deviation represents standard error across n = 5 independent training runs. Pruning many neurons with short delays has a relatively little effect, whereas removing a small set of the longest delays causes a sharp drop in accuracy for the well-controlled datasets (SHD and NTIDIGITS). DVS Gesture shows minimal sensitivity for long delays, consistent with the lower temporal complexity of the task. b . Time constant trade-off. With plastic delays, increasing the layer-shared time constant ( $\tau$ ) systematically shifts the learned delay distributions towards shorter values and reduces dispersion. A non-trivial long tail persists, especially in deeper layers, indicating that longer $\tau$ only partially substitutes for long-range delays. Accuracy decreases at excessively large $\tau$ . c . Relationship between delay and $L_{2}$ regularisation (strength $\lambda$ ). As $\lambda$ increases, the mean delay magnitude and long-delay fraction decrease while test accuracy remains high. The vertical dashed line marks 89% accuracy (higher to the left). Earlier-layer delays contract first, whereas long delays in later layers are preferentially preserved. The horizontal dashed lines correspond to the zero-delay network.

Figure 4: Delay learning, quantization, regularization, and time constants jointly optimize resource efficiency on SHD dataset. (Top) Test accuracy (%) with error bars representing 3 $\sigma$ ; delay-enabled configurations achieve similar accuracies. (Middle) Memory footprint (Mbits) demonstrates parameter-efficient gains from delay heterogeneity, enabling compact models without sacrificing accuracy. (Bottom) Learned axonal delay distributions per layer under distinct configurations. Starting from weight-only feedforward baselines, direct weight quantization reduces memory but degrades accuracy. Adding learned delays with regularization and longer time constants ( $\tau$ ) restores high accuracy while achieving substantially lower memory footprint. The optimal solution combines 1.58-bit weight quantization, 5-step delay quantization, delay regularization, and $\tau=2$ , achieving 90% accuracy with only 0.174 Mbits and sparse long-delay distributions.

Symbol	Description
$u_{i}^{l}(t)$	Membrane potential of neuron $i$ in layer $l$ at time $t$
$s_{i}^{l}(t)$	Spike output of neuron
$W_{ij}^{\,l-1}$	Synaptic weight
$\theta_{u}$	Membrane potential threshold
$\tau_{s}$ , $\tau_{r}$	Synaptic and refractory time constants
$\tau_{\mathrm{scale}},\tau_{\theta}$	scaling factor and sharpness around $\theta_{u}$

Table 1: Notation summary. Key variables used in the spiking neuron model, quantization, and loss functions.

Dataset	$\tau_{s}$	$\tau_{r}$	$\tau_{\mathrm{scale}},\tau_{\theta}$	$\theta_{u}$	$\Delta_{d}$	$\delta_{d}$	$\theta_{d}$
SHD	1	2	(0.1, 1)	10	$[1,70]$	0/learnable	$+\infty$
SSC	1	1	(0.1, 1)	10	$[1,70]$	0/learnable	$+\infty$
NTIDIGITS	1	1	(0.1, 1)	10	$[1,70]$	0/learnable	$+\infty$
DVS Gesture	5	5	(0.1, 1)	10	$[1,70]$	0/learnable	$+\infty$

Table 2: Hyperparameter settings across datasets.

Dataset	SHD	SSC	NTIDIGITS	DVS Gesture
Input size	700	700	64	128 $\times$ 128 $\times$ 2
Number of hidden layers	2	2	2	3
Hidden spiking units	128	128	256	-
Output spiking neurons	20	35	11	11

Table 3: Network configurations across datasets. Input size refers to cochleagram channels (SHD/SSC), auditory features (NTIDIGITS), or event-camera resolution (DVS Gesture). For DVS Gesture, we use two spiking convolutional layers (16 and 32 channels) followed by one spiking fully connected layer (512 neurons).

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Neuromorphic Computing

Neuromorphic computing uses brain-inspired, event-driven algorithms and hardware to process information with low power and latency.

See all 194 papers

Design and Analysis of Temporal Representations in Neural Networks
Oct 14, 2026Alban Colas-Gattepaille, Alexandre MuzyNeural Encoding Spiking Neural Networks
I3S · ILLS
Controlling the latency in temporal coded spiking models for neuromorphic datasets
Oct 14, 2026Alban Gattepaille, Alexandre MuzySpiking Neural Networks
I3S · ILLS
Congestion-Aware Dynamic Axonal Delay for Spiking Neural Networks
May 2, 2026Dewei Bai, Hongxiang Peng, Yunyun Zeng +2Dynamic Neural Networks Spiking Neural Networks
University of Electronic Science and Technology of China

Oct 14, 2026J/K move · Enter open · S save

Design and Analysis of Temporal Representations in Neural Networks

Alban Colas-Gattepaille, Alexandre Muzy

I3S · ILLS

Neural Encoding Spiking Neural Networks Neural Network Activation Functions

Open paper AskPDF arXiv

The ability to represent temporal information is often hypothesized to underlie the efficiency and learning capabilities of biological neural networks. This motivates the development of spiking neural networks. However, it remains unclear how temporal information is actually represented in neural networks and how different forms of temporal representation affect performance and computational efficiency. We introduce the notion of the degree of temporal representation of a neural network, defined by whether its constituent functions (activation, learning, encoding, and decoding) explicitly manipulate time as a computational variable. This perspective yields a continuum ranging from non-temporal representations, where time does not explicitly appear as a computational variable in the network functions, to fully temporal representations, where computations explicitly operate on temporal variables. Using a fixed deep architecture, we compare different encoding, activation mechanisms and decoding with increasing temporal expressiveness: non-temporal weighted-sum activations used in usual artificial neural networks, non-temporal binary spiking activations used in spiking neural networks, and spike-time-based activations used in delay neural networks. Experiments on event-based datasets show that increasing the degree of temporal representation improves both performance and computational efficiency, allowing state-of-the-art performances on the Google Speech Commands dataset. We further analyze how temporal representations induce distinct forward and backward computational trajectories, leading to gradients that depend explicitly on the history of event times.

Abstract

Neuromorphic Computing

Explore similar work

Design and Analysis of Temporal Representations in Neural Networks

Controlling the latency in temporal coded spiking models for neuromorphic datasets

Congestion-Aware Dynamic Axonal Delay for Spiking Neural Networks