The input space of single-spike neural networks decomposes into causal pieces, and the number of causal pieces has been shown to provide a lower bound on the approximation error for some spiking neural networks. While this decomposition is applicable to a wide range of spiking neuron models, the limits of how many causal pieces can be realised by spiking neural networks are still unknown. In this work, we derive two complementary upper bounds on the number of causal pieces in feedforward single-spike neural networks. First, we establish a combinatorial bound given by the poly-Bernoulli numbers, which applies to a broad class of spiking neuron models. The bound is of order e2(L−1)NlogN+O(N), with network depth L and maximum layer width N. Second, for a specific set of neuron models, we derive a bound based on hyperplane arrangements. Applying the latter framework, we obtain bounds of order O(N(4L−3)(N0−1)) for the non-leaky integrate-and-fire model with exponential synapses and the ReLU-type spike response model, and O(NL(N0−1)) for the leaky integrate-and-fire model with Dirac-δ synapses, where N0 is the input dimension. For the non-leaky integrate-and-fire model with Dirac-δ synapses, additional temporal structure allows us to sharpen the general combinatorial bound to one that depends only on N0, independently of the network depth and widths.
Figures & tables
Figure 1: A. The input space (here: two-dimensional) of a SNN partitions into separate regions with, potentially, curved boundaries. Each region (causal pieces) is denoted by a different colour. B. Every causal piece corresponds to a different subnetwork (coloured) that produces, in that input region, the output. Corresponding regions in (A) are coded here by colour. Thus, counting causal pieces corresponds to counting possible subnetworks. C. Not all synaptic activity patterns between two layers in a SNN are allowed. We show here one such forbidden pattern: t0′ is caused by input neurons {0,1}=C0 , and t1′ by neurons {0,2}=C1 . However, if t1′≥t0′ , then C0 has to be a subset of C1 . Similarly, if t1′<t0′ , C1 has to be a subset of C0 . Neither is the case, and thus the pattern is not allowed. D. For specific neuron models, there exists a change of variables through which causal pieces become polyhedra.
Figure 2: Upper bounds as stated in Proposition 4.2 and Theorem 4.6 for general SNNs, and for specific SNNs using hyperplane arrangement arguments ( Theorems 6.1 , 6.2 , 6.4 and 6.6 ), evaluated for different neural architectures. We denote here “input dimension; hidden dimension; hidden dimension; … ; output dimension”. As some of our bounds also apply to the neuron model studied in [ 26 ] , we include here the upper bound stated in [ 26 , Theorem 7.1] . The first set of architectures is taken from [ 24 ] .
Model
Asymptotic upper bound for causal pieces
nLIF with exponential synapses ( Theorem 6.1 )
O(N(4L−3)(N0−1))
ReLU response ( Theorem 6.2 )
O(N(4L−3)(N0−1))
LIF with Dirac- δ synapses ( Theorem 6.4 )
O(NL(N0−1))
nLIF with Dirac- δ synapses ( Theorem 6.6 )
O(1)
Combinatorial bound ( Theorem 4.6 )
e2(L−1)NlogN+O(N)
Table 1: Asymptotic behaviour of the upper bounds w.r.t. maximum width N for fixed input dimension N0 and depth L .
Spiking neural networks process information through discrete, time-dependent neural events, aiming for efficient computation and biologically inspired learning.
We introduce "causal pieces", a novel concept for analysing spiking neural networks (SNNs), inspired by "linear pieces" used to study expressivity and trainability in artificial neural networks (ANNs). Causal pieces partition the input and parameter space of a feedforward SNN with single-spike coding into distinct regions where the same subnetwork causes the output spikes. For networks of current-based leaky integrate-and-fire (LIF) neurons with large membrane time constants, we show that within each causal piece, output spike times are locally Lipschitz continuous with respect to inputs and network parameters. We further prove a lower bound on the approximation error that depends on the number of causal pieces. Thus, the number of causal pieces is a measure of the approximation capabilities of SNNs, which is valid despite spike-time discontinuities and applies to networks with both excitatory and inhibitory synapses. Empirically, we find that parameter initialisations yielding more causal pieces on the training set strongly correlate with SNN training success across multiple benchmarks, including Yin-Yang, Fashion-MNIST, and EuroSAT. Moreover, simulations with standard single-spike LIF neurons indicate that our findings extend beyond the theoretically analysed regime. These results establish causal pieces as a powerful and principled tool for analysing and improving the computational capabilities of SNNs.
We study the expressivity of spiking neural networks, which provide a natural framework for asynchronous, event-driven computation complementary to conventional feedforward neural networks. We consider the time-to-first-spike model in a setting for which the input-output map is continuous and piecewise linear, with affine pieces governed by causal feasibility constraints that determine which presynaptic spikes occur before a neuron fires. We first show that each neuron's firing time admits a maxout-like representation with exponentially many, highly constrained affine pieces. We then formalize causal regions as polyhedral regions with fixed causal sets and derive upper and lower bounds on the maximal number of causal regions in both shallow and multilayer feedforward spiking networks. Our theoretical and experimental results show that spiking networks can generate richer partitions of the input space than conventional feedforward ReLU networks.
Manjot Singh, Guido Montúfar, Gitta Kutyniok
Department of Mathematics, Ludwig-Maximilians-Universität München · Munich Center for Machine Learning · Departments of Mathematics and Statistics & Data Science, University of California, Los Angeles +2
In a spiking neural network, is it enough for each neuron to spike at most once? In recent work, approximation bounds for spiking neural networks have been derived, quantifying how well they can fit target functions. However, these results are only valid for neurons that spike at most once, which is commonly thought to be a strong limitation. Here, we show that the opposite is true for a large class of spiking neuron models, including the commonly used leaky integrate-and-fire model with subtractive reset: for every approximation bound that is valid for a set of multi-spike neural networks, there is an equivalent set of single-spike neural networks with only linearly more (or less) neurons, in the maximum number of spikes, for which the bound holds. The same is true for the reverse direction too, showing that regarding their approximation capabilities in general machine learning tasks, single-spike and multi-spike neural networks are equivalent. Consequently, many approximation results in the literature for single-spike neural networks also hold for the multi-spike case.