Central limit theorems for the outputs of fully convolutional neural networks with time series input
Authors: Annika Betken, Giorgio Micali, Johannes Schmidt-Hieber
Organizations: University of Twente, Faculty of Electrical Engineering, Mathematics, and Computer Science (EEMCS), Drienerlolaan 5, 7522 NB Enschede, Netherlands
Deep learning is widely deployed for time series learning tasks such as classification and forecasting. Despite the empirical successes, only little theory has been developed so far in the time series context. In this work, we prove that if the network inputs are generated from short-range dependent linear processes, the outputs of fully convolutional neural networks (FCNs) with global average pooling (GAP) are asymptotically Gaussian and the limit is attained if the length of the observed time series tends to infinity. The proof leverages existing tools from the theoretical time series literature. Based on our theory, we propose a generalization of the GAP layer by considering a global weighted pooling step with slowly varying, learnable coefficients.
Figures & tables
Figure 1: FCN architecture with Residual connection.
Figure 2: Empirical autocorrelation matrices RGAP(θ) for the one-layer GAP network with m1=10 under fixed Gaussian initialization and input generated from an AR (1) process with parameters θ=0 (left panel) and θ=0.99 (right panel). The numbering of the output neurons has been chosen to maximize the correlation of successive neurons, thereby revealing the strong clustering of the outputs.
Figure 3: Empirical correlation matrices with dependence parameter θ=0.65 in the Gaussian AR(1) input time series. The numbering of the output neurons has been chosen to maximize the correlation of successive neurons.
Figure 4: QQ plots and histograms (with fitted Gaussian densities) of the standardized statistic Sn for two representative configurations.
Dataset
GAP
WGAP
reg_WGAP
Median λ
Adiac
0.53 ± 0.12
0.77 ± 0.03
0.80 ± 0.01
1.024
ArrowHead
0.79 ± 0.03
0.76 ± 0.02
0.78 ± 0.03
0.128
BeetleFly
0.78 ± 0.06
0.67 ± 0.08
0.72 ± 0.13
0.032
BirdChicken
0.92 ± 0.03
0.72 ± 0.04
0.80 ± 0.07
1.024
Car
0.63 ± 0.23
0.76 ± 0.05
0.81 ± 0.08
0.512
Meat
0.59 ± 0.17
0.86 ± 0.04
0.92 ± 0.01
2.048
Table 1: Completed simulation by dataset. For each dataset, the best mean accuracy among GAP, WGAP, and reg_WGAP is highlighted in red, while the second-best is highlighted in orange. The last column reports the median selected regularization parameter λ across the five runs. Each method is trained for 200 epochs using the same batch size, while the regularized variant reg_WGAP selects λ by 5 -fold cross-validation, with 80 training epochs in each fold.
We establish central and non-central limit theorems for sequences of functionals of the Gaussian output of an infinitely-wide random neural network on the d-dimensional sphere . We show that the asymptotic behaviour of these functionals as the depth of the network increases depends crucially on the fixed points of the covariance function, resulting in three distinct limiting regimes: convergence to the same functional of a limiting Gaussian field, convergence to a Gaussian distribution, convergence to a distribution in the Qth Wiener chaos. Our proofs exploit tools that are now classical (Hermite expansions, Diagram Formula, Stein-Malliavin techniques), but also ideas which have never been used in similar contexts: in particular, the asymptotic behaviour is determined by the fixed-point structure of the iterative operator associated with the covariance, whose nature and stability governs the different limiting regimes.
Simmaco Di Lillo, Leonardo Maini, Domenico Marinucci
RoMaDS - Department of Mathematics, University of Rome Tor Vergata, Rome, Italy
We consider the infinite-width limit of a fully connected deep neural network with general weights, and we prove quantitative general bounds on the 2-Wasserstein distance between the network and its infinite-width Gaussian limit, under appropriate regularity assumptions on the activation function. Our main tool is a Lindeberg principle for Deep Neural Networks, which we use to successively replace the weights on each layer by Gaussian random variables.
Filippo Giovagnini, Sotirios Kotitsas, Marco Romito
We investigate the convergence guarantee of two-layer neural network training with Gaussian randomly masked inputs. This scenario corresponds to Gaussian dropout at the input level, or noisy input training common in sensor networks, privacy-preserving training, and federated learning, where each user may have access to partial or corrupted features. Using a Neural Tangent Kernel (NTK) analysis, we demonstrate that training a two-layer ReLU network with Gaussian randomly masked inputs achieves linear convergence up to an error region proportional to the mask's variance. A key technical contribution is resolving the randomness within the non-linear activation, a problem of independent interest.
Afroditi Kolomvaki, Fangshuo Liao, Evan Dramko +2
Computer Science Dept., Rice University, Houston, TX, USA