Central limit theorems for the outputs of fully convolutional neural networks with time series input
Authors: Annika Betken, Giorgio Micali, Johannes Schmidt-Hieber
Organizations: University of Twente, Faculty of Electrical Engineering, Mathematics, and Computer Science (EEMCS), Drienerlolaan 5, 7522 NB Enschede, Netherlands
Deep learning is widely deployed for time series learning tasks such as classification and forecasting. Despite the empirical successes, only little theory has been developed so far in the time series context. In this work, we prove that if the network inputs are generated from short-range dependent linear processes, the outputs of fully convolutional neural networks (FCNs) with global average pooling (GAP) are asymptotically Gaussian and the limit is attained if the length of the observed time series tends to infinity. The proof leverages existing tools from the theoretical time series literature. Based on our theory, we propose a generalization of the GAP layer by considering a global weighted pooling step with slowly varying, learnable coefficients.
Figures & tables
Figure 1: FCN architecture with Residual connection.
Figure 2: Empirical autocorrelation matrices RGAP(θ) for the one-layer GAP network with m1=10 under fixed Gaussian initialization and input generated from an AR (1) process with parameters θ=0 (left panel) and θ=0.99 (right panel). The numbering of the output neurons has been chosen to maximize the correlation of successive neurons, thereby revealing the strong clustering of the outputs.
Figure 3: Empirical correlation matrices with dependence parameter θ=0.65 in the Gaussian AR(1) input time series. The numbering of the output neurons has been chosen to maximize the correlation of successive neurons.
Figure 4: QQ plots and histograms (with fitted Gaussian densities) of the standardized statistic Sn for two representative configurations.
Dataset
GAP
WGAP
reg_WGAP
Median λ
Adiac
0.53 ± 0.12
0.77 ± 0.03
0.80 ± 0.01
1.024
ArrowHead
0.79 ± 0.03
0.76 ± 0.02
0.78 ± 0.03
0.128
BeetleFly
0.78 ± 0.06
0.67 ± 0.08
0.72 ± 0.13
0.032
BirdChicken
0.92 ± 0.03
0.72 ± 0.04
0.80 ± 0.07
1.024
Car
0.63 ± 0.23
0.76 ± 0.05
0.81 ± 0.08
0.512
Meat
0.59 ± 0.17
0.86 ± 0.04
0.92 ± 0.01
2.048
Table 1: Completed simulation by dataset. For each dataset, the best mean accuracy among GAP, WGAP, and reg_WGAP is highlighted in red, while the second-best is highlighted in orange. The last column reports the median selected regularization parameter λ across the five runs. Each method is trained for 200 epochs using the same batch size, while the regularized variant reg_WGAP selects λ by 5 -fold cross-validation, with 80 training epochs in each fold.