Robustness in sparse artificial neural networks trained with adaptive topology
Authors: Bendegúz Sulyok, Gergely Palla, Filippo Radicchi, Santo Fortunato
Organizations: Dept. of Biological Physics, Eötvös Loránd University, Budapest, Hungary · Semmelweis University, Faculty of Health and Public Administration, Health Services Management Training Centre, Budapest, Hungary · Center for Complex Networks and Systems Research Luddy School of Informatics, Computing, and Engineering Indiana University Bloomington, USA
We investigate the robustness of sparse artificial neural networks trained with adaptive topology. We focus on a simple yet effective architecture consisting of three sparse layers with 99% sparsity followed by a dense layer, applied to image classification tasks such as MNIST and Fashion MNIST. By updating the topology of the sparse layers between each epoch, we achieve competitive accuracy despite the significantly reduced number of weights. Our primary contribution is a detailed analysis of the robustness of these networks, exploring their performance under various perturbations including random link removal, adversarial attack, and link weight shuffling. Through extensive experiments, we demonstrate that adaptive topology not only enhances efficiency but also maintains robustness. This work highlights the potential of adaptive sparse networks as a promising direction for developing efficient and reliable deep learning models.
Figures & tables
Figure 1: Illustration of the network architecture. The input layer (shown on the left) contains 28 x 28 pixels, followed by 3 sparsely connected neuron layers with 1000 neurons each. Blue coloured connections have a positive link weight, whereas red colour indicates a negative weight. The last layer, providing the readout is densely connected to the third sparse layer.
Figure 2: Duration of the topology update. We show the bar chart of the average duration in seconds for a single topology update over 4 training instances and 50 epochs. The results for the CH3L3 method are shown in light blue, whereas for the Random Link Regrowth method in purple, where the dataset is marked under the corresponding bars.
Figure 3: Accuracy during training. We show the accuracy ( fraction of correctly classified inputs, where the shaded area indicates the standard deviation) as a function of the number of epochs (on logarithmic scale). The results for the CH3L3 method are shown in blue, for the Random Link Regrowth method in red. Results are displayed for the following datasets: a) MNIST, b) FashionMNIST, c) KMNIST, and d) EMNIST Letters. Each setup was run 10 times with a different random seed and each training instance run for 1000 epochs with a topology update between consecutive epochs.
Figure 4: Robustness of trained sparse networks against link removal. Average accuracy over 32 networks as a function of the fraction of removed links (perturbation value) for systems trained according to the CHCL3 method (continous lines) and the RLR method (dashed lines) for a) the MNIST dataset, b) the Fashion MNIST dataset, c) the KMNIST dataset, and d) the EMNIST letters dataset. Results for Reverse Weight Order Pruning are shown in red, for Random Pruning in green and for Weight Order Pruning in gray.
Figure 5: Robustness of trained sparse networks against perturbing the weights by shuffling or adding noise. Accuracy as a function of the perturbation value for networks trained with the CH3L3 Link Regrowth method (continuous lines) and with the RLR method (dashed lines) for a) the MNIST dataset, b) the Fashion MNIST dataset, c) the KMNIST dataset, and d) the EMNIST letters dataset. Results for Weight Shuffling are shown in purple, for Weight Modification in light blue.
Figure 6: Weight magnitude distribution in the trained networks. We plot the distribution density for the absolute value of the connection weights at the end of the sparse training process, where results for networks trained according to the CH3L3 algorithm are shown in light blue and results for networks trained with RLR are shown in red. Panels depict the distributions observed for a) the MNIST dataset, b) the FashionMNIST dataset, c) the KMNIST dataset, and d) the EMNIST letters dataset.
Figure 7: The three stages of the topology update. In the first stage (a) a fraction of links (dotted red lines) is removed in each sparse layer. In the second optional stage (b) all links that are no longer connected to the input and/or output layer (dotted red lines) are also removed. In the final stage (c) new links (green) are drawn and initialized such that the total number of links in each layer will be unchanged after the procedure.
College of Information and Communication Technology, Can Tho University, Vietnam · College of Engineering and Computer Science, VinUniversity, Hanoi, Vietnam · Center of Digital Transformation and Communication, Can Tho University, Vietnam