Understanding the geometry of deep learning with decision boundary volume
Authors: Matthew Burfitt, Jacek Brodzki, Pawel Dłotko
Organizations: Beijing Key Laboratory of Topological Statistics and Applications for Complex Systems, Beijing Institute of Mathematical Sciences and Applications (BIMSA) · Mathematical Sciences, University of Southampton · Dioscuri Centre for Topological Data Analysis, Institute of Mathematics, Polish Academy of Sciences
For classification tasks, the performance of a deep neural network is determined by the structure of its decision boundary, whose geometry directly affects essential properties of the model, including accuracy and robustness. Motivated by a classical tube formula due to Weyl, we introduce a method to measure the decision boundary of a neural network through local surface volumes, providing a theoretically justifiable and efficient measure enabling a geometric interpretation of the effectiveness of the model applicable to the high dimensional feature spaces considered in deep learning. A smaller surface volume is expected to correspond to lower model complexity and better generalisation. We verify, on a number of image processing tasks with convolutional architectures that decision boundary volume is inversely proportional to classification accuracy. Meanwhile, the relationship between local surface volume and generalisation for fully connected architecture is observed to be less stable between tasks. Therefore, for network architectures suited to a particular data structure, we demonstrate that smoother decision boundaries lead to better performance, as our intuition would suggest.
Figures & tables
Figure 1. The volumes of a simpler decision boundary (orange) and its tubular neighborhood (yellow) should be smaller than the volumes of a more complex decision boundary and its tubular neighbourhood (purple) when measured in the vicinity of two data manifolds (red and blue).
Figure 2. To the left, a demonstration of layer-wise operations in a convolutional neural network taking grayscale images as an input. On the right, a neural network trained to classify two classes of 2 -dimensional points. Class 0 points are indicated by blue squares and the class 1 points by orange triangles. The decision boundary is formed by the black line in-between the blue and orange decision regions of the two label classes.
Figure 3. Frequency of minj∥x−xj∥2maxj∥x−xj∥2−mini,j∥x−xj∥2 values for data points x=xj in the MNIST and CIFAR-10 data sets, respectively. In particular, the width of the distribution for the higher dimensional CIFAR-10 data is smaller than width of the MNIST data distribution.
Figure 4. Sample space of δ -neighbourhoods about training points in which to estimate the TrainBvol ε -neighbourhood volume of the network decision boundary.
Figure 5. The points sampled on the boundary linearly between data classes in the first step (orange) within a δ neighbourhood of which the second step estimate of LAdvBvol , the ε -neighbourhood volume of the network decision boundary is made.
Figure 6. Value of the bound given in equation ( 13 ) over varying dimensions in the case of 100 cubes for each dimension, with curves for varying sizes of ζ between 0.1 and 0.9 . The bound consistently vanishes for even modestly high dimensions.
MNIST
Fashion MNIST
CIFAR-10
Conv 16,5×5+1
Conv 16,5×5+1
Conv 16,3×3+1
Pool 2×2+2
Pool 2×2+2
Pool 2×2+2
Conv 32,5×5+1
Conv 32,5×5+1
Conv 32,3×3+1
Pool 2×2+2
Pool 2×2+2
Pool 2×2+2
FC 100
FC 100
Conv 64,3×3+1
Pool 2×2+2
Table 1. Convolutional architectures used for training on data sets MNIST, Fashion MNIST, and CIFAR-10. The notation Conv a,b×b+1 means a 2 -dimensional convolutional layer with a square filters of size b , a stride of 1 , and same padding. All pooling layers Pool 2×2+2 used square filters of size 2 , a stride of 2 , and had no padding. The notation FC c denotes a fully connected layer with c hidden units.
Data
Method
ε
δ
(Nework)
Bvol
0.001
-
MNIST
TrainBvol
0.003
0.2
(FC)
LAdvBvol
0.001
0.2
Fashion
Bvol
0.001
-
MNIST
TrainBvol
0.003
0.2
Table 2. The parameter values of ε and δ for each neighbourhood boundary volume measurement using the FGSM introduced in Section 2.5 .
Data
Minimal
Average
Fraction
Fraction
Fraction
(network)
distance
distance
<0.2
<0.1
<0.05
MNIST (Fc)
0.0102
0.587
0.0034
0.0008
0.0004
Fashion MNIST (Fc)
0.000400
0.4601
0.0269
0.0181
0.0145
CIFAR-10 (Fc)
0.000270
0.388
0.0669
0.039
0.0313
MNIST (Conv)
0.148
0.577
0.0002
0
0
Fashion MNIST (Conv)
0.000463
0.465
0.0215
0.0057
0.0026
Table 3. Statistics on the minimal d∞ distances between pairs of 10000 linear adversarial points sampled using the method described in Sections 6.4 for constructing points lying linearly between two training examples as part of the LAdvBvol method. For networks trained with each combination of data sets and network architecture, the minimum distance and average minimum distances between pairs of the linear adversarial points are given alongside the fraction of those pairs of points that were less than 0.2 , 0.1 , or 0.05 apart.
Figure 7. Values of Bvol , TrainBvol , and LAdvBvol over a range of sample sizes plotted alongside 95% confidence intervals obtained treating the Monte Carlo method as an estimation of a Bernoulli parameter (see Section 6.1 ). The network used was fully connected and trained on MNIST with the Adam optimiser for 100 epochs. In each plot the boundary volume measurements stabilise with reasonably sized confidence intervals when about 6000 or more samples are taken.
Figure 8. Values of ε -neighbourhood boundary volume measurements over a range of ε values for 10 networks trained on MNIST. The stability of the measurements appears good over the entire range of ε . We expect the rate of growth of the neighbourhood boundary volume measurements to decrease slightly for large values of ε , as the adversarial attack becomes less accurate at a greater distance. This feature is clearly apparent in the Bvol and LAdvBvol plots. However, in the TrainBvol plot there is an initial increase in the rate of growth as ε increases. Following the discussion in Sections 4.1 and 4.2 , we would expect this to occur if curvature is present in this region of the decision boundary since the lower degree terms in Weyl’s tube formula are curvature dependent.
Data
Architecture
Optimiser
Epochs
+epochs per 0.1 dropout
MNIST
Fc
SGD
100
0
MNIST
Conv
SGD
200
50
MNIST
Fc
Adam
70
30
MNIST
Conv
Adam
50
15
Fashion MNIST
Fc
Adam
100
0
Fashion MNIST
Conv
Adam
50
15
Table 4. Number of training epochs and increase in the number of training epochs per increase of 0.1 of dropout rate on the final hidden layer. Epochs are given for fully connected and convolutional neural networks for each data set used in experiments in this section.
Figure 9. Boundary volume over varying dropout rate in the final fully connected layer applied to convolutional neural networks. The top two rows are trained on MNIST with the SGD and Adam optimisers, respectively. The third row is trained on Fashion MNIST, and the bottom row on CIFAR-10, both with the Adam optimiser. We see a local minima in TrainBvol around or just before the optimal test accuracy, while the LAdvBvol measurements generally increase.
Figure 10. Boundary volume and test accuracy over varying dropout rates applied to fully connected neural networks. The top two rows are trained on MNIST with the SGD and Adam optimisers, respectively. The third row is trained on Fashion MNIST, and the bottom row is trained on CIFAR-10, both with the Adam optimiser. Optimal test accuracy is achieved around a critical point of either TrainBvol or LAdvBvol , while the other of the two measurements changes monotonically.
Medical Data Privacy and Privacy-preserving Machine Learning (MDPPML), University of Tübingen. · Institute for Bioinformatics and Medical Informatics (IBMI), University of Tübingen.