GLL: A Differentiable Graph Learning Layer for Neural Networks
Organizations: Department of Mathematics University of California, Los Angeles Los Angeles, CA 90095, USA · Computing + Mathematical Sciences (CMS) Department California Institute of Technology Pasadena, CA 91125, USA · School of Mathematics University of Minnesota Minneapolis, MN 55455, USA
Abstract
Standard deep learning architectures used for classification generate label predictions with a projection head and softmax activation function. Although successful, these methods fail to leverage the relational information between samples for generating label predictions. In recent works, graph-based learning techniques, namely Laplace learning, have been heuristically combined with neural networks for both supervised and semi-supervised learning (SSL) tasks. However, prior works approximate the gradient of the loss function with respect to the graph learning algorithm or decouple the processes; end-to-end integration with neural networks is not achieved. In this work, we derive backpropagation equations, via the adjoint method, for inclusion of a general family of graph learning layers into a neural network. The resulting method, distinct from graph neural networks, allows us to precisely integrate similarity graph construction and graph Laplacian-based label propagation into a neural network layer, replacing a projection head and softmax activation function for general classification task. Our experimental results demonstrate smooth label transitions across data, improved generalization and robustness to adversarial attacks, and improved training dynamics compared to a standard softmax-based approach.
Figures & tables
| Structure | Layer | Details | Feature Size |
| Input Image | - | From FashionMNIST Dataset | 28 28 1 |
| Feature Encoder | Conv1 | 1 64, kernel size: 3, padding: 1 | 28 28 64 |
| Conv2 | 64 128, kernel size: 3, padding: 1 | 28 28 128 | |
| MaxPool | kernel size: 2, stride: 2 | 14 14 128 | |
| Conv3 | 128 256, kernel size: 3, padding: 1 | 14 14 256 | |
| MaxPool | kernel size: 2, stride: 2 | 7 7 256 |
| Dataset | Trial | Epochs | Test error (%) |
|---|---|---|---|
| FashionMNIST | MLP only (MLP) | 100 MLP | 45.42 |
| FashionMNIST | MLP only (GLL) | 100 MLP | 14.66 |
| FashionMNIST | GLL only (GLL-0) | 50 GLL | 8.90 |
| FashionMNIST | GLL-50 | 50 MLP + 50 GLL | 8.68 |
| FashionMNIST | GLL-75 | 75 MLP + 25 GLL | 8.91 |
| Network | Params | ||
|---|---|---|---|
| Encoder | MLP head | Total | |
| VGG11 | 9,223,232 | 332,778 | 9,556,010 |
| VGG13 | 9,407,936 | 332,778 | 9,740,714 |
| ResNet20 | 271,824 | 16,938 | 288,762 |
| ResNet32 | 466,256 | 16,938 | 483,194 |
| ResNet44 | 660,688 | 16,938 | 677,626 |
| Train Num | Whole (50000) | 20% (10000) | 2% (1000) | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Network | MLP | WNLL | GLL | MLP | WNLL | GLL | MLP | WNLL | GLL |
| VGG11 | 7.77 0.22 | 7.71 0.10 | 7.48 0.19 | 15.16 0.58 | 14.74 0.27 | 13.14 0.09 | 34.76 1.38 | 34.97 1.57 | 29.59 0.12 |
| VGG13 | 6.16 0.31 | 6.20 0.23 | 5.80 0.03 | 12.68 0.48 | 12.27 0.19 | 11.03 0.04 | 31.61 1.17 | 31.31 0.49 | 26.90 0.18 |
| ResNet20 | 9.25 0.96 | 7.99 0.33 | 7.49 0.24 | 15.60 0.74 | 14.76 0.68 | 14.00 0.31 | 33.35 1.81 | 32.04 0.58 | 31.25 0.19 |
| ResNet32 | 8.45 0.96 | 7.26 0.29 | 6.61 0.15 | 14.78 0.58 | 13.63 0.24 | 13.04 0.13 | 33.93 1.76 | 30.82 0.79 | 32.96 1.44 |
| ResNet44 | 8.07 0.78 | 6.57 0.11 | 6.12 0.12 | 14.38 0.82 | 13.23 0.33 | 12.51 0.34 | 33.76 2.07 | 31.85 1.46 | 31.69 0.70 |
| Train Num | Whole (112800) | 20% (22560) | 2% (2256) | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Network | MLP | WNLL | GLL | MLP | WNLL | GLL | MLP | WNLL | GLL |
| VGG11 | 10.72 0.14 | 12.18 0.79 | 10.76 0.09 | 12.06 0.13 | 13.46 0.69 | 11.96 0.08 | 15.86 0.35 | 16.64 0.78 | 14.98 0.13 |
| VGG13 | 10.50 0.06 | 10.73 0.15 | 10.60 0.08 | 11.85 0.08 | 15.43 1.15 | 11.94 0.10 | 15.91 0.38 | 16.48 0.82 | 15.08 0.20 |
| ResNet20 | 9.72 0.07 | 10.77 0.29 | 10.34 0.05 | 11.45 0.06 | 12.33 0.21 | 11.60 0.07 | 15.79 0.33 | 17.10 1.01 | 14.63 0.12 |
| ResNet32 | 9.73 0.06 | 10.78 0.37 | 10.15 0.14 | 11.41 0.05 | 12.26 0.38 | 11.54 0.11 | 15.85 0.46 | 17.52 0.90 | 14.30 0.08 |
| ResNet44 | 9.92 0.06 | 11.30 0.36 | 10.10 0.09 | 11.55 0.16 | 12.04 0.28 | 11.43 0.14 | 15.87 0.35 | 17.81 0.98 | 14.98 0.18 |
| Train Num | Whole (112800) | 20% (22560) | 2% (2256) | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Network | MLP | WNLL | GLL | MLP | WNLL | GLL | MLP | WNLL | GLL |
| VGG11 | 35.88 0.48 | 36.92 1.15 | 35.91 0.28 | 38.51 0.42 | 36.57 0.55 | 38.10 0.18 | 42.91 0.76 | 45.05 0.33 | 42.14 0.33 |
| VGG13 | 34.88 0.32 | 36.05 0.37 | 35.38 0.34 | 37.90 0.23 | 38.68 0.74 | 38.11 0.33 | 44.51 1.48 | 45.28 0.53 | 42.29 0.36 |
| ResNet20 | 31.66 0.21 | 35.76 0.59 | 33.56 0.38 | 35.80 0.59 | 36.00 0.36 | 35.53 0.23 | 43.17 1.29 | 46.61 0.96 | 41.45 0.41 |
| ResNet32 | 31.69 0.29 | 34.47 0.81 | 32.87 0.49 | 36.09 0.32 | 36.72 0.60 | 36.09 0.42 | 43.66 1.35 | 45.80 0.62 | 39.94 0.33 |
| ResNet44 | 31.84 0.27 | 35.61 0.56 | 32.83 0.25 | 37.01 0.56 | 36.47 0.44 | 35.98 0.57 | 43.70 1.44 | 46.68 0.39 | 41.51 0.51 |
| Batch Size | Metric | GLL | MLP |
|---|---|---|---|
| 128 | Training time / batch (s) | ||
| Testing time / batch (s) | |||
| Testing error (%) | 0.76 | 2.11 | |
| 256 | Training time / batch (s) | ||
| Testing time / batch (s) | |||
| Testing error (%) | 1.33 | 1.69 |
| Batch Size | Metric | GLL | MLP |
|---|---|---|---|
| 128 | Training time / batch (s) | ||
| Testing time / batch (s) | |||
| Testing error (%) | 0.84 | 0.86 | |
| 256 | Training time / batch (s) | ||
| Testing time / batch (s) | |||
| Testing error (%) | 0.94 | 1.04 |
| Batch Size | Metric | GLL | MLP |
|---|---|---|---|
| 128 | Training time / batch (s) | ||
| Testing time / batch (s) | |||
| Testing error (%) | 1.65 | 2.39 | |
| 256 | Training time / batch (s) | ||
| Testing time / batch (s) | |||
| Testing error (%) | 2.10 | 1.18 |
| Dataset | Model | Natural | FGSM | IFGSM | CW |
|---|---|---|---|---|---|
| MNIST | GL Robust | 0.55 | 2.28 | 2.30 | 1.16 |
| WNLL Robust | 0.58 | 2.48 | 2.97 | 2.72 | |
| MLP Robust | 0.80 | 4.83 | 6.43 | 2.59 | |
| GL Natural | 0.47 | 3.49 | 3.56 | 1.29 | |
| WNLL Natural | 0.50 | 3.54 | 4.64 | 3.19 | |
| MLP Natural | 0.75 | 5.74 | 8.48 | 2.76 |
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.