How deep is your network? Deep vs. shallow learning of transfer operators
Organizations: Maxwell Institute for Mathematical Sciences, University of Edinburgh and Heriot–Watt University, Edinburgh, UK · School of Mathematics, University of Edinburgh, Edinburgh, UK · School of Mathematical and Computer Sciences, Heriot–Watt University, Edinburgh, UK
Abstract
We introduce RaNNDy, a randomized neural network framework for learning transfer operators and their spectral decompositions. By fixing the weights of the hidden layers and training only the output layer, RaNNDy achieves significant reductions in computational costs -- often by two orders of magnitude -- with similar accuracy compared to state-of-the-art deep learning methods. The method is less sensitive to hyperparameter tuning and provides a closed-form solution for the output layer weights that directly represent eigenfunctions of the learned operator. Moreover, it is possible to estimate uncertainties associated with the computed spectral properties via ensemble learning. We demonstrate the efficacy of the proposed approach using diverse benchmark problems, including stochastic dynamical systems and protein folding processes, highlighting the strengths but also weaknesses of RaNNDy.
Figures & tables
| Computational time (in sec.) | Identified folded frames | Identified unfolded frames | ||||
| Protein | RaNNDy | VAMPnets | RaNNDy | VAMPnets | RaNNDy | VAMPnets |
| Chignolin | 0.52 | 60.44 | 8282 | 8296 | 2412 | 2398 |
| NuG2 | 1.34 | 139.98 | 11706 | 11728 | 6879 | 6857 |
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
| Computational time (in sec.) | ||
|---|---|---|
| System | RaNNDy | VAMPnets (100 epochs) |
| OU process | 0.32 | 121.35 |
| Lemon-slice | 0.33 | 117.65 |
| Triple-well | 0.65 | 125.70 |
| Weight Distribution | Bias Distribution | Time (s) | VAMP Score |
|---|---|---|---|
| normal | normal | 0.3186 | 2.1902 |
| normal | uniform | 0.2529 | 2.1782 |
| lecun_normal | normal | 0.7938 | 2.1018 |
| lecun_normal | uniform | 0.8749 | 2.0938 |
| uniform | normal | 0.3097 | 0.4270 |
| uniform | uniform | 0.3197 | 0.4332 |
| Weight Distribution | Bias Distribution | Time (s) | VAMP Score |
|---|---|---|---|
| normal | normal | 1.5467 | 2.7475 |
| normal | uniform | 0.1705 | 2.7879 |
| normal | zeros | 0.6861 | 2.7802 |
| lecun_normal | normal | 0.2509 | 2.0616 |
| lecun_normal | uniform | 0.1862 | 2.0510 |
| lecun_normal | zeros | 0.1554 | 2.0708 |
| Weight Scale ( ) | Bias Scale ( ) | Time (s) | VAMP Score |
|---|---|---|---|
| 0.001 | 0.001 | 0.3179 | 1.0000 |
| 0.001 | 0.5 | 0.2962 | 1.0000 |
| 0.001 | 3.0 | 0.2718 | 1.0000 |
| 0.001 | 10.0 | 0.2725 | 1.0000 |
| 0.05 | 0.001 | 0.4051 | 1.6128 |
| 0.05 | 0.5 | 0.4017 | 1.6128 |
| System | Avg. Search Time (s) | Best Initialization | Best VAMP Score |
|---|---|---|---|
| Chignolin | 5–6 | normal weights + normal biases | 2.57 |
| NuG2 | 9–10 | LeCun normal weights + Zero biases | 4.61 |
| Ensemble Size | Sum of Average Eigenvalues (VAMP Score) |
|---|---|
| 50 | 2.182460 |
| 100 | 2.182810 |
| 200 | 2.182152 |
| 400 | 2.182144 |
| 600 | 2.182425 |
| Method | Feature Configuration | Scale ( ) | VAMP Score | Time (s) |
|---|---|---|---|---|
| RFF | 2048 | 1.0 | 3.98 | 6.47 |
| RFF | 4096 | 1.0 | 3.50 | 84.47 |
| RFF | 2048 | 2.0 | 7.25 | 7.72 |
| RFF | 4096 | 2.0 | 7.11 | 72.47 |
| RFF | 8192 | 2.0 | 6.59 | 422.49 |
| RFF | 2048 | 3.0 | 8.17 | 7.28 |
Operator Learning
Operator learning studies models that map functions or fields to functions, enabling fast prediction across scientific and dynamical systems.
See all 112 papers