Susceptibilities for Neural Networks Learning from Physical Data
Organizations: Resolution · CSIRO · University of Melbourne · Monash University
Abstract
Neural networks that are trained on data arising from a physical system must somehow learn regularities induced by the underlying physical laws. In this setting, concepts from statistical physics provide powerful tools for analysing how physical laws influence the learning process. In this paper we explain how susceptibilities --- which measure the response of a neural network to perturbations of a parameter of the data distribution --- can be applied when learning from physical data. We show that such susceptibilities can identify the conditions under which specific input features are most informative for learning, guiding data selection, a prediction we validate experimentally. For distributions lacking a natural parameter, we propose introducing one via a physically meaningful scalar function of the physical state, and demonstrate that susceptibilities track the emergence of specific capabilities (such as collision detection) during training.
Figures & tables
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Hyperparameter | Value |
|---|---|
| Total trainable parameters | 14 839 809 |
| Number of layers | 8 |
| Residual stream dimension | 256 |
| Number of heads | 4 |
| MLP width | 1024 |
| Optimizer | AdamW |
| Hyperparameter | Value |
| Sampler step size | |
| 50 | |
| 0.5 | |
| RMSProp stability parameter | |
| RMSProp exponential decay rate | 0.99 |
| Sampler batch size | 256 |
| Hyperparameter | Value |
|---|---|
| Number of particles | 64 |
| Particle diameter | 1 |
| Particle mass | 1 |
| Collision spring constant | 1000 |
| Collision damping coefficient | 10 |
| Acceleration due to gravity | 9.8 |
| Hyperparameter | Value |
|---|---|
| Total trainable parameters | 432 989 |
| Number of layers | 4 |
| Residual stream dimension | 128 |
| Number of heads | 8 |
| MLP width | 128 |
| Optimizer | Adam |
| Hyperparameter | Value |
| Sampler step size | |
| 1385 | |
| 1 | |
| RMSProp stability parameter | |
| RMSProp exponential decay rate | 0.99 |
| Sampler batch size | 128 |
Statistical Physics of Learning
Statistical physics of learning uses tools from physics to explain how data, model structure, and training dynamics shape learning and generalization.
See all 60 papers