Variational Functional Neural Networks (vfNN)
Organizations: Department of Physics, Virginia Polytechnic Institute and State University, Blacksburg, VA 24061, USA. · Center for Soft Matter and Biological Physics, Virginia Polytechnic Institute and State University, Blacksburg, VA 24061, USA. · Department of Mechanical Engineering, Virginia Polytechnic Institute and State University, Blacksburg, VA 24061, USA. · Center for the Mathematics of Biosystems, Virginia Polytechnic Institute and State University, Blacksburg, VA 24061, USA.
Abstract
We present a variational functional neural network (vfNN) algorithm that optimizes physically motivated continuum functionals in integral form (e.g., Lagrangian, Hamiltonian, or free energy) to find the states of a system. That way, vfNN bypasses solving the equations of motion (a.k.a. the Euler-Lagrange equations), which are in the form of differential equations. To this end, our approach is fundamentally different from conventional physics-informed neural networks (PINN) that solve differential equations by minimizing a phenomenological loss function involving higher order derivatives and ad hoc weight factors. In contrast, vfNN merely evaluates lower order derivatives of the generating functional and replaces the weight factors with rigorous physical scales. This also fundamentally changes the loss landscape, and the desired solutions may become unstable. For this emergent nonconvex optimization problem, we develop a quasi-Newton optimization algorithm with simulated annealing that converges to the solutions of the corresponding Euler-Lagrange equations of the functional regardless of their stability properties. Our simulations using vfNN give accurate results for benchmark systems governed by linear or nonlinear ordinary differential equations.
Figures & tables
| Boundary conditions | |
|---|---|
| vfNN | PINN | |||
| SR1 | ADAM | SR1 | ADAM | |
| Number of iterations | ||||
| 28 | 603 | 120 | 442 | |
| 28 | 262 | 68 | 832 | |
| 456 | 854 | 761 | 499 | |
| 5 | 229 | 54 | 413 | |
Neural Network Optimization
Neural network optimization studies algorithms and training dynamics that help models learn efficiently, reliably, and generalize well.
See all 355 papers