Organizations: Department of Physics, Virginia Polytechnic Institute and State University, Blacksburg, VA 24061, USA. · Center for Soft Matter and Biological Physics, Virginia Polytechnic Institute and State University, Blacksburg, VA 24061, USA. · Department of Mechanical Engineering, Virginia Polytechnic Institute and State University, Blacksburg, VA 24061, USA. · Center for the Mathematics of Biosystems, Virginia Polytechnic Institute and State University, Blacksburg, VA 24061, USA.
We present a variational functional neural network (vfNN) algorithm that optimizes physically motivated continuum functionals in integral form (e.g., Lagrangian, Hamiltonian, or free energy) to find the states of a system. That way, vfNN bypasses solving the equations of motion (a.k.a. the Euler-Lagrange equations), which are in the form of differential equations. To this end, our approach is fundamentally different from conventional physics-informed neural networks (PINN) that solve differential equations by minimizing a phenomenological loss function involving higher order derivatives and ad hoc weight factors. In contrast, vfNN merely evaluates lower order derivatives of the generating functional and replaces the weight factors with rigorous physical scales. This also fundamentally changes the loss landscape, and the desired solutions may become unstable. For this emergent nonconvex optimization problem, we develop a quasi-Newton optimization algorithm with simulated annealing that converges to the solutions of the corresponding Euler-Lagrange equations of the functional regardless of their stability properties. Our simulations using vfNN give accurate results for benchmark systems governed by linear or nonlinear ordinary differential equations.
Figures & tables
Figure 1: Variational functional neural network (vfNN) . We use a multilayer perceptron (MLP) deep neural network with an input x and output u that minimizes the variational loss J. The derivative u′(x) is computed using automatic differentiation at each input point x. The Dirichlet boundary conditions are implemented by transforming the output of the network uN with known functions l~(x) and g(x). The Neumann boundary conditions are implemented by including a corresponding boundary term in the functional J. For the definitions of σ,l,W,o, see Eqs. 4 , 5 .
f(u)
Boundary conditions
0
u(−1.0)=−0.06,u(0)=0.
0.25
u(−1)=0.06,u(1.0)=0.06.
u
u(−1)=e−1,u(2)=e2.
−(2π)2u
u(0)=1,u(1)=0.
−(23π)2u
u(0)=1,u(1)=0.
Table 1: Functions f(u) and corresponding boundary conditions for Eqs. 22 , 23 . The results are shown in Fig. 2 .
Figure 2: Linear ODEs using vfNN. The function f(u) in Eq. 23 is chosen as (a) f(u)=0, (b) f(u)=0.25, (c) f(u)=u, (d) f(u)=−(π/2)2u, and (e) f(u)=−(3π/2)2u∗. (f) shows the variational loss J for f(u)=u in (c) as a function of the iterations of our SR1 update (Algorithm 2 ). The dashed red lines show the solutions obtained using vfNN, and the solid blue lines indicate the solutions obtained using PINN. ∗ The domain in (e) contains a conjugate point that makes the loss landscape nonconvex; see Appendices A , B .
f(u)
vfNN
PINN
SR1
ADAM
SR1
ADAM
Number of iterations
0
28
603
120
442
0.25
28
262
68
832
u
456
854
761
499
−(2π)2u
5
229
54
413
Table 2: Convergence rates of vfNN vs. PINN . By specifying different f(u) for the problem given in Eq. 22 , we optimize PINN and vfNN via two algorithms: the first-order optimization algorithm ADAM [ 36 ] and the second-order optimization algorithm SR1 (Sec. II ). For f(u)=−(3π/2)2u, the use of ADAM optimizer with vfNN leads to the divergence of the loss function since the domain contains a conjugate point that makes the loss landscape nonconvex; see Appendices A , B . For all cases, a deep neural network with three layers and 20 neurons per layer is used. The number of iterations are averaged over 30 independent initializations.
Figure 3: Nonlinear ODEs using vfNN. Solutions to the functionals whose ELE are (a) the minimal surface of revolution (Eqs. 24 , 25 ), (b) steady-state Sine-Gordon equation (Eq. 26 ), and (c) the Poisson-Boltzmann equation (Eqs. 27 – 29 ). The dashed red lines indicate the solutions using vfNN, and all other colors correspond to different nonunique solutions (a) from the exact solution (Eq. 25 ) and (b), (c) found by using FEM. For all panels, a deep neural network with three layers and 20 neurons at each layer is used.
Figure 4: Nonlinear ODEs using vfNN (continued). Minimization of the elastic energy of a colloidal membrane composed of hard rods (Eqs. 30 – 34 ) [ 44 ] . (a) θ(x) the tilt angle with respect to membrane normal. (b) ψ(x) is the scalar order parameter describing the orientation of the rods with respect to the membrane normal. The x axis is nondimensionalized by the twist penetration depth λt. The membrane interior is at x=0, whereas x=L=10 is the position of the membrane edge. The dashed red lines indicate the solutions using vfNN, and the blue curves correspond to the solutions from FEM. Two separate deep neural networks (three layers with 20 neurons at each layer) for each of the coupled dependent variables θ(x) , ψ(x) were used.
Physics-Informed Neural Networks (PINNs) solve differential equations by minimizing the residual of a nonlinear operator over a neural parameterization of the solution. However, monolithic PINNs often suffer from ill-conditioning, spectral bias, and optimization instability. We introduce a variational boosting framework in which solutions are constructed additively in function space. Each stage trains a weak learner whose converged correction satisfies a local orthogonality condition, equivalent to a projected functional gradient descent step onto the tangent space of the network's function manifold. Because each correction network is deliberately small, the restricted minimization admits full Newton or conjugate gradient updates, which are typically infeasible in large PINNs. The resulting method separates global nonlinear refinement into a sequence of well-conditioned subproblems while preserving the full variational structure of the operator. This framework provides a geometric interpretation of multi-stage PINNs as projected functional gradient descent and enables stable second-order optimization for nonlinear differential equations.
Efficient and robust optimization is essential for neural networks, enabling scientific machine learning models to converge rapidly to very high accuracy -- faithfully capturing complex physical behavior governed by differential equations. In this work, we present advanced optimization strategies to accelerate the convergence of physics-informed neural networks (PINNs) for challenging partial (PDEs) and ordinary differential equations (ODEs). Specifically, we provide efficient implementations of the Natural Gradient (NG) optimizer, Self-Scaling BFGS and Broyden optimizers, and demonstrate their performance on problems including the Helmholtz equation, Stokes flow, inviscid Burgers equation, Euler equations for high-speed flows, and stiff ODEs arising in pharmacokinetics and pharmacodynamics. Beyond optimizer development, we also propose new PINN-based methods for solving the inviscid Burgers and Euler equations, and compare the resulting solutions against high-order numerical methods to provide a rigorous and fair assessment. Finally, we address the challenge of scaling these quasi-Newton optimizers for batched training, enabling efficient and scalable solutions for large data-driven problems.
Anas Jnini, Elham Kiyani, Khemraj Shukla +5
Department of Information Engineering and Computer Science, University of Trento, Trento, Italy · Division of Applied Mathematics, Brown University, Providence, RI, USA · Departament de Física, Universitat d’Alacant, Spain +3
Evolution equations with time-dependent coefficients force a neural solver to spend capacity on a scale that the equation already determines. We show that a single classical object, the Ermakov-Pinney scale function ρ, can be written into the input layer of a physics-informed neural network as a parameter-free embedding. One construction covers three problem families through the two signs of the Pinney constant c: linear diffusion (c=−1, ω≡0), diffusion in a time-dependent confining trap (c=−1, ω=0), and the quantum oscillator with time-dependent frequency (c=+1). An exact moment identity gives ρ in closed form from the initial data alone, so no auxiliary network or knowledge of the solution is needed. Our central finding is comparative: each family requires a different component of the construction and has a different honest control. In family I, the diffusive-time reparametrisation alone reproduces most of the gain, so that family is a poor test of the idea and shows why an ablation is mandatory. In family II no time reparametrisation exists, the control degrades accuracy, and the amplitude prefactor becomes essential (40.8x). In family III the adiabatic approximation also fails, and the quadratic phase factor is what matters (5.7x). The method does not help when the reduced solution does not converge to a fixed point: its extrapolation advantage then disappears. The reduction itself is the classical scaling-variable change of the self-similar-asymptotics literature; what is new is the moment identity that makes it architectural for time-dependent coefficients, and the comparative anatomy above.
Roberto Bernal-Jaquez, Roberto Romero Huerta, Alexander Schaum
Departamento de Matemáticas Aplicadas y Sistemas, Universidad Autónoma Metropolitana–Cuajimalpa, Mexico City, Mexico · Posgrado en Ciencias Naturales e Ingeniería, Universidad Autónoma Metropolitana–Cuajimalpa, Mexico City, Mexico · Department for Process Analytics, University of Hohenheim, Stuttgart, Germany