Upper Generalization Bounds for Neural Oscillators
Authors: Zifeng Huang, Konstantin M. Zuev, Yong Xia, Michael Beer
Organizations: Institute for Risk and Reliability, Leibniz University Hannover, Callinstraße 34, Hannover, 30167, Germany · Department of Computing and Mathematical Sciences, California Institute of Technology, Pasadena, California, United States · Joint Research Centre for Marine Infrastructure, Department of Civil and Environmental Engineering, The Hong Kong Polytechnic University, Kowloon, Hong Kong, China · Department of Civil and Environmental Engineering, University of Liverpool, Liverpool, L69 3GH, United Kingdom · International Joint Research Center for Resilient Infrastructure & International Joint Research Center for Engineering Reliability and Stochastic Mechanics, Tongji University, Shanghai, 200092, China
Neural oscillators that originate from second-order ordinary differential equations (ODEs) have shown competitive performance in learning mappings between dynamic loads and responses of complex nonlinear structural systems. Despite this empirical success, theoretically quantifying the generalization capacities of their neural network architectures remains undeveloped. In this study, the neural oscillator consisting of a second-order ODE followed by a multilayer perceptron (MLP) is considered. Its upper probably approximately correct (PAC) generalization bound for approximating causal and uniformly continuous operators between continuous temporal function spaces and that for approximating the uniformly asymptotically incrementally stable second-order dynamical systems are derived by leveraging the Rademacher complexity framework. These bounds are further extended to the squared Wasserstein-1 distances between the probability measures of quantities of interest calculated from target causal operators and the corresponding learned neural oscillators. The theoretical results show that the estimation errors grow polynomially with respect to both MLP sizes and the time length, thereby avoiding the curse of parametric complexity. Furthermore, the derived error bounds demonstrate that constraining the Lipschitz constants of the MLPs via loss function regularization can improve the generalization ability of the neural oscillator. Numerical studies considering a Bouc-Wen nonlinear system under stochastic seismic excitation validates the theoretically predicted power laws of the estimation errors with respect to the sample size and time length, and confirms the effectiveness of constraining MLPs' matrix and vector norms in enhancing the performance of the neural oscillator under limited training data.
Neural operators provide a framework for learning solution operators of partial differential equations (PDEs), enabling efficient surrogate modeling for complex systems. While universal approximation results are now well understood, quantitative approximation analysis specific to nonlinear reaction-diffusion systems remains limited. In this paper, we study Laplacian-based neural operators applied to the solution mapping from initial conditions to time-dependent solutions of nonlinear reaction-diffusion systems. By exploiting the Laplacian spectral representation of the diffusion semigroup, we obtain polynomial complexity bounds on the network depth, width, and spectral rank, demonstrating that Laplacian eigenfunction-based neural operators can exploit the structure of the underlying PDE while avoiding the curse of parametric complexity encountered in generic operator learning. Numerical experiments support the theoretical findings and illustrate the benefit of Laplacian eigenfunction representations compared with Fourier-based spectral representations.
Takashi Furuya, Ryo Ozawa, Jenn-Nan Wang
Doshisha University, RIKEN AIP · Tohoku University · National Taiwan University
Training modern neural networks often relies on large learning rates, operating at the edge of stability, where the optimization dynamics exhibit oscillatory and chaotic behavior. Empirically, this regime often yields improved generalization performance, yet the underlying mechanism remains poorly understood. In this work, we represent stochastic optimizers as random dynamical systems, which often converge to a fractal attractor set (rather than a point) with a smaller intrinsic dimension. Building on this connection and inspired by Lyapunov dimension theory, we introduce a novel notion of dimension, coined the `sharpness dimension', and prove a generalization bound based on this dimension. Our results show that generalization in the chaotic regime depends on the complete Hessian spectrum and the structure of its partial determinants, highlighting a complexity that cannot be captured by the trace or spectral norm considered in prior work. Experiments across various MLPs and transformers validate our theory while also providing new insights into the recently observed phenomenon of grokking.
Mario Tuci, Caner Korkmaz, Umut Şimşekli +1
INRIA, CNRS, Département d’Informatique de l’Ecole Normale Supérieure / PSL, France · Department of Computing, Imperial College London, United Kingdom
Deep neural networks (DNNs) have significantly advanced machine learning, with model depth playing a central role in their successes. The dynamical system modeling approach has recently emerged as a powerful framework, offering new mathematical insights into the structure and learning behavior of DNNs. In this work, we establish generalization error bounds for both discrete- and continuous-time residual networks (ResNets) by combining Rademacher complexity, flow maps of dynamical systems, and the convergence behavior of ResNets in the deep-layer limit. The resulting bounds are of order O(1/S) with respect to the number of training samples S, and include a structure-dependent negative term, yielding depth-uniform and asymptotic generalization bounds under milder assumptions. These findings provide a unified understanding of generalization across both discrete- and continuous-time ResNets, helping to close the gap in both the order of sample complexity and assumptions between the discrete- and continuous-time settings.
Jinshu Huang, Mingfei Sun, Chunlin Wu
School of Mathematical Sciences, Nankai University, Tianjin, China