cs.LGJul 8, 2026

An optimal control approach for neural network architecture adaptation with a posteriori error estimation

Authors: C G Krishnanunni, Thomas Scott, Tan Bui-Thanh

Organizations: Department of Aerospace Engineering & Engineering Mechanics, UT Austin · Oden Institute for Computational Engineering and Sciences, UT Austin.

Abstract

This work presents a novel approach for adapting neural network architecture along the depth based on a posteriori error estimation. By formulating neural network training as a continuous-time optimal control problem, we derive rigorous error estimates that quantify how approximation error distributes across network layers. This error decomposition enables a principled depth adaptation strategy: new layers are inserted at locations of maximum estimated error, allowing the network to efficiently capture complex, nonlinear variations in the underlying problem. Our framework introduces a novel network architecture that treats weights and biases as piecewise linear functions varying across layers, with the error estimator bounding the discrepancy between this discrete representation and the true continuous optimal control solution. The approach leverages dual weighted residual methodology from finite element analysis to derive computable upper bounds on the functional error. A key theoretical contribution is the derivation of explicit error bounds that decompose the total approximation error into interval-wise contributions, providing a rigorous basis for targeted architecture refinement. We demonstrate the effectiveness of our method on scientific datasets, including learning the observable-to-parameter map for the Navier-Stokes equation. Numerical results reveal that our approach consistently outperforms existing architecture adaptation methods in terms of generalization performance.

Explore similar work

CardsList
Jul 2, 2026stat.ML

Born Discrete, Made Smooth: Variational Formulation of Shallow Neural Networks

Although neural networks are remarkably effective, their underlying optimization principles remain theoretically elusive, often characterized by non-convex landscapes and stochastic heuristics. In this work, we propose a paradigm shift by replacing the discrete training problem of shallow neural networks with a well-posed continuum variational surrogate. We identify a family of λλ-convex functionals over parameter densities in weighted Sobolev spaces and prove that these variational problems are globally well-posed, stable, and exhibit unexpected almost C3C^3 regularity. Unlike existing Wasserstein-based or Mean-Field approaches, which often face limited regularity and discretization challenges, our formulation provides direct access to elliptic regularity and convex analysis. This allows us to prove that the optimal parameter density can be obtained by solving a single linear system, bypassing iterative optimization entirely. We establish explicit generalization error controls at a rate of 1/α1/α relative to the regularization parameter, and prove that finite-width networks of size NN achieve the continuum optimum at an O(1/N)O(1/N) rate. This perspective bridges the gap between the Neural Tangent Kernel (NTK) and feature-learning regimes, providing a principled framework for understanding over-parameterization through the lens of variational calculus.
Matej Benko, Pierre Bousquet, Iwona Chlebicka +1
Two-Layer Neural NetworksVariationalNeural NetworkRegularization TechniquesBayesian Quadrature
AskPDF
Mar 19, 2026cs.LG

Rigorous Error Certification for Neural PDE Solvers: From Empirical Residuals to Solution Guarantees

Uncertainty quantification for partial differential equations is traditionally grounded in discretization theory, where solution error is controlled via mesh/grid refinement. Physics-informed neural networks fundamentally depart from this paradigm: they approximate solutions by minimizing residual losses at collocation points, introducing new sources of error arising from optimization, sampling, representation, and overfitting. As a result, the generalization error in the solution space remains an open problem. Our main theoretical contribution establishes generalization bounds that connect residual control to solution-space error. We prove that when neural approximations lie in a compact subset of the solution space, vanishing residual error guarantees convergence to the true solution. We derive deterministic and probabilistic convergence results and provide certified generalization bounds translating residual, boundary, and initial errors into explicit solution error guarantees.
Amartya Mukherjee, Maxwell Fitzsimmons, David C. Del Rey Fernández +1
Neural Partial Differential Equation SolversNeural-Network ApproximationDiscretizationGeneralization Bounds
AskPDF
Apr 22, 2026cs.LG

Layer-wise Geometric Approximation Rates for Deep Networks

Depth is widely viewed as a central contributor to the success of deep neural networks, whereas standard neural network approximation theory typically provides guarantees only for the final output and leaves the role of intermediate layers largely unclear. We address this gap by developing a quantitative framework in which depth admits a precise scale-dependent interpretation. Specifically, we design a single shared mixed-activation architecture of fixed width 2dN+d+22dN+d+2 and any prescribed finite depth such that each intermediate readout ΦℓΦ_\ell is itself an approximant to the target function ff. For f∈Lp([0,1]d)f\in L^p([0,1]^d) with p∈[1,∞)p\in [1,\infty), the approximation error of ΦℓΦ_\ell is controlled by (2d+1)(2d+1) times the LpL^p modulus of continuity at the geometric scale N−ℓN^{-\ell} for all ℓ\ell. The estimate reduces to the geometric rate (2d+1)N−ℓ(2d+1)N^{-\ell} if ff is 11-Lipschitz. Our network design is inspired by multigrade deep learning, where depth serves as a progressive refinement mechanism. For every prescribed terminal depth, the construction yields a finite nested family of prefix readouts whose earlier correction terms remain embedded in later readouts. Thus the approximation may be truncated within the prescribed depth range once the desired certified accuracy is reached.
Shijun Zhang, Zuowei Shen, Yuesheng Xu
Neural-Network ApproximationDeep NetworkApproximationGeometric Deep LearningDeep Learning
AskPDF