stat.MLSep 27, 2026

Neural Scaling Laws of Transformer Operator Network

Authors: Haoran Yan, Zhongjie Shi, Yuanzhe Xi, Peng Chen, Wenjing Liao

Organizations: School of Mathematics, Georgia Institute of Technology, Atlanta, GA 30332, USA · Department of Mathematics, Emory University, Atlanta, GA 30322, USA · School of Computational Science and Engineering, Georgia Institute of Technology, Atlanta, GA 30332, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Transformers have emerged as powerful architectures for learning solution operators of physical systems. Empirically the prediction error has been observed to decrease when the data size and model size increase, suggesting neural scaling behavior. Yet a theoretical understanding of such scaling laws for transformer-based operator learning remains limited. In this work, we develop a theoretical framework for characterizing the approximation and generalization errors of transformer-based operator learning. Our analysis builds on a local-to-global approximation principle that is naturally aligned with the softmax attention mechanism and yields discretization-invariant output functions. On approximation theory, we derive a universal approximation error of transformer-based operator learning for Hölder-regular operators. On generalization theory, we establish a power scaling law between the prediction error and the training data size. The rate of convergence represented by the scaling exponent explicitly reflects the dimensions of the input and output domains, the regularity of the underlying functions and operators, and crucially, the intrinsic dimension of the input function class. By exploiting this intrinsic low-dimensional structure, our analysis yields a power-law generalization rate for operator learning, in contrast to the logarithmic-type power-law rates appearing in existing analyses of operator learning with feedforward neural networks. Numerical experiments validate the predicted power-law scaling and confirm that the convergence rate varies systematically with the intrinsic dimension of the input function class.

Figures & tables

Figure 1 : Illustration of the operator-learning data-generation setting. Each input function is observed on a shared input grid, while output values are queried at independently sampled locations.
Figure 2: Illustration of the two-level Softmax POU approximation. The input-space POU localizes uu over function-space anchors {uk}k=1CU\{u_{k}\}_{k=1}^{C_{U}} , while the output-domain POU localizes the query y\mathbf{y} over anchors {zl}l=1CV\{\mathbf{z}_{l}\}_{l=1}^{C_{V}} . Their combination yields the oracle approximation in ( 23 ).
Figure 3: Empirical data scaling for Burgers’ (left) and KdV (right). Markers show the geometric mean test MSE over five repeats, with error bars corresponding to one sample standard deviation of the log MSE; dashed lines show the fitted power law MSE≈Cn−p\mathrm{MSE}\approx Cn^{-p} .
Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

 Burgers’KdV
Initial learning rate10−310^{-3}3×10−43\times 10^{-4}
Batch size25664
Random output queries per training function816
Maximum epochs12,0006,000
Test functions128128
Test spatial grid1024 uniform points1024 uniform points
Appendix
Table 1: Training and evaluation settings for the Burgers’ and KdV experiments.
Figure 4: Representative Burgers’ predictions. The top and bottom rows correspond to n=32n=32 and n=1024n=1024 , respectively; the left and right columns correspond to dU=2d_{\mathcal{U}}=2 and 88 . Dotted curves show inputs, solid curves reference solutions, and dashed curves model predictions. Each column uses the same held-out test input across both sample sizes.
Figure 5: Representative KdV predictions. The top and bottom rows correspond to n=32n=32 and n=1024n=1024 , respectively; the left and right columns correspond to dU=4d_{\mathcal{U}}=4 and 88 . Dotted curves show inputs, solid curves exact solutions, and dashed curves model predictions. Each column uses the same held-out test input across both sample sizes.

Explore similar work

CardsList

  1. Function graph transformers universally approximate operators between function spaces

    May 18, 2026Takashi Furuya, David Mis, Ivan Dokmanić +2Transformer AttentionNeural Network Approximation Theory

    Doshisha University, RIKEN AIP · Rice University · University of Basel +2

  2. Generalization Guarantees for Multi-Input Neural Operator Learning in Sobolev Spaces

    Jun 16, 2026Yahong Yang, Zecheng Zhang, Wei Zhu +2Neural Network Approximation TheoryOperator Learning

    School of Mathematics, Georgia Institute of Technology, 686 Cherry Street, Atlanta, GA 30332-0160, USA. · Department of Applied and Computational Mathematics and Statistics, University of Notre Dame, Notre Dame, IN 46556, USA. · Department of Mathematics, Hong Kong Baptist University, FSC1202, Fong Shu Chuen Building, Hong Kong Baptist University, Kowloon Tong, Hong Kong.

  3. Learning Theory of Transformers: Local-to-Global Approximation via Softmax Partition of Unity

    May 9, 2026Zhongjie Shi, Wenjing LiaoNeural Network GeneralizationTransformer

    School of Mathematics, Georgia Institute of Technology, Atlanta, GA 30332, United States