cs.LGApr 27, 2026

A Limit Theory of Foundation Models: A Mathematical Approach to Understanding Emergent Intelligence and Scaling Laws

Authors: Jun Shu, Junxiong Jia, Deyu Meng, Zongben Xu

Organizations: School of Mathematics and Statistics, Xi’an Jiaotong University.

Abstract

Emergent intelligence have played a major role in the modern AI development. While existing studies primarily rely on empirical observations to characterize this phenomenon, a rigorous theoretical framework remains underexplored. This study attempts to develop a mathematical approach to formalize emergent intelligence from the perspective of limit theory. Specifically, we introduce a performance function E(N, P, K), dependent on data size N, model size P and training steps K, to quantify intelligence behavior. We posit that intelligence emerges as a transition from finite to effectively infinite knowledge, and thus recast emergent intelligence as existence of the limit lim⁡N,P,K→∞E(N,P,K)\lim_{N,P,K \to \infty} \mathcal{E}(N,P,K), with emergent abilities corresponding to the limiting behavior. This limit theory helps reveal that emergent intelligence originates from the existence of a parameter-limit architecture (referred to as the limit architecture), and that emergent intelligence rationally corresponds to the learning behavior of this limit system. By introducing tools from nonlinear Lipschitz operator theory, we prove that the necessary and sufficient conditions for existence of the limit architecture. Furthermore, we derive the scaling law of foundation models by leveraging tools of Lipschitz operator and covering number. Theoretical results show that: 1) emergent intelligence is governed by three key factors-training steps, data size and the model architecture, where the properties of basic blocks play a crucial role in constructing foundation models; 2) the critical condition Lip(T)=1 for emergent intelligence provides theoretical support for existing findings. 3) emergent intelligence is determined by an infinite-dimensional system, yet can be effectively realized in practice through a finite-dimensional architecture. Our empirical results corroborate these theoretical findings.

Explore similar work

CardsList
Jun 16, 2026cs.AI

Small Initialization Matters for Large Language Models

Large language models provide a tractable system for asking how intelligence itself emerges, rather than only how LLMs can be engineered. Although progress is usually attributed to scale, data and architecture, we show that parameter initialization is a gene-like determinant of training and, in particular, of model capacity. Reducing the initialization scale consistently improves pretraining, with the largest gains on reasoning-demanding tasks. We identify two widely used empirical settings that restrain the advantage of small initialization, and show how relaxing them restores favorable scaling. We further uncover a critical initialization that balances the reasoning and training. Mechanistically, small initialization drives a distinct developmental trajectory: parameters first condense into low-complexity structures and later expand into richer representations, giving concrete form to the idea that compression is intelligence. Token-level analyses show that the gains concentrate on non-trivial, context-constrained predictions rather than all tokens uniformly. These results motivate a simple γγ-initialization rule: expose initialization rage as an explicit knob and use small initialization by default, an almost cost-free intervention that improves pretraining and strengthens reasoning across model scales.
Liangkai Hang, Junjie Yao, Zhiyu Li +3
Capacity
AskPDF
Jul 1, 2026cs.LG

From Approximation to Emergence: A Theory of Deep Learning

Deep learning has outgrown any single mathematical explanation. From Approximation to Emergence develops a unified, proof-oriented account of modern deep learning theory, tracing a path from the classical foundations of approximation, optimization, and generalization to the contemporary mechanisms of overparameterization, robustness, generative modeling, transformers, in-context learning, scaling laws, interpretability, alignment, and emergence. Rather than presenting isolated results, the book organizes a broad literature into a coherent research narrative: each theory is examined through the object it controls, the assumptions that make it valid, and the phenomena it leaves unexplained. Written for researchers, graduate students, and mathematically trained practitioners, this monograph offers a rigorous map of deep learning theory as it stands today: powerful, incomplete, and increasingly centered on the question of how learned mechanisms arise from scale, data, architecture, and training.
Zhilin Zhao
Over-Parameterized Neural NetworksDeep LearningTheoretical FoundationsApproximationEmergenceModel Interpretability+1
AskPDF
Aug 3, 2026cs.AI

Emergence Invariance: From Symbolized Thought to Structural Control

Language-first intelligence is constrained by which distinctions enter its symbolic record, which mappings its language--interpreter--environment complex can execute, and which possibilities can be realized with finite resources. We formalize these limits through an effective interface φφ, executable support Πφ,HΠ_{φ,H}, a realization profile MM, and resource-indexed families Fs(J,M)\mathcal F_s(\mathcal J,M). They induce four nested capability levels: current reachability, budget realizability, asymptotic realizability, and structural capability, each with literal and risk-equivalent forms. On finite task spaces, universal bounded-loss dominance is characterized by inclusion of closed convexified risk envelopes; directed deficiencies and resource transformations quantify approximate simulation and realization burden. The decomposition Rs,J,M∗=RJ∗+Cs(J,M)\mathcal R^*_{s,\mathcal J,M}=\mathcal R^*_{\mathcal J}+C_s(\mathcal J,M) separates structural limits from finite-resource compensation gaps. We develop dynamic control over these boundaries. Target risk identifies the lowest capability level that must change. Ockhamian control operates within inherited structural possibilities; Chattonian-containing control can produce endpoint structural novelty beyond the inherited decision class. Diminishing returns to within-structure computation and sustained structural-enrichment value yield a switching threshold, with switching costs creating hysteresis. Matched DeepSeek V4-Flash evidence shows: thinking raises pointer chasing from 0/160/16 to 14/1614/16, exact observational and memory twins remain at their 0.50.5 floors, restoring decisive memory raises performance from 0.50.5 to 1.01.0, and executable support determines whether reasoning can become effective. The framework organizes LLM emergence limits through structural capability, finite realizability, boundary control, and recursive meta-control.
Yi Liu
Cognitive ScienceEnrichmentInvariance
AskPDF