Landau theory of quenched criticality in linear in-context learning
Authors: Daesik Kim, Sumin Choi, Hyojae Jeon, Jung Hoon Han
Organizations: Department of Physics, Sungkyunkwan University, Suwon 16419, South Korea · Center for Quantum Dynamics of Angular Momentum, Pohang University of Science and Technology, Pohang 37673, Korea
In-context learning (ICL) allows a pretrained model to infer a new task from examples supplied in its prompt without updating its parameters. In linear models of ICL, the prediction error develops a double-descent singularity when the number of pretraining samples becomes comparable to the number of learnable parameters. We formulate this interpolation singularity as a critical phenomenon of a quenched disordered system. By comparing annealed and quenched descriptions of the same linear ICL model, we identify the connected sample-to-sample fluctuations of the learned parameters as the microscopic origin of the singular error. A Landau potential is constructed by integrating the cavity self-consistency equation for the renormalized ridge parameter ξ. The role of (magnetization) order parameter is played by ξ, while the bare ridge parameter λ becomes its conjugate magnetic field. The normalized sample complexity τ acts as a temperature and the double-descent singularity occurs at the critical temperature τc=1. The Landau susceptibility is precisely the quantity that diverges in the fluctuation contribution to the prediction error. The order parameter is closely related to the fraction of zero eigenvalues of the empirical relaxation matrix in the ridgeless limit, which define flat directions in the learning dynamics. The Landau theory is generically cubic in the order parameter with critical exponents (βcr,δcr,γcr)=(1,2,1). In the large-context regime, there appears a pseudogap-like regime characterized by suppressed order parameter. Predictions of the Landau theory are independently confirmed from numerical solutions of the original learning problem with good quantitative agreement. Our results pave the way for solid statistical-physics understanding of the interpolation criticality in linear in-context learning.
Figures & tables
Figure 1: Order parameter ξ∗ as a function of τ below the critical temperature τc=1 for several values of κ<1 and γ≪1 . The crossover temperature τ=κ is indicated by hollow circles. The residual order in the crossover region κ<τ<1 diminishes as γ→0 .
Figure 2: Schematic phase diagram for γ≪1 in the (κ,τ) plane. The transition between the ordered phase ξ∗>0 and disordered phase ξ∗=0 occurs at τc=1 (solid line). Above the crossover temperature τ=κ (dashed line), the ordered phase exhibits a small residual order ξ∗=O(γ) characteristic of a pseudogap regime. When we take the γ→0 limit first, the crossover line becomes the exact phase transition. Furthermore, the nonanalytic critical point (NCP) at (κ,τ)=(1,1) develops new critical exponents (βcr,δcr,γcr)=(2,3/2,1) different from the rest of the phase diagram where (βcr,δcr,γcr)=(1,2,1) .
Figure 3: Plot of the order parameter ξ∗ in the (κ,τ) plane for (a) γ=10−2 and (b) γ=10−1 . The solid and dashed white lines are the true critical line τ=1 and crossover line τ=κ , respectively. Other parameters used are d=30 , λ=10−5 , and σ=0 . A small shift ξ∗→ξ∗+10−5 was made to avoid numerical instability of the logarithm at small ξ∗ . Depression of the order parameter in the pseudogap region is highly visible for very small value of γ used in (a). (c) Comparison of the order parameter extracted from Eq. ( 86 ) (hollow marks) with the theoretical curves obtained by solving the self-consistency equation ( 58 ) at κ=0.6 .
Figure 4: Map of the numerically extracted power-law exponents p in the assumed scaling relation ξ∗∝γp . The ordered phase τ<1 separates into two distinct scaling regimes with p≈0 and p≈1 (pseudogap region), divided by the theoretically predicted crossover line τ=κ (dashed line).
Statistical Physics of Computation Laboratory, École Polytechnique Fédérale de Lausanne (EPFL) · University of Chinese Academy of Sciences · Information, Learning and Physics Laboratory, École Polytechnique Fédérale de Lausanne (EPFL)