Landau theory of quenched criticality in linear in-context learning
Authors: Daesik Kim, Sumin Choi, Hyojae Jeon, Jung Hoon Han
Organizations: Department of Physics, Sungkyunkwan University, Suwon 16419, South Korea · Center for Quantum Dynamics of Angular Momentum, Pohang University of Science and Technology, Pohang 37673, Korea
In-context learning (ICL) allows a pretrained model to infer a new task from examples supplied in its prompt without updating its parameters. In linear models of ICL, the prediction error develops a double-descent singularity when the number of pretraining samples becomes comparable to the number of learnable parameters. We formulate this interpolation singularity as a critical phenomenon of a quenched disordered system. By comparing annealed and quenched descriptions of the same linear ICL model, we identify the connected sample-to-sample fluctuations of the learned parameters as the microscopic origin of the singular error. A Landau potential is constructed by integrating the cavity self-consistency equation for the renormalized ridge parameter ξ. The role of (magnetization) order parameter is played by ξ, while the bare ridge parameter λ becomes its conjugate magnetic field. The normalized sample complexity τ acts as a temperature and the double-descent singularity occurs at the critical temperature τc=1. The Landau susceptibility is precisely the quantity that diverges in the fluctuation contribution to the prediction error. The order parameter is closely related to the fraction of zero eigenvalues of the empirical relaxation matrix in the ridgeless limit, which define flat directions in the learning dynamics. The Landau theory is generically cubic in the order parameter with critical exponents (βcr,δcr,γcr)=(1,2,1). In the large-context regime, there appears a pseudogap-like regime characterized by suppressed order parameter. Predictions of the Landau theory are independently confirmed from numerical solutions of the original learning problem with good quantitative agreement. Our results pave the way for solid statistical-physics understanding of the interpolation criticality in linear in-context learning.
Figures & tables
Figure 1: Order parameter ξ∗ as a function of τ below the critical temperature τc=1 for several values of κ<1 and γ≪1 . The crossover temperature τ=κ is indicated by hollow circles. The residual order in the crossover region κ<τ<1 diminishes as γ→0 .
Figure 2: Schematic phase diagram for γ≪1 in the (κ,τ) plane. The transition between the ordered phase ξ∗>0 and disordered phase ξ∗=0 occurs at τc=1 (solid line). Above the crossover temperature τ=κ (dashed line), the ordered phase exhibits a small residual order ξ∗=O(γ) characteristic of a pseudogap regime. When we take the γ→0 limit first, the crossover line becomes the exact phase transition. Furthermore, the nonanalytic critical point (NCP) at (κ,τ)=(1,1) develops new critical exponents (βcr,δcr,γcr)=(2,3/2,1) different from the rest of the phase diagram where (βcr,δcr,γcr)=(1,2,1) .
Figure 3: Plot of the order parameter ξ∗ in the (κ,τ) plane for (a) γ=10−2 and (b) γ=10−1 . The solid and dashed white lines are the true critical line τ=1 and crossover line τ=κ , respectively. Other parameters used are d=30 , λ=10−5 , and σ=0 . A small shift ξ∗→ξ∗+10−5 was made to avoid numerical instability of the logarithm at small ξ∗ . Depression of the order parameter in the pseudogap region is highly visible for very small value of γ used in (a). (c) Comparison of the order parameter extracted from Eq. ( 86 ) (hollow marks) with the theoretical curves obtained by solving the self-consistency equation ( 58 ) at κ=0.6 .
Figure 4: Map of the numerically extracted power-law exponents p in the assumed scaling relation ξ∗∝γp . The ordered phase τ<1 separates into two distinct scaling regimes with p≈0 and p≈1 (pseudogap region), divided by the theoretically predicted crossover line τ=κ (dashed line).
In-context learning (ICL) enables a pretrained model to infer a task from demonstrations without updating its parameters. While much of the existing theory focuses on linear target functions, in this paper we study nonlinear cases by comparing two one-layer attention architectures on the same family of single-index tasks. A kernel learner first maps inputs through a fixed nonlinear feature map and then applies linear attention, whereas a feature learner applies attention to the original input, followed by a learned nonlinear readout. We derive predictions for their memorization and generalization errors using the replica method, retaining the effects of pretraining size, task-pool diversity, and training and inference context lengths. The resulting predictions closely match numerical experiments across a broad range of regimes. Our analysis yields phase diagrams that characterize when each architecture is advantageous as the amount of pretraining data, task diversity, and context lengths vary. We further identify qualitatively different context-length scalings for the two learners. Together, these results clarify how architectural choices interact with the dataset and govern nonlinear in-context learning.
Haotian Gu, Yizhou Xu, Lenka Zdeborová
Statistical Physics of Computation Laboratory, École Polytechnique Fédérale de Lausanne (EPFL) · University of Chinese Academy of Sciences · Information, Learning and Physics Laboratory, École Polytechnique Fédérale de Lausanne (EPFL)
This paper investigates context stickiness in in-context learning (ICL), a phenomenon where earlier examples in a prompt interfere with a transformer's ability to adapt to later tasks. Using synthetic regression tasks over linear and quadratic functions, we examine how models trained under sequential, mixed, and random curricula handle abrupt task switches during inference. By sweeping over structured combinations of misleading linear examples followed by recovery quadratic examples, we quantify how prior context biases prediction error and how quickly models realign. Our results show strong evidence of persistent interference: more preceding linear examples reliably degrade quadratic predictions, while additional quadratic examples reduce error but with diminishing returns. We further find that training curricula significantly modulate resilience, with sequential training on the target function class yielding the fastest recovery, and surprisingly, random training producing the least robust behavior.
Invariant-learning objectives pursue similar goals yet produce qualitatively different regularization paths, leaving unclear when shortcuts can be suppressed without damaging stable modes. Our starting point is simple: in the desired shortcut-suppressed regime, shortcut loading is small, so the objective's low-order expansion governs local stability and residual amplitude. This brings the problem into the domain of Landau phase-transition theory. We establish a mathematical isomorphism between the near-critical normal form of predictive-mode learning and Landau theory, identifying the learning objective as an effective free energy and critical-mode amplitude as an order parameter. The resulting low-order objective signatures predict distinct regularization phenotypes: quadratic R2 terms shift phase boundaries and enable finite-strength elimination, whereas quartic R4 terms continuously attenuate acquired modes while leaving nonzero residual loading at finite strengths. Higher-order terms may further shape nonlinear tails. In a canonical bilinear model, we derive exact phase boundaries and equilibrium loadings, identifying conditions for a selective-retention window in which shortcut suppression preserves stable structure. Controlled bilinear and ReLU experiments, including an MNIST construction, support the predicted signature-phenotype relation, while coupled-feature experiments validate its extension to collective modes. The framework connects the mathematical structure of invariant-learning objectives to what models learn as regularization varies.
Pinli Wang, Yue He, Peng Cui
Department of Physics, Tsinghua University, Beijing, China · Department of Computer Science and Technology, Tsinghua University, Beijing, China