cs.LGNov 10, 2026

You Don't Need Feature Learning to Do In-Context Learning

Authors: Joseph Turnbull, Jamie Simon

Organizations: UC Berkeley · Imbue

Abstract

How should we understand memorization vs. generalization in in-context learning (ICL) tasks, and when does one happen vs. the other? To investigate, we study how rotation-invariant kernel machines learn the ICL linear regression task proposed by Raventos et al. (2023). By decomposing the task in the kernel eigenbasis, we obtain a spectral picture of learning in this ICL setting. This picture reveals that as sample size grows, kernel machines learn successively-higher-degree components of the memorizing solution, which depend upon successively-higher-degree statistics of the training task set. The generalizing solution is approximately a low-degree truncation of the memorizing solution, and so the model first generalizes, then degrades into memorization as training proceeds. The takeaways are threefold. First, if this case is indicative, learning a "generalizing" solution on an ICL task is really just underfitting the memorizing solution! Second, the fact that kernel machines can generalize on this task means that feature learning is inessential to ICL. Third, transformers empirically trained to do ICL are shown to memorize tasks through the higher-order statistics of the training tasks that the kernel-based theory predicts, suggesting it applies even to models capable of feature learning.

Figures & tables

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. In-context Learning of Single-index Targets: Comparing Kernel and Feature Learners

    Oct 1, 2026Haotian Gu, Yizhou Xu, Lenka ZdeborováRepresentation LearningSingle-Index Models

  2. The Effect of Training Task Diversity on In-Context Learning through the Lens of Low-Dimensional Subspaces

    Jun 5, 2026Soo Min Kwon, Alec S. Xu, Can Yaras +3OOD GeneralizationTransformer Attention

  3. Transformers as Adaptive Kernel Learners across Tasks for In-Context Learning

    Oct 17, 2026Zhaiming Shen, Alexander Cloninger, Rongjie Lai +1Transformer AttentionIn-Context Learning