In-Context Learning

Also known as ICL

In-context learning (ICL) studies how a model can infer a task, rule, or preference from information in its prompt and apply it to new inputs, making adaptation possible without task-specific weight updates. Common approaches include few-shot demonstrations, retrieval and ordering strategies, synthetic-task pretraining for prior-fitted models, and test-time optimization of prompts or memory; evaluations range from synthetic regression and Markov tasks to language, vision, time-series, graph, and robotics benchmarks. Theory and mechanistic studies examine what algorithms attention and recurrent models implement, and how context length, training-task diversity, and architecture affect generalization. A central tension is that models may encode contextual information yet fail to use it reliably: performance can be sensitive to example quality, order, noise, and conflict with pretrained priors, while longer contexts increase inference costs.

Latest papers

402 papers64 papers in the last four weeks, up 52% on the four weeks before. 0.4% of all new papers.

Test time training enhances in-context learning of nonlinear functions
Sep 30, 2025Kento Kuwataka, Taiji SuzukiSingle-Index Models Test-Time Training
Department of Mathematical Engineering and Information Physics The University of Tokyo Hongo 7-3-1, Bunkyo-ku, Tokyo
GraphPFN: A Prior-Data Fitted Graph Foundation Model
Sep 25, 2025Dmitry Eremeev, Oleg Platonov, Gleb Bazhenov +2Synthetic Data Pretraining In-Context Learning
1HSE University · 2Yandex Research
Discovering New Theorems via LLMs with In-Context Proof Learning in Lean
Sep 16, 2025Kazumi Kasaura, Naoto Onda, Yuta Oriike +3Automated Theorem Proving In-Context Learning
OMRON SINIC X Corporation · RIKEN AIP · AutoRes +7
ICR-RL: Deep Reinforcement Learning via In-Context Regression
Sep 14, 2025David Schiff, Ofir Lindenbaum, Yonathan EfroniReinforcement Learning In-Context Learning
Bar-Ilan University, Ramat Gan, Israel · Tel Aviv University, Tel Aviv, Israel
LLMs are Bayesian, In Expectation, Not in Realization
Jul 15, 2025Leon Chlon, Fatima Sheaib, Zein Khamis +3Bayesian Inference In-Context Learning
Hassana Labs · Harvard University
Context Tuning for In-Context Optimization
Jul 6, 2025Jack Lu, Ryan Teehan, Zhenbang Yang +1Few-Shot Learning In-Context Learning
Agentic Learning AI Lab, New York University
InComeS: Integrating Compression and Selection Mechanisms into LLMs for Efficient Model Editing
May 28, 2025Shuaiyi Li, Zhisong Zhang, Yang Deng +6In-Context Learning Model Editing
The Chinese University of Hong Kong · City University of Hong Kong · Singapore Management University +1
In-context Learning vs. Instruction Tuning: The Case of Small and Multilingual Language Models
Mar 3, 2025David Ponce, Thierry EtchegoyhenMultilingual Language Models Multilingual Language Model Evaluation
Vicomtech Foundation, Basque Research and Technology Alliance (BRTA) · University of the Basque Country - EHU
From Markov to Laplace: How Mamba In-Context Learns Markov Chains
Feb 14, 2025Marco Bondaschi, Nived Rajaraman, Xiuying Wei +5Mamba Selective SSMs
EPFL · UC Berkeley · Google DeepMind +1
LLM Program Optimization via Retrieval Augmented Search
Jan 31, 2025Sagnik Anupam, Alexander Shypula, Osbert BastaniLarge Language Model-Guided Optimization In-Context Learning
University of Pennsylvania
Learning in the Recurrent State: Gradient Descent with Linear Recurrent Networks
Oct 15, 2024Yudou Tian, Neeraj Mohan Sushma, Harshvardhan Mestha +3In-Context Learning Linear RNNs
Center for Cognitive Interaction Technology CITEC, Universität Bielefeld, Germany · Department of Electrical and Electronics Engineering, Birla Institute of Technology and Science Pilani, India · Department of Computer Science, Royal Holloway, University of London, United Kingdom
From system models to class models: An in-context learning paradigm
Aug 25, 2023Marco Forgione, Filippo Pura, Dario PigaMeta-Learning In-Context Learning
IDSIA Dalle Molle Institute for Artificial Intelligence USI-SUPSI, Via la Santa 1, CH-6962 Lugano-Viganello, Switzerland.

Sep 30, 2025J/K move · Enter open · S save

Test time training enhances in-context learning of nonlinear functions

Kento Kuwataka, Taiji Suzuki

Department of Mathematical Engineering and Information Physics The University of Tokyo Hongo 7-3-1, Bunkyo-ku, Tokyo

Single-Index Models Test-Time Training In-Context Learning Statistical Learning Theory

Open paper AskPDF arXiv

Test-time training (TTT) enhances model performance by explicitly updating designated parameters prior to each prediction to adapt to the test data. While TTT has demonstrated considerable empirical success, its theoretical underpinnings remain limited, particularly for nonlinear models. In this paper, we investigate the combination of TTT with in-context learning (ICL), where the model is given a few examples from the target distribution at inference time. We analyze this framework in the setting of single-index models, where the feature vector is drawn from a hidden low-dimensional subspace. For single-layer transformers trained with gradient-based algorithms and adopting TTT, we establish an upper bound on the prediction risk. Our theory reveals that TTT enables the single-layer transformers to adapt to both the feature vector and the link function, which vary across tasks. This creates a sharp contrast with ICL alone, which is theoretically difficult to adapt to shifts in the link function. Moreover, we provide the convergence rate with respect to the data length, showing the predictive error can be driven arbitrarily close to the noise level as the context size and the network width grow.