In-Context Learning

Also known as ICL

In-context learning (ICL) studies how a model can infer a task, rule, or preference from information in its prompt and apply it to new inputs, making adaptation possible without task-specific weight updates. Common approaches include few-shot demonstrations, retrieval and ordering strategies, synthetic-task pretraining for prior-fitted models, and test-time optimization of prompts or memory; evaluations range from synthetic regression and Markov tasks to language, vision, time-series, graph, and robotics benchmarks. Theory and mechanistic studies examine what algorithms attention and recurrent models implement, and how context length, training-task diversity, and architecture affect generalization. A central tension is that models may encode contextual information yet fail to use it reliably: performance can be sensitive to example quality, order, noise, and conflict with pretrained priors, while longer contexts increase inference costs.

Latest papers

404 papers64 papers in the last four weeks, up 52% on the four weeks before. 0.4% of all new papers.

Test-Time Adaptation via Many-Shot Prompting: Benefits, Limits, and Pitfalls
Mar 6, 2026Shubhangi Upasani, Chen Wu, Jay Rainton +4Inference-Time Adaptation of Language Models LLM Prompting
SambaNova Systems, Inc · Stanford University
Non-Parametric Structural Priors for Geometry Theorem Prediction
Mar 5, 2026Junbo Zhao, Ting Zhang, Can Li +3Mathematical Reasoning Automated Theorem Proving
School of Artificial Intelligence, Beijing Normal University, Beijing, China · Engineering Research Center of Intelligent Technology and Educational Application, Ministry of Education, Beijing, China · Beijing Key Laboratory of Artificial Intelligence for Education, Beijing, China +1
Relational In-Context Learning via Synthetic Pre-training with Structural Prior
Mar 4, 2026Yanbo Wang, Jiaxuan You, Chuan Shi +1Relational ICL Relational Foundation Models
Institute for Artificial Intelligence, Peking University · University of Illinois at Urbana-Champaign · Institute of Computing Technology, Beijing University of Post and Telecommunication +1
Understanding the Dynamics of Demonstration Conflict in In-Context Learning
Mar 3, 2026Difan Jiao, Di Wang, Lijie HuIn-Context Learning Implicit Reasoning in Language Models
University of Toronto · King Abdullah University of Science and Technology · Mohamed bin Zayed University of Artificial Intelligence
Fine-Tuning Without Forgetting In-Context Learning: A Theoretical Analysis of Linear Attention Models
Feb 26, 2026Chungpa Lee, Jy-yong Sohn, Kangwook LeeLLM Fine-Tuning In-Context Learning
Yonsei University, Korea · University of Wisconsin–Madison, USA · KRAFTON, Korea +1
Causality $\neq$ Invariance: Function and Concept Vectors in LLMs
Feb 25, 2026Gustaw Opiełka, Hannes Rosenbusch, Claire E. StevensonRepresentation Geometry in Language Models In-Context Learning
University of Amsterdam
In-context Pre-trained Time-Series Foundation Models adapt to Unseen Tasks
Feb 23, 2026Shangqing Xu, Harshavardhan Kamarthi, Haoxin Liu +1Time-Series Foundation Models In-Context Learning
Georgia Institute of Technology, Atlanta, GA, USA
Transformers for dynamical systems learn transfer operators in-context
Feb 21, 2026Anthony Bao, Jeffrey Lai, William GilpinIn-Context Learning Dynamical Systems
Department of Electrical & Computer Engineering, The University of Texas at Austin, Austin, Texas 78712, USA · The Oden Institute, The University of Texas at Austin, Austin, Texas 78712, USA · Department of Physics, The University of Texas at Austin, Austin, Texas 78712, USA
Bayesian Optimality of In-Context Learning with Selective State Spaces
Feb 19, 2026Di Zhang, Jiaqi XingIn-Context Learning Selective SSMs
School of AI and Advanced Computing Xi’an Jiaotong-Liverpool University Suzhou, Jiangsu, China
Bigger Is Safer: Provable Robustness in In-Context Learning Scales with Capacity
Feb 19, 2026Di Zhang, Ningxu Zhang, Zimeng LiuIn-Context Learning Distribution Shift Robustness
School of AI and Advanced Computing Xi’an Jiaotong-Liverpool University
In-Context Learning in Linear vs. Quadratic Attention Models: An Empirical Study on Regression Tasks
Feb 19, 2026Ayush Goel, Arjun Kohli, Sarvagya SomvanshiIn-Context Learning Transformer Attention
University of California, Berkeley
Transforming Behavioral Neuroscience Discovery with In-Context Learning and AI-Enhanced Tensor Methods
Feb 19, 2026Paimon Goulart, Jordan Steinhauser, Dawon Ahn +4AI for Science Tensor Decomposition
UC Riverside Computer Science & Engineering Riverside, CA, USA · UC Riverside Psychology Riverside, CA, USA · UC Riverside Electrical & Computer Engineering Riverside, CA, USA
Improving Interactive In-Context Learning from Natural Language Feedback
Feb 17, 2026Martin Klissarov, Jonathan Cook, Diego Antognini +5In-Context Learning Language Model Self-Improvement
Google DeepMind
Beyond the Prompt in Large Language Models: Comprehension, In-Context Learning, and Chain-of-Thought
Feb 16, 2026Yuling Jiao, Yanming Lai, Huazhen Lin +3LLM Prompting In-Context Learning
School of Artificial Intelligence, Wuhan University · National Center for Applied Mathematics in Hubei, Wuhan University · Hubei Key Laboratory of Computational Science, Wuhan University +6
MAS-on-the-Fly: In-Context Structural Adaptation of LLM-Based Multi-Agent Systems
Feb 14, 2026Guangyi Liu, Haojun Lin, Huan Zeng +2Multi-Agent LLM Systems Multi-Agent Orchestration
Department of Electronic Engineering, Tsinghua University · Ant Group
TabSieve: Explicit In-Table Evidence Selection for Tabular Prediction
Feb 12, 2026Yongyao Wang, Ziqi Miao, Lu Yang +4In-Context Example Selection Reinforcement Learning
Gaoling School of Artificial Intelligence, Renmin University of China · Shanghai AI Laboratory
SHINE: A Scalable In-Context Hypernetwork for Mapping Context to LoRA in a Single Pass
Feb 6, 2026Yewei Liu, Xiyuan Wang, Yansheng Mao +3Knowledge Augmentation for Language Models Low-Rank Adaptation
Institute for Artificial Intelligence, Peking University · School of Electronics Engineering and Computer Science, Peking University · University of Oxford +1
Language Models Struggle to Use Representations Learned In-Context
Feb 4, 2026Michael A. Lepori, Tal Linzen, Ann Yuan +1In-Context Learning LLM World Models
Brown University · Google Research · New York University +1
Language Steering for Multilingual In-Context Learning
Feb 2, 2026Neeraja Kirtane, Kuan-Hao HuangMultilingual Language Models Cross-Lingual Transfer
Texas A&M University
Functional Subspace, where language models can use vector algebra to solve problems
Feb 2, 2026Jung H. Lee, Sujith VijayanLLM Interpretability In-Context Learning
Pacific Northwest National Laboratory Seattle, WA · School of Neuroscience Virginia Tech Blacksburg, VA
Rethinking Zero-Shot Time Series Classification: From Task-specific Classifiers to In-Context Inference
Jan 31, 2026Juntao Fang, Shifeng Xie, Shengbin Nie +7Time Series Classification In-Context Learning
Guangdong University of Technology, Guangzhou, China · Huawei Noah’s Ark Lab, Paris, France · Paris Descartes University, Paris, France +2
In-Context Multi-Operator Learning with DeepOSets
Dec 18, 2025Shao-Ting Chiu, Aditya Nambiar, Ali Syed +2Operator Learning In-Context Learning
Department of Electrical and Computer Engineering, Texas A&M University · Department of Mathematics, Texas A&M University
Equivalence of Context and Parameter Updates in Modern Transformer Blocks
Nov 22, 2025Adrian Goldwaser, Michael Munn, Javier Gonzalvo +1Transformer Interpretability Transformer FFNs
University of Cambridge, UK · Google Research
Predicting the Emergence of Induction Heads in Language Model Pretraining
Nov 21, 2025Tatsuya Aoyama, Ethan Gotlieb Wilcox, Nathan SchneiderLanguage Model Pretraining In-Context Learning
Department of Linguistics, Georgetown University.
T2T-VICL: Cross-Task Visual In-Context Learning via Implicit Text-Driven VLMs
Nov 20, 2025Shao-Jun Xia, Huixin Zhang, Zhengzhong TuVision-Language Models In-Context Learning
Duke University · Texas A&M University
PRISM: Prompt-Refined In-Context System Modelling for Financial Retrieval
Nov 18, 2025Chun Chet Ng, Jia Yu Lim, Zhen Hao Chu +2Information Retrieval LLM Reranking
AI Lens, Kuala Lumpur, Malaysia
In-Context Learning Beyond Copying: A Training-Time Analysis of Abstractive ICL
Nov 7, 2025Kerem Sahin, Sheridan Feucht, Adam Belfki +4In-Context Learning Emergent Abilities in Language Models
Northeastern University · University of Mannheim · Boston University
Chain of Time: In-Context Physical Simulation with Image Generation Models
Oct 30, 2025YingQiao Wang, Eric Bigelow, Boyi Li +1Physics-Based Simulation Visual Reasoning
Department of Psychology, Harvard University · Sakana AI · NTT Research +3
Beyond Semantics: How Temporal Biases Shape Retrieval in Transformer and State-Space Models
Oct 26, 2025Anooshka Bajaj, Deven Mahesh Mistry, Sahaj Singh Maini +2In-Context Learning Induction Heads
Department of Computer Science Indiana University Bloomington
How Does the Pretraining Distribution Shape In-Context Learning? A Fundamental Trade-Off
Oct 1, 2025Waïss Azizian, Ali HasanLanguage Model Pretraining OOD Generalization
Univ. Grenoble Alpes, CNRS, Inria, Grenoble INP, LJK, 38000 Grenoble, France · Work done during an internship at Morgan Stanley Machine Learning Research. · Machine Learning Research, Morgan Stanley, New York, USA

Mar 6, 2026J/K move · Enter open · S save

Test-Time Adaptation via Many-Shot Prompting: Benefits, Limits, and Pitfalls

Shubhangi Upasani, Chen Wu, Jay Rainton, Bo Li+3

SambaNova Systems, Inc · Stanford University

Inference-Time Adaptation of Language Models LLM Prompting In-Context Learning

Open paper AskPDF arXiv

Test-time adaptation enables large language models (LLMs) to modify their behavior at inference without updating model parameters. A common approach is many-shot prompting, where large numbers of in-context learning (ICL) examples are injected as an input-space test-time update. Although performance can improve as more demonstrations are added, the reliability and limits of this update mechanism remain poorly understood, particularly for open-source models. We present an empirical study of many-shot prompting across tasks and model backbones, analyzing how performance varies with update magnitude, example ordering, and selection policy. We further study Dynamic and Reinforced ICL as alternative test-time update strategies that control which information is injected and how it constrains model behavior. We find that many-shot prompting is effective for structured tasks where demonstrations provide high information gain, but is highly sensitive to selection strategy and often shows limited benefits for open-ended generation tasks. Overall, we characterize the practical limits of prompt-based test-time adaptation and outline when input-space updates are beneficial versus harmful.