In-Context Learning

Also known as ICL

In-context learning (ICL) studies how a model can infer a task, rule, or preference from information in its prompt and apply it to new inputs, making adaptation possible without task-specific weight updates. Common approaches include few-shot demonstrations, retrieval and ordering strategies, synthetic-task pretraining for prior-fitted models, and test-time optimization of prompts or memory; evaluations range from synthetic regression and Markov tasks to language, vision, time-series, graph, and robotics benchmarks. Theory and mechanistic studies examine what algorithms attention and recurrent models implement, and how context length, training-task diversity, and architecture affect generalization. A central tension is that models may encode contextual information yet fail to use it reliably: performance can be sensitive to example quality, order, noise, and conflict with pretrained priors, while longer contexts increase inference costs.

Latest papers

402 papers64 papers in the last four weeks, up 52% on the four weeks before. 0.4% of all new papers.

An Algorithmic Account of Few-Shot Concept Induction in Language Models
Oct 26, 2026Michael A. Lepori, Laila Johnston, Aalok Sathe +3Inductive Reasoning in Language Models In-Context Learning
Dept. of Computer Science, Brown University, Providence, Rhode Island · Dept. of Cognitive and Psychological Sciences, Brown University, Providence, Rhode Island
World-in-Context: Interactive Video World Modeling with In-Context Conditioning
Oct 23, 2026Harold Haodong Chen, Rui Peng, Rongjin Guo +4Interactive World Models Camera-Controlled Video Generation
Alibaba Token Hub, Alibaba Group
From Noisy Retrieval to Decision Evidence: Silver Evidence Demonstrations for In-Context Learning
Oct 23, 2026Masoud Monajatipoor, Fazlolah Mohaghegh, Hamid Reza Hassanzadeh +2Retrieval-Augmented ICL In-Context Learning
Optum AI
POS-Signature Demonstration Retrieval for In-Context Dependency Parsing
Oct 22, 2026Muhammad Azam Afridi, Rajab Ali, Marco PassarottiIn-Context Example Selection Dependency Parsing
Università Cattolica del Sacro Cuore, Milan, Italy · University of Zaragoza, Zaragoza, Spain · Universidade NOVA de Lisboa, Lisbon, Portugal +1
How Depth Growth and Looping Shape In-Context Learning in Language Models
Oct 22, 2026Ferdinand Kapl, Kaitlin Maile, Stefan Bauer +1In-Context Learning Parameter Sharing
Technical University of Munich · Helmholtz AI, Munich · Munich Center for Machine Learning (MCML) +1
Transformers with CoT Discover Heavy-Ball Acceleration Through Training
Oct 22, 2026Yuxuan Cai, Yuan CaoCoT Reasoning Neural Algorithmic Reasoning
School of Computing & Data Science, The University of Hong Kong
Watch and Learn: Teaching Visual In-Context Learning by Removing Textual Shortcuts
Oct 22, 2026Nimrod Shabtay, Asaf Yehudai, Avihu Dekel +2Multimodal ICL In-Context Learning
IBM Research · Tel-Aviv University
Collaborative In-Context Learning for Indoor Radio Map Estimation
Oct 21, 2026Yuanzhe Peng, Jie XuIn-Context Learning
University of North Florida, Jacksonville, FL, USA · University of Florida, Gainesville, FL, USA
When Thoughts Become Memory: Unifying In-Context Learning and Fine-Tuning
Oct 21, 2026Leon Chlon, Julie HuangIn-Context Learning Data Valuation
Hassana Labs.
Cognitively Decoupled Meta-Induction for Out-of-Distribution Generalization
Oct 19, 2026Yuqian Wang, Yuanzheng Wang, Chaojun Xiao +7Inductive Reasoning in Language Models OOD Generalization
Beijing Institute of Technology · University of Chinese Academy of Sciences · Department of Computer Science and Technology, Tsinghua University +4
One Skill Takes Over: Clashing Agent Skills and a One-File Fix
Oct 18, 2026Parsa Razmara, Seyedarmin Azizi, Ardalan AryashadAI Coding Agents LLM Agent Skill Evaluation
University of Southern California, Los Angeles, USA
Training with Ciphers to Enhance In-Context Terminology Learning For Translation from Rare Languages
Oct 17, 2026Deevyanshu Malu, Deeptanshu Malu, Sunita SarawagiMachine Translation In-Context Learning
IIT Bombay
Transformers as Adaptive Kernel Learners across Tasks for In-Context Learning
Oct 17, 2026Zhaiming Shen, Alexander Cloninger, Rongjie Lai +1Transformer Attention In-Context Learning
Department of Mathematics and Statistics, Utah State University, Logan, UT 84322 · Department of Mathematics, Purdue University, West Lafayette, IN 47907 · School of Mathematics, Georgia Institute of Technology, Atlanta, GA 30332
Out of Support, Not Out of Reach: In-Context Learning with Unsupported Instructions and Composed Tasks
Oct 16, 2026Xuhan Tong, Yuchen Zeng, Jiawei ZhangIn-Context Learning
Department of Computer Sciences, University of Wisconsin–Madison. · Microsoft Research.
Agentic In-Context Learning: What Algorithms Can Transformers Learn in Closed-Loop Interaction?
Oct 16, 2026Shiguang Wu, Yaqing Wang, Quanming YaoIn-Context Learning In-Context RL
Tsinghua University · Beijing Institute of Mathematical Sciences and Applications
In-Context Learning Before Specialisation: Theory of Learning in a Solvable Topic Model
Oct 15, 2026Francesco Cagnetta, Casper L. ChristensenIn-Context Learning Language Model Pretraining
Simplex · Simplex, Astera Institute
In-Context Preference-Conditioned Pareto Set Learning from Limited Offline Data
Oct 15, 2026Yiming Yao, Liang Zhao, Qingfu ZhangMulti-Objective Optimization Pareto Optimization
City University of Hong Kong
From Observation to Intervention: Distilling Self-Generated In-Context Demonstrations via Executable Causal Intervention
Oct 15, 2026Minming Ye, Min Liu, Haoang Chi +2In-Context Learning In-Context Example Selection
Shanghai Jiaotong University · Academy of Military Science · National University of Defense Technology
Provable Transformer Context Compression for In-Context Reinforcement Learning
Oct 14, 2026Yajie Zhao, Yingjun Mo, Zhiyong Wang +3In-Context RL In-Context Learning
Southern University of Science and Technology · University of Electronic Science and Technology of China · Harbin Institute of Technology
Test-Time Forecasting: Recurrent Depth Enables In-Context Dynamics Prediction
Oct 14, 2026Jeffrey B. Lai, John Quinn, Anthony Bao +1Dynamical Systems In-Context Learning
Oden Institute University of Texas at Austin · Department of Physics University of Texas at Austin
From Copying to Composition: How Language Models Develop In-Context Learning
Oct 13, 2026Minglai Yang, Xinyu Guo, Renxiong Wang +3In-Context Learning Compositional Generalization
Scale AI · University of Arizona
From Relation-Specific Heads to Shared Interventions: Contextual Entrainment in Language Models
Oct 13, 2026Jinghan Zheng, Jiajun ShiCircuits in Language Models Causal Interventions in Language Models
Dalian University of Technology
RT-J: Large-Scale Pretraining of Relational Transformers for Context-Efficient Predictions
Oct 13, 2026Rishabh Ranjan, Vignesh Kothapalli, Harshvardhan Agarwal +5Relational ICL Relational Foundation Models
Stanford University · SAP
Foundation Models for Event Prediction Across Heterogeneous Event Logs
Oct 13, 2026Fabian Spaeh, Sean Jaffe, Shandian Zhe +1Next-Event Prediction Predictive Process Monitoring
Celonis
Towards Universal Calibration for In-Context Classification
Oct 12, 2026Awik Dhar, Sunita Sarawagi, Arpit AgarwalLanguage Model Calibration Post-Hoc Calibration
Indian Institute of Technology Bombay
In-context classification: Calibration and noise in linear attention with softmax readout
Oct 12, 2026Arman Rysmakhanov, Mary Letey, Yue M. Lu +2In-Context Learning Linear Attention
Applied Mathematics, Harvard University · Williams College · Kempner Institute, Harvard University +1
The Value of Being Wrong: Self-Mined Visual In-Context Learning from Errors
Oct 12, 2026Shu Zhao, Tan YuIn-Context Learning Test-Time Adaptation of VLMs
NVIDIA
Label Collapse in Diffusion Language Model In-Context Learning
Oct 12, 2026Ziye Guo, Ye YuanIn-Context Learning LLM Evaluation
University of Connecticut · 2McGill University
One-Layer Transformers Provably Learn In-Context K-Nearest Neighbor Prediction with Chain-of-Thought
Oct 12, 2026Lyumin Wu, Yuan CaoTransformer In-Context Learning
School of Computing & Data Science, The University of Hong Kong
CB-Agent: Concept-Based In-Context Learning for Sequential Action Selection
Oct 10, 2026Qingyue Zhang, Huaze Tang, Minyu Zhang +3LLM Agents In-Context Learning
Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China

Oct 26, 2026J/K move · Enter open · S save

An Algorithmic Account of Few-Shot Concept Induction in Language Models

Michael A. Lepori, Laila Johnston, Aalok Sathe, Zhuonan Yang+2

Dept. of Computer Science, Brown University, Providence, Rhode Island · Dept. of Cognitive and Psychological Sciences, Brown University, Providence, Rhode Island

Inductive Reasoning in Language Models In-Context Learning Few-Shot Learning

Open paper AskPDF arXiv

Both models and minds not only encode features that describe an entity, but can specifically represent which features of that entity are relevant for a given context. This core capability underlies in-context learning in language models and a variety of cognitive phenomena in minds, yet it is unclear how intelligent systems identify and promote relevant features. In this work, we propose Amortized Direction Accentuation (ADA), an algorithmic hypothesis that provides an explanation for how relevance is determined in the context of few-shot concept induction --- the ability to infer a latent concept from a small number of positive examples. Under ADA, concept induction proceeds by forming a summary representation of the examples in a feature space that is shaped by amortized knowledge about which features tend to be useful for finding commonalities between items. We investigate and validate ADA in language models using a few-shot concept induction task. This work provides a novel perspective on the mechanisms supporting contextualization in language models and a promising hypothesis for a core aspect of cognition.