In-Context Learning

Also known as ICL

In-context learning (ICL) studies how a model can infer a task, rule, or preference from information in its prompt and apply it to new inputs, making adaptation possible without task-specific weight updates. Common approaches include few-shot demonstrations, retrieval and ordering strategies, synthetic-task pretraining for prior-fitted models, and test-time optimization of prompts or memory; evaluations range from synthetic regression and Markov tasks to language, vision, time-series, graph, and robotics benchmarks. Theory and mechanistic studies examine what algorithms attention and recurrent models implement, and how context length, training-task diversity, and architecture affect generalization. A central tension is that models may encode contextual information yet fail to use it reliably: performance can be sensitive to example quality, order, noise, and conflict with pretrained priors, while longer contexts increase inference costs.

Latest papers

404 papers64 papers in the last four weeks, up 52% on the four weeks before. 0.4% of all new papers.

In-Context Fixation: When Demonstrated Labels Override Semantics in Few-Shot Classification
May 8, 2026Ming LiuLLM Interpretability Few-Shot Learning
Amazon
GRaSp: Automatic Example Optimization for In-Context Learning in Low-Data Tasks
May 8, 2026Simen Bihaug-Frøyland, Henrik BrådlandIn-Context Example Selection In-Context Learning
Centre for Artificial Intelligence Research, University of Agder · School of Computing and Information, University of Pittsburgh · Norkart AS
Mitigating Many-shot Jailbreak Attacks with One Single Demonstration
May 8, 2026Kejia Chen, Jiawen Zhang, Boheng Li +6Jailbreak Attacks In-Context Learning
Zhejiang University · Nanyang Technological University · University of Science and Technology of China +2
Convergence and Emergence of In-Context Reinforcement Learning with Chain of Thought
May 8, 2026Zixuan Xie, Xinyu Liu, Rohan Chandra +1Reinforcement Learning Transformer
University of Virginia
Transformers Efficiently Perform In-Context Logistic Regression via Normalized Gradient Descent
May 7, 2026Chenyang Zhang, Yuan CaoLogistic Regression Self-Attention
School of Computing & Data Science, The University of Hong Kong
In-Context Black-Box Optimization with Unreliable Feedback
May 7, 2026Nicolas Samuel Blumer, Julien Martinelli, Samuel KaskiBayesian Optimization Black-Box Optimization
ELLIS Institute Finland · Aalto University · University of Manchester
Understanding In-Context Learning for Nonlinear Regression with Transformers: Attention as Featurizer
May 6, 2026Alexander Hsu, Zhaiming Shen, Wenjing Liao +1Transformer Attention In-Context Learning
Department of Mathematics, Purdue University · School of Mathematics, Georgia Institute of Technology
Adaptivity Under Realizability Constraints: Comparing In-Context and Agentic Learning
May 6, 2026Anastasis Kratsios, A. Martina Neuman, Philipp PetersenReLU Neural Networks In-Context Learning
McMaster University and the Vector Institute, Main St., Hamilton, L8S 4L8, Ontario, Canada · University of Vienna, Faculty of Mathematics, Kolingasse 14-16, 1090 Wien, Austria
LLMs with in-context learning for Algorithmic Theoretical Physics
May 6, 2026Anamaria Hell, Leander ThieleIn-Context Learning LLM Tool Use
Kavli IPMU (WPI), UTIAS, The University of Tokyo, and Center for Data-Driven Discovery, 5-1-5 Kashiwanoha, Kashiwa, Chiba 277-8583, Japan
LIVEditor-14B: Lightning Unified Video Editing via In-Context Sparse Attention
May 6, 2026Shitong Shao, Zikai Zhou, Haopeng Li +4Video Editing In-Context Learning
Hong Kong University of Science and Technology (GZ), Hong Kong · University of Arizona, USA
Task Vector Geometry Underlies Dual Modes of Task Inference in Transformers
May 5, 2026Hao Yan, Haolin Yang, Yiqiao ZhongTransformer Interpretability Neural Representation Geometry
University of Wisconsin–Madison · University of Chicago
AdapShot: Adaptive Many-Shot In-Context Learning with Semantic-Aware KV Cache Reuse
May 5, 2026Jie Ou, Jinyu Guo, Shiyao Guo +5Long-Context Language Model Inference KV-Cache Management
School of Information and Software Engineering, University of Electronic Science and Technology of China
Membership Inference Attacks for Retrieval Based In-Context Learning for Document Question Answering
May 5, 2026Tejas Kulkarni, Antti Koskela, Laith ZumotIn-Context Retrieval Membership Inference Attacks
Nokia Bell Labs · Nokia
Enhancing Multimodal In-Context Learning via Inductive-Deductive Reasoning
May 4, 2026Haoyu Wang, Haonan Wang, Yuyan Chen +5Multimodal ICL In-Context Learning
Tencent QQ · Fudan University · Cornell University
Embedding-based In-Context Prompt Training for Enhancing LLMs as Text Encoders
May 2, 2026Ailiang Lin, Zhuoyun Li, Keyu Mao +2Text Embeddings In-Context Learning
Institute of Science Tokyo · Tencent
From Context to Skills: Can Language Models Learn from Context Skillfully?
Apr 30, 2026Shuzheng Si, Haozhe Zhao, Yu Lei +10In-Context Learning Agent Skill Learning
THU · UIUC · DeepLang AI +2
In-Context Examples Suppress Scientific Knowledge Recall in LLMs
Apr 30, 2026Chaemin Jang, Woojin Park, Hyeok Yun +2Knowledge Conflicts in Language Models Scientific Reasoning
Korea Advanced Institute of Science and Technology · Shanghai Jiao Tong University
CL-bench Life: Can Language Models Learn from Real-Life Context?
Apr 29, 2026Shihan Dou, Yujiong Shen, Chenhao Huang +35LLM Evaluation LLM Grounding
Hunyuan Team, Tencent · Fudan University
QCalEval: Benchmarking Vision-Language Models for Quantum Calibration Plot Understanding
Apr 28, 2026Shuxiang Cao, Zijian Zhang, Abhishek Agarwal +29VLM Evaluation In-Context Learning
1NVIDIA · University of Toronto · 13Vector Institute for Artificial Intelligence +10
Investigation into In-Context Learning Capabilities of Transformers
Apr 28, 2026Rushil Chandrupatla, Leo Bangayan, Sebastian LengNeural Network Generalization Transformer
Quantifying Aleatoric Uncertainty of In-Context Learning for Robust Measure of LLM Prediction Confidence
Apr 28, 2026Jinseok Chung, Minkyoung Song, Hyunji Jung +1Uncertainty Quantification In-Context Learning
POSTECH
What Makes Good Instruction-Tuning Data? An In-Context Learning Perspective
Apr 28, 2026Guangzeng Han, Xiaolei HuangInstruction-Tuning Data Selection Training Data Selection
University of Memphis
Where to Place the Query? Unveiling and Mitigating Positional Bias in In-Context Learning for Diffusion LLMs via Decoding Dynamics
Apr 26, 2026Zhengheng Li, Panrui Li, Xuyang Liu +1In-Context Learning Diffusion Language Model Decoding
Southeast University
When Context Sticks: Studying Interference in In-Context Learning
Apr 25, 2026Hanna Rød, Dagny Streit, Nils Valseth Selte +1In-Context Learning
Fine-tuning vs. In-context Learning in Large Language Models: A Formal Language Learning Perspective
Apr 25, 2026Bishwamittra Ghosh, Soumi Das, Till Speicher +5LLM Evaluation LLM Fine-Tuning
Max Planck Institute for Software Systems, Germany · Boston University, USA
Adaptive Test-Time Compute Allocation with Evolving In-Context Demonstrations
Apr 22, 2026Bowen Zuo, Dongruo Zhou, Yinglun ZhuEfficient Inference Test-Time Optimization
University of California, Riverside · 2Indiana University Bloomington
Diagnosing CFG Interpretation in LLMs
Apr 22, 2026Hanqi Li, Lu Chen, Kai YuLLM Evaluation In-Context Learning
1X-LANCE Lab, School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · 3Shanghai Innovation Institution, Shanghai, China · 4Jiangsu Key Lab of Language Computing, Suzhou, China +2
The Structural Attention Tax: How Retrieval Format Hijacks In-Context Learning Independent of Content
Apr 21, 2026Yuqi Zhang, Di ZhangRetrieval-Augmented Generation Self-Attention
Xi’an Jiaotong-Liverpool University
Learning Posterior Predictive Distributions for Node Classification from Synthetic Graph Priors
Apr 21, 2026Jeongwhan Choi, Jongwoo Kim, Woosung Kang +1Synthetic Data Pretraining OOD Generalization
KAIST Daejeon, Republic of Korea
Syntax as a Rosetta Stone: Universal Dependencies for In-Context Coptic Translation
Apr 20, 2026Abhishek Purushothama, Emma Thronson, Alexia Guo +1In-Context Learning Low-Resource MT
Corpling Lab Georgetown University

May 8, 2026J/K move · Enter open · S save

In-Context Fixation: When Demonstrated Labels Override Semantics in Few-Shot Classification

Ming Liu

Amazon

LLM Interpretability Few-Shot Learning In-Context Learning Classification

Open paper AskPDF arXiv

While random demonstration labels barely hurt in-context learning (Min et al., 2022), we show that homogeneous labels--even semantically valid ones--collapse accuracy to <=12% across six models (Pythia, Llama, Qwen; 0.8B--8B) and four tasks. The trigger is label-slot content: the model treats tokens occupying the label position as an exhaustive answer vocabulary, with homogeneity as the maximally collapsed case. A novel set-level fixation finding confirms this: when demonstrations carry varied nonsense tokens from {foo,bar,vex,nit,orb}, the model places 42--67% of probability on the demonstrated set while P(dog) remains below 0.2%. This is inconsistent with latent-concept Bayesian accounts (Xie et al., 2022) and reveals that ICL output is constrained vocabulary retrieval--the model binds its output to the demonstrated token inventory regardless of semantic plausibility. The effect generalizes to 4-way classification (0% accuracy across three models, 1B--8B) and multi-token verbalizers ("very positive"), where we decompose fixation into format-level (template adoption) and content-level (polarity override) components that are experimentally dissociable. Mechanistically, per-item paired activation patching on Pythia-1B recovers 98.4% of the gap (95% CI [84%, 112%]), localizing fixation to a layer-7-centered circuit (rank 2/560, 99.8th percentile; 4-fold CV mean 103%). Cross-architecture logit lens on Llama-3.2-1B replicates the encode-then-override trajectory with causal confirmation (top-5 layers: 89% recovery).