In-Context Learning
Also known as ICL
Overview
In-context learning (ICL) studies how a model can infer a task, rule, or preference from information in its prompt and apply it to new inputs, making adaptation possible without task-specific weight updates. Common approaches include few-shot demonstrations, retrieval and ordering strategies, synthetic-task pretraining for prior-fitted models, and test-time optimization of prompts or memory; evaluations range from synthetic regression and Markov tasks to language, vision, time-series, graph, and robotics benchmarks. Theory and mechanistic studies examine what algorithms attention and recurrent models implement, and how context length, training-task diversity, and architecture affect generalization. A central tension is that models may encode contextual information yet fail to use it reliably: performance can be sensitive to example quality, order, noise, and conflict with pretrained priors, while longer contexts increase inference costs.
Latest papers
429 papers46 papers in the last four weeks, down 26% on the four weeks before. 0.3% of all new papers.
When Should an In-Context Learner Expand Its Hypothesis Space?
Spatial Induction Heads: In-Context Learning of Multidimensional Cellular Automata
Adaptive Mean Estimation by In-Context Learning: A Gradient-Flow Analysis
Adversarially Trained Linear Transformers Are Optimal Robust In-Context Learners for Gaussian Mixtures
Learning Decision-Stump Thresholds in Context: Dynamics of Softmax Attention
Learning to Learn a Language
RepICL: Reusable In-Context Prediction Across Heterogeneous Representation Spaces
Retrieval-Based In-Context Learning: A Domain Adaptation Framework
Verb-ICL: Rethinking In-Context Learning for Structured Prediction
Generalization of Transformer-Based Neural Quantum States via In-Context Learning
When History Fails to Become Experience: Action Calibration in Language Agents
Pooling Helps, Learned Weighting Hurts In-Context: Decomposing Group Attention
In-context Learning of Single-index Targets: Comparing Kernel and Feature Learners
Skeleton-and-Strategy Prompting: Training-Free Negation Understanding for Vision-Language Models
Capturing In-Context Learning Dynamics with Task Operators
Efficient Task Adaptation in Large Language Models: A Survey of Weight-Based, Prompt-Based, and Embedding-Based Adaptations
Rules Amortize, Pairings Don't: Linguistic Structure Determines What Latent Task Representations Can Replace In-Context Learning
Message Passing Does More with Less for In-Context Learning on Graphs
Theory on Attention Dynamics for Out-of-Distribution In-Context Learning
Support-Set Target Leakage in Relational Foundation Models during In-Context Learning: Model Dependence and Evaluation Reliability
Support-Set Target Leakage in Relational Foundation Models during In-Context Learning: Impact, Detection, and Mitigation
In-Context Learning Amplifies a Latent Symbolic Circuit
In-Context Learning for Robots: Methods and Applications
ALICE: In-context, Zero-shot, Mutual Information Estimation
Unifying Video Tasks via Spatiotemporal Analogy
In-Context Adaptation of Encoder-Decoder Models in Speech Recognition
Learning to Learn from Context: Synthetic Training from Perturbed Public Documents
You Only Edit Once: Incentivizing In-Context Capability of LLMs via Local Demonstration Refinement
Keep, Delete, or Replace elements, all within a framework of reinforcement learning with verifiable rewards. At test time, Jev-LDE executes a single edit of the retrieved demonstration set, followed by one inference from the target LLM, avoiding the need for iterative context scoring or subset searches. Across standard classification benchmarks, various target LLMs with Jev-LDE as the plug-and-play module consistently improve ICL performance, and Jev-LDE shows transferability to held-out benchmarks and models without retraining. These findings indicate that the LDE approach offers an efficient and adaptable method for harnessing the ICL capabilities of target LLMs.