In-Context Learning

Also known as ICL

Overview

In-context learning (ICL) studies how a model can infer a task, rule, or preference from information in its prompt and apply it to new inputs, making adaptation possible without task-specific weight updates. Common approaches include few-shot demonstrations, retrieval and ordering strategies, synthetic-task pretraining for prior-fitted models, and test-time optimization of prompts or memory; evaluations range from synthetic regression and Markov tasks to language, vision, time-series, graph, and robotics benchmarks. Theory and mechanistic studies examine what algorithms attention and recurrent models implement, and how context length, training-task diversity, and architecture affect generalization. A central tension is that models may encode contextual information yet fail to use it reliably: performance can be sensitive to example quality, order, noise, and conflict with pretrained priors, while longer contexts increase inference costs.

Latest papers

CardsList

429 papers46 papers in the last four weeks, down 26% on the four weeks before. 0.3% of all new papers.

  1. One Skill Takes Over: Clashing Agent Skills and a One-File Fix

    Oct 18, 2026Parsa Razmara, Seyedarmin Azizi, Ardalan AryashadAI Coding AgentsLLM Agent Skill Evaluation

  2. Transformers as Adaptive Kernel Learners across Tasks for In-Context Learning

    Oct 17, 2026Zhaiming Shen, Alexander Cloninger, Rongjie Lai +1Transformer AttentionIn-Context Learning

  3. Out of Support, Not Out of Reach: In-Context Learning with Unsupported Instructions and Composed Tasks

    Oct 16, 2026Xuhan Tong, Yuchen Zeng, Jiawei ZhangIn-Context Learning

  4. Agentic In-Context Learning: What Algorithms Can Transformers Learn in Closed-Loop Interaction?

    Oct 16, 2026Shiguang Wu, Yaqing Wang, Quanming YaoIn-Context LearningIn-Context RL

  5. In-Context Learning Before Specialisation: Theory of Learning in a Solvable Topic Model

    Oct 15, 2026Francesco Cagnetta, Casper L. ChristensenIn-Context LearningLanguage Model Pretraining

  6. From Observation to Intervention: Distilling Self-Generated In-Context Demonstrations via Executable Causal Intervention

    Oct 15, 2026Minming Ye, Min Liu, Haoang Chi +2In-Context LearningIn-Context Example Selection

  7. Provable Transformer Context Compression for In-Context Reinforcement Learning

    Oct 14, 2026Yajie Zhao, Yingjun Mo, Zhiyong Wang +3In-Context RLIn-Context Learning

  8. Test-Time Forecasting: Recurrent Depth Enables In-Context Dynamics Prediction

    Oct 14, 2026Jeffrey B. Lai, John Quinn, Anthony Bao +1Dynamical SystemsIn-Context Learning

  9. From Copying to Composition: How Language Models Develop In-Context Learning

    Oct 13, 2026Minglai Yang, Xinyu Guo, Renxiong Wang +3In-Context LearningCompositional Generalization

  10. RT-J: Large-Scale Pretraining of Relational Transformers for Context-Efficient Predictions

    Oct 13, 2026Rishabh Ranjan, Vignesh Kothapalli, Harshvardhan Agarwal +5Relational ICLRelational Foundation Models

  11. Towards Universal Calibration for In-Context Classification

    Oct 12, 2026Awik Dhar, Sunita Sarawagi, Arpit AgarwalLanguage Model CalibrationPost-Hoc Calibration

  12. In-context classification: Calibration and noise in linear attention with softmax readout

    Oct 12, 2026Arman Rysmakhanov, Mary Letey, Yue M. Lu +2In-Context LearningLinear Attention

  13. Label Collapse in Diffusion Language Model In-Context Learning

    Oct 12, 2026Ziye Guo, Ye YuanIn-Context LearningLLM Evaluation

  14. One-Layer Transformers Provably Learn In-Context K-Nearest Neighbor Prediction with Chain-of-Thought

    Oct 12, 2026Lyumin Wu, Yuan CaoTransformerIn-Context Learning

  15. CB-Agent: Concept-Based In-Context Learning for Sequential Action Selection

    Oct 10, 2026Qingyue Zhang, Huaze Tang, Minyu Zhang +3LLM AgentsIn-Context Learning

  16. Transformer-based In-Context Score Operator Learning for Diffusion Models

    Oct 9, 2026Wuzhe Xu, Zhaiming Shen, Yulong Lu +2Diffusion ModelsIn-Context Learning

  17. In-Context Learning for System Identification: Probabilistic Formulation, Long-Sequence Handling, and Benchmark Validation

    Oct 9, 2026Matteo Rufolo, Dario Piga, Filippo Pura +1System IdentificationIn-Context Learning

  18. PDE-PFN: Prior-Data Fitted Neural PDE Solver

    Oct 9, 2026Jaehyeon Park, Mingu Kang, Dongseok Lee +5Neural PDE SolversPDE Solving

  19. Softmax Attention on Gaussian Mixtures: Linear When It Can, Selective When It Must

    Oct 8, 2026Simon Gabet, Etienne Boursier, Claire BoyerClassificationSoftmax Attention

  20. Imitation Dominates Reinforcement: Direct In-Context RL Is Closer to ICL Than RL

    Oct 8, 2026Minchan Kwon, Seunghee Koh, Sunghyun Baek +2In-Context RLContextual RL

  21. What can linear attention learn from nonlinear teachers in-context?

    Oct 7, 2026Mary Letey, Arman Rysmakhanov, Yue M. Lu +2In-Context LearningLinear Attention

  22. Continual Learning without Continual Training

    Oct 7, 2026Nikita Narayanan, Ritham Majumdarr, Sonali ParbhooContinual LearningClass-Incremental Learning

  23. When to Unpair: Regulating Pairing Dependence in Medical Visual In-Context Learning

    Oct 7, 2026Cheng Wan, Chenjun Li, Qingyu ZhaoIn-Context LearningCurriculum Learning