In-Context Learning (ICL) enables pretrained LLMs to adapt to downstream tasks by conditioning on a small set of input-output demonstrations, without any parameter updates. Although there have been many theoretical efforts to explain how ICL works, most either rely on strong architectural or data assumptions, or fail to capture the impact of key practical factors such as demonstration selection, Chain-of-Thought (CoT) prompting, the number of demonstrations, and prompt templates. We address this gap by establishing a theoretical analysis of ICL under mild assumptions that links these design choices to generalization behavior. We derive an upper bound on the ICL test loss, showing that performance is governed by (i) the quality of selected demonstrations, quantified by Lipschitz constants of the ICL loss along paths connecting test prompts to pretraining samples, (ii) an intrinsic ICL capability of the pretrained model, and (iii) the degree of distribution shift. Within the same framework, we analyze CoT prompting as inducing a task decomposition and show that it is beneficial when demonstrations are well chosen at each substep and the resulting subtasks are easier to learn. Finally, we characterize how ICL performance sensitivity to prompt templates varies with the number of demonstrations. Together, our study shows that pretraining equips the model with the ability to generalize beyond observed tasks, while CoT enables the model to compose simpler subtasks into more complex ones, and demonstrations and instructions enable it to retrieve similar or complex tasks, including those that can be composed into more complex ones, jointly supporting generalization to unseen tasks. All theoretical insights are corroborated by experiments.
Figures & tables
Figure 1: Empirical validation of (a) Theorem 2.3 , (b) Theorem 3.1 , and (c) Theorems 4.3 and 4.7 , respectively.
Task
Demonstration
Qwen-30B-A3B
Qwen-235B-A22B
Sport
Ambiguous
0.164±0.039
0.202±0.037
Identifying
0.559 ± 0.074
0.602 ± 0.043
People
Ambiguous
0.820±0.027
0.648±0.008
Identifying
0.990 ± 0.001
0.845 ± 0.044
Table 1: Test accuracy ( ↑ ) of ICL under ambiguous and identifying demonstrations ( k=8 ). Identifying demonstrations consistently yield higher accuracy across both tasks and models, indicating more precise task identification and better in-context generalization.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 2: Effect of structured distractors introduced during pretraining on CoT performance. Shakespeare less/mid/more denote progressively larger proportions of Shakespearean corpus mixed into the pretraining data. At evaluation time, the model is prompted to solve addition problems using CoT. Despite substantial irrelevant context, the model retains strong CoT performance at inference, indicating an ability to suppress task-irrelevant information.
Demo number
Ambiguous Acc (235B)
Identifying Acc (235B)
Ambiguous Acc (30B)
Identifying Acc (30B)
2
0.265±0.073
0.421±0.080
0.122±0.029
0.176±0.062
4
0.267±0.084
0.542±0.048
0.151±0.078
0.410±0.059
8
0.202±0.037
0.602±0.043
0.164±0.039
0.559±0.074
Appendix
Table 2: Demonstration Experiment. Task: sports, repeated 5 times with different random seeds.
Demo number
Ambiguous Acc (235B)
Identifying Acc (235B)
Ambiguous Acc (30B)
Identifying Acc (30B)
2
0.713±0.074
0.940±0.049
0.735±0.083
0.908±0.059
4
0.708±0.096
0.931±0.027
0.775±0.037
0.950±0.058
8
0.648±0.008
0.845±0.044
0.820±0.027
0.990±0.001
Appendix
Table 3: Demonstration Experiment. Task: People, repeated 5 times with different random seeds.
Figure 3: Posterior confidence under instruction variation. ( Qwen-235B-A22B-Instruct ) (a) Equivalent instructions yield stable, high-confidence predictions. (b) Incorrect but consistent instructions allow confidence to recover as more demonstrations are added. (c) Incorrect and inconsistent instructions lead to persistently low and unstable confidence despite increasing context length.
Figure 4: Posterior confidence under instruction variation. ( Qwen-30B-A3B-Instruct ) (a) Equivalent instructions yield stable, high-confidence predictions. (b) Incorrect but consistent instructions allow confidence to recover as more demonstrations are added. (c) Incorrect and inconsistent instructions lead to persistently low and unstable confidence despite increasing context length.
Figure 5: Discrete gradients of log probability. ( Qwen-235B-A22B-Instruct ) (a) Equivalent instructions yield small and stable gradients. (b) Incorrect but consistent instructions show gradual stabilization.(c) Incorrect and inconsistent instructions produce large and irregular gradients.
Figure 6: Discrete gradients of log probability. ( Qwen-30B-A3B-Instruct ) (a) Equivalent instructions yield small and stable gradients. (b) Incorrect but consistent instructions show gradual stabilization.(c) Incorrect and inconsistent instructions produce large and irregular gradients.
In-Context Learning (ICL) enables pretrained LLMs to adapt to downstream tasks from a small set of input-output demonstrations, without parameter updates. Yet existing theory primarily explains ICL as identifying a latent task already represented in pretraining, leaving unclear whether ICL can succeed when either the instruction or the target task falls outside the pretraining support. We develop a theoretical framework under mild assumptions that separates these two forms of distribution shift. For target tasks represented in pretraining, we show that demonstrations can identify the task and reduce sensitivity to pretrain-supported prompt templates as their number increases. For an unsupported instruction, we show that under a fixed model-induced reference extension, demonstration evidence can dominate an N-independent instruction shift while preserving the exponential identification rate. We then use the same ICL framework as an interpretive lens for long-horizon target tasks that are not represented in pretraining but admit a plan that decomposes them into pretraining-supported subtasks. Applying the supported-task analysis step by step yields a standard accumulated-error bound and clarifies how invoking familiar subtasks along an intermediate-state trajectory can avoid directly selecting the unsupported complete task. Together, our results characterize when demonstrations, instructions, and task decomposition can support ICL beyond the corresponding pretraining supports.
Xuhan Tong, Yuchen Zeng, Jiawei Zhang
Department of Computer Sciences, University of Wisconsin–Madison. · Microsoft Research.
In-context learning (ICL) is an emerging paradigm that employs the semantic information inherent in large language models (LLMs) for generating answers to user queries. While the remarkable performance of ICL has been widely known, a general modeling and a rigorous theoretical analysis of this paradigm are still lacking. This work presents a probabilistic model for ICL and derives the performance of ICL for both general parametric distributions and exponential families. Based on the derived results, the work explains the impact of multiple factors such as the number of demonstrations, the sensitivity of the probabilistic model to the variation of its parameters, as well as the similarity between the demonstrations and the query on the performance of ICL.
Zhenyu Liu, Huaze Tang, Shao-Lun Huang
Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China
While many-shot ICL achieves remarkable performance, prior studies of its scaling behavior have mainly focused on non-reasoning tasks. In this work, we study many-shot ICL on reasoning tasks, with a particular focus on many-shot chain-of-thought in-context learning (CoT-ICL). Analyzing across non-reasoning and reasoning tasks and across non-reasoning and reasoning-oriented LLMs, we identify several distinctive properties of many-shot CoT-ICL. We further interpret these findings by viewing many-shot CoT-ICL as in-context test-time learning rather than scaled pattern matching, and suggest two principles: (i) demonstrations should be easy for the target model to understand, and (ii) they should be ordered to support a smooth conceptual progression. Guided by the principle, we propose Curvilinear Demonstration Selection (CDS), a simple ordering method that yields up to a 5.42 percentage-point gain on a math task with 64 demonstrations. Overall, our results reframe the long context window from a retrieval buffer into a structured curriculum for in-context test-time learning.
Tsz Ting Chung, Lemao Liu, Mo Yu +1
Hong Kong University of Science and Technology · Fudan University · Wechat AI, Tencent.