Test-time adaptation enables large language models (LLMs) to modify their behavior at inference without updating model parameters. A common approach is many-shot prompting, where large numbers of in-context learning (ICL) examples are injected as an input-space test-time update. Although performance can improve as more demonstrations are added, the reliability and limits of this update mechanism remain poorly understood, particularly for open-source models. We present an empirical study of many-shot prompting across tasks and model backbones, analyzing how performance varies with update magnitude, example ordering, and selection policy. We further study Dynamic and Reinforced ICL as alternative test-time update strategies that control which information is injected and how it constrains model behavior. We find that many-shot prompting is effective for structured tasks where demonstrations provide high information gain, but is highly sensitive to selection strategy and often shows limited benefits for open-ended generation tasks. Overall, we characterize the practical limits of prompt-based test-time adaptation and outline when input-space updates are beneficial versus harmful.
Figures & tables
Figure 1: A unified view of prompt-based test-time adaptation. Update design determines whether added context provides signal or noise.
Figure 2: Scaling behavior of prompt-based test-time updates. (a) Many-shot accuracy vs. update magnitude (b) Dynamic ICL update policies (c) Model capacity effects (d) Reinforced ICL scaling
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
n per-class shots
N Total demonstrations
1
77
2
154
3
231
4
308
5
385
50
3850
Appendix
Table 1: Comparison of per-class shots with total demonstrations in input prompt for Banking 77
Figure 3: Dynamic ICL Selection Strategies on Banking 77
Per-class shots n
LLaMA-3.1-8B
LLaMA-3.1-8B-Instruct
50
33.9
80.8
70
35.9
82.3
Appendix
Table 2: Many-shot prompting for Llama-3.1-8B on Banking 77 (accuracy in %). Prompts contain N total demonstrations, constructed using n shots per class ( N=n×C , C=77 ).
Generator Prompt
Generator agent for creating step-by-step reasoning. Rules:
– Use at most 10 steps.
– No explanations.
– No restating the question.
– End immediately after the answer.
– Do not reference the order of the answers in the question (e.g., option a, b, c).
Appendix
Table 3: Prompt template used for the Generator agent during data generation for Reinforced ICL.
Verifier Prompt
Verified agent for step by step reasoning.
For the text within the <a> tags, make sure it matches exactly to the correct answer, word for word.
Appendix
Table 4: Prompt template used for the Verifier agent during data generation for Reinforced ICL.
Task
Baseline
4-Shot
16-Shot
32-Shot
Domain-Specific / Information-Heavy Tasks
DROP
10.8
13.1
14.2
15.8
FDA
39.65
86.8
89.4
89.7
SWDE
74.17
92.9
95.1
96.4
Structured Reasoning Tasks (Constrained Output)
ARC-Challenge
38.82
93.72
93.48
93.45
Appendix
Table 5: Task-dependent performance of many-shot test-time updates (LLaMA-3.3-70B-Instruct)
Figure 4: Context length scaling with update magnitude