Retrieval-Based In-Context Learning: A Domain Adaptation Framework

Yilun Zhu; Naihao Deng; Yingcong Li; Naichen Shi; Clayton Scott

Figure 7 : Test error on

Q

vs. number of demonstrations

n

, with

P=Q

(no domain shift). Color : ICL (blue) vs. ICR (orange). Line style : TabPFN (solid, •) vs.

k

-NN (dashed,

\square

). Corpus size

m=20{,}000

,

n_{\text{test}}=400

. Results averaged over 5 independent runs.

$n$	TabPFN-ICL	TabPFN-ICR	$k$ -NN-ICL	$k$ -NN-ICR
4	$0.395$ $\pm$ $0.094$	$0.082$ $\pm$ $0.018$	$0.350$ $\pm$ $0.114$	$0.085$ $\pm$ $0.011$
16	$0.229$ $\pm$ $0.073$	$0.048$ $\pm$ $0.006$	$0.272$ $\pm$ $0.026$	$0.041$ $\pm$ $0.010$
32	$0.165$ $\pm$ $0.039$	$0.048$ $\pm$ $0.008$	$0.230$ $\pm$ $0.053$	$0.039$ $\pm$ $0.012$
128	$0.073$ $\pm$ $0.017$	$0.031$ $\pm$ $0.004$	$0.123$ $\pm$ $0.028$	$0.027$ $\pm$ $0.009$
256	$0.038$ $\pm$ $0.010$	$0.012$ $\pm$ $0.008$	$0.102$ $\pm$ $0.027$	$0.027$ $\pm$ $0.009$

Appendix

Table 1 : Experiment 1,

d=8

: test error on

Q

, we report mean

\pm

std over 5 trials.

$n$	TabPFN-ICL	TabPFN-ICR	$k$ -NN-ICL	$k$ -NN-ICR
4	$0.469$ $\pm$ $0.054$	$0.139$ $\pm$ $0.022$	$0.397$ $\pm$ $0.071$	$0.149$ $\pm$ $0.026$
16	$0.382$ $\pm$ $0.086$	$0.101$ $\pm$ $0.011$	$0.374$ $\pm$ $0.039$	$0.089$ $\pm$ $0.012$
32	$0.266$ $\pm$ $0.023$	$0.100$ $\pm$ $0.017$	$0.327$ $\pm$ $0.058$	$0.066$ $\pm$ $0.013$
128	$0.099$ $\pm$ $0.012$	$0.080$ $\pm$ $0.020$	$0.196$ $\pm$ $0.040$	$0.045$ $\pm$ $0.012$
256	$0.067$ $\pm$ $0.012$	$0.062$ $\pm$ $0.010$	$0.138$ $\pm$ $0.017$	$0.041$ $\pm$ $0.009$

Appendix

Table 2 : Experiment 1,

d=16

: test error on

Q

, we report mean

\pm

std over 5 trials.

Figure 8 : Test error on

Q

vs. posterior drift parameter

\varepsilon

, with

n=4

fixed. Color : ICL (blue) vs. ICR (orange). Line style : TabPFN (solid) vs.

k

-NN (dashed). ICL baselines (horizontal) are the means over 5 runs

\times

11

\varepsilon

values (since

\varepsilon

only affects the ICR corpus; not affecting ICL). ICR mean error grows monotonically with

\varepsilon

. Results averaged over 5 independent runs.

$\varepsilon$	TabPFN-ICL	TabPFN-ICR	$k$ -NN-ICL	$k$ -NN-ICR
0.0	$0.394$ $\pm$ $0.094$	$0.082$ $\pm$ $0.017$	$0.350$ $\pm$ $0.114$	$0.085$ $\pm$ $0.011$
0.1	$0.411$ $\pm$ $0.060$	$0.127$ $\pm$ $0.018$	$0.354$ $\pm$ $0.071$	$0.128$ $\pm$ $0.019$
0.2	$0.404$ $\pm$ $0.065$	$0.233$ $\pm$ $0.033$	$0.373$ $\pm$ $0.047$	$0.225$ $\pm$ $0.043$
0.3	$0.458$ $\pm$ $0.074$	$0.328$ $\pm$ $0.042$	$0.389$ $\pm$ $0.137$	$0.318$ $\pm$ $0.045$
0.4	$0.411$ $\pm$ $0.075$	$0.400$ $\pm$ $0.032$	$0.333$ $\pm$ $0.057$	$0.405$ $\pm$ $0.039$
0.5	$0.414$ $\pm$ $0.092$	$0.496$ $\pm$ $0.041$	$0.380$ $\pm$ $0.085$	$0.506$ $\pm$ $0.044$

Appendix

Table 3 : Experiment 2,

d=8

,

n=4

: test error on

Q

vs. posterior drift

\varepsilon

, we report mean

\pm

std over 5 trials.

$\varepsilon$	TabPFN-ICL	TabPFN-ICR	$k$ -NN-ICL	$k$ -NN-ICR
0.0	$0.468$ $\pm$ $0.055$	$0.139$ $\pm$ $0.023$	$0.397$ $\pm$ $0.071$	$0.149$ $\pm$ $0.026$
0.1	$0.447$ $\pm$ $0.072$	$0.185$ $\pm$ $0.032$	$0.440$ $\pm$ $0.064$	$0.184$ $\pm$ $0.024$
0.2	$0.465$ $\pm$ $0.048$	$0.264$ $\pm$ $0.033$	$0.451$ $\pm$ $0.030$	$0.267$ $\pm$ $0.025$
0.3	$0.477$ $\pm$ $0.037$	$0.336$ $\pm$ $0.026$	$0.430$ $\pm$ $0.071$	$0.353$ $\pm$ $0.027$
0.4	$0.468$ $\pm$ $0.027$	$0.416$ $\pm$ $0.018$	$0.429$ $\pm$ $0.033$	$0.428$ $\pm$ $0.020$
0.5	$0.480$ $\pm$ $0.058$	$0.496$ $\pm$ $0.034$	$0.468$ $\pm$ $0.034$	$0.491$ $\pm$ $0.024$

Appendix

Table 4 : Experiment 2,

d=16

,

n=4

: test error on

Q

vs. posterior drift

\varepsilon

, we report mean

\pm

std over 5 trials.

Figure 9 : Examples of the Weierstrass class-probability function

\eta_{\alpha}(x)

for different values of the smoothness parameter

\alpha

.

Figure 10 : Target excess risk vs. number of demonstrations

n

on the 1-D Weierstrass family (

P=Q

;

\beta=1

; oracle

k_{\alpha}

). Color : ICL (blue) vs. ICR (orange). Line style : TabPFN (solid, •) vs.

k

-NN (dashed,

\square

). Both axes are log-scaled. Corpus size

m=10^{5}

,

n_{\text{test}}=400

; each curve is the mean over 5 seeds (42–46).

$n$	TabPFN-ICL	TabPFN-ICR	$k$ -NN-ICL	$k$ -NN-ICR
4	$0.151$ $\pm$ $0.036$	$0.066$ $\pm$ $0.006$	$0.147$ $\pm$ $0.040$	$0.058$ $\pm$ $0.005$
16	$0.151$ $\pm$ $0.013$	$0.040$ $\pm$ $0.005$	$0.129$ $\pm$ $0.046$	$0.023$ $\pm$ $0.002$
32	$0.134$ $\pm$ $0.023$	$0.024$ $\pm$ $0.003$	$0.092$ $\pm$ $0.019$	$0.014$ $\pm$ $0.001$
64	$0.103$ $\pm$ $0.035$	$0.018$ $\pm$ $0.004$	$0.058$ $\pm$ $0.022$	$0.008$ $\pm$ $0.001$
128	$0.059$ $\pm$ $0.009$	$0.012$ $\pm$ $0.002$	$0.029$ $\pm$ $0.013$	$0.005$ $\pm$ $0.001$
256	$0.036$ $\pm$ $0.020$	$0.007$ $\pm$ $0.001$	$0.020$ $\pm$ $0.011$	$0.002$ $\pm$ $0.001$

Appendix

Table 5 : Weierstrass sample-size sweep,

\alpha=1.0

: target excess risk, mean

\pm

std over 5 seeds.

$n$	TabPFN-ICL	TabPFN-ICR	$k$ -NN-ICL	$k$ -NN-ICR
4	$0.160$ $\pm$ $0.028$	$0.077$ $\pm$ $0.006$	$0.157$ $\pm$ $0.027$	$0.067$ $\pm$ $0.008$
16	$0.154$ $\pm$ $0.015$	$0.043$ $\pm$ $0.006$	$0.121$ $\pm$ $0.060$	$0.024$ $\pm$ $0.002$
32	$0.127$ $\pm$ $0.025$	$0.027$ $\pm$ $0.004$	$0.096$ $\pm$ $0.017$	$0.014$ $\pm$ $0.002$
64	$0.107$ $\pm$ $0.034$	$0.019$ $\pm$ $0.005$	$0.071$ $\pm$ $0.036$	$0.008$ $\pm$ $0.001$
128	$0.069$ $\pm$ $0.028$	$0.011$ $\pm$ $0.003$	$0.055$ $\pm$ $0.018$	$0.003$ $\pm$ $0.001$
256	$0.037$ $\pm$ $0.024$	$0.006$ $\pm$ $0.001$	$0.043$ $\pm$ $0.017$	$0.002$ $\pm$ $0.000$

Appendix

Table 6 : Weierstrass sample-size sweep,

\alpha=0.5

: target excess risk, mean

\pm

std over 5 seeds.

Figure 11 : Target excess risk vs. posterior drift

\varepsilon=Q_{X}(B)

on the Weierstrass family at

\alpha=1.0

(smooth

\eta

), for

n\in\{16,512\}

over 5 seeds.

B

is a union of

N=40

periodic sub-brackets, each covering the leftmost

\varepsilon

-fraction of one width-

2/N

interval of

[-1,1]

, so

B

samples every macro-cycle of

\eta_{Q}

uniformly and

Q_{X}(B)=\varepsilon

holds exactly. Color : ICL (blue) vs. ICR (orange). Line style : TabPFN (solid, •) vs.

k

-NN (dashed,

\square

). ICL curves are exactly flat because ICL uses

Q

labels and does not see the drift; ICR curves rise almost linearly in

\varepsilon

across the whole range, saturating near

1-2R_{Q}^{\ast}\approx 0.306

at

\varepsilon=1

. At small

n

(left,

n=16

) ICR beats ICL up to

\varepsilon\approx 0.4

–

0.5

; at large

n

(right,

n=512

) ICR is roughly an order of magnitude below ICL at

\varepsilon=0

(variance-limited regime), but ICL overtakes it already by

\varepsilon\approx 0.08

. In both regimes the ICR bias tracks

\varepsilon

almost linearly, matching the

\varepsilon+(\lambda/\kappa)(R_{P}-R_{P}^{\ast})

reduction in Lemma 1 .

$\varepsilon$	TabPFN-ICL	TabPFN-ICR	$k$ -NN-ICL	$k$ -NN-ICR
$0.0$	$0.151$ $\pm$ $0.013$	$0.040$ $\pm$ $0.005$	$0.129$ $\pm$ $0.046$	$0.023$ $\pm$ $0.002$
$0.1$	$0.151$ $\pm$ $0.013$	$0.063$ $\pm$ $0.003$	$0.129$ $\pm$ $0.046$	$0.049$ $\pm$ $0.002$
$0.2$	$0.151$ $\pm$ $0.013$	$0.083$ $\pm$ $0.003$	$0.129$ $\pm$ $0.046$	$0.073$ $\pm$ $0.003$
$0.3$	$0.151$ $\pm$ $0.013$	$0.105$ $\pm$ $0.002$	$0.129$ $\pm$ $0.046$	$0.098$ $\pm$ $0.002$
$0.4$	$0.151$ $\pm$ $0.013$	$0.129$ $\pm$ $0.004$	$0.129$ $\pm$ $0.046$	$0.124$ $\pm$ $0.001$
$0.5$	$0.151$ $\pm$ $0.013$	$0.149$ $\pm$ $0.007$	$0.129$ $\pm$ $0.046$	$0.151$ $\pm$ $0.002$

Appendix

Table 7 : Experiment 2,

\alpha=1.0

,

n=16

: target excess risk vs. posterior drift

\varepsilon=Q_{X}(B)

, mean

\pm

std over

5

seeds (Weierstrass family,

A=0.24

,

b=0.85

,

f=2

,

J=12

; oracle

k_{\alpha}=\mathrm{round\_odd}(0.30\cdot n^{2\alpha/(2\alpha+1)})

;

m_{\mathrm{corpus}}=10^{5}

;

n_{\mathrm{test}}=400

). Region

B

is the leftmost

\varepsilon

-fraction of each of

N=40

equal-width periodic brackets partitioning

[-1,1]

, so

Q_{X}(B)=\varepsilon

exactly and

B

samples every macro-cycle of

\eta_{Q}

uniformly;

\eta_{P}=1-\eta_{Q}

on

B

; ICL uses

Q

labels (so ICL columns are

\varepsilon

-constant).

$\varepsilon$	TabPFN-ICL	TabPFN-ICR	$k$ -NN-ICL	$k$ -NN-ICR
$0.0$	$0.019$ $\pm$ $0.014$	$0.003$ $\pm$ $0.001$	$0.018$ $\pm$ $0.008$	$0.001$ $\pm$ $0.000$
$0.1$	$0.019$ $\pm$ $0.014$	$0.032$ $\pm$ $0.001$	$0.018$ $\pm$ $0.008$	$0.020$ $\pm$ $0.005$
$0.2$	$0.019$ $\pm$ $0.014$	$0.060$ $\pm$ $0.001$	$0.018$ $\pm$ $0.008$	$0.060$ $\pm$ $0.001$
$0.3$	$0.019$ $\pm$ $0.014$	$0.090$ $\pm$ $0.001$	$0.018$ $\pm$ $0.008$	$0.090$ $\pm$ $0.001$
$0.4$	$0.019$ $\pm$ $0.014$	$0.120$ $\pm$ $0.001$	$0.018$ $\pm$ $0.008$	$0.120$ $\pm$ $0.001$
$0.5$	$0.019$ $\pm$ $0.014$	$0.151$ $\pm$ $0.001$	$0.018$ $\pm$ $0.008$	$0.150$ $\pm$ $0.001$

Appendix

Table 8 : Experiment 2,

\alpha=1.0

,

n=512

: target excess risk vs. posterior drift

\varepsilon=Q_{X}(B)

, mean

\pm

std over

5

seeds (same setup as Table 7 ).

Figure 12 : Target excess risk vs. smoothness exponent

\alpha

on the Weierstrass family at

n=256

, with

\alpha\in\{1,\,10^{-1},\,10^{-2},\,10^{-3}\}

(log-spaced, 5 seeds). Color : ICL (blue) vs. ICR (orange). Line style : TabPFN (solid, •) vs.

k

-NN (dashed,

\square

). The horizontal axis is logarithmic; the

\alpha

-axis is reversed so rougher

\eta_{\alpha}

is on the right. Both

k

-NN-ICL and TabPFN-ICL trend upward as

\alpha

decreases, matching the

n^{-2\alpha/(2\alpha+1)}

prediction of Theorem 1 ; the

k

-NN-ICL curve saturates near

\alpha\to 0

at the finite-

n

noise floor. ICR sits about an order of magnitude below ICL and is essentially flat in

\alpha

, consistent with the

\alpha

-independent

n^{-(\beta+1)/2}

rate of Theorem 1 ; the ICL/ICR gap widens as

\alpha\to 0

.

$\alpha$	TabPFN-ICL	TabPFN-ICR	$k$ -NN-ICL	$k$ -NN-ICR
$1$	$0.036$ $\pm$ $0.020$	$0.007$ $\pm$ $0.001$	$0.020$ $\pm$ $0.011$	$0.002$ $\pm$ $0.001$
$10^{-1}$	$0.038$ $\pm$ $0.019$	$0.004$ $\pm$ $0.001$	$0.080$ $\pm$ $0.013$	$0.001$ $\pm$ $0.000$
$10^{-2}$	$0.046$ $\pm$ $0.031$	$0.005$ $\pm$ $0.002$	$0.089$ $\pm$ $0.009$	$0.001$ $\pm$ $0.000$
$10^{-3}$	$0.080$ $\pm$ $0.025$	$0.004$ $\pm$ $0.002$	$0.104$ $\pm$ $0.009$	$0.001$ $\pm$ $0.001$

Appendix

Table 9 : Experiment 3,

n=256

: target excess risk vs. smoothness

\alpha

, mean

\pm

std over

5

seeds (Weierstrass family,

A=0.24

,

b=0.85

,

f=2

,

J=12

; oracle

k_{\alpha}=\mathrm{round\_odd}(0.30\cdot n^{2\alpha/(2\alpha+1)})

;

m_{\mathrm{corpus}}=10^{5}

;

n_{\mathrm{test}}=400

).

Figure 13 : Excess error on

Q

versus corpus size

m

, with

n

fixed and

\alpha_{P}=1

. Color : ICL (blue) versus ICR (orange). Line style : TabPFN (solid, •) vs.

k

-NN (dashed,

\square

). Per

(n,\text{seed})

we draw one corpus of size

m_{\max}

and use nested prefixes

\mathcal{D}_{m}=\mathcal{D}_{m_{\max}}[:m]

; the ICL demo set is drawn once per

n

from that corpus and reused across every

m

. Consequently ICL is exactly constant along the

m

axis for each seed, and the horizontal ICL baselines are the across-seed means of those per-seed constants. Both ICR curves collapse sharply around the reference scale and remain near their asymptotic value thereafter. The dotted grey line marks the unit-constant reference

n^{1+1/(2\alpha_{P})}=n^{3/2}

(

512

for

n=64

,

\approx 1448

for

n=128

); the actual saturation threshold is

C_{\star}\,n^{1+1/(2\alpha_{P})}

for some unknown constant. All 5 seeds (42–46). Setup: 1-D Weierstrass

\eta

-family with

A=0.24

,

b=0.85

,

f=2

,

J=12

, oracle

k_{\alpha}=\mathrm{round\_odd}(0.30\cdot n^{2\alpha/(2\alpha+1)})

, linspace

n_{\text{test}}=400

, and joint demonstrations.

$n$	$m$	TabPFN-ICL	TabPFN-ICR	$k$ -NN-ICL	$k$ -NN-ICR
64	64	$0.097$ $\pm$ $0.033$	$0.133$ $\pm$ $0.019$	$0.079$ $\pm$ $0.016$	$0.145$ $\pm$ $0.042$
	128		$0.073$ $\pm$ $0.022$		$0.140$ $\pm$ $0.028$
	256		$0.038$ $\pm$ $0.011$		$0.109$ $\pm$ $0.032$
	512		$0.020$ $\pm$ $0.005$		$0.019$ $\pm$ $0.009$
	1024		$0.021$ $\pm$ $0.006$		$0.009$ $\pm$ $0.003$
	2048		$0.021$ $\pm$ $0.006$		$0.008$ $\pm$ $0.004$

Appendix

Table 10 : Corpus-size sweep: excess error on

Q

, mean

\pm

sample standard deviation over five seeds. ICL entries are constant along the

m

axis by construction (see Fig. 13 caption) and are shown once per

n

; ICR entries vary per

m

.

Figure 14 : Test error vs. number of demonstrations

n

on the age-biased sentiment dataset (878 test annotators, mean over 5 seeds), for Qwen3.5-4B (left) and Qwen3.5-27B (right). Blue : ICL with demonstrations sampled uniformly without replacement from the annotator’s own data. Yellow

\to

red : ICR under four corpus designs with increasing posterior drift. The stronger model improves ICR only where the corpus is aligned with the target; under severe and extreme drift its error is essentially unchanged—the drift-induced bias is irreducible. Per-cell mean values are in Tables 11 and 12 .

Figure 15 : Test error vs. number of demonstrations

n

on the age-biased sentiment dataset (878 test annotators, Qwen3.5-4B). Each point is the mean over

5

independent runs (seeds 42–46). Blue : Qwen-ICL with demonstrations sampled uniformly without replacement from the annotator’s own training data. Yellow

\to

red : Qwen-ICR under four levels of posterior drift, as a result of expanding the source corpus: no drift (

P=Q

, self only), mild (self

+

100 random others), severe (all 1,481 annotators), and extreme (all other annotators, self excluded). ICR sharply outperforms ICL when the source corpus is well-aligned with the target distribution and degrades as posterior drift increases. Uncertainty estimates are reported in Table 11 .

$n$	Qwen-ICL	ICR ( $P=Q$ )	ICR (mild)	ICR (severe)	ICR (extreme)
4	$0.538$ $\pm$ $0.007$	$0.263$ $\pm$ $0.000$	$0.265$ $\pm$ $0.009$	$0.535$ $\pm$ $0.004$	$0.633$ $\pm$ $0.004$
8	$0.481$ $\pm$ $0.008$	$0.169$ $\pm$ $0.002$	$0.225$ $\pm$ $0.009$	$0.504$ $\pm$ $0.002$	$0.628$ $\pm$ $0.007$
16	$0.348$ $\pm$ $0.011$	$0.099$ $\pm$ $0.002$	$0.162$ $\pm$ $0.012$	$0.491$ $\pm$ $0.003$	$0.615$ $\pm$ $0.006$
32	$0.117$ $\pm$ $0.004$	$0.072$ $\pm$ $0.001$	$0.142$ $\pm$ $0.010$	$0.480$ $\pm$ $0.002$	$0.619$ $\pm$ $0.008$

Appendix

Table 11 : Sentiment classification with Qwen3.5-4B: test error (mean

\pm

std over

5

seeds,

878

test annotators) vs. number of demonstrations

n

across the four corpus designs.

Figure 16 : Test error vs. number of demonstrations

n

on the age-biased sentiment dataset (878 test annotators, Qwen3.5-27B). Each point is the mean over

5

independent runs (seeds 42–46). Blue : Qwen-ICL with demonstrations sampled uniformly without replacement from the annotator’s own training data. Yellow

\to

red : Qwen-ICR under four levels of posterior drift, as a result of expanding the source corpus: no drift (

P=Q

, self only), mild (self

+

100 random others), severe (all 1,481 annotators), and extreme (all other annotators, self excluded). Compared to Qwen3.5-4B ( Figure 15 ), the stronger model lowers the ICR error under no/mild drift but does not remove the irreducible bias under severe or extreme drift, matching the theory. Uncertainty estimates are reported in Table 12 .

$n$	Qwen-ICL	ICR ( $P=Q$ )	ICR (mild)	ICR (severe)	ICR (extreme)
4	$0.514$ $\pm$ $0.005$	$0.118$ $\pm$ $0.001$	$0.165$ $\pm$ $0.003$	$0.522$ $\pm$ $0.008$	$0.619$ $\pm$ $0.006$
8	$0.459$ $\pm$ $0.005$	$0.094$ $\pm$ $0.000$	$0.154$ $\pm$ $0.006$	$0.507$ $\pm$ $0.003$	$0.620$ $\pm$ $0.005$
16	$0.341$ $\pm$ $0.006$	$0.083$ $\pm$ $0.000$	$0.155$ $\pm$ $0.005$	$0.506$ $\pm$ $0.003$	$0.606$ $\pm$ $0.008$
32	$0.132$ $\pm$ $0.007$	$0.097$ $\pm$ $0.001$	$0.163$ $\pm$ $0.005$	$0.502$ $\pm$ $0.003$	$0.612$ $\pm$ $0.007$

Appendix

Table 12 : Sentiment classification with Qwen3.5-27B: test error (mean

\pm

std over

5

seeds,

878

test annotators) vs. number of demonstrations

n

across the four corpus designs.

Retrieval-Based In-Context Learning: A Domain Adaptation Framework

Abstract

Figures & tables

Appendix

Explore similar work

A Theoretical Interpretation of In-Context Learning via Probabilistic Modeling

How Does the Pretraining Distribution Shape In-Context Learning? A Fundamental Trade-Off

Contrast Matters: Understanding Robustness of In-Context Fine-Tuning to Target-Context Relatedness