Organizations: College of Computer Science and Technology, Jilin University, China · Department of Information Engineering and Computer Science, University of Trento, Italy
Although studies have demonstrated that Large Language Models (LLMs) can perform well on Out-of-Distribution (OOD) tasks, their advantage tends to diminish as the distribution shift becomes more severe. Consequently, researchers aim to retrieve distributionally similar and informative demonstrations from the available source domain to boost the inference capabilities of LLMs. However, in practical scenarios where the target domain is inaccessible, evaluating the unknown distribution is challenging, which indirectly impacts the quality of the selected demonstrations. To address this problem, we propose DOPA, a demonstration search framework that incorporates an OOD proxy to approximate the inaccessible target domain and guide the retrieval process. Building on proxy-based evaluation, DOPA further introduces a Mahalanobis distance-based global diversity constraint to ensure sufficient diversity among the retrieved demonstrations. Experimental results on multiple LLMs and tasks demonstrate that DOPA effectively enhances robustness in OOD settings\footnote{https://github.com/bort64/ood\_code}.
Figures & tables
Figure 1: The model architecture of DOPA based on the sentiment analysis task. First, DOPA performs task-specific instruction tuning on the source domain to obtain a source proxy based on any given LLM. Correspondingly, an identical LLM without fine-tuning, which preserves the prior knowledge of the target domain, is employed as the target-domain proxy. For the same input, the ratio between the two proxies is employed as an OOD proxy estimation, which is further combined with similarity and diversity to support multi-granularity demonstration search.
Methods
SA
TD
NLI
dynasent
semeval
sst
avg
implicit
adv
toxigen
avg
wanli
anli
cnli
avg
GPT2-xl
Random
36.33
49.28
47.70
44.44
50.47
50.20
50.60
50.42
34.23
32.23
39.22
35.23
KNN
35.89
45.92
51.17
44.33
47.67
47.50
48.50
47.89
33.57
33.50
45.05
37.37
DrICL
37.00
47.66
52.76
45.81
49.70
51.38
48.23
49.77
32.03
33.33
47.20
37.52
Rewrite
36.00
45.84
50.61
44.15
46.90
45.72
49.37
47.33
34.00
32.77
45.38
37.38
Table 1: The performance (accuracy %) on classification tasks, * indicates that the results based on the LLM among all the datasets are statistically significant under the Wilcoxon Signed-Rank Test ( p≤0.05 ). The calculation process of significance is presented in Appendix B.3 .
Methods
SA
TD
NLI
dynasent
semeval
sst
avg
implicit
adv
toxigen
avg
wanli
anli
cnli
avg
GPT4o-mini
KNN
67.67
60.50
78.17
68.78
57.63
57.13
82.00
65.58
37.67
39.67
26.83
34.72
Rewrite
64.33
56.17
75.83
65.44
54.63
54.75
72.25
60.54
36.83
37.67
27.67
34.06
InfICL
63.83
54.17
80.67
66.22
59.38
62.88
84.15
68.79
38.38
38.00
32.83
36.39
DICL
68.67
62.00
78.00
69.56
57.38
54.75
80.63
64.25
37.00
36.67
20.50
31.39
Table 2: The performance (accuracy %) on classification tasks based on closed-source models.
LLMs
Methods
wnut
ener
avg
GPT2-xl
Random
22.85
32.83
27.84
KNN
51.03
54.47
52.75
DrICL
19.65
23.29
21.47
DICL
22.95
31.76
27.36
DOPA
51.32
56.82
54.07
Qwen3-1.7B
Random
25.92
28.71
27.32
Table 3: The performance on NER tasks.
Variants
SA
TD
NLI
NER
avg
LLaMA3.2-3B
DOPA -mah
56.59
58.67
38.66
37.13
47.76 ↓1.64
DOPA -sim
56.15
57.62
37.24
36.24
46.81 ↓2.59
DOPA -pro
55.81
59.36
38.41
37.21
47.70 ↓1.70
DOPA uni
57.46
59.67
38.52
34.57
47.56 ↓1.84
DOPA
59.29
59.87
39.15
39.29
49.40
Table 4: Ablation study results on LLaMA3.2-3B and Qwen3-1.7B .
Figure 2: Performance influence of k on LLaMA3.2-3B and Qwen3-1.7B across tasks.
Figure 3: Performance influence of N on DOPA and KNN based on LLaMA3.2-3B and Qwen3-1.7B , the shaded areas with corresponding colors indicate the performance differences.
Figure 4: Different visualization results on sst .
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Different KDE visualization results on all classification tasks.
Source Proxy
Target Proxy
dynasent
semeval
sst
average
Llama3.2-3B
Llama3.2-3B
55.71
53.28
68.88
59.29
Llama3.2-3B
Llama3.1-8B
55.38
50.28
66.45
57.37
Llama3.2-3B
Qwen3-1.7B
56.54
45.86
67.76
56.72
KNN
52.63
45.76
65.42
54.60
Appendix
Table 5: Performance comparison under different target proxy settings.
Model
Method
advQA
searchQA
newsQA
Avg
EM
F1
EM
F1
EM
F1
LLaMA3.2-3B
KNN
30.5
54.94
33.1
52.46
32.0
58.61
43.60
DOPA
31.5
55.49
34.5
53.72
33.2
59.97
44.73
Qwen3-1.7B
KNN
31.7
51.21
39.35
51.96
37.3
61.45
45.50
DOPA
32.7
52.65
40.95
52.56
37.8
61.49
46.35
Appendix
Table 6: Experimental results on QA tasks.
Figure 6: Case study on sst and implicit.
Figure 7: Euclidean distance comparison to target domain samples for retrieval results with and without the diversity constraint (with MahDist and w/o MahDist) on all tasks.