Can Jev Improve Demonstration Retrieval for In-Context Learning?
Organizations: College of Computer Science and Technology, Jilin University, Changchun, China
Abstract
Demonstration selection plays a crucial role in the effectiveness of in-context learning (ICL) for large language models (LLMs). Extensive research has explored various retrieval strategies to identify informative demonstrations and improve downstream performance. Recently, the emergence of Jev, a decision-oriented language model capable of producing probabilistic judgments, offers a new perspective on evaluating demonstration utility. This motivates us to investigate a fundamental question: Can Jev improve demonstration retrieval for in-context learning? In this paper, we demonstrate that Jev can effectively enhance demonstration retrieval by directly assessing the utility of candidate demonstrations for a given query. However, applying Jev to large-scale retrieval introduces substantial computational overhead, as each candidate requires an additional decision-making inference. To address this limitation, we propose a simple yet effective knowledge distillation approach that transfers Jev's utility estimation capability to a lightweight multilayer perceptron (MLP). By learning to approximate Jev's probabilistic judgments, the distilled model enables efficient demonstration ranking while maintaining competitive retrieval performance. Our study highlights the potential of decision-oriented language models for demonstration selection and provides an efficient framework for integrating their decision-making capabilities into ICL.
Figures & tables
| Method | TREC | SST-2 | SST-5 | AG News | MNLI | Avg. |
|---|---|---|---|---|---|---|
| Llama3.2-1B | ||||||
| NN | 72.60 | 58.83 | 38.10 | 86.65 | 42.05 | 59.65 |
| ConE | 75.20 | 60.55 | 34.15 | 85.90 | 40.65 | 59.29 |
| DiSP | 69.40 | 55.77 | 39.10 | 87.64 | 45.42 | 59.47 |
| DisNeoJEV | 80.60 | 57.91 | 41.85 | 91.05 | 45.00 | 63.28 |
| DisJEV | 80.00 | 58.14 | 41.90 | 88.88 | 44.29 | 62.64 |
| Method | TREC | SST-2 | SST-5 | AG News | MNLI | Avg. |
|---|---|---|---|---|---|---|
| Llama3.2-1B | ||||||
| NN | 72.60 | 58.83 | 38.10 | 86.65 | 42.05 | 59.65 |
| DisJEV-trg | 76.60 | 57.80 | 42.40 | 86.55 | 42.70 | 61.21 |
| DisJEV-score | 76.05 | 58.94 | 40.86 | 88.80 | 42.69 | 61.47 |
| DisJEV-rank | 75.17 | 58.19 | 42.13 | 87.43 | 43.44 | 61.27 |
| DisJEV | 80.00 | 58.14 | 41.90 | 88.88 | 44.29 | 62.64 |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Llama Models | Qwen Models | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Method | TREC | SST-2 | SST-5 | AG News | MNLI | Avg. | Method | TREC | SST-2 | SST-5 | AG News | MNLI | Avg. | ||
| Llama3.2-1B | Qwen3-1.7B | ||||||||||||||
| Zero-shot | 0 | 40.00 | 89.68 | 34.05 | 74.15 | 41.80 | 55.94 | Zero-shot | 0 | 53.60 | 81.54 | 31.00 | 33.95 | 50.45 | 50.11 |
| NN | 1 | 78.40 | 72.13 | 39.25 | 74.00 | 39.35 | 60.63 | NN | 1 | 65.00 | 87.96 | 40.40 | 83.95 | 67.05 | 68.87 |
| NeoJevICL | 1 | 80.60 | 65.94 | 38.50 | 74.70 | 37.55 | 59.46 | NeoJevICL | 1 | 77.60 | 85.78 | 43.85 | 84.80 | 70.30 | 72.47 |
| JevICL | 1 | 78.20 | 64.91 | 36.30 | 74.75 | 34.50 | 57.73 | JevICL | 1 | 78.20 | 88.53 | 43.45 | 84.00 | 69.85 | 72.81 |
In-Context Example Selection
In-context example selection chooses demonstrations that help language and multimodal models perform a query-specific task without changing model weights.
See all 30 papers