cs.AIOct 31, 2026

Can Jev Improve Demonstration Retrieval for In-Context Learning?

Authors: Rui Song, Yingji Li, Hao Xu

Organizations: College of Computer Science and Technology, Jilin University, Changchun, China

Abstract

Demonstration selection plays a crucial role in the effectiveness of in-context learning (ICL) for large language models (LLMs). Extensive research has explored various retrieval strategies to identify informative demonstrations and improve downstream performance. Recently, the emergence of Jev, a decision-oriented language model capable of producing probabilistic judgments, offers a new perspective on evaluating demonstration utility. This motivates us to investigate a fundamental question: Can Jev improve demonstration retrieval for in-context learning? In this paper, we demonstrate that Jev can effectively enhance demonstration retrieval by directly assessing the utility of candidate demonstrations for a given query. However, applying Jev to large-scale retrieval introduces substantial computational overhead, as each candidate requires an additional decision-making inference. To address this limitation, we propose a simple yet effective knowledge distillation approach that transfers Jev's utility estimation capability to a lightweight multilayer perceptron (MLP). By learning to approximate Jev's probabilistic judgments, the distilled model enables efficient demonstration ranking while maintaining competitive retrieval performance. Our study highlights the potential of decision-oriented language models for demonstration selection and provides an efficient framework for integrating their decision-making capabilities into ICL.

Figures & tables

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Easier to Judge than to Find: Predicting In-Context Learning Success for Demonstration Selection

    May 18, 2026Haochun Wang, Chaofen Yang, Jiatong Liu +5LLM Inference EfficiencyIn-Context Example Selection

  2. Data Efficient Sample Selection for In-Context Learning

    Sep 6, 2026V Venktesh, Cem Levi, Avishek AnandIn-Context Example SelectionIn-Context Learning