Data Efficient Sample Selection for In-Context Learning

V Venktesh; Cem Levi; Avishek Anand

cs.LGSep 6, 2026

Data Efficient Sample Selection for In-Context Learning

Authors: V Venktesh, Cem Levi, Avishek Anand

Organizations: Stockholm University · SKIM Group B.V. · TU Delft

Abstract

The In-context learning (ICL) paradigm aids large language models (LLMs) to adapt to new tasks without need for fine-tuning. However, selecting an optimal combination of demonstration examples from a large pool of example subsets is a challenging problem. Existing approaches for selection do not model the complex relationship between ICL samples and downstream LLM performance. They typically perform static task-level selection, choosing subsets once offline, which can fail to generalize to unseen queries. We introduce DearICL (Data Efficient Algorithm for Ranking) ICL samples, a new framework that models demonstration example selection as a subset ranking problem. DearICL employs a non-linear surrogate employing a differentiable sorting objective within a gap-index bandit algorithm. The gap-index based approach enables fine-grained separation of good arms and borderline arms, which is used as an auxiliary objective to train the non-linear surrogate through sufficient sampling of borderline arms, supporting instance-level subset ranking. On exemplar selection benchmarks with open-source LLMs, DearICL achieves 8.08-15.9% accuracy gains over strong linear bandit baselines, with low sample complexity. Code and data: https://github.com/VenkteshV/DearICL.

Figures & tables

Panel of Figure 1 — Figure 1: Top- $m$ arm identification by DEARICL , CASE for AquaRAT. (a) Gap Index ( $B_{t}(ch_{t},b_{t})$ ) comparison and (b) Simple regret comparison, (c) subset selection time, (d) Average inference time/query.

Method	GSM8K	AquaRat	WMT19
Task level
Zero-shot-COT Kojima et al. (2022)	37.37	36.61	51.22
Few-Shot COT Wei et al. (2022b)	63.22	40.94	61.64
LENS Li and Qiu (2023)	64.97	44.88	64.09
EXPLORA Purohit et al. (2024)	69.92	47.24	66.26
BRIDGE Wan et al. (2025)	64.52	48.81	64.96

Table 1: Results using llama3.2:3b

\dagger

indicates statistical significance (t-test) over

CASE_{dynamic}

at 0.05 level. We report performance using the official metrics: Exact Match (EM) (AquaRAT,GSM8K) and BertScore Zhang et al. (2020) (WMT19) for the respective datasets.

Method	GSM8K	AquaRat	WMT19
$CASE_{dynamic}$	1554.19	2378.04	106.54
Neural NDCG	10.59	18.51	5.20
PiRank	6.10	11.59	1.80
DearICL dynamic	6.04	11.59	1.68

Table 2: Inference Time Comparison (in seconds)

Total # LLM calls	GSM8K	AquaRat	WMT19
CASE	10200	4360	87
DEARICL ${}_{\text{static}}$	2600	1580	48

Table 3: Results comparing the number of LLM calls.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Framework	Loss function	Hyperparameters	Network architecture
PiRank	PiRank surrogate loss	Presented in Section 4	(256, 256,128,64)
Other LTR baselines	Neural NDCG	N = 2, $d_{ff}$ = 384, h = 1, dropout = 0.1	(768,96)
	ListNet	N = 4, $d_{ff}$ = 512, h = 2, dropout = 0.3	(768,128)
	Neural NDCG	N = 4, $d_{ff}$ = 512, h = 4, dropout = 0.3	(768,96)
	Neural NDCG With Normalized data	N = 2, $d_{ff}$ = 384, h = 1, dropout = 0.1	(768,96)
	NDCGLoss 2++	-	(256, 512, 1024, 512, 256)

Appendix

Table 4: Details of hyperparameters used in different LTR model configurations. Categorized by loss function and framework.

Task performance	GSM8K	AquaRat	WMT19
NDCGLoss 2++	72.11	49.00	74.06
NeuralNDCG	69.83	48.30	72.75
NeuralNDCG (norm. data)	72.06	49.21	72.97
UDR	71.63	48.03	72.65
ListNet	71.79	46.06	73.64
DEARICL dynamic	75.66	55.11	78.89

Appendix

Table 5: Ablation results: Training LTR approaches on training data collected across rounds by DEARICL leveraging llama3.2:3b

Method	GSM8K	AquaRat	WMT
Task level
EXPLORA Purohit et al. (2024)	93.63	69.29	84.55
LENS ( Li and Qiu, 2023 )	76.19	64.56	83.57
Static CASE	91.13	73.23	83.49
$\textsc{DEARICL}{}_{static}$	94.84	77.16	86.25
Instance level

Appendix

Table 6: Results across datasets (we use 5-shot for all methods) using gpt-4o-mini.

\dagger

indicates statistical significance (t-test) over

CASE_{dynamic}

at 0.05 level

Method	GSM8K	AquaRat	WMT
Task level
EXPLORA Purohit et al. (2024)	82.63	68.10	78.59
LENS ( Li and Qiu, 2023 )	77.33	57.87	76.98
Static CASE	83.09	69.29	78.26
$\textsc{DEARICL}{}_{static}$	86.12	70.47	79.27
Instance level

Appendix

Table 7: Results across datasets using Deepseek-R1:7b (we use 5-shot for all methods).

\dagger

indicates statistical significance (t-test) over

CASE_{dynamic}

at 0.05 level

Panel of Figure 2 — Figure 2: Top- $m$ arm identification by DEARICL , CASE for GSM8k, WMT19. (a,b) Gap Index ( $B_{t}(ch_{t},b_{t})$ ) comparison and (c,d) Simple regret comparison

Question	A travel company wants to charter a plane to the Bahamas. Chartering the plane costs $5,000. So far, 12 people have signed up for the trip. If the company charges$ 200 per ticket, how many more passengers must sign up for the trip before the company can make any profit on the charter?
Method	Exemplars
NeuralNDCG	Question: A group consists of 4 couples in which each of the 4 boys has one girlfriend. In how many ways can they be arranged in a straight line such that boys and girls occupy alternate positions?
	Question: A man cycling along the road noticed that every 12 minutes a bus overtakes him and every 4 minutes he meets an oncoming bus. If all buses and the cyclist move at a constant speed, what is the time interval between consecutive buses?
	Question: Find large number from below question. The difference of two numbers is 1365. On dividing the larger number by the smaller, we get 6 as quotient and the 15 as remainder?
	Question: 64, 81, 100, 144, 196, ?, Find the missing number(?).
DearICL dynamic	Question: A certain sum is invested at simple interest at 18% p.a. for two years instead of investing at 12% p.a. for the same time period. Therefore the interest received is more by Rs. 840. Find the sum?

Appendix

Table 8: Exemplar subsets selected by NeuralNDCG and DearICL dynamic using LLama3.2:3b for a particular AquaRat test instance where DearICL dynamic had a correct answer and the NeuralNDCG model did not.

Panel of Figure 3 — Figure 3: Prompt for Aqua

Panel of Figure 4 — Figure 4: Prompt for GSM8K

Explore similar work

You Only Edit Once: Incentivizing In-Context Capability of LLMs via Local Demonstration Refinement
Sep 27, 2026Jiarong Wen, Qi Wang, Yun Qu +8LLM Inference Efficiency In-Context Example Selection
Department of Automation, Tsinghua University
UCS: Estimating Unseen Coverage for Improved In-Context Learning
Apr 13, 2026Jiayi Xin, Xiang Li, Evan Qiang +4In-Context Example Selection In-Context Learning
University of Pennsylvania
PLR: Plackett-Luce for Reordering In-Context Learning Examples
Mar 22, 2026Pawel Batorski, Paul SwobodaIn-Context Learning Few-Shot Learning
Heinrich Heine Universität Düsseldorf

Sep 27, 2026J/K move · Enter open · S save

You Only Edit Once: Incentivizing In-Context Capability of LLMs via Local Demonstration Refinement

Jiarong Wen, Qi Wang, Yun Qu, Yixiu Mao+7

Department of Automation, Tsinghua University

LLM Inference Efficiency In-Context Example Selection In-Context Learning Retrieval-Augmented ICL

Open paper AskPDF arXiv

In-context learning (ICL) is crucial for boosting the inference performance of large language models (LLMs). However, the effectiveness of ICL in LLMs is greatly influenced by the choice of demonstration sets. Exhaustive searches over these sets are combinatorial, and existing selectors often rely on relevance or likelihood proxies to implicitly assess ICL quality. Making repeated queries to the target LLM with these strategies can incur substantial costs. This work simplifies selection by framing it as a constrained local search problem and presents local demonstration editing (LDE). Starting with an initially retrieved set of demonstrations, LDE employs a single structured edit to explore its surrounding neighborhood while balancing performance gains with search costs. Technically, LDE is reduced to a policy search problem, for which we train a small LLM, referred to as Jev-LDE. This model as the System-1 modifies the retrieved demonstration set by performing actions such as Keep, Delete, or Replace elements, all within a framework of reinforcement learning with verifiable rewards. At test time, Jev-LDE executes a single edit of the retrieved demonstration set, followed by one inference from the target LLM, avoiding the need for iterative context scoring or subset searches. Across standard classification benchmarks, various target LLMs with Jev-LDE as the plug-and-play module consistently improve ICL performance, and Jev-LDE shows transferability to held-out benchmarks and models without retraining. These findings indicate that the LDE approach offers an efficient and adaptable method for harnessing the ICL capabilities of target LLMs.

Abstract

Explore similar work

You Only Edit Once: Incentivizing In-Context Capability of LLMs via Local Demonstration Refinement

UCS: Estimating Unseen Coverage for Improved In-Context Learning

PLR: Plackett-Luce for Reordering In-Context Learning Examples