ICLHerd: Herding-Based Data Selection for LLMs with In-Context Learning as a Proxy

Yuqiao Wen; Lili Mou

cs.CLOct 19, 2026

ICLHerd: Herding-Based Data Selection for LLMs with In-Context Learning as a Proxy

Authors: Yuqiao Wen, Lili Mou

Organizations: Dept. Computing Science & Alberta Machine Intelligence Institute (Amii), University of Alberta · Canada CIFAR AI Chair

Abstract

Data selection plays a central role in modern large language model training, as the quality of training data strongly influences downstream performance. A common approach is to curate a development set that encodes the desired skills (e.g., reasoning or multilinguality) and select data that are similar to them. Existing methods compute similarity using cosine distance over either the model's hidden states or gradients. However, we observe that this max-similarity strategy induces a mode-seeking bias: the selected data clusters tightly around the development samples. To address this, we propose ICLHerd, a data selection framework that leverages the von Mises-Fisher kernel to balance mode-seeking with mode-averaging. We further introduce in-context learning as a validation proxy, enabling efficient hyperparameter search over the balance trade-off. Results show that ICLHerd outperforms existing LLM data selection algorithms, highlighting the importance of balancing mode-seeking and mode-averaging.

Figures & tables

Selected on Qwen3-1.7B, fine-tuned on Qwen3-1.7B
Size	Method	GSM8K	MLQA	DROP	XSum	OASST1	Avg
1.7B	Random	$40.33$	$48.27$	$8.13$	$15.31$	$20.37$	$26.48$
	Top-PPL ( Ankner et al., 2024 )	$13.57$	51.63	16.72	$6.24$	$4.71$	$18.57$
	Mid-PPL ( Marion et al., 2023 )	$42.08$	$21.30$	$9.06$	$13.60$	$19.77$	$21.16$
	LESS ( Xia et al., 2024 )	$42.53$	$38.47$	$9.03$	$22.58$	$21.06$	$26.73$
	RDS+ ( Ivison et al., 2025 )	58.91	$51.31$	$7.84$	24.47	21.41	32.79

Table 1: Main results on five tasks with 10K selected samples per task. Training data are selected using Qwen3-1.7B representations and used to finetune Qwen3-1.7B, 4B, and 8B. Due to resource constraints, Top-PPL and Mid-PPL are only evaluated at 1.7B as their performance is relatively low. The best and second-best results within each model size are bold and underlined , respectively. ICLHerd significantly outperforms RDS+, the strongest baseline, under a two-sided binomial test across tasks and model sizes (

p\approx 0.007

Task	Random	Mean-similarity	ICLHerd (ours), $\kappa$						Max-similarity	Round-robin
Task	Random	Mean-similarity	0.01 $\times$	1 $\times$	3 $\times$	10 $\times$	30 $\times$	100 $\times$	Max-similarity	Round-robin
GSM8K	$40.33$	$54.28$	$56.56$	$56.25$	57.47	58.83	$55.88$	$55.88$	$54.74$	$53.53$
MLQA	$48.27$	60.34	60.84	$59.99$	$59.71$	$43.13$	$44.63$	$45.29$	$47.20$	$59.31$

Table 2: Analysis of the concentration parameter

\kappa

using the fusion kernel (

\beta=0.5

, no diversity term). Mean- and max-similarity on concatenated features serve as the approximate extremes for small and large

\kappa

, respectively. Random and round-robin (on fused features) lie outside the

\kappa

continuum and are shown as references.

\kappa

is given as a multiple of the value from the median heuristic ( Gretton et al., 2012 ) .

Method	GSM8K	MLQA	Avg
RDS+	$58.91$	$51.31$	$55.11$
LESS	$42.53$	$38.47$	$40.50$
ICLHerd (hidden states)	61.56	$51.05$	$56.31$
ICLHerd (gradients)	$52.77$	60.19	56.48
ICLHerd (fused)	59.14	60.61	59.88

Table 3: Comparison of feature spaces. RDS+ and LESS use hidden states and gradients, respectively, while ICLHerd (fused) combines both in the kernel.

Method	Arabic	German	English	Spanish	Hindi	Vietnamese	Chinese	Avg
Random	49.18	$46.61$	$59.87$	$44.13$	$35.59$	$53.37$	$49.11$	$48.27$
Top-PPL ( Ankner et al., 2024 )	$46.89$	$48.76$	$63.65$	$50.73$	44.54	54.90	51.97	51.63
Mid-PPL ( Marion et al., 2023 )	$38.98$	$14.39$	$14.60$	$16.20$	$25.76$	$19.70$	$19.47$	$21.30$
LESS ( Xia et al., 2024 )	$47.06$	$38.85$	$35.83$	$26.98$	$33.31$	$48.53$	$38.75$	$38.47$
RDS+ ( Ivison et al., 2025 )	$47.56$	51.57	65.71	52.23	38.89	$52.81$	$50.39$	$51.31$
ICLHerd (ours)	56.82	61.58	75.85	65.57	$36.48$	65.05	62.90	60.61

Table 4: Per-language F1 on MLQA using Qwen3-1.7B with 10k selected samples. The average matches the MLQA column of Table 1 . The best and second-best results are bold and underlined , respectively.

Method	GSM8K			MLQA
	Proxy		Test	Proxy		Test
	ICL (ours)	Train–Val	Test	ICL (ours)	Train–Val	Test
Random	$-3.76$	$46.51$	$40.33$	$-6.93$	$46.39$	$48.27$
Top-PPL	$-3.86$	$16.21$	$13.57$	$-4.93$	$52.25$	51.63
Mid-PPL	$-3.61$	$48.66$	$42.08$	$-7.31$	$21.54$	$21.30$
LESS ( Xia et al., 2024 )	$-3.25$	$47.81$	$42.53$	$-7.93$	$37.53$	$38.47$

Table 5: Comparison of the ICL and Train–Val proxies against test performance. Train–Val involves fine-tuning on selected data followed by dev-set evaluation. The final row shows the Spearman’s

\rho

between each proxy and the test score across the six methods.

Method	GSM8K					MLQA
Method	Train	Infer	Total	Speedup	$\rho$	Train	Infer	Total	Speedup	$\rho$
ICL (ours), $M=1$	–	$0.4$	$0.4$	157 $\times$	$1.00$	–	$0.5$	$0.5$	68 $\times$	$0.89$
ICL (ours), $M=5$	–	$1.9$	$1.9$	31 $\times$	$1.00$	–	$2.5$	$2.5$	14 $\times$	$0.94$
ICL (ours), $M=10$	–	$3.8$	$3.8$	16 $\times$	$1.00$	–	$5.0$	$5.0$	7 $\times$	$0.94$
Training–validation	$33.0$	$26.1$	$59.1$	1 $\times$	$0.94$	$30.0$	$4.3$	$34.3$	1 $\times$	$1.00$

Table 6: Efficiency comparison. Times are in minutes, with inference measured on 1K samples. Speedup is the total time of training–validation divided by that of each method, computed from unrounded times.

\rho

denotes the Spearman correlation relative to the rankings in Table 1 . Experiments use an NVIDIA L40S GPU.

Kernel	GSM8K	MLQA	Avg
Euclidean	$47.84$	$54.97$	$51.41$
Cosine	$49.58$	$55.56$	$52.57$
RBF	58.91	60.04	59.48
vMF (ours)	59.14	60.61	59.88

Table 7: Comparison of kernels within our herding framework, all using fused hidden-state and gradient features.

# Dev samples	GSM8K	MLQA
10	58.98	$60.51$
100	$58.61$	61.23
1,000	$58.61$	$60.52$
Full	59.14	60.61

Table 8: Effect of the dev set size on ICLHerd. Full denotes the entire dev set (7,473 samples for GSM8K and 3,500 for MLQA).

Task	Skill	Dev	Test
GSM8K	Math reasoning	7,473	1,319
MLQA	Multilingual QA	3,500	7,000
DROP	Reading comprehension	5,541	9,536
XSum	Summarization	1,000	1,000
OASST1	Instruction following	500	500

Table 9: Statistics of the evaluation tasks. Dev samples define the target skill for data selection, and test samples are used for the reported results.

Training Data Selection

Training data selection studies how to choose, filter, weight, or schedule examples to improve learning while reducing data and compute costs.

See all 186 papers

BLADE: Scalable Bi-level Adaptive Data Selection for LLM Training
Jun 17, 2026Jiaxing Wang, Deping Xiang, Jin Xu +9Training Data Selection Bilevel Optimization
University of Oxford
Data Efficient Sample Selection for In-Context Learning
Sep 6, 2026V Venktesh, Cem Levi, Avishek AnandIn-Context Example Selection In-Context Learning
Stockholm University · SKIM Group B.V. · TU Delft
Dr. Post-Training: A Data Regularization Perspective on LLM Post-Training
May 8, 2026Pingbang Hu, Xueshen Liu, Z. Morley Mao +1Supervised Fine-Tuning Training Data Selection
University of Illinois Urbana–Champaign · University of Michigan

Jun 17, 2026J/K move · Enter open · S save

BLADE: Scalable Bi-level Adaptive Data Selection for LLM Training

Jiaxing Wang, Deping Xiang, Jin Xu, Zirui Liu+8

University of Oxford

Training Data Selection Bilevel Optimization LLM Training Frank-Wolfe Algorithms

Open paper AskPDF arXiv

As Large Language Model (LLM) datasets scale to trillions of tokens, data selection has emerged as a critical frontier to filter out uninformative noise and construct adaptive learning trajectories. Beyond static heuristic filtering, advanced data selection methods for LLM training largely follow two paradigms, each with fundamental limitations. Influence-based methods provide principled bi-level objectives but require intractable inverse-Hessian computations, while excess-loss methods are computationally efficient but rely on a static reference model that becomes misaligned with the evolving proxy model during training. We propose BLADE (Bi-Level Adaptive Data sElection), a Hessian-free framework for data selection. BLADE reformulates the bi-level optimization problem underlying influence-based methods as a penalized single-level objective via Lagrange multipliers, avoiding inverse-Hessian computation while revealing a principled connection to excess-loss based data selection. The resulting objective recovers an excess-loss form but replaces the static reference model with a dynamic one that stays synchronized with training. Theoretically, we prove that this penalized formulation guarantees first-order convergence. For efficient online batch selection, we instantiate BLADE as a memoryless randomized block-coordinate Frank-Wolfe algorithm. Extensive experiments show that BLADE consistently outperforms state-of-the-art data selection baselines, providing a practical recipe for LLM training.

Abstract

Training Data Selection

Explore similar work

BLADE: Scalable Bi-level Adaptive Data Selection for LLM Training

Data Efficient Sample Selection for In-Context Learning

Dr. Post-Training: A Data Regularization Perspective on LLM Post-Training