Pick the Mixture, Not the Samples: Combinatorial Data Selection for LLMs

Luyang Zhang; Jialu Wang; Bebei Li

cs.LGSep 30, 2026

Pick the Mixture, Not the Samples: Combinatorial Data Selection for LLMs

Authors: Luyang Zhang, Jialu Wang, Bebei Li

Organizations: Carnegie Mellon University · University of California, Santa Cruz

Abstract

Data selection for LLM fine-tuning faces two facts about real training: data pools draw from many distinct domains, and evaluation spans diverse tasks. As a result, the same sample helps different tasks differently, and samples interact during training so that a subset's value is not the sum of its parts. Current data-selection methods leave these realities largely unaddressed. We formalize them as Combinatorial Data Selection (CDS), a framework that separates attribution (how samples are scored) from selection (which subset to pick) and models the non-additive interactions among samples. On a real instruction-tuning data pool, we find substantial cross-domain complementarity (joint utility exceeds the sum of singletons), alongside redundancy within domains. Inspired by this, we propose Mixture-then-Select, a two-stage algorithm that first decides how many samples to take from each domain, then ranks samples within each domain by attribution score. Theoretically, the gap to the optimal subset is linear in the selection size and independent of data pool size. Empirically, Mixture-then-Select outperforms strong baselines by 20 to 54%, is robust across attribution methods, and works across different models and data pool sizes. The framework also extends to the data purchasing problem under a budget constraint when sample prices vary. Our findings suggest that allocating selection across domains offers a promising path toward more efficient and effective data selection for modern LLMs.

Figures & tables

Direction	Method	Interactions	Size	Budget	Guarantee	Mixture
Data attribution	Data Shapley ( Ghorbani and Zou, 2019 )	✗	✗	✗	✗	✗
Data attribution	Per-sample scores	✗	✗	✗	✗	✗
Data selection and distillation	Dataset distillation ( Wang et al., 2020 ; Zhao et al., 2021 )	✗	✓	✗	✗	✗
	$k$ -DPP ( Kulesza and Taskar, 2011 )	✗	✓	✗	✗	✗
	DoReMi ( Xie et al., 2023 ) / RegMix ( Liu et al., 2025 )	✗	✗	✗	✗	✓
	Stratified	✗	✓	✗	✗	✓

Table 1: Comparison with related methods. Columns indicate whether a method models sample-pair interactions, supports cardinality or nonuniform-price budgets, provides a subset-utility approximation guarantee, and optimizes domain allocation. 2 2 2 Similarity kernels and gradient surrogates do not count as Interactions; convergence guarantees on inner optimization do not count as Guarantee.

Figure 1: Mixture-then-Select first uses a learned utility predictor to choose how many samples to take from each domain, then selects samples ranked by attribution score within each domain. The selected subset is used for fine-tuning.

Panel A. Full leaderboard: MtS vs. 11 baselines, each with its native attribution.
Selection	Attribution	Qwen3-8B	Qwen3-8B	Llama-3.2-3B
		$N{=}50K$	$N{=}200K$	$N{=}50K$
		$m{=}1K$	$m{=}3.5K$	$m{=}1K$
MtS	Oracle	$\mathbf{0.694}$	$\mathbf{0.718}$	$\mathbf{0.281}$
Stratified	Oracle	$0.666$	$0.694$	$0.190$
	Oracle	$0.603$	$0.632$	$-0.203$

Table 2: Performance comparison for utility (higher is better). Base models are Qwen3-8B and Llama-3.2-3B-Instruct, both LoRA fine-tuned at rank

r{=}16

. All MtS rows significantly outperform Stratified (paired

t

-test,

p{<}0.05

); per-cell seed counts and

p

-values are in Appendix B .

Method	IFEval	MMLU	GSM8K	ARC-C	HumanEval
MtS ∗	0.5048	0.7254	0.8062	0.5857	0.8049
RegMix	0.4822	0.7136	0.7959	0.5558	0.7703
DoReMi	0.4802	0.7149	0.8016	0.5316	0.7622
LESS †	0.4484	0.7213	0.7930	0.5486	0.7512

Table 3: Downstream benchmarks on Qwen3-8B and the 50K pool. Metrics: IFEval instruction-level strict accuracy; MMLU and GSM8K accuracy; ARC-Challenge normalized accuracy; HumanEval pass@1.

Method	$\boldsymbol{m}$	$\boldsymbol{u}$	GPU-hr
Stratified	1k	$0.666$	$0.58$
Stratified	5k	$0.678$	$2.92$
EL2N	1k	$0.567$	$1.43$
MtS	1k	$\mathbf{0.694}$	$\mathbf{16.58}$
MtS	5k	$0.691$	$18.92$
Full data	50k	$\mathbf{0.548}$	$\mathbf{29.17}$

Table 4: Compute-utility table at

N{=}50K

on Qwen3-8B LoRA

r{=}16

. Total compute

=

selection-side

+

5

-seed training. Per-row breakdown: Table 10 ; visual scatter: Figure 3 .

Figure 2: Diagnostic panels on Qwen3-8B. (A) Domain proportions across the four selection rules at

N{=}50K

m{=}1000

. (B) MtS

-

Stratified utility gap vs. LoRA rank

r

, with (

m{=}1K

N{=}50K

;

m{=}3.5K

N{=}200K

). For other setting, see Appendix B .

Method	$B{=}15\%$	$B{=}20\%$	$B{=}25\%$
Additive LP	$0.197$	$0.146$	$0.137$
MtS	$\mathbf{0.229}$	$\mathbf{0.242}$	$\mathbf{0.249}$
Oracle	$0.236$	$0.246$	$0.250$
$\Delta$ MtS $-$ Additive LP	$0.033$	$0.097$	$0.111$
Optimality recovery	$89.5\%$	$95.6\%$	$98.3\%$

Table 5: Data purchasing under a budget: aggregate utility

u(S)

on Llama-3.2-3B-Instruct, averaged over

10

pricing instances.

B

is the monetary budget as a fraction of total cost. Recovery rate is the share of the optimality gap closed by MtS over the additive baseline. Details: Appendix B.14 .

Appendix figures & tables32 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Description
$\mathcal{D}=\{1,\ldots,N\}$	Data catalog
$p_{j}$	Cost of sample $j$
$B$	Selection budget
$u(S)$	True utility of subset $S$
$v_{j}=u(\{j\})$	Individual value of sample $j$
$\varphi_{ij}$	Pairwise interaction ( $<0$ : substitutes, $>0$ : complements)

Appendix

Table 8

Form / quantity	Form	Raw value	Ratio vs. entrywise
Entrywise (centered)	$(\beta_{w}+\delta_{p})\,m(m-1)\,v_{\max}$	$\approx 1255\,v_{\max}$	$1\times$ (baseline)
Spec-Main	$\sum_{c}\beta_{w,\text{op}}^{(c),\text{cent}}n_{c}^{*}v_{\max}+\delta_{\text{op}}mv_{\max}$	$\approx 25\,v_{\max}$	$\mathbf{51\times}$
$\mathcal{R}(m)$ component breakdown	$\nu_{\text{op}}^{\perp}mv_{\max}+\zeta_{\Delta}^{\text{loc}}+2\zeta_{\text{pred}}v_{\max}$	$\approx 37\,v_{\max}$	N/A
$S_{\text{HMtS}}$ vs. $S_{\text{MtS}}$ on benchmarks	$\arg\max\{u(S_{\text{MtS}}),u(S_{\text{IA}})\}$	$S_{\text{HMtS}}=S_{\text{MtS}}$ (all benchmarks)	N/A

Appendix

Table 6: Spectral versus entrywise additive bounds for the empirical interaction matrix

\hat{\Phi}

m{=}1000

\mathcal{R}(m)

is the finite-sample error in Eq. ( 4 ); dashes indicate terms with no entrywise counterpart.

Variant	Evaluations	$\hat{g}$ loss vs. exact	Guarantee
Exact IA-Greedy	$O(N^{2}m)$	0%	$(1{-}e^{-\gamma_{\text{op}}})$
Lazy IA-Greedy	$\boldsymbol{O(Nk)}$ avg	0 to 1.3%	empirical
Stochastic IA-Greedy	$O(Nk\log N)$	2 to 5%	$(1{-}e^{-\gamma_{\text{op}}}{-}\varepsilon)$

Appendix

Table 7: IA-Greedy variants on the 40-sample pilot. Columns report evaluations per step, loss in

\tilde{u}

relative to exact IA-Greedy, and the approximation guarantee.

Setting	Specification
Data	Alpaca ( Taori et al., 2023 ) , GSM8K ( Cobbe et al., 2021 ) , HumanEval ( Chen et al., 2021 ) , ARC-Easy ( Clark et al., 2018 ) .
Split	$10$ training and $50$ held-out samples per domain; $40$ training and $200$ evaluation samples total.
Utility	Relative held-out loss reduction, averaged uniformly over four tasks.
Pythia-1B	Full fine-tuning; $3$ epochs; learning rate $2\times 10^{-5}$ .
Qwen3-1.7B/8B	LoRA rank $16$ , $\alpha{=}32$ , $q/v$ projections; $3$ epochs; learning rate $2\times 10^{-4}$ .
Precision / hardware	bf16; NVIDIA L40S ( $48$ GB).

Appendix

Table 8: Pilot and scaled-proxy settings.

Method	Wall-clock	GPU-hours
Mixture-then-Select (calibration step, 8 domain fine-tunes in parallel)	$\sim 2$ hr	$\sim 16$
LESS extract_train (80 chunks, 8 GPUs parallel)	$\sim 35$ hr	$\sim 280$
DEITA (scoring)	$5.7$ hr	$5.7$
EL2N (scoring)	$3.4$ hr	$3.4$
Facility Location (embedding + greedy)	$3.0$ hr	$3.0$
Superfiltering (scoring)	$0.8$ hr	$0.8$

Appendix

Table 9: Selection compute on Qwen3-8B at

N{=}200K

, using up to eight NVIDIA L40S GPUs. Final LoRA training is excluded and uses the same recipe across methods at each selection size

m

Figure 3: Compute and utility comparison on Qwen3-8B at

N{=}50K

with five representative baselines. Total compute includes selection and five-seed training ( Table 10 ). The dashed grey line is the Pareto frontier; the dotted red line marks MtS at

m{=}1K

Method	$\boldsymbol{m}$	$\boldsymbol{u}$	select	train (5-seed)	total	$\boldsymbol{\Delta}$ vs. MtS m=1000
			(GPU-hr)	(GPU-hr)	(GPU-hr)
Stratified (oracle)	$1000$	$0.666$	$0.00$	$0.58$	$0.58$	$-0.028$
Greedy (oracle)	$1000$	$0.598$	$0.00$	$0.58$	$0.58$	$-0.096$
Random	$1000$	$0.549$	$0.00$	$0.58$	$0.58$	$-0.145$
$k$ -DPP	$1000$	$0.581$	$0.01$	$0.58$	$0.59$	$-0.113$
Superfiltering	$1000$	$0.534$	$0.20$	$0.58$	$0.78$	$-0.160$

Appendix

Table 10: Full compute and utility comparison on Qwen3-8B at

N{=}50K

, LoRA

r{=}16

. Total compute includes selection and five-seed training at

0.117

GPU-hr per seed per 1K samples. Selection costs from Table 9 are scaled to 50K for per-sample methods; MtS calibration is unchanged.

Base model	Cross-domain $\delta{>}0$	$\boldsymbol{\beta}$	$2/(2+\beta)$	Tasks complementary
Pythia-1B (full FT)	0/60 (0%)	$0.0$	$1.000$	$0/4$
Qwen3-1.7B (LoRA)	0/60 (0%) agg; 60/60 HE	${\approx}0$	${\approx}1$	$1/4$ (HumanEval)
Qwen3-8B (LoRA)	38/60 (63%)	$\mathbf{1.32{\pm}0.60}$	$\mathbf{0.60}$	$\mathbf{3/4}$

Appendix

Table 11: Interaction structure by base model in the 40-sample pilot.

\beta

measures pairwise complementarity;

2/(2+\beta)

is a singleton-based diagnostic, not a certified submodularity-ratio bound without conditional-marginal control. Complementary tasks have positive aggregate

\delta^{(t)}

. Qwen3-8B

\beta

is the mean

\pm

standard deviation over four seeds and 30 strategic pairs.

Eval task	Positive %	Mean $\delta^{(t)}$	Per-task $\beta$
Alpaca (instruction)	$76.7\%$	$0.0027$	$3.06$
GSM8K (math)	$70.0\%$	$0.0018$	$6.66$
HumanEval (code)	$73.3\%$	$0.0033$	$3.27$
ARC (reasoning)	$43.3\%$	$-0.0038$	$4.45$

Appendix

Table 12: Cross-domain complementarity by task in the 40-sample Qwen3-8B LoRA pilot. Each task uses 60 cross-domain pairs;

\delta^{(t)}

is measured on that task’s held-out data.

Figure 4: Complementarity in the 40-sample pilot. (A) Pairwise complementarity

\beta

across three base models; Qwen3-8B shows the mean

\pm

standard deviation over four seeds and 30 strategic pairs. (B) Fraction of pairs with positive interaction on each Qwen3-8B evaluation task, using 60 pairs per task. The dashed line marks 0.5.

Method	$m{=}5$	$m{=}10$	$m{=}15$	$m{=}20$
Random	$.143{\scriptstyle\pm.001}$	$.217{\scriptstyle\pm.003}$	$.271{\scriptstyle\pm.003}$	$.328{\scriptstyle\pm.002}$
Greedy-by-Value	$.146{\scriptstyle\pm.001}$	$.227{\scriptstyle\pm.003}$	$.282{\scriptstyle\pm.003}$	$.317{\scriptstyle\pm.003}$
EL2N ( Paul et al., 2021 )	$.107{\scriptstyle\pm.006}$	$.183{\scriptstyle\pm.005}$	$.215{\scriptstyle\pm.003}$	$.297{\scriptstyle\pm.005}$
Facility Location	$.141{\scriptstyle\pm.002}$	$.205{\scriptstyle\pm.003}$	$.260{\scriptstyle\pm.003}$	$.313{\scriptstyle\pm.004}$
CRAIG ( Mirzasoleiman et al., 2020 )	$.138{\scriptstyle\pm.005}$	$.219{\scriptstyle\pm.007}$	$.267{\scriptstyle\pm.006}$	$.319{\scriptstyle\pm.003}$
LESS ( Xia et al., 2024 )	$.141{\scriptstyle\pm.001}$	$.217{\scriptstyle\pm.002}$	$.268{\scriptstyle\pm.003}$	$.318{\scriptstyle\pm.005}$

Appendix

Table 13: Selection-algorithm comparison in the 40-sample Qwen3-8B LoRA pilot, with uniform task weights. Entries are means

\pm

standard deviations over five seeds. Best values are bold; second-best values are underlined.

Diagnostic	Result
Triplet variance explained (approximately 100 triplets)	90%
Singleton-only prediction MAE	0.014
Singleton + pairwise prediction MAE	0.004
Cross-task-group Pearson correlation (4,998 pair edges)	0.903

Appendix

Table 14: Direct interaction diagnostics. Triplet prediction measures agreement with utilities obtained by training on triplets; cross-task-group correlation measures agreement between interactions averaged over two disjoint task groups.

Proxy	$\rho_{\text{all}}$	$\rho_{\text{cross}}$	$\rho_{\text{within}}$	AUC
Gradient cosine ( $7.7$ M dim)	$0.182$	$0.020$	$0.769$	N/A
Calibrated predictor (Ridge)	$\mathbf{0.832}$	$\mathbf{0.791}$	$\mathbf{0.925}$	$\mathbf{0.907}$
Calibrated predictor (GBR)	$0.837$	$0.802$	$0.913$	$0.909$

Appendix

Table 15: Prediction of pairwise interactions at

N{=}864

, using five-fold cross-validation. Spearman

\rho

measures ranking agreement over all, cross-domain, and within-domain pairs. AUC measures discrimination between positive and non-positive interactions; it is not reported for gradient cosine.

Method	$m{=}500$	$m{=}750$	$m{=}1000$	$m{=}1250$	$m{=}2500$	$m{=}5000$	$m{=}10000$
Mixture-then-Select	$0.601{\scriptstyle\pm.039}$	$0.671{\scriptstyle\pm.020}$	$0.694{\scriptstyle\pm.012}$	$0.700{\scriptstyle\pm.011}$	$0.686{\scriptstyle\pm.020}$	$0.691{\scriptstyle\pm.018}$	$0.658{\scriptstyle\pm.022}$
D1 (set-level)	$0.605{\scriptstyle\pm.022}$	$0.640{\scriptstyle\pm.014}$	$0.693{\scriptstyle\pm.017}$	$0.699{\scriptstyle\pm.006}$	$0.687{\scriptstyle\pm.015}$	$0.690{\scriptstyle\pm.018}$	$0.654{\scriptstyle\pm.021}$
Stratified	$0.666{\scriptstyle\pm.011}$	${\approx}0.666$	$0.666{\scriptstyle\pm.011}$	${\approx}0.680$	$0.693{\scriptstyle\pm.014}$	$0.664{\scriptstyle\pm.009}$	$0.629{\scriptstyle\pm.013}$
$\Delta$ (Mixture-then-Select $-$ Strat)	$-0.065$	${\approx}{+}0.005$	$\mathbf{+0.028}$	${\approx}{+}0.020$	$-0.007$	$+0.027$	$+0.029$
$p$ (paired $t$ )	$0.018$ (loss)	$\approx$ tie	$\mathbf{0.0013}$	$0.052$	$0.53$ (tie)	$0.41$ (tie)	$0.31$ (tie)

Appendix

Table 16: Selection-budget sweep on Qwen3-8B with

N{=}50K

and LoRA

r{=}16

. Entries are means

\pm

standard deviations over five seeds, except MtS at

m{=}1K

(ten seeds). Bottom rows give MtS’s utility difference and paired

p

-value against Stratified at the same budget.

Variant	Features	mean $\pm$ std	$\Delta$ vs. Full
D1 Full ( $11$ features)	domain-hist + $H(\pi)$ + $v$ -mean + $v$ -std	$\mathbf{0.6927{\scriptstyle\pm.017}}$	$\mathbf{0}$
D1 $\pi$ -only	domain-histogram only ( $8$ features)	$0.6044{\scriptstyle\pm.013}$	$-0.088$
D1 $v$ -stats only	mean + std of $v_{\text{agg}}$ ( $2$ features)	$0.6299{\scriptstyle\pm.018}$	$-0.063$

Appendix

Table 17: Predictor-feature ablation for the D1 set-level variant on Qwen3-8B, with LoRA

r{=}16

N{=}50K

, and

m{=}1K

. Entries are means

\pm

standard deviations over five seeds;

\Delta

is the signed utility difference from the full-feature variant.

Figure 5: Robustness across capacity, attribution, and model scale. (A) MtS, Stratified, and LESS across LoRA ranks on Qwen3-8B (

N{=}50K

m{=}1K

); five seeds, except MtS at

r{=}16

(ten). (B) Three attribution signals crossed with three selection rules on Qwen3-8B (

N{=}50K

m{=}5K

). (C) MtS’s utility difference from the strongest baseline across pools and models. Llama-3.2-3B uses the Qwen mixture with Llama values ( native ) or Qwen values ( transfer ); 200K Greedy uses five seeds.

{}^{*}:p{<}0.05

;

{}^{**}:p{<}0.01

; ns:

p{\geq}0.05

Condition	$\Delta$	$\boldsymbol{p}$	Status
Oracle, $m{=}1000$ , $r{=}8$	$0.008$	$0.49$	tie
Oracle, $m{=}1000$ , $r{=}16$	$0.028$	$0.0013$	✓sig
Oracle, $m{=}1000$ , $r{=}32$	$\mathbf{+0.046}$	$\mathbf{0.003}$	✓sig
Oracle, $m{=}1000$ , $r{=}16$ , lr ${=}10^{-4}$	$0.012$	$0.27$	trending
Oracle, $m{=}5000$	$0.027$	$0.008$	✓sig
LESS, $m{=}1000$	$0.015$	$0.020$	✓sig (paired, $n{=}10$ )

Appendix

Table 18: MtS versus Stratified across 15 combinations of model, rank, budget, pool size, and attribution. The default is Qwen3-8B with LoRA

r{=}16

\Delta

is the utility difference within each condition;

p

is from a paired

t

-test.

EL2N attribution ( $n{=}10$ seeds per cell)
Budget	MtS	Strat	Greedy	$\boldsymbol{\Delta}_{\text{MtS}-\text{Strat}}$	$\boldsymbol{\Delta}_{\text{MtS}-\text{Greedy}}$
$m{=}3500$ ( $1.75\%$ )	$\mathbf{0.6774\scriptstyle\pm 0.026}$	$0.6387\scriptstyle\pm 0.030$	$0.5983\scriptstyle\pm 0.024$	$\mathbf{+0.039}$	$\mathbf{+0.079}$
$m{=}10000$ ( $5\%$ )	$\mathbf{0.6447\scriptstyle\pm 0.027}$	$0.6296\scriptstyle\pm 0.024$	$0.6312\scriptstyle\pm 0.016$	$0.015$	$0.014$
$m{=}20000$ ( $10\%$ )	$\mathbf{0.6230\scriptstyle\pm 0.022}$	$0.5688\scriptstyle\pm 0.020$	$0.6108\scriptstyle\pm 0.024$	$\mathbf{+0.054}$	$0.012$
LESS attribution ( $n{=}5$ seeds per cell)
$m{=}3500$ ( $1.75\%$ )	$\mathbf{0.6251\scriptstyle\pm 0.033}$	$0.6145\scriptstyle\pm 0.010$	$0.5796\scriptstyle\pm 0.027$	$0.011$	$\mathbf{+0.046}$

Appendix

Table 19: Attribution and selection rules on Qwen3-8B with

N{=}200K

and LoRA

r{=}16

. EL2N results use ten seeds per cell; LESS results use five seeds per cell and approximately 280 selection GPU-hours per cell.

Domain assignment	Measurement	Result
Original eight domains	Utility gain over Stratified	4.3%
Four coarsened domains	Utility gain over Stratified	3.5%
Eight utility-derived clusters	Utility gain over Stratified	3.2%
20% corrupted labels	Four-task composite change	$-0.7\%$

Appendix

Table 20: Robustness to alternative and perturbed domain assignments. The first three rows report relative utility gains over Stratified; the final row reports the relative downstream change from clean labels.

Assignment approach	Mean Jaccard
Perturbed labels, no reassignment	0.403
Combined embedding and utility similarity	0.806
Embedding similarity to domain averages	0.762
Utility-profile similarity to domain averages	0.749

Appendix

Table 21: Recovery of the clean selection when 30% of samples have two possible domain labels. Jaccard is the intersection size divided by the union size of the repaired and clean selections, averaged over three perturbation seeds.

Ambiguous samples	Method	Held-out utility	Equal4
10%	Stratified	0.2886	0.6336
	MtS, $\lambda=0.50$	0.2957	0.6506
30%	Stratified	0.28856	0.62933
	MtS, $\lambda=0.50$	0.29746	0.64454

Appendix

Table 22: End-to-end evaluation of selected overlap conditions with embedding-based reassignment and fresh calibration. Entries are three-seed means. Equal4 is the mean of IFEval, MMLU, GSM8K, and normalized ARC-C. The tested variant is lambda-0.50, not original MtS.

Method	$T$	$n$	$u(S)$	$\Delta$ vs. Stratified
Mixture-then-Select	8	15	$0.6943\scriptstyle\pm 0.014$	$\mathbf{+0.0288}$ ( $p{=}0.041$ )
Stratified	8	5	$0.6662\scriptstyle\pm 0.012$	N/A
Mixture-then-Select- $T{=}4$	4	5	$0.6461\scriptstyle\pm 0.016$	$\mathbf{+0.0217}$ ( $p{=}0.132$ )
Stratified- $T{=}4$	4	5	$0.6245\scriptstyle\pm 0.020$	N/A

Appendix

Table 23: Sensitivity to domain count on Qwen3-8B with LoRA

r{=}16

N{=}50K

, and

m{=}1K

. Results use five paired seeds; headline MtS at

T{=}8

uses 15 with augmented data. Four-domain calibration uses fresh per-domain fine-tunes on the top-125 samples by

v_{\mathrm{agg}}

\Delta

is relative to Stratified; reference cells are N/A.

Method	math	code	QA	dlg	rsn	nlp	cpx	crv	agg
Mixture-then-Select	$\mathbf{+0.766}$	$\mathbf{+0.732}$	$\mathbf{+0.717}$	$\mathbf{+0.703}$	$\mathbf{+0.692}$	$\mathbf{+0.651}$	$\mathbf{+0.698}$	$\mathbf{+0.582}$	$\mathbf{+0.693}$
Stratified	$0.736$	$0.653$	$0.622$	$0.610$	$0.662$	$0.510$	$0.629$	$0.540$	$0.620$
Greedy (oracle)	$0.646$	$0.609$	$0.611$	$0.601$	$0.632$	$0.517$	$0.641$	$0.524$	$0.598$
Greedy + EL2N	$0.687$	$0.655$	$0.622$	$0.599$	$0.610$	$0.563$	$0.593$	$0.507$	$0.605$
LESS	$0.675$	$0.617$	$0.607$	$0.577$	$0.611$	$0.462$	$0.601$	$0.485$	$0.579$
Superfiltering	$0.658$	$0.615$	$0.595$	$0.547$	$0.565$	$0.466$	$0.565$	$0.423$	$0.554$

Appendix

Table 24: Per-domain relative loss reduction

u_{c}

on Qwen3-8B at

N{=}50K

m{=}1K

, averaged over seeds. Per-method seed counts follow Table 2 ; the mean column gives aggregate utility.

Method	Uniform	math 3 $\times$	code 3 $\times$	reason 3 $\times$	complex 3 $\times$
Mixture-then-Select	$\mathbf{+0.693}$	$\mathbf{+0.707}$	$\mathbf{+0.700}$	$\mathbf{+0.693}$	$\mathbf{+0.694}$
Stratified	$0.620$	$0.643$	$0.627$	$0.628$	$0.622$
Greedy (oracle)	$0.598$	$0.608$	$0.600$	$0.605$	$0.606$
Greedy + EL2N	$0.605$	$0.621$	$0.615$	$0.606$	$0.602$
LESS	$0.579$	$0.598$	$0.587$	$0.586$	$0.584$
Superfiltering	$0.554$	$0.575$	$0.566$	$0.556$	$0.556$

Appendix

Table 25: Aggregate utility under five task-weighting schemes on Qwen3-8B at

N{=}50K

m{=}1K

. A scheme labeled

x

3{\times}

gives domain

x

weight 3 and each other domain weight 1, then normalizes the weights to sum to 8.

Model	IFEval	MMLU	GSM8K	ARC-C
Qwen3-8B	0.4894	0.7061	0.7973	0.5660
Qwen2-72B base	0.3777	0.8376	0.8764	0.6348

Appendix

Table 26: Benchmark evaluation of the frozen Qwen3-8B selection on Qwen2-72B, without recalibration or reselection. The models differ in both size and instruction alignment.

Method	Mean	Std	$\Delta$ vs. Strat	$p$ (paired)
Mixture-then-Select (Qwen $\pi^{*}$ + Llama $v_{\text{agg}}$ )	$\mathbf{+0.281}$	$0.035$	$\mathbf{+0.091}$	$\mathbf{0.0067}$
Stratified (Llama $v_{\text{agg}}$ within-domain)	$0.190$	$0.051$	N/A	N/A
Random (Qwen-selection-transfer)	$0.076$	$0.056$	$-0.114$	$0.012$
Facility Location (transfer)	$-0.016$	$0.051$	$-0.206$	$<0.001$
DEITA (transfer) †	$-0.109$	$0.089$	$-0.299$	$<0.001$
$k$ -DPP (transfer)	$-0.101$	$0.038$	$-0.291$	$<0.001$

Appendix

Table 27: Utility on Llama-3.2-3B with LoRA

r{=}16

and

m{=}1K

. Entries are means

\pm

standard deviations over five seeds (

\dagger

: ten seeds). Utility differences and paired tests use Stratified as the reference.

Method	math	code	gen. qa	dialog	reason.	nlp	cmplx	creative	agg
Mixture-then-Select	$0.325$	$0.228$	$0.284$	$0.206$	$0.295$	$0.244$	$0.218$	$0.210$	$\mathbf{+0.251}$
Stratified	$0.080$	$0.173$	$0.176$	$0.126$	$0.168$	$0.177$	$0.158$	$0.158$	$0.152$
LESS	$-0.325$	$-0.112$	$-0.156$	$-0.184$	$-0.141$	$-0.137$	$-0.168$	$-0.120$	$-0.168$
EL2N	$-0.306$	$-0.188$	$-0.297$	$-0.254$	$-0.207$	$-0.252$	$-0.191$	$-0.187$	$-0.235$

Appendix

Table 28: Per-domain utility on Llama-3.2-3B for seed 42. Utility is

u_{c}=(L_{c}^{\mathrm{base}}-L_{c}^{S})/L_{c}^{\mathrm{base}}

, where the two losses are measured on held-out domain-

c

data before and after fine-tuning on subset

S

Figure 6: Per-domain utility on Llama-3.2-3B at

N{=}50K

m{=}1K

, for seed 42. Numerical values are in Table 28 .

Figure 7: Selected domain proportions on Qwen3-8B at

N{=}200K

m{=}3.5K

. The dashed line shows uniform allocation (

1/8

); the corresponding 50K comparison is in Figure 2 A.

$N$	LoRA $r$	$\Delta$ (MG $-$ Strat)	paired $p$	$n$
50K	16	$+$ 0.099	$<$ 0.001	10
100K	8	$+$ 0.036	0.062	5
200K	8	$+$ 0.053	0.001	5

Appendix

Table 29: MtS’s utility difference from Stratified across pool sizes. The 50K condition uses LoRA

r{=}16

; 100K and 200K use

r{=}8

, so pool size and rank vary together.

Setting	Specification
Pool	Math, code, general QA; five vendors per cluster, $50$ samples per indivisible bundle ( $750$ samples total).
Prices / budgets	One price per cluster, $p_{c}\sim\mathcal{U}[0.5,1.5]$ independently; budgets $15\%,20\%,25\%$ of $\sum_{c}5p_{c}$ ; ten price/utility shuffles shared across methods.
Training	Llama-3.2-3B-Instruct, LoRA rank $16$ , three epochs, learning rate $2\times 10^{-4}$ ; five training seeds.
Candidate allocations	All feasible cluster counts $n_{c}\in\{0,\ldots,5\}$ , selecting the top $n_{c}$ vendors by individual utility within each cluster.
Additive LP	Maximizes $\sum_{c}V_{c}(n_{c})$ , where $V_{c}$ is five-seed held-out utility from training on that cluster alone.
Joint-aware MtS	Scores the same feasible allocations by measured joint-training utility.

Appendix

Table 30: Bundle-pricing settings and comparison definitions.

inst	$B\%$	Add. LP	MtS (joint-aware)	Oracle	$\Delta$ MtS $-$ Add	Recovery
$0$	$15$	$0.172$	$\mathbf{0.201{\pm}0.014}$	$0.229$	$+0.029$	$50.9\%$
$0$	$20$	$0.139$	$\mathbf{0.235{\pm}0.009}$	$0.236$	$+0.096$	$99.7\%$
$0$	$25$	$0.140$	$\mathbf{0.240{\pm}0.022}$	$0.240$	$+0.100$	$100\%$
$1$	$15$	$0.180$	$\mathbf{0.215{\pm}0.015}$	$0.234$	$+0.034$	$64.0\%$
$1$	$20$	$0.136$	$\mathbf{0.215{\pm}0.015}$	$0.234$	$+0.079$	$80.3\%$
$1$	$25$	$0.136$	$\mathbf{0.247{\pm}0.014}$	$0.247$	$+0.112$	$100\%$

Appendix

Table 31: Per-instance data-pricing results on Llama-3.2-3B-Instruct with LoRA

r{=}16

. Entries are means

\pm

standard deviations over five seeds, covering 10 instances at each of three budgets.

Data Selection

Data selection chooses informative, representative, or reliable examples to improve model training and evaluation while reducing data, compute, or labeling costs.

See all 85 papers

May 28, 2026

The Long-Term Effects of Data Selection in LLM Fine-Tuning

Data selection is increasingly used to reduce the cost of large language model (LLM) fine-tuning, with recent methods prioritizing samples by current utility, diversity, quality, or influence. This paper studies a different question: when fine-tuning occurs over multiple stages, can selection strategies that look optimal now make the model less adaptable later? We introduce a long-horizon view of LLM data selection in which a selector is evaluated not only by immediate task performance, but also by future adaptation speed, forgetting, capability imbalance, and out-of-distribution robustness. We compare representative random, loss-based, gradient-based, diversity-based, quality-based, and utility-diversity selection families under a unified multi-stage protocol. Through controlled experiments designed to instantiate this protocol, we show how short-term selectors can exhibit rank reversal: they improve the current stage while slowing subsequent learning and increasing forgetting. We formalize this behavior as myopic selection, provide a simple local analysis of why it can occur, and propose a diagnostic Long-Horizon Aware Selection (LHAS) objective that augments immediate utility with coverage, future-proxy transfer, and anti-concentration terms. The study argues that data selection should be evaluated as a training intervention that shapes the model's learning trajectory, rather than only as a local data-efficiency mechanism.

Yuxin Yang, Aoxiong Zeng, Xiangquan Yang

Shanghai University · East China Normal University

Continual Learning for LLMs LLM Fine-Tuning Data Selection Training Data Selection Selective Fine-Tuning Data Selection for Language Model Fine-Tuning

Ask PDF

Jun 17, 2026

BLADE: Scalable Bi-level Adaptive Data Selection for LLM Training

As Large Language Model (LLM) datasets scale to trillions of tokens, data selection has emerged as a critical frontier to filter out uninformative noise and construct adaptive learning trajectories. Beyond static heuristic filtering, advanced data selection methods for LLM training largely follow two paradigms, each with fundamental limitations. Influence-based methods provide principled bi-level objectives but require intractable inverse-Hessian computations, while excess-loss methods are computationally efficient but rely on a static reference model that becomes misaligned with the evolving proxy model during training. We propose BLADE (Bi-Level Adaptive Data sElection), a Hessian-free framework for data selection. BLADE reformulates the bi-level optimization problem underlying influence-based methods as a penalized single-level objective via Lagrange multipliers, avoiding inverse-Hessian computation while revealing a principled connection to excess-loss based data selection. The resulting objective recovers an excess-loss form but replaces the static reference model with a dynamic one that stays synchronized with training. Theoretically, we prove that this penalized formulation guarantees first-order convergence. For efficient online batch selection, we instantiate BLADE as a memoryless randomized block-coordinate Frank-Wolfe algorithm. Extensive experiments show that BLADE consistently outperforms state-of-the-art data selection baselines, providing a practical recipe for LLM training.

Jiaxing Wang, Deping Xiang, Jin Xu +9

University of Oxford

Training Data Selection Bilevel Optimization LLM Training Frank-Wolfe Algorithms

Ask PDF

Mar 25, 2026

Data Mixing for Large Language Models Pretraining: A Survey and Outlook

Large language models (LLMs) rely on pretraining on massive and heterogeneous corpora, where training data composition has a decisive impact on training efficiency and downstream generalization under realistic compute and data budget constraints. Unlike sample-level data selection, data mixing optimizes domain-level sampling weights to allocate limited budgets more effectively. In recent years, a growing body of work has proposed principled data mixing methods for LLM pretraining; however, the literature remains fragmented and lacks a dedicated, systematic survey. This paper provides a comprehensive review of data mixing for LLM pretraining. We first formalize data mixture optimization as a bilevel problem on the probability simplex and clarify the role of data mixing in the pretraining pipeline, and briefly explain how existing methods make this formulation tractable in practice. We then introduce a fine-grained taxonomy that organizes existing methods along two dimensions: static versus dynamic mixing. Static mixing is further categorized into rule-based and learning-based methods, while dynamic mixing is grouped into adaptive and externally guided families. For each class, we summarize representative approaches and analyze their strengths and limitations from a performance-cost trade-off perspective. Building on this analysis, we highlight challenges that cut across methods, including limited transferability across data domains, optimization objectives, models, and validation sets, as well as unstandardized evaluation protocols and benchmarks, and the inherent tension between performance gains and cost control in learning-based methods. Finally, we outline several exploratory directions, including finer-grained domain partitioning and inverse data mixing, as well as pipeline-aware designs, aiming to provide conceptual and methodological insights for future research.

Zhuo Chen, Yuxuan Miao, Supryadi +1

The International Joint Institute of Tianjin University, Tianjin University, Tianjin 300072, China · TJUNLP Lab, School of Computer Science and Technology, Tianjin University, Tianjin 300350, China

Data Mixture Optimization Language Model Pretraining

Ask PDF

Abstract

Data Selection

Explore similar work

The Long-Term Effects of Data Selection in LLM Fine-Tuning

BLADE: Scalable Bi-level Adaptive Data Selection for LLM Training

Data Mixing for Large Language Models Pretraining: A Survey and Outlook