Supervised fine-tuning over a large pool of heterogeneous tasks is often computationally expensive, leading practitioners to train language models on a fixed training budget of carefully chosen SFT tasks. Thus, achieving strong downstream performance hinges on selecting an effective subset of tasks, or equivalently, identifying an optimal task mixture under a fixed budget constraint. In practice, supervised finetuning mixtures are often determined using simple heuristics (e.g., uniform or size-proportional sampling) that ignore task interactions, which can hurt transfer and waste budget on redundant task sources. We introduce TASKMIXPGM, a framework for learning continuous task mixtures through energy-based modelling over SFT tasks. Each task forms a node in a Markov random field, with unary potentials capturing task utility and pairwise potentials modelling inter-task relationships via behavioral divergences between predictive distributions of single-task fine-tuned models. Optimizing this objective yields task mixtures that balance utility, coverage, and redundancy. We further show that the induced set function defined over the set of such tasks is weakly submodular under budget constraints, enabling approximation guarantees for discrete task selection. Empirically, we provide a fixed, benchmark-independent mixture that improves over heuristic mixing across multiple model families and different training budgets. TASKMIXPGM also uncovers interpretable structure in task interactions, revealing complementarities and redundancies that inform effective SFT mixtures.
Figures & tables
Figure 1: Illustrative example. Left: a six-task MRF whose edges are labelled with similarities Sij (thicker edges for high Sij , dashed for low). Right: the optimal mixture p∗ from Eq. ( 3 ) for these similarities; darker shading marks larger weight in both panels. The most connected task T6 receives the most mass, while the strongly redundant pairs ( T1,T2 and T4,T5 , both with Sij=0.8 ) split theirs.
Figure 2: Effect of the spectrum shift τ in S+τI on the optimal mixture ( n=309 , ρ=0.02424 ). Shaded: τ≤τc , where the objective is not convex on Δn . Dashed lines show the local minimum reached from p∗(τ0) . At τ=0 the solution is a single task. Per-collection mass is in Figure 9 ; discussion in Section C.3 .
Figure 3: HeatMap of PMI Affinity Matrix across all 309 tasks as specified in Appendix K
Figure 4: Incremental composition. Solid: median over five random insertion orders (band: range). Dashed: greedy order. TV curves are a 15-step moving average. Markers: steps at which the unshifted ( τ=0 ) mixture changes, each moving all mass to the new task. Grey dashed line in (c): all k tasks active.
Table 1: Instruct-tuning performance on Leaderboard benchmarks (25K and 50K samples).
Size ( B )
Method
Reasoning
Knowledge
Math
Instr.
BBH
MuSR
GPQA
MMLU-Pro
MATH
IFEval
Avg.
25K
Random ( 3 )
57.89
38.76
36.07
31.10
27.02
46.16
39.50
Uniform ( 1 )
53.08
39.02
31.29
23.74
28.25
45.08
36.74
EPM ( 2 )
55.86
39.42
30.62
25.65
25.73
44.48
36.96
LESS ( xia2024less )
61.62
44.90
36.58
51.78
29.68
45.29
44.98
SMART ( renduchintala-etal-2024-smart )
40.37
39.02
29.11
12.77
27.04
44.12
32.07
Table 2: Qwen3.5-9B
Figure 5: The 15 highest-weighted tasks, coloured by collection, with each task’s rank by degree S[i] . The full structure of the mixture, including the affinity matrix reordered by weight, is analysed in Section E.1 .
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Weak submodularity of f . (a) Gain of five fixed tasks along greedy prefixes at τ0 (legend: greedy position of j ). (b) Gap f(V)−f(Ak) for exact greedy (solid) and random orders (dashed, band over 5 random orders). (c) γ^A,S ( 1 ) on the 180 random-set pairs per τ ( 12 draws of A per size, ∣S∣∈{2,5,10} ). Pairs with f(Aˉ∪Sˉ)=f(Aˉ) are omitted ( 0 , 1 and 105 for τ0 , 50 , 0 ). (d) mA/MA for (S+τI)A along the τ0 greedy order, for τ∈{τ0,50,τc,0} , with τc=17.09 the critical shift of ( 4 ). This principal-submatrix ratio is a conservative proxy for the zero-sum constants of Theorem 1 ; below zero (shaded) the proxy gives no guarantee.
Figure 7: Eigen-structure of the PMI affinity matrix S ( n=309 ). (a) Sorted eigenvalues (blue: positive, orange: negative), with an inset on ranks 6 – 304 . S is strongly indefinite, with one outlier λmin=−τ0=−186.3 ; the next eigenvalue is −15.4 . (b) Fraction of ∥S∥F2 captured by the k largest-magnitude eigenpairs. The outlier alone carries 86.8% ; 2 eigenpairs give 90% and 13 give 99% (entropy effective rank 28.5 ). (c) The 10 smallest eigenvalues of S and of its restriction B⊤SB to 1⊥ . Restriction removes the outlier and raises the threshold from −τ0=−186.3 to −τc=−17.1 .
Figure 8: (a) cos2(vi,1) against λi for all 309 eigenvectors of S (dashed: 1/n ). Only vmin ( 0.78 ) and vmax ( 0.21 ) are substantially aligned with 1 . (b) Loadings of vmin and vmax by source collection (dashed: 1/n , the loading of a uniform vector). vmin is close to uniform for T0, Flan, CoT and Tulu (larger on SuperGLUE), while vmax separates SuperGLUE from the other collections.
Figure 9: Share of mixture mass per source collection as the shift decreases. Down to τc+ the collection-level allocation is stable (T0 0.62 – 0.65 ), with Flan2021 growing from 0.22 to 0.35 . Without the shift all mass goes to one Flan2021 task.
ρ
nρ
active
eH
interaction effect
0.0060
1.85
302
289.8
0.097
0.0100
3.09
275
250.4
0.065
0.0150
4.63
246
218.8
0.050
0.0200
6.18
225
194.5
0.039
0.0220
6.80
210
186.1
0.036
0.02424
7.49
206
179.6
0.035
Appendix
Table 3: The ρ path at τ=τ0+10−3 . ρ alone controls breadth. “Interaction effect” is the TV between p∗ and the solution of the same problem with S removed from the quadratic term, ΠΔn(ρ(d+τ1)/τ) ; it grows as nρ decreases. The operating point is in bold.
Figure 10: S reordered by learned weight: the 206 active tasks in decreasing p[i]∗ , followed by the 103 inactive tasks (average-linkage clustering). Side strips show the source collection and p[i]∗ . Colour scale clipped at −2 .
nρ
ρ
∣supp∣
eH
TV no-pair
TV deg.
1.2
0.0039
308
306.8
0.120
0.003
2
0.0065
299
285.2
0.093
0.009
4
0.0129
252
229.5
0.054
0.008
7.49
0.0242
206
179.6
0.035
0.011
10
0.0324
190
157.4
0.027
0.009
20
0.0647
131
109.5
0.012
0.005
Appendix
Table 4: Effect of the pairwise term along the ρ path ( τ=τ0+10−3 ). TV is measured from p∗(ρ) to the pairwise-removed solution at the same ρ (no-pair) and to the degree-only allocation with the same support size (deg.).
Dataset
Leaderboard
Size / Method
BBH
GPQA
IFEval
Math
MMLU-Pro
MUSR
25K
Random
0.3482 ±0.0059
0.2626 ±0.0128
0.3465 ±N/A
0.0098 ±0.0027
0.1877 ±0.0036
0.3677 ±0.0172
Uniform
0.3501 ±0.0059
0.2701 ±0.0129
0.3501 ±N/A
0.0151 ±0.0034
0.1768 ±0.0035
0.4027 ±0.0175
EPM
0.3593 ±0.0059
0.2601 ±0.0127
0.3405 ±N/A
0.0151 ±0.0033
0.1836 ±0.0035
0.4286 ±0.0177
LESS
0.4059±0.0055
0.2412±0.0088
0.3417±N/A
0.0001±0.0004
0.1816±0.0062
0.4157±0.0175
Appendix
Table 5: Llama-2-7b: Instruction-tuning performance on Leaderboard subsets using batch size 8. Scores are reported as mean ± std. over three fine-tuning repeats on the same mixture with different seeds).
Figure 11: Comparison of task similarity metrics using cosine similarity and PMI-based heatmaps. Cosine scores are generally low and fail to distinguish task structure. PMI highlights asymmetric task alignment and reveals clearer task groupings across models.
Benchmark
Eval. ex.
Exact
Exact rate
13-gram cand.
13-gram rate
BBH
5,761
0
0.000%
30
0.521%
GPQA
1,192
0
0.000%
1
0.084%
IFEval
541
0
0.000%
5
0.924%
MATH-hard
1,324
0
0.000%
275
20.770%
MMLU-Pro
12,032
0
0.000%
112
0.931%
MuSR
756
0
0.000%
1
0.132%
Appendix
Table 6: First-stage overlap audit between the TaskMixPGM source pool (309 tasks) and evaluation benchmarks. “13-gram candidates” are lexical hits requiring adjudication; a shared 13-gram may arise from repeated templates, mathematical expressions, or standard instructional language and does not by itself establish leakage.
#
Task
Description
COT
1
cot_creak
Chain-of-thought reasoning on the CREAK entity-based commonsense fact-verification dataset.
2
cot_creak_ii
Second prompt variant of cot_creak with alternate instruction formatting.
3
cot_ecqa
Chain-of-thought on ECQA: generates explanations for commonsense QA answer choices.
Supervised fine-tuning performance for large language models depends strongly on how training budget is distributed across a heterogeneous set of tasks. In practice, mixtures are often fixed using simple heuristics (e.g., uniform or size-proportional sampling) that ignore task interactions, which can hurt transfer and waste budget on redundant sources. We introduce TaskPGM, a framework for learning continuous task mixtures via an energy-based model over tasks. Tasks form the nodes of a Markov random field: unary potentials capture per-task utility, and pairwise potentials encode inter-task relationships using behavioral divergences computed from predictive distributions of single-task fine-tuned models (e.g., Jensen--Shannon divergence and pointwise mutual information). Optimizing this objective yields mixtures that balance coverage against redundancy. We show that the resulting set function is weakly submodular under budget constraints, enabling approximation guarantees for discrete selection variants. Across multiple model families (LLaMA-7B, Qwen2-7B) and evaluation suites (BIG-Bench Hard), TaskPGM improves over standard mixing strategies and provides interpretable structure over task interactions.
Current language model training commonly applies multi-task Supervised Fine-Tuning (SFT) using a homogeneous compute budget across all sub-datasets. This approach is fundamentally sub-optimal: heterogeneous learning dynamics cause faster-learning tasks to overfit early while slower ones remain under-fitted. To address this, we introduce mSFT, an iterative, overfitting-aware search algorithm for multi-task data mixtures. mSFT trains the model on an active mixture, identifies and excludes the earliest overfitting sub-dataset, and reverts to that specific optimal checkpoint before continuing. Extensive evaluations demonstrate that mSFT consistently outperforms 4 baselines across 10 benchmarks and 6 base models. Further analysis confirms mSFT maintains robust gains across diverse dataset sizes, task granularities, and is insensitive to its single new hyperparameter (compute budget). Notably, at low compute budget, mSFT can improve performance while lowering training FLOPs. Ultimately, mSFT establishes a practical overfitting-aware algorithm for multi-task SFT that maximizes the potential of models across diverse data mixtures.
Woosung Koh, Jeyoung Jeon, Youngjin Song +4
KAIST AI · Yonsei University · Samsung Advanced Institute of Technology
Adapting a language model to a specialized corpus means choosing which instruction-tuning tasks to train on under a fixed budget, and testing one choice costs a fine-tuning run. Common heuristics add more source tasks or pick sources similar to the target. The first assumes transfer is never negative; the second, that it is symmetric. We show that both assumptions fail: task A can help task B while B hurts A, so helpfulness is a signed property of ordered source--target pairs. We introduce the transfer map, a signed estimate of how much each source helps or hurts each held-out target. We fit the map in hundreds of fine-tuning runs on Qwen3 and Mistral models from 0.6B to 32B parameters, with all sources drawn from one corpus and no training examples from the target. The map predicts a held-out target's accuracy on unseen mixtures: recorded before those runs, its predictions have less than half the error of a mixture-agnostic baseline. The map is specific to its target and corpus but transfers across model scale: a mixture selected in advance at one size beats training on all source tasks at every other size we tested. Transfer is thus a property of the data. The map selects the tasks that help and drops the one that interferes: accuracy on the reasoning targets (causal explanation, multi-hop questions and methodological critique) rises by up to 14 percentage points over training on all source tasks.
Nima H. Siboni, Vahid Rostami
Juna.ai, Kastanienallee 32, 10435 Berlin, Germany · Computational Systems Neuroscience, Institute of Zoology, University of Cologne, Germany