Supervised fine-tuning over a large pool of heterogeneous tasks is often computationally expensive, leading practitioners to train language models on a fixed training budget of carefully chosen SFT tasks. Thus, achieving strong downstream performance hinges on selecting an effective subset of tasks, or equivalently, identifying an optimal task mixture under a fixed budget constraint. In practice, supervised finetuning mixtures are often determined using simple heuristics (e.g., uniform or size-proportional sampling) that ignore task interactions, which can hurt transfer and waste budget on redundant task sources. We introduce TASKMIXPGM, a framework for learning continuous task mixtures through energy-based modelling over SFT tasks. Each task forms a node in a Markov random field, with unary potentials capturing task utility and pairwise potentials modelling inter-task relationships via behavioral divergences between predictive distributions of single-task fine-tuned models. Optimizing this objective yields task mixtures that balance utility, coverage, and redundancy. We further show that the induced set function defined over the set of such tasks is weakly submodular under budget constraints, enabling approximation guarantees for discrete task selection. Empirically, we provide a fixed, benchmark-independent mixture that improves over heuristic mixing across multiple model families and different training budgets. TASKMIXPGM also uncovers interpretable structure in task interactions, revealing complementarities and redundancies that inform effective SFT mixtures.
Figures & tables
Figure 1: Illustrative example. Left: a six-task MRF whose edges are labelled with similarities Sij (thicker edges for high Sij , dashed for low). Right: the optimal mixture p∗ from Eq. ( 3 ) for these similarities; darker shading marks larger weight in both panels. The most connected task T6 receives the most mass, while the strongly redundant pairs ( T1,T2 and T4,T5 , both with Sij=0.8 ) split theirs.
Figure 2: Effect of the spectrum shift τ in S+τI on the optimal mixture ( n=309 , ρ=0.02424 ). Shaded: τ≤τc , where the objective is not convex on Δn . Dashed lines show the local minimum reached from p∗(τ0) . At τ=0 the solution is a single task. Per-collection mass is in Figure 9 ; discussion in Section C.3 .
Figure 3: HeatMap of PMI Affinity Matrix across all 309 tasks as specified in Appendix K
Figure 4: Incremental composition. Solid: median over five random insertion orders (band: range). Dashed: greedy order. TV curves are a 15-step moving average. Markers: steps at which the unshifted ( τ=0 ) mixture changes, each moving all mass to the new task. Grey dashed line in (c): all k tasks active.
Table 1: Instruct-tuning performance on Leaderboard benchmarks (25K and 50K samples).
Size ( B )
Method
Reasoning
Knowledge
Math
Instr.
BBH
MuSR
GPQA
MMLU-Pro
MATH
IFEval
Avg.
25K
Random ( 3 )
57.89
38.76
36.07
31.10
27.02
46.16
39.50
Uniform ( 1 )
53.08
39.02
31.29
23.74
28.25
45.08
36.74
EPM ( 2 )
55.86
39.42
30.62
25.65
25.73
44.48
36.96
LESS ( xia2024less )
61.62
44.90
36.58
51.78
29.68
45.29
44.98
SMART ( renduchintala-etal-2024-smart )
40.37
39.02
29.11
12.77
27.04
44.12
32.07
Table 2: Qwen3.5-9B
Figure 5: The 15 highest-weighted tasks, coloured by collection, with each task’s rank by degree S[i] . The full structure of the mixture, including the affinity matrix reordered by weight, is analysed in Section E.1 .
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Weak submodularity of f . (a) Gain of five fixed tasks along greedy prefixes at τ0 (legend: greedy position of j ). (b) Gap f(V)−f(Ak) for exact greedy (solid) and random orders (dashed, band over 5 random orders). (c) γ^A,S ( 1 ) on the 180 random-set pairs per τ ( 12 draws of A per size, ∣S∣∈{2,5,10} ). Pairs with f(Aˉ∪Sˉ)=f(Aˉ) are omitted ( 0 , 1 and 105 for τ0 , 50 , 0 ). (d) mA/MA for (S+τI)A along the τ0 greedy order, for τ∈{τ0,50,τc,0} , with τc=17.09 the critical shift of ( 4 ). This principal-submatrix ratio is a conservative proxy for the zero-sum constants of Theorem 1 ; below zero (shaded) the proxy gives no guarantee.
Figure 7: Eigen-structure of the PMI affinity matrix S ( n=309 ). (a) Sorted eigenvalues (blue: positive, orange: negative), with an inset on ranks 6 – 304 . S is strongly indefinite, with one outlier λmin=−τ0=−186.3 ; the next eigenvalue is −15.4 . (b) Fraction of ∥S∥F2 captured by the k largest-magnitude eigenpairs. The outlier alone carries 86.8% ; 2 eigenpairs give 90% and 13 give 99% (entropy effective rank 28.5 ). (c) The 10 smallest eigenvalues of S and of its restriction B⊤SB to 1⊥ . Restriction removes the outlier and raises the threshold from −τ0=−186.3 to −τc=−17.1 .
Figure 8: (a) cos2(vi,1) against λi for all 309 eigenvectors of S (dashed: 1/n ). Only vmin ( 0.78 ) and vmax ( 0.21 ) are substantially aligned with 1 . (b) Loadings of vmin and vmax by source collection (dashed: 1/n , the loading of a uniform vector). vmin is close to uniform for T0, Flan, CoT and Tulu (larger on SuperGLUE), while vmax separates SuperGLUE from the other collections.
Figure 9: Share of mixture mass per source collection as the shift decreases. Down to τc+ the collection-level allocation is stable (T0 0.62 – 0.65 ), with Flan2021 growing from 0.22 to 0.35 . Without the shift all mass goes to one Flan2021 task.
ρ
nρ
active
eH
interaction effect
0.0060
1.85
302
289.8
0.097
0.0100
3.09
275
250.4
0.065
0.0150
4.63
246
218.8
0.050
0.0200
6.18
225
194.5
0.039
0.0220
6.80
210
186.1
0.036
0.02424
7.49
206
179.6
0.035
Appendix
Table 3: The ρ path at τ=τ0+10−3 . ρ alone controls breadth. “Interaction effect” is the TV between p∗ and the solution of the same problem with S removed from the quadratic term, ΠΔn(ρ(d+τ1)/τ) ; it grows as nρ decreases. The operating point is in bold.
Figure 10: S reordered by learned weight: the 206 active tasks in decreasing p[i]∗ , followed by the 103 inactive tasks (average-linkage clustering). Side strips show the source collection and p[i]∗ . Colour scale clipped at −2 .
nρ
ρ
∣supp∣
eH
TV no-pair
TV deg.
1.2
0.0039
308
306.8
0.120
0.003
2
0.0065
299
285.2
0.093
0.009
4
0.0129
252
229.5
0.054
0.008
7.49
0.0242
206
179.6
0.035
0.011
10
0.0324
190
157.4
0.027
0.009
20
0.0647
131
109.5
0.012
0.005
Appendix
Table 4: Effect of the pairwise term along the ρ path ( τ=τ0+10−3 ). TV is measured from p∗(ρ) to the pairwise-removed solution at the same ρ (no-pair) and to the degree-only allocation with the same support size (deg.).
Dataset
Leaderboard
Size / Method
BBH
GPQA
IFEval
Math
MMLU-Pro
MUSR
25K
Random
0.3482 ±0.0059
0.2626 ±0.0128
0.3465 ±N/A
0.0098 ±0.0027
0.1877 ±0.0036
0.3677 ±0.0172
Uniform
0.3501 ±0.0059
0.2701 ±0.0129
0.3501 ±N/A
0.0151 ±0.0034
0.1768 ±0.0035
0.4027 ±0.0175
EPM
0.3593 ±0.0059
0.2601 ±0.0127
0.3405 ±N/A
0.0151 ±0.0033
0.1836 ±0.0035
0.4286 ±0.0177
LESS
0.4059±0.0055
0.2412±0.0088
0.3417±N/A
0.0001±0.0004
0.1816±0.0062
0.4157±0.0175
Appendix
Table 5: Llama-2-7b: Instruction-tuning performance on Leaderboard subsets using batch size 8. Scores are reported as mean ± std. over three fine-tuning repeats on the same mixture with different seeds).
Figure 11: Comparison of task similarity metrics using cosine similarity and PMI-based heatmaps. Cosine scores are generally low and fail to distinguish task structure. PMI highlights asymmetric task alignment and reveals clearer task groupings across models.
Benchmark
Eval. ex.
Exact
Exact rate
13-gram cand.
13-gram rate
BBH
5,761
0
0.000%
30
0.521%
GPQA
1,192
0
0.000%
1
0.084%
IFEval
541
0
0.000%
5
0.924%
MATH-hard
1,324
0
0.000%
275
20.770%
MMLU-Pro
12,032
0
0.000%
112
0.931%
MuSR
756
0
0.000%
1
0.132%
Appendix
Table 6: First-stage overlap audit between the TaskMixPGM source pool (309 tasks) and evaluation benchmarks. “13-gram candidates” are lexical hits requiring adjudication; a shared 13-gram may arise from repeated templates, mathematical expressions, or standard instructional language and does not by itself establish leakage.
#
Task
Description
COT
1
cot_creak
Chain-of-thought reasoning on the CREAK entity-based commonsense fact-verification dataset.
2
cot_creak_ii
Second prompt variant of cot_creak with alternate instruction formatting.
3
cot_ecqa
Chain-of-thought on ECQA: generates explanations for commonsense QA answer choices.