SLIM: Simplex-Lattice Interpolation Merging

Seongcheol Jeong; Masahiro Suzuki; Yutaka Matsuo

cs.LGOct 1, 2026

SLIM: Simplex-Lattice Interpolation Merging

Authors: Seongcheol Jeong, Masahiro Suzuki, Yutaka Matsuo

Organizations: The University of Tokyo

Abstract

Optimizing merging coefficients for large language models can require many costly benchmark evaluations. We propose \textbf{Simplex-Lattice Interpolation Merging (SLIM)}, which constructs a quadratic surrogate of aggregate performance on the coefficient simplex using a classical mixture design. Evaluations of individual experts and equal-weight pairs determine the surrogate with the minimum number of measurements needed to identify a general quadratic on this domain. SLIM then optimizes the surrogate without further target-metric evaluations. Experiments on two model architectures demonstrate accurate prediction of unseen multi-expert mixtures and competitive merge performance under limited evaluation budgets. Matched-budget comparisons show that structured evaluation points improve prediction fidelity over random designs, including those using regularized fitting.

Figures & tables

Figure 1: A quadratic surrogate constructed from individual experts and equal-weight pairs accurately predicts AVG at unseen mixtures of all eight experts. For each architecture, 36 construction evaluations are followed by testing on 150 held-out interior vectors sampled from $\operatorname{Dirichlet}(1,\ldots,1)$ . Axes show predicted (horizontal) and measured (vertical) scores; colored markers show domain scores and black stars show AVG. AVG zoom panels report Spearman’s $\rho$ and MAE; dashed lines indicate $y=x$ .

Figure 2: Simplex-lattice designs for $T=3$ . Each point is a coefficient vector with nonnegative entries summing to one. The quadratic design contains three vertices and three edge midpoints; the cubic design contains the vertices, two points on each edge with weights $(2/3,1/3)$ and $(1/3,2/3)$ , and the barycenter $(1/3,1/3,1/3)$ . Colors and marker shapes indicate the number of experts with nonzero coefficients. For $T=8$ , the corresponding designs contain 36 and 120 points.

Domain	ID (used for selection)	OOD (held out)
Instruction	IFEval	AlpacaEval 2.0
Math	GSM8K	MATH-500
Coding	MBPP	HumanEval
Multilingual	FLORES+ dev	FLORES+ devtest
Safety	HarmBench	WildGuardTest

Table 1: ID benchmarks for coefficient selection and held-out OOD benchmarks across five domains. Reported ID scores use the selection benchmarks. Metrics, sample counts, and prompting settings appear in Table 6 .

Figure 3: ID and held-out OOD AVG versus the evaluation budget for coefficient selection (ID left, OOD right within each architecture). #Evals counts cumulative ID evaluations per run; upper axes estimate GPU-hours from measured per-evaluation costs. Evolutionary merging evaluates 10 candidates per generation. ID scores are shown after every generation and OOD scores after 10, 30, 40, 50, 100, 150, and 200 evaluations, always for the candidate with the highest ID AVG observed so far. Quadratic and cubic SLIM use 36 and 120 evaluations; dotted lines mark quadratic SLIM’s AVG. MAP and evolutionary merging show means over three seeds; shaded bands indicate $\pm 1$ standard deviation.

Figure 4: Selected merging coefficients reveal differences in expert allocation and overall scale. Cells show raw coefficients; bars show their sums, with dotted lines marking the sum-to-one constraint. MAP uses 135 evaluations per seed. Evolutionary merging shows the candidate with the highest ID AVG observed within 200 evaluations. Cell labels are rounded, so displayed zeros need not be exact zeros; sums use unrounded values. Expert identities are listed in Table 5 .

	Gemma-2-9B				Llama-3.1-8B
$n$	$\rho_{\mathrm{lattice}}$	$\rho_{\mathrm{random}}$	$\mathrm{MAE}_{\mathrm{random}}$	Lattice win	$\rho_{\mathrm{lattice}}$	$\rho_{\mathrm{random}}$	$\mathrm{MAE}_{\mathrm{random}}$	Lattice win
36	0.935	0.465 / 0.847	33.64 / 2.90	99.5 / 99.5	0.958	0.330 / 0.758	30.15 / 2.24	100.0 / 100.0
45	0.935	0.902 / 0.884	2.46 / 2.50	68.9 / 86.2	0.958	0.819 / 0.800	1.81 / 1.99	99.9 / 99.9
60	0.934	0.954 / 0.948	1.63 / 1.70	12.1 / 18.2	0.957	0.918 / 0.915	1.11 / 1.16	95.6 / 95.9
75	0.933	0.966 / 0.965	1.38 / 1.39	1.7 / 2.4	0.956	0.941 / 0.941	0.93 / 0.94	79.1 / 79.3
90	0.931	0.970 / 0.970	1.27 / 1.27	0.5 / 0.5	0.954	0.950 / 0.950	0.83 / 0.84	57.9 / 57.1

Table 2: The lattice improves AVG prediction at the matched budget of 36 evaluations. Random budgets are

n

; the lattice always uses 36. Values average 1,000 resamples with shared held-out points; pairs show LS / ridge. Lattice win is the percentage with

\rho_{\mathrm{lattice}}>\rho_{\mathrm{random}}

. Lattice MAEs average approximately 1.93 (Gemma) and 1.07 (Llama).

		Gemma-2-9B		Llama-3.1-8B
Variant	#Evals	ID	OOD	ID	OOD
Best vertex	8	58.51	49.73	54.80	46.41
Best lattice point	36	65.37	51.85	56.56	42.93
SLIM (quadratic)	36	66.15	50.30	56.61	47.27
SLIM (cubic)	120	66.35	48.41	57.04	46.27

Table 3: Effects of evaluating expert pairs, optimizing the surrogate, and increasing its degree on merge performance. The first two rows select by measured ID AVG among eight vertices or 36 lattice points; SLIM optimizes its surrogate. #Evals counts ID evaluations for selection; OOD is held out. Bold marks column maxima.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning
$T$	Number of expert models.
$\theta_{\mathrm{pre}},\theta_{k}$	Parameters of the shared base model and expert $k$ .
$\tau_{k}$	Task vector $\theta_{k}-\theta_{\mathrm{pre}}$ .
$\lambda,\lambda_{k}$	Merging coefficient vector and its $k$ -th component.
$\theta(\lambda)$	Merged parameters $\theta_{\mathrm{pre}}+\sum_{k}\lambda_{k}\tau_{k}$ .
$\Lambda$	General feasible set of merging coefficients.

Appendix

Table 4: Symbols defining the merging coefficients, target metric, and surrogate construction.

Role	Hugging Face identifier
Gemma-2-9B
Base	google/gemma-2-9b
Instruction A	MergeBench/gemma-2-9b_instruction
Instruction B	google/gemma-2-9b-it
Math A	MergeBench/gemma-2-9b_math
Math B	POME-OPTIM/Gemma2-9B-MetaMathQA

Appendix

Table 5: Shared bases and eight experts per architecture, with identifiers linked to their releases. A and B distinguish experts in the same domain. WildGuard is the external evaluator and is not merged.

Benchmark	Role	Examples	Shots	Scoring
IFEval ( Zhou et al., 2023 )	ID	541	0	Prompt-level strict accuracy
AlpacaEval 2.0 ( Dubois et al., 2024 )	OOD	805	0	Length-controlled win rate
GSM8K-CoT ( Cobbe et al., 2021 )	ID	1,319	8	Strict-match exact match
MATH-500 ( Hendrycks et al., 2021 ; Lightman et al., 2024 )	OOD	500	4	Verified accuracy ( math_verify )
MBPP ( Austin et al., 2021 )	ID	500	3	pass@1
HumanEval ( Chen et al., 2021 )	OOD	164	0	pass@1

Appendix

Table 6: Sample counts, prompting, and scoring for ID selection and held-out OOD evaluation. Shots denotes in-context examples. FLORES+ counts are sentences per target language times eight languages. Refusal rates use WildGuard.

		Gemma-2-9B			Llama-3.1-8B
Method	#Evals	GPU-h	ID AVG	OOD AVG	GPU-h	ID AVG	OOD AVG
Weight Averaging	0	0	50.1	40.0	0	43.3	33.2
MetaGPT	0	0	62.4	50.6	0	47.1	35.5
Task Arithmetic	21	3.0	52.0	39.5	2.7	50.2	37.6
Evol. Merge (gen $=$ 1)	10	1.4	$51.5_{\pm 0.8}$	$32.7_{\pm 1.1}$	1.3	$54.0_{\pm 1.0}$	$38.3_{\pm 0.4}$
Evol. Merge (gen $=$ 3)	30	4.1	$60.1_{\pm 0.8}$	$38.6_{\pm 1.3}$	3.9	$58.4_{\pm 1.7}$	$42.1_{\pm 2.0}$

Appendix

Table 7: ID and held-out OOD performance across coefficient-selection budgets (

T=8

; Figure 3 ). AVG is the unweighted mean of five domain scores on a 0–100 scale. MAP and evolutionary merging report mean

\pm

standard deviation over three seeds; their GPU-hours are means. Costs cover coefficient selection per run, excluding final reporting and fidelity measurements. Zero cost denotes no target-metric evaluations. Bold marks the highest AVG in each column before rounding. On Gemma OOD, MetaGPT (50.62) and MAP with

n=135

(50.60) both round to 50.6. The cubic variant is a degree ablation.

	Gemma-2-9B			Llama-3.1-8B
Domain	$\rho$	MAE	IQR	$\rho$	MAE	IQR
Instruction	0.97	2.60	15.9	0.96	2.46	10.9
Math	$-0.24$	4.88	5.0	0.85	2.33	6.5
Coding	0.86	1.89	4.2	0.70	1.93	3.9
Multilingual	0.63	1.40	1.1	0.56	1.22	0.6
Safety	0.94	9.20	47.2	0.95	4.39	19.9

Appendix

Table 8: AVG is accurately predicted despite variation in domain-wise fidelity. On 150 held-out mixtures, Spearman’s

\rho

compares predicted and measured rankings; MAE and the IQR of measured scores are in score points. All interpolants use the same 36 construction evaluations.

Figure 5: Quadratic predictions and measured ID AVG along all 28 simplex edges for Gemma-2-9B. Blue circles mark construction points and blue curves show the quadratic surrogate. Orange squares mark points held out from quadratic construction; orange segments show prediction errors. Panel titles list the experts at $t=0$ and $t=1$ . Y-axis ranges vary across panels.

Figure 6: Quadratic predictions and measured ID AVG along all 28 simplex edges for Llama-3.1-8B. Blue circles mark construction points and blue curves show the quadratic surrogate. Orange squares mark points held out from quadratic construction; orange segments show prediction errors. Panel titles list the experts at $t=0$ and $t=1$ . Y-axis ranges vary across panels.

Explore similar work

Rethinking Heterogeneous LLM Merging: A Weighted Model Averaging Perspective
Jul 20, 2026Jiahe Fan, Yinghao Hou, Si Chen +3Continual Model Merging Federated Averaging
University of Science and Technology of China · School of Information Science and Technology, Department of Automation, University of Science and Technology of China · School of Computer Science and Technology, University of Science and Technology of China
Are we Merging the Right Models? Impact of Expert Training Duration on Model Merging for LLMs
Jul 13, 2026Nikita Kozodoi, Zainab Afolabi, Jack ButlerContinual Model Merging Loss Function
1Amazon Web Services
Extra-Merge: Tracing the Rank-1 Subspace of Model Merging in Language Model Pre-Training
May 26, 2026Wenjie Zhou, Bohan Wang, Hongtao Zhang +3Large Language Model Pretraining Merge
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, China · University of Chinese Academy of Sciences, China · Alibaba Group, China +2

Jul 20, 2026, cs.AIJ/K move · Enter open · S save

Rethinking Heterogeneous LLM Merging: A Weighted Model Averaging Perspective

Jiahe Fan, Yinghao Hou, Si Chen, Aiyuan Zhang+2

University of Science and Technology of China · School of Information Science and Technology, Department of Automation, University of Science and Technology of China · School of Computer Science and Technology, University of Science and Technology of China

Continual Model Merging Federated Averaging Perspective

Open paper AskPDF arXiv

Can large language models with substantially different parameter spaces be merged by direct weighted averaging, without training or semantic alignment? Existing heterogeneous fusion methods typically introduce distillation, adapters, learned latent spaces, routing, or feature alignment, leaving open whether a simpler recipe can work for genuinely different billion-parameter checkpoints. We revisit this counterintuitive question through training-free dimensional adaptation followed by ratio-controlled interpolation. In union-style merging, we expand the smaller model into the larger parameter space; in intersection-style merging, we truncate the larger model into the smaller parameter space. Across Qwen-family model pairs and benchmarks covering mathematical reasoning, code generation, language understanding, commonsense reasoning, knowledge, and instruction following, deterministic expansion largely preserves the source model function, and small-ratio interpolation can improve over strong source checkpoints by transferring complementary capabilities. However, near-balanced interpolation often collapses, and task-level results reveal a seesaw effect in which gains on some capabilities coexist with regressions on others. These results show that simple parameter averaging, when paired with lightweight dimensional adaptation and carefully controlled ratios, is a surprisingly strong baseline for heterogeneous LLM merging, suggesting that the limits of direct weighted fusion may also bound what more complex heterogeneous merging methods can achieve at scale.

Abstract

Explore similar work

Rethinking Heterogeneous LLM Merging: A Weighted Model Averaging Perspective

Are we Merging the Right Models? Impact of Expert Training Duration on Model Merging for LLMs

Extra-Merge: Tracing the Rank-1 Subspace of Model Merging in Language Model Pre-Training