SLEM: Spectrum-Aware Latent Expert Merging

Dohyeon Kim; Bedionita Soro; Bruno Andreis; Sung Ju Hwang

cs.LGOct 12, 2026

SLEM: Spectrum-Aware Latent Expert Merging

Authors: Dohyeon Kim, Bedionita Soro, Bruno Andreis, Sung Ju Hwang

Organizations: KAIST · University of Oxford · Slater Labs

Abstract

Mixture-of-Experts (MoE) models scale model capacity efficiently by activating only a small subset of experts for each input, while still requiring all parameters to be stored in memory, resulting in substantial memory overhead. Expert merging techniques have recently emerged as a promising compression paradigm that reduces this overhead by consolidating multiple experts into a smaller set. However, existing approaches have two key limitations: (i) direct convex interpolation in parameter space is not generally guaranteed to yield meaningful interpolation of expert functions; and (ii) estimating expert similarity or merging coefficients often relies on calibration data, introducing dependence on the calibration distribution. To address these limitations, we propose Spectrum-Aware Latent Expert Merging (SLEM), a calibration-free expert merging framework that occurs in a structured latent representation of expert weights using a VAE. SLEM groups experts within and across layers based on latent similarities and derives merging coefficients from expert weight spectra. It then performs geometry-aware latent interpolation and decodes the resulting representations into merged expert weights. Compared with existing expert merging methods, SLEM achieves comparable or better performance on general knowledge benchmarks and higher average performance on generation benchmarks involving mathematical reasoning and code generation.

Figures & tables

Figure 1: Average performance comparison of expert merging methods on generation-based benchmarks involving mathematical reasoning and code generation (GSM8K, MATH500, MBPP, and HumanEval). SLEM consistently achieves the highest average performance under all evaluated settings.

Figure 2: Overall framework of Spectrum-Aware Latent Expert Merging (SLEM).

Panel of Figure 3 — Figure 3: Culmulative explained variance ratio (EVR) of expert projection weights across layers in LFM2-8B-A1B. The EVR curves show that variance is concentrated along a subset of principal directions across all projection types, motivating the use of dominant spectral structure for estimating subnetwork importance. Results for the other evaluated models are provided in Appendix G .

LFM2-8B-A1B
		General Knowledge (MCQA)				Mathematics & Coding (Generation)
Expert Reduction	Method	MMLU	ARC-E	ARC-C	Average $\uparrow$	GSM8K	MATH500	MBPP	HumanEval	Average $\uparrow$
0.0%	Vanilla	$63.11$	$75.59$	$47.63$	$62.11$	$81.73$	$72.40$	$45.60$	$70.73$	$67.62$
0.0%	VAE-reconstructed	$62.93_{\mathrlap{\pm 0.21}}$	$75.61_{\mathrlap{\pm 0.45}}$	$47.41_{\mathrlap{\pm 0.61}}$	$61.98_{\mathrlap{\pm 0.32}}$	$82.35_{\mathrlap{\pm 0.30}}$	$72.20_{\mathrlap{\pm 0.64}}$	$45.52_{\mathrlap{\pm 1.14}}$	$71.83_{\mathrlap{\pm 3.17}}$	$67.97_{\mathrlap{\pm 0.72}}$
12.5%	F-Merging	$55.64$	$68.18$	$42.32$	$55.38$	$69.14$	$30.60$	$03.00$	$03.66$	$26.60$
	M-SMoE	$54.72$	$72.97$	$44.11$	$57.27$	$57.24$	$20.40$	$13.20$	$25.61$	$29.11$

Table 1: Downstream performance comparison with existing expert merging methods. All merging methods are evaluated under the same expert reduction rate. The best and second-best results within each expert reduction setting are highlighted in bold and underlined , respectively.

Figure 4: Comparison of parameter-space (PS) and latent-space (LS) merging under identical expert pairs and merging coefficients. Across both models and task categories, LS Merging consistently outperforms PS Merging for every evaluated pair.

Figure 5: Effect of the truncation rank

r

for spectrum-aware merging coefficients. Spectrum-aware coefficients consistently outperform the uniform-coefficient baseline across models and tasks.

LFM2-8B-A1B
		General Knowledge (MCQA)				Mathematics & Coding (Generation)
Max Group Size	Method	MMLU	ARC-E	ARC-C	Average $\uparrow$	GSM8K	MATH500	MBPP	HumanEval	Average $\uparrow$
$\infty$	M-SMoE	$46.07$	$66.89$	$40.31$	$51.09$	$02.05$	$02.20$	$00.00$	$00.00$	$01.06$
	HC-SMoE	$50.98$	$60.47$	$37.15$	$49.53$	$54.13$	$45.20$	$30.40$	$44.51$	$43.56$
	Sub-MoE	$49.91$	$64.82$	$37.75$	$50.83$	$43.59$	$34.20$	$08.20$	$28.05$	$28.51$
2	M-SMoE	$34.55$	$65.99$	$38.99$	$46.51$ $\mathrlap{\downarrow 4.58}$	$08.34$	$03.20$	$00.00$	$00.00$	$02.89$ $\mathrlap{\uparrow 1.83}$

Table 2: Downstream performance under constrained maximum expert group size. Limiting the group size generally degrades performance compared with unconstrained grouping.

Figure 6: Token-level Jensen–Shannon divergence (JSD) and perplexity (PPL) of merged models on sequences generated by the original model. We evaluate 100 sequences only over the response tokens. Lower values indicate better preservation of the original model’s generative behavior.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Metric	Few-shot	Maximum Sequence Length	Maximum Generation Tokens
General Knowledge (MCQA)
MMLU ( Hendrycks et al., 2021a )	Accuracy	0-shot	$4{,}096$	N/A
ARC-Easy ( Clark et al., 2018 )	Accuracy	0-shot	$4{,}096$	N/A
ARC-Challenge ( Clark et al., 2018 )	Accuracy	0-shot	$4{,}096$	N/A
Mathematics (Generation)
GSM8K ( Cobbe et al., 2021 )	Exact Match	8-shot (single-turn)	$4{,}096$	$1{,}024$

Appendix

Table 3: Benchmark-specific evaluation settings, including the evaluation metric, few-shot setting, maximum sequence length, and maximum generation tokens.

Model	# layers (enc./dec.)	$D_{\mathrm{\{emb,\,hidden,\,latent\}}}$	$D_{\mathrm{FFN}}$	# attn. heads	Pre-LN architecture
LFM2-8B-A1B	6	$3{,}072$	$4\times 3{,}072$	24	yes
OLMoE-1B-7B-0125-Instruct	6	$3{,}072$	$4\times 3{,}072$	24	yes
ERNIE-4.5-21B-A3B-PT	6	$3{,}840$	$4\times 3{,}840$	30	yes

Appendix

Table 4: Architecture configurations of the hyperspherical expert VAE for each evaluated model. Training and optimization hyperparameters are provided in Appendix C .

LFM2-8B-A1B
		General Knowledge (MCQA)				Mathematics & Coding (Generation)
Expert Reduction	Method	MMLU	ARC-E	ARC-C	Average $\uparrow$	GSM8K	MATH500	MBPP	HumanEval	Average $\uparrow$
25.0%	SLEM ( $r$ =14)	51.10	70.62	42.75	54.82	70.36	58.80	36.20	61.59	56.74
	$\;\llcorner$ within-layer adaptive	53.12	67.85	41.81	54.26 $\mathrlap{\downarrow 0.56}$	68.84	60.00	22.00	43.90	48.69 $\mathrlap{\downarrow 8.05}$
	$\;\llcorner$ cross-layer adaptive	52.98	69.19	43.60	55.26 $\mathrlap{\uparrow 0.44}$	68.39	60.40	34.40	55.49	54.67 $\mathrlap{\downarrow 2.07}$
	$\;\llcorner$ uniform coefficient	50.48	68.01	41.47	53.32 $\mathrlap{\downarrow 1.50}$	68.92	58.80	35.40	57.32	55.11 $\mathrlap{\downarrow 1.63}$

Appendix

Table 5: Comparison of SLEM design variants for expert grouping and merging coefficients. We evaluate alternative grouping strategies, uniform and row-norm coefficients, and projection-specific spectral importance using only gate, up, or down projections. All variants are compared with the default SLEM under the same expert reduction setting.

LFM2-8B-A1B
		General Knowledge (MCQA)				Mathematics & Coding (Generation)
Expert Reduction	Method	MMLU	ARC-E	ARC-C	Average $\uparrow$	GSM8K	MATH500	MBPP	HumanEval	Average $\uparrow$
0.0%	Vanilla	$63.11$	$75.59$	$47.63$	$62.11$	$81.73$	$72.40$	$45.60$	$70.73$	$67.62$
	VAE-reconstructed	$62.93_{\mathrlap{\pm 0.21}}$	$75.61_{\mathrlap{\pm 0.45}}$	$47.41_{\mathrlap{\pm 0.61}}$	$61.98_{\mathrlap{\pm 0.32}}$	$82.35_{\mathrlap{\pm 0.30}}$	$72.20_{\mathrlap{\pm 0.64}}$	$45.52_{\mathrlap{\pm 1.14}}$	$71.83_{\mathrlap{\pm 3.17}}$	$67.97_{\mathrlap{\pm 0.72}}$
	$\;\llcorner$ VAE with $\beta$ =0	$62.94_{\mathrlap{\pm 0.19}}$	$75.14_{\mathrlap{\pm 0.50}}$	$47.25_{\mathrlap{\pm 0.61}}$	$61.78_{\mathrlap{\pm 0.19}}$	$82.02_{\mathrlap{\pm 0.49}}$	$72.68_{\mathrlap{\pm 1.84}}$	$45.56_{\mathrlap{\pm 1.51}}$	$69.39_{\mathrlap{\pm 1.18}}$	$67.41_{\mathrlap{\pm 0.62}}$
12.5%	SLEM ( $r$ =14)	58.37	74.79	47.87	60.34	77.10	67.60	44.20	62.80	62.93

Appendix

Table 6: Downstream performance comparison with and without KL regularization during Expert VAE training. The VAE with

\beta=0

variant removes the KL divergence term while keeping the remaining training settings unchanged.

Expert Reduction	GSM8K	MATH500	MBPP	HumanEval	Average
	$n=1{,}319$	$n=500$	$n=500$	$n=164$
12.5%	[9.10, 14.33]	[7.60, 15.60]	[3.40, 11.00]	[-3.05, 12.80]	[6.38, 11.31]
25.0%	[13.19, 18.65]	[9.40, 17.80]	[1.00, 9.80]	[7.93, 23.78]	[10.07, 15.32]

Appendix

Table 7: Paired bootstrap 95% confidence intervals (pp) of the performance difference between SLEM and HC-SMoE on LFM2-8B-A1B generation benchmarks. We use 200,000 bootstrap resamples. Positive intervals favor SLEM.

Panel of Figure 7 — Figure 7: Culmulative explained variance ratio (EVR) of expert projection weights across layers in other evaluated models.

Panel of Figure 8 — Figure 8: Normalized spectral importance across subnetworks under different truncation ranks $r$ . Lower ranks produce more concentrated importance distributions, while larger ranks distribute importance more uniformly across subnetworks.

Language Model Merging

Language model merging combines capabilities from multiple models or adapters by composing their weights, updates, or outputs.

See all 110 papers

REAM: Merging Improves Pruning of Experts in LLMs
Apr 6, 2026Saurav Jha, Maryam Hashemzadeh, Ali Saheb Pasand +3LLM Compression Mixture-of-Experts Pruning
Mila – Quebec AI Institute · Polytechnique Montréal · Université de Montréal +3
ResMerge: Residual-based Spectral Merging of Large Language Models
Jun 1, 2026Yandu Sun, Zhiyan Hou, Haokai Ma +6Task Vector Merging Language Model Merging
1Southeast University, Nanjing, China · Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences +4
Speculating Experts Accelerates Inference for Mixture-of-Experts
Mar 9, 2026Vivan Madan, Prajwal Singhania, Abhinav Bhatele +2Mixture-of-Experts Inference LLM Inference Acceleration
University of Maryland · TogetherAI

Apr 6, 2026J/K move · Enter open · S save

REAM: Merging Improves Pruning of Experts in LLMs

Saurav Jha, Maryam Hashemzadeh, Ali Saheb Pasand, Ali Parviz+2

Mila – Quebec AI Institute · Polytechnique Montréal · Université de Montréal+3

LLM Compression Mixture-of-Experts Pruning Mixture-of-Experts Language Models

Open paper AskPDF arXiv

Mixture-of-Experts (MoE) large language models (LLMs) are among the top-performing architectures. The largest models, often with hundreds of billions of parameters, pose significant memory challenges for deployment. Traditional approaches to reduce memory requirements include weight pruning and quantization. Motivated by the Router-weighted Expert Activation Pruning (REAP) that prunes experts, we propose a novel method, Router-weighted Expert Activation Merging (REAM). Instead of removing experts, REAM groups them and merges their weights, better preserving original performance. We evaluate REAM against REAP and other baselines across multiple MoE LLMs on diverse multiple-choice (MC) question answering and generative (GEN) benchmarks. Our results reveal a trade-off between MC and GEN performance that depends on the mix of calibration data. By controlling the mix of general, math and coding data, we examine the Pareto frontier of this trade-off and show that REAM often outperforms the baselines and in many cases is comparable to the original uncompressed models.

Abstract

Language Model Merging

Explore similar work

REAM: Merging Improves Pruning of Experts in LLMs

ResMerge: Residual-based Spectral Merging of Large Language Models

Speculating Experts Accelerates Inference for Mixture-of-Experts