MoEITS: A Green AI approach for simplifying MoE-LLMs
Authors: Luis Balderas, Miguel Lastra, José M. Benítez
Organizations: Department of Computer Science and Artificial Intelligence, DiCITS, iMUDS, DaSCI, University of Granada, Granada, Spain 18071 · Department of Software Engineering, DiCITS, iMUDS, DaSCI, University of Granada, Granada, Spain 18071
Large language models are transforming all areas of academia and industry, attracting the attention of researchers, professionals, and the general public. In the trek for more powerful architectures, Mixture-of-Experts, inspired by ensemble models, have emerged as one of the most effective ways to follow. However, this implies a high computational burden for both training and inference. To reduce the impact on computing and memory footprint as well as the energy consumption, simplification methods has arisen as very effective procedures. In this paper, an original algorithm, MoEITS, for MoE-LLMs simplification is presented. The algorithm is characterized by a refined simplicity, underpinned by standardized Information Theoretic frameworks. MoEITS is analyzed in depth from theoretical and practical points of view. Its computational complexity is studied. Its performance on the accuracy of the simplified LLMs and the reduction rate achieved is assessed through a thoroughly designed experimentation. This empirical evaluation includes a comparison with state-of-the-art MoE-LLM pruning methods applied on Mixtral 8×7B, Qwen1.5-2.7B, and DeepSeek-V2-Lite. The extensive experimentation conducted demonstrates that MoEITS outperforms state-of-the-art techniques by generating models that are both effective across all benchmarks and computationally efficient. The code implementing the method will be available at https://github.com/luisbalru/MoEITS.
Figures & tables
Figure 1: Diagram of MoEITS. Starting from a block of model experts, the redundancy analysis metric between experts based on normalized mutual information is applied. Next, the most relevant experts are simplified and a new version of the model is generated by simplifying, inheriting the knowledge of the original model by means of the weight matrices. Note that the layers immediately preceding and following the simplified expert block must be adjusted.
Benchmark
Task Description
Key Focus
BoolQ
Question answering where the answer is either “yes” or “no.” It tests a model’s ability to understand a passage and answer a simple boolean question about it.
Reading comprehension, logical reasoning.
HellaSwag
Commonsense reasoning. Given a context, the model must choose the most likely ending from four options. It’s designed to be challenging for AI by including adversarial examples.
Commonsense inference, context understanding.
WinoGrande
A large-scale dataset of pronoun resolution problems. The model must determine the referent of a pronoun in a sentence, requiring fine-grained understanding of context and world knowledge.
A set of multiple-choice science questions designed for elementary school level. It tests a model’s ability to understand and reason about scientific concepts.
Scientific reasoning, knowledge application.
ARC-c
A more difficult subset of the ARC dataset, designed to require deeper reasoning and knowledge than ARC-e.
Table 1: Overview of LLM benchmarks used for evaluating reasoning and comprehension capabilities for simplification methods, showing task descriptions and key focus. For all benchmarks, accuracy is the metric used.
Method
BoolQ
ARC-e
ARC-c
WinoG
HellaS
Av
P ↓
MoE- I2 [ 59 ]
76.79
71.8
42.58
67.64
55.16
62.79
53.98%
MoE-Pruner [ 58 ]
76.61
71.89
40.02
67.64
50.94
61.42
-
DERN [ 65 ]
68.29
70.9
48.14
53.51
44.47
57.06
22.56%
MoEITS
80.03
77.61
43.77
67.72
70.8
67.99
55.42%
Table 2: Results of pruning DeepSeek-V2 Lite on BoolQ, ARC-e, ARC-c, WinoG (WinoGrande) and HelloS (HellaSwag) benchmarks. Av represents the average of the metrics. P ↓ represents the percentage reduction in the number of expert parameters. Best results are highlighted in bold.
Method
BoolQ
ARC-e
ARC-c
WinoG
HellaS
Av
P ↓
MoE- I2 [ 59 ]
75.08
71.68
41.13
66.54
53.08
61.5
53.98%
MoE-Pruner [ 58 ]
69.14
52.02
29.1
59.12
42.99
50.47
50%
MoP [ 26 ]
68.56
59.76
44.97
52.57
56.4
56.45
33%
MoEITS
75.2
72.3
36.01
67.46
61.27
62.45
52.92%
Table 3: Results of pruning Qwen1.5-2.7B on BoolQ, ARC-e, ARC-c, WinoG (WinoGrande) and HelloS (HellaSwag) benchmarks. Av represents the average of the metrics. P ↓ represents the percentage reduction in the number of expert parameters. Best results are highlighted in bold.
Method
BoolQ
ARC-e
ARC-c
WinoG
HellaS
Av
P ↓
S
MoE- I2 [ 59 ]
82.6
78.2
52.2
71.5
61.1
69.1
49%
-
MoE-Pruner [ 58 ]
86.0
81.9
53.3
75.5
62.3
71.8
0%
50%
STUN [ 34 ]
-
80.0
51.5
-
59.9
63.8
0%
65%
MC-MoE [ 27 ]
80.6
73.1
48.4
71.3
74.9
69.6
-
-
Lu et al. [ 39 ]
84.8
79.9
53.9
73.8
60.1
70.5
40%
-
C-GNN-PRUNE [ 35 ]
83.73
80.03
-
72.37
56.16
73.1
50%
-
Table 4: Results of pruning Mixtral 8×7 B on BoolQ, ARC-e, ARC-c, WinoG (WinoGrande) and HelloS (HellaSwag) benchmarks. Av represents the average of the metrics. P ↓ represents the percentage reduction in the number of expert parameters. S (sparsity) refers to the portion of parameters that are zero. Best results are highlighted in bold.
Figure 2: Evolution of the simplified Qwen1.5-2.7B model with MoEITS for different values of τ ( number of parameters , not retrained model accuracy and retrained model accuracy ). Low τ values facilitate substantial simplification (up to 70% parameter reduction for τ=0.5 ). The subsequent retraining phase proves critical, restoring accuracy by as much as 20% relative to the simplification level.