Mixture of Expert

Mixture-of-Experts (MoE) models aim to improve the efficiency and scalability of large language and other models by using multiple specialized "expert" networks, each handling a subset of the input data. Current research focuses on improving routing algorithms to efficiently assign inputs to experts, developing heterogeneous MoE architectures with experts of varying sizes and capabilities, and optimizing training methods to address challenges like load imbalance and gradient conflicts. This approach holds significant promise for creating larger, more powerful models with reduced computational costs, impacting various fields from natural language processing and computer vision to robotics and scientific discovery.

Papers

January 11, 2024

DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models
Damai Dai, Chengqi Deng, Chenggang Zhao, R. X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y. K. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, Wenfeng Liang
Language Model Mixture of Expert Expert Specialization

January 5, 2024

Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Tasks
Haoyuan Wu, Haisheng Zheng, Zhuolun He, Bei Yu
Instruction Tuning Mixture of Expert Sparse Model Parameter Efficient Sparsity General Task

January 4, 2024

k-Winners-Take-All Ensemble Neural Network
Abien Fred Agarap, Arnulfo P. Azcarraga
Mixture of Expert Model Ensembling Ensemble Neural Network Neural Network Performance

December 28, 2023

Fast Inference of Mixture-of-Experts Language Models with Offloading
Artyom Eliseev, Denis Mazur
Mixture of Expert Fast Inference Mixed Quantization Offloading Robot Functionality

December 27, 2023

Efficient Deweather Mixture-of-Experts with Uncertainty-aware Feature-wise Linear Modulation
Rongyu Zhang, Yulin Luo, Jiaming Liu, Huanrui Yang, Zhen Dong, Denis Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, Yuan Du, Shanghang Zhang
Mixture of Expert Multiple Expert

December 13, 2023

SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention
Róbert Csordás, Piotr Piękos, Kazuki Irie, Jürgen Schmidhuber
Transformer Megatron Decepticons Transformer Model Mixture of Expert Self Attention Layer Baseline Transformer

December 11, 2023

Mixture-of-Linear-Experts for Long-term Time Series Forecasting
Ronghao Ni, Zinan Lin, Shuaiqi Wang, Giulia Fanti
Mixture of Expert Object Centric Long Term Time Series Forecasting

December 2, 2023

Improving Normative Modeling for Multi-modal Neuroimaging Data using mixture-of-product-of-experts variational autoencoders
Sayantan Kumar, Philip Payne, Aristeidis Sotiras
Variational Autoencoder Variational Autoencoders Mixture of Expert Joint Generative Normative Modeling

December 1, 2023

Omni-SMoLA: Boosting Generalist Multimodal Models with Soft Mixture of Low-rank Experts
Jialin Wu, Xia Hu, Yaqing Wang, Bo Pang, Radu Soricut
Mixture Component Mixture of Expert Low Rank Expert Model Large Multi Modal Model Vision and Language Multimodal Generalization

November 29, 2023

Gene-MOE: A sparsely gated prognosis and classification framework exploiting pan-cancer genomic information
Xiangyu Meng, Xue Li, Qing Yang, Huanhuan Dai, Lian Qiao, Hongzhen Ding, Long Hao, Xun Wang
Mixture of Expert Prognosis Prediction

November 16, 2023

Towards Improving Robustness Against Common Corruptions using Mixture of Class Specific Experts
Shashank Kotyan, Danilo Vasconcellos Vargas
Native Robustness Mixture Component Mixture of Expert Class Specific Common Corruption

November 8, 2023

DAMEX: Dataset-aware Mixture-of-Experts for visual understanding of mixture-of-datasets
Yash Jain, Harkirat Behl, Zsolt Kira, Vibhav Vineet
Mixture of Expert Visual Understanding Object Detection Benchmark X Klaim Representation Collapse Universal Detection Dataset Mixture

November 1, 2023

Mixture-of-Experts for Open Set Domain Adaptation: A Dual-Space Detection Approach
Zhenbang Du, Jiayu An, Jiahao Hong, Dongrui Wu
Mixture of Expert Feature Space Open Set Domain Adaptation Class Sample

October 29, 2023

SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Models
Zhixu Du, Shiyu Li, Yuhao Wu, Xiangyu Jiang, Jingwei Sun, Qilin Zheng, Yongkai Wu, Ang Li, Hai "Helen" Li, Yiran Chen
High Efficiency Mixture of Expert Potential Scalability Efficient Inference Efficient Model Model Capacity Sparsity Aware

October 22, 2023

A General Theory for Softmax Gating Multinomial Logistic Mixture of Experts
Huy Nguyen, Pedram Akbarian, TrungTin Nguyen, Nhat Ho
Expert Knowledge Mixture of Expert Maximum Likelihood Softmax Gating

October 21, 2023

When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applications
Qidong Liu, Xian Wu, Xiangyu Zhao, Yuanshao Zhu, Derong Xu, Feng Tian, Yefeng Zheng
Large Language Model Multi Task Learning Mixture of Expert Efficient Fine Tuning Multi Task Medical

October 15, 2023

Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizer
Boan Liu, Liang Ding, Li Shen, Keqin Peng, Yu Cao, Dazhao Cheng, Dacheng Tao
Language Model Mixture of Expert Large Scale Language Model Inference Task Representation Diversity Orthogonal Gradient

October 11, 2023

Adaptive Gating in Mixture-of-Experts based Language Models
Jiamin Li, Qiang Su, Yitao Yang, Yimin Jiang, Cong Wang, Hong Xu
Large Language Model Language Model NLP Task Mixture of Expert Gating Mechanism

October 6, 2023

Exploiting Activation Sparsity with Dense to Dynamic-k Mixture-of-Experts Conversion
Filip Szatkowski, Bartosz Wójcik, Mikołaj Piórczyński, Simone Scardapane
Transformer Model Mixture of Expert Activation Sparsity

October 4, 2023

BA-MoE: Boundary-Aware Mixture-of-Experts Adapter for Code-Switching Speech Recognition
Peikun Chen, Fan Yu, Yuhao Lian, Hongfei Xue, Xucheng Wan, Naijun Zheng, Huan Zhou, Lei Xie
Mixture of Expert Language Specific Code Switching Speech Recognition Long Short Range Adapter Language Modeling Loss Mandarin English Code Switching

Mixture of Expert

Papers

DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models

Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Tasks

k-Winners-Take-All Ensemble Neural Network

Fast Inference of Mixture-of-Experts Language Models with Offloading

Efficient Deweather Mixture-of-Experts with Uncertainty-aware Feature-wise Linear Modulation

SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention

Mixture-of-Linear-Experts for Long-term Time Series Forecasting

Improving Normative Modeling for Multi-modal Neuroimaging Data using mixture-of-product-of-experts variational autoencoders

Omni-SMoLA: Boosting Generalist Multimodal Models with Soft Mixture of Low-rank Experts

Gene-MOE: A sparsely gated prognosis and classification framework exploiting pan-cancer genomic information

Towards Improving Robustness Against Common Corruptions using Mixture of Class Specific Experts

DAMEX: Dataset-aware Mixture-of-Experts for visual understanding of mixture-of-datasets

Mixture-of-Experts for Open Set Domain Adaptation: A Dual-Space Detection Approach

SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Models

A General Theory for Softmax Gating Multinomial Logistic Mixture of Experts

When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applications

Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizer

Adaptive Gating in Mixture-of-Experts based Language Models

Exploiting Activation Sparsity with Dense to Dynamic-k Mixture-of-Experts Conversion

BA-MoE: Boundary-Aware Mixture-of-Experts Adapter for Code-Switching Speech Recognition