Mixture-of-Experts Models

Follow this topic Ask about this topic

Momentum

41 papers in the last four weeks, up 116% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 297

All topics

TileQ: Efficient Low-Rank Quantization of Mixture-of-Experts with 2D Tiling
May 10, 2026Hongyaoxing Gu, Xinzhe Chen, Lijuan Hu +1Efficient Inference Mixture-of-Experts Inference
Institute of Software Chinese Academy of Sciences · University of Chinese Academy of Sciences
VECTOR-Drive: Tightly Coupled Vision-Language and Trajectory Expert Routing for End-to-End Autonomous Driving
May 9, 2026Rui Zhao, Jianlin Yu, Zhenhai Gao +2VLMs for Autonomous Driving End-to-End Autonomous Driving
College of Automotive Engineering, Jilin University · 2The National Key Laboratory of Automotive Chassis Integration and Bionics, Jilin University · 3ReeFocus AI Technology
ReLibra: Routing-Replay-Guided Load Balancing for MoE Training in Reinforcement Learning
May 9, 2026Chao Jin, Xinming Wei, Yinmin Zhong +6RL for Language Models Expert Load Balancing
School of Computer Science, Peking University · Independent Researcher
Uncovering Intra-expert Activation Sparsity for Efficient Mixture-of-Expert Model Execution
May 9, 2026Jongseok Park, Sunga Kim, Zhenyu Gu +2Activation Sparsity Mixture-of-Experts Inference
University of California, Berkeley · Work done during an internship at Advanced Micro Devices, Inc. · Advanced Micro Devices, Inc.
SDG-MoE: Signed Debate Graph Mixture-of-Experts
May 8, 2026Stepan Kulibaba, Kirill Labzin, Artem Dzhalilov +4Mixture-of-Experts Language Models Mixture of Experts
Innopolis University, Innopolis · Sirius University, Sirius · Innopolis University, Innopolis; Sirius University, Sirius +1
Hierarchical Mixture-of-Experts with Two-Stage Optimization
May 8, 2026Gleb Molodtsov, Alexander Miasnikov, Aleksandr BeznosikovExpert Load Balancing Mixture of Experts
BRAIn Lab, Moscow, Russia
When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Models
May 8, 2026Youngsik Yoon, Siwei Wang, Wei Chen +1Mixture-of-Experts Language Models LLM Routing
Department of Computer Science and Engineering, POSTECH, South Korea · Microsoft Research Asia, Beijing, China · Graduate School of Artificial Intelligence, POSTECH, South Korea
Geometric Asymmetry in MoE Specialization: Functional Decorrelation and Representational Overlap
May 8, 2026Feilong LiuTransformer Interpretability Neural Representation Geometry
Scene-Adaptive Continual Learning for CSI-based Human Activity Recognition with Mixture of Experts
May 7, 2026Wenhan Zheng, Yuyi Mao, Ivan Wang-Hei HoAdaptive Inference Human Activity Recognition
Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University, Hong Kong · School of Computer Science and Engineering, The Macau University of Science and Technology, Macau
E = T*H/(O+B): A Dimensionless Control Parameter for Mixture-of-Experts Ecology
May 7, 2026Qingjun ZhangExpert Load Balancing Mixture of Experts
School of Integrated Circuits, Wuxi Taihu University Department of Communication Engineering · Department of Communication Engineering
Expert Routing for Communication-Efficient MoE via Finite Expert Banks
May 6, 2026Mohammad Reza Deylam Salehi, Ali KhalesiMixture of Experts Information-Theoretic Generalization Bounds
IEEE Graduate Member · Nice, France · Institut Polytechnique des Sciences Avancées (IPSA) +1
Piper: Efficient Large-Scale MoE Training via Resource Modeling and Pipelined Hybrid Parallelism
May 6, 2026Sajal Dash, Feiyi WangCommunication-Efficient Distributed Training High-Performance Computing
Oak Ridge National Laboratory Oak Ridge, USA
Adaptive Inverted-Index Routing for Granular Mixtures-of-Experts
May 6, 2026Klaus-Rudolf Kladny, Maximilian Mordig, Bernhard Schölkopf +1Mixture-of-Experts Inference Expert Routing
Max Planck Institute for Intelligent Systems, Tübingen · Tübingen AI Center · ETH Zurich +1
AxMoE: Characterizing the Impact of Approximate Multipliers on Mixture-of-Experts DNN Architectures
May 6, 2026Omkar B Shende, Marcello Traiola, Gayathri AnanthanarayananEfficient Neural Network Inference Edge Inference
Dept. of Computer Science and Engineering · Univ Rennes, CNRS, Inria, IRISA
YOTOnet: Zero-Shot Cross-Domain Fault Diagnosis via Domain-Conditioned Mixture of Experts
May 6, 2026Zesen Wang, Zihao Wu, Yue Hu +2Cross-Domain Transfer Learning Bearing Fault Diagnosis
School of Mechanical and Power Engineering, East China University of Science and Technology, Shanghai, China
GEM: Graph-Enhanced Mixture-of-Experts with ReAct Agents for Dialogue State Tracking
May 6, 2026Ziqi Zhu, Adithya Suresh, Tomal Deb +1Spoken Dialogue Systems Mixture-of-Experts Models
Generative AI Innovation Center, Amazon Web Services
MILE: Mixture of Incremental LoRA Experts for Continual Semantic Segmentation across Domains and Modalities
May 5, 2026Shishir Muralidhara, Didier Stricker, René SchusterDomain-Incremental Learning Incremental Learning
German Research Center for Artificial Intelligence (DFKI), Trippstadter Str. 122, 67663 Kaiserslautern, Germany · University of Kaiserslautern-Landau (RPTU), Gottlieb-Daimler-Str. 47, 67663 Kaiserslautern, Germany
Ortho-Hydra: Orthogonalized Experts for DiT LoRA
May 5, 2026Seunghyun JiFine-Tuning Diffusion Transformer
RouteHijack: Routing-Aware Attack on Mixture-of-Experts LLMs
May 1, 2026Zhiyuan Xu, Joseph Gardiner, Sana Belguith +1Mixture-of-Experts Language Models Adversarial Attacks on LLMs
University of Bristol Bristol, United Kingdom
Prediction-powered Inference by Mixture of Experts
Apr 30, 2026Yanwu Gu, Linglong Kong, Dong XiaUncertainty Quantification Mixture-of-Experts Models
Department of Mathematics, Hong Kong University of Science and Technology · Department of Mathematical and Statistical Sciences, University of Alberta
Advancing multi-site emission control: A physics-informed transfer learning framework with mixture of experts for carbon-pollutant synergy
Apr 29, 2026Yuxuan Ying, Hanqing Yang, Kaige Wang +6Cross-Domain Transfer Learning Physics-Informed ML
College of Mechanical Engineering, Zhejiang University of Technology, Hangzhou, 310023, China. · Alibaba Group, Hangzhou, 311100, China. · School of Computer Science and Technology, Zhejiang Normal University, Hangzhou, 311231, China. +2
Adaptive and Fine-grained Module-wise Expert Pruning for Efficient LoRA-MoE Fine-Tuning
Apr 29, 2026Weihang Li, Jianchun Liu, Hongli XuMixture-of-Experts Pruning LLM Fine-Tuning
School of Computer Science and Technology, University of Science and Technology of China, China · Suzhou Institute for Advanced Research, University of Science and Technology of China, China
Mixture of Experts Framework in Machine Learning Interatomic Potentials for Atomistic Simulations
Apr 28, 2026Gabriel de Miranda Nascimento, Marc L. Descoteaux, Laura Zichi +9Machine Learning Interatomic Potentials Multifidelity Modeling
Department of Materials Science and Engineering, Massachusetts Institute of Technology, Cambridge, MA, USA · John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA, USA · Robert Bosch LLC Research and Technology Center, Watertown, MA, USA
SMoES: Soft Modality-Guided Expert Specialization in MoE-VLMs
Apr 27, 2026Zi-Hao Bo, Yaqian Li, Anzhou Hou +6Vision-Language Models Efficient VLM Inference
Li Auto Inc.
Scaling Multi-Node Mixture-of-Experts Inference Using Expert Activation Patterns
Apr 25, 2026Abhimanyu Bambhaniya, Geonhwa Jeong, Jason Park +6Expert Load Balancing Mixture-of-Experts Inference
1Georgia Institute of Technology, USA · 2Meta Platforms, Inc.
Preserving Long-Tailed Expert Information in Mixture-of-Experts Tuning
Apr 24, 2026Haoze He, Xingyuan Ding, Xuan Jiang +5Supervised Fine-Tuning Fine-Tuning
Carnegie Mellon University · MIT · UCSD +1
Towards Adaptive Continual Model Merging via Manifold-Aware Expert Evolution
Apr 24, 2026Haiyun Qiu, Xingyu Wu, Kay Chen TanAdaptive Model Routing Mixture-of-Experts Models
Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University, Hong Kong, China
On Bayesian Softmax-Gated Mixture-of-Experts Models
Apr 22, 2026Nicola Bariletto, Huy Nguyen, Nhat Ho +1Bayesian Mixture Models Bayesian Inference
The University of Texas at Austin
Temporally Extended Mixture-of-Experts Models
Apr 22, 2026Zeyu Shen, Peter HendersonMixture-of-Experts Inference Memory-Efficient Inference
1Princeton University
Expert Upcycling: Shifting the Compute-Efficient Frontier of Mixture-of-Experts
Apr 21, 2026Chaitanya Dwivedi, Binxuan Huang, Himanshu Gupta +3Efficient Language Model Training Language Model Scaling
1Amazon Stores Foundation AI · 3Anthropic · 2Carnegie Mellon University

May 10, 2026, cs.LGJ/K move · Enter open · S save

TileQ: Efficient Low-Rank Quantization of Mixture-of-Experts with 2D Tiling

Hongyaoxing Gu, Xinzhe Chen, Lijuan Hu, Fangfang Liu

Institute of Software Chinese Academy of Sciences · University of Chinese Academy of Sciences

Efficient Inference Mixture-of-Experts Inference Mixture-of-Experts Quantization Mixture-of-Experts Models Post-Training Quantization Neural Network Quantization

Open paper AskPDF arXiv

Mixture-of-Experts (MoE) models achieve remarkable performance by sparsely activating specialized experts, yet their massive parameters in experts pose significant challenges for deployment. While low-rank quantization offers a promising route to compress MoE models, existing methods still incur nonnegligible memory overhead and inference latency. To address these limitations, we propose \textsc{TileQ}, a fine-tuning-free post-training quantization (PTQ) method that employs 2D-tiling structured low-rank quantization to share low-rank factors across both input and output dimensions of MoE experts. Furthermore, we introduce an efficient inference technique for \textsc{TileQ} that fuses multiple low-rank expert computations into a single-pass operation, significantly improving hardware utilization. Experiments show that \textsc{TileQ} cuts down additional memory usage up to 10

\times

and reduces inference latency to

\sim

5% while preserving state-of-the-art accuracy.

Momentum

Related topics

Latest papers 297