Mixture-of-Experts Models

Follow this topic Ask about this topic

Momentum

41 papers in the last four weeks, up 116% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 297

All topics

SAMoRA: Semantic-Aware Mixture of LoRA Experts for Task-Adaptive Learning
Apr 21, 2026Boyan Shi, Wei Chen, Shuyuan Zhao +4Multi-Task Learning Low-Rank Adaptation
School of Computer Science and Technology, Beijing Jiaotong University, China · 2Guangxi Key Lab of Trusted Software, Guilin University of Electronic Technology, China · 3Beijing Key Lab of Traffic Data Mining and Embodied Intelligence, China +2
Multi-Domain Learning with Global Expert Mapping
Apr 20, 2026Pourya Shamsolmoali, Masoumeh Zareapoor, Huiyu Zhou +3Vision Foundation Model Adaptation Expert Routing
Department of Computer Science, University of York, UK · School of Computing and Mathematical Sciences, University of Leicester, UK · Centre for Vision Speech and Signal Processing, University of Surrey, UK +2
Domain-Specialized Object Detection via Model-Level Mixtures of Experts
Apr 20, 2026Svetlana Pavlitska, Malte Stüven, Beyza Keskin +1Object Detection Mixture-of-Experts Models
FZI Research Center for Information Technology · Karlsruhe Institute of Technology (KIT)
Polysemantic Experts, Monosemantic Paths: Routing as Control in MoEs
Apr 20, 2026Charles Ye, Bo Yuan, Lee SharkeyLLM Interpretability Neural Network Interpretability
Independent · Georgia Institute of Technology · Goodfire
IMA-MoE: An Interpretable Modality-Aware Mixture-of-Experts Framework for Characterizing the Neurobiological Signatures of Binge Eating Disorder
Apr 18, 2026Lin Zhao, Qiaohui Gao, Elizabeth Martin +5Multimodal Classification Mixture-of-Experts Models
Department of Biomedical Engineering, New Jersey Institute of Technology, Newark, 07102, NJ, USA · College of Engineering, Northeastern University, Boston, 02115, MA, USA · Department of Psychiatry, Icahn School of Medicine at Mount Sinai, New York, 30602, NY, 10029 +1
CoGR-MoE: Concept-Guided Expert Routing with Consistent Selection and Flexible Reasoning for Visual Question Answering
Apr 18, 2026Xiyin Zeng, Yi Lu, Hao WangVisual Question Answering Expert Routing
Hong Kong University of Science and Technology (Guangzhou)
Application of a Mixture of Experts-based Foundation Model to the GlueX DIRC Detector
Apr 17, 2026Cristiano Fanelli, James Giroux, Cole Granger +1High-Energy Physics Mixture-of-Experts Models
William & Mary, Department of Data Science, Williamsburg, VA 23185, USA · William & Mary, Department of Physics, Williamsburg, VA 23185, USA
FL-MHSM: Spatially-adaptive Fusion and Ensemble Learning for Flood-Landslide Multi-Hazard Susceptibility Mapping at Regional Scale
Apr 17, 2026Aswathi Mundayatt, Jaya Sreevalsan-NairMixture-of-Experts Models
Graphics-Visualization-Computing Lab, International Institute of Information Technology Bangalore, Karnataka 560100, India.
Geometric Metrics for MoE Specialization: From Fisher Information to Early Failure Detection
Apr 16, 2026Dongxin Guo, Jikun Wu, Siu Ming YiuInformation Geometry Mixture of Experts
The University of Hong Kong · Hong Kong, China · Brain Investing Limited
Symbiotic-MoE: Unlocking the Synergy between Generation and Understanding
Apr 9, 2026Xiangyue Liu, Zijian Zhang, Miles Yang +3Cross-Modal Representation Learning Unified Multimodal Models
HKUST · Tencent Hunyuan
StanceMoE: Mixture-of-Experts Architecture for Stance Detection
Apr 1, 2026Abdullah Al Shafi, Md. Milon Islam, Sk. Imran Hossain +1Mixture-of-Experts Models
Department of Computer Science and Engineering, Khulna University of Engineering & Technology
Self-Routing: Parameter-Free Expert Routing from Hidden States
Apr 1, 2026Jama Hussein Mohamud, Drew Wagner, Mirco RavanelliParameter-Free Mixture-of-Experts Routing LLM Routing
Mila - Quebec AI Institute · Universite de Montreal · Concordia University
ExFusion: Efficient Transformer Training via Multi-Experts Fusion
Mar 30, 2026Jiacheng Ruan, Daize Dong, Xiaoye Qu +5Transformer Mixture of Experts
School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University, No 800, Dongchuan Road, Minhang District, Shanghai, China, 200240 · Shanghai Artificial Intelligence Laboratory, Shanghai, China, 200240 · Huazhong University of Science and Technology, Hubei, China, 430074 +3
MoE-ACT: Scaling Multi-Task Bimanual Manipulation with Sparse Task-Conditioned Mixture-of-Experts Transformers
Mar 16, 2026Kangjun Guo, Haichao Liu, Yanji Sun +3Bimanual Robotic Manipulation Visuomotor Policy Learning
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou 511453, China · The Hong Kong University of Science and Technology, Hong Kong SAR, China
Feature-level Interaction Explanations in Multimodal Transformers
Mar 4, 2026Yeji Kim, Housam Khalifa Bashier Babiker, Mi-Young Kim +1Transformer Interpretability Feature Attribution
University of Alberta, Edmonton, AB, Canada · University of Waikato, Hamilton, New Zealand · University of Alberta, Camrose, AB, Canada
Modality-Guided Mixture of Structured Experts with Entropy-Triggered Routing for Multimodal Recommendation
Feb 24, 2026Ji Dai, Quan Fang, DeSheng CaiSparse Mixture-of-Experts Recommender Systems
Beijing University of Posts and Telecommunications, China · Beijing University of Posts and Telecommunications, Beijing, China · Tianjin University of Technology, China
OmniMoE: An Efficient MoE by Orchestrating Atomic Experts at Scale
Feb 5, 2026Jingze Shi, Zhangyang Peng, Yizhang Zhu +3Mixture-of-Experts Inference LLM Inference Acceleration
The Hong Kong University of Science and Technology (Guangzhou) · Beijing Academy of Artificial Intelligence
L2R: Low-Rank and Lipschitz-Controlled Routing for Mixture-of-Experts
Jan 29, 2026Minghao Yang, Ren Togo, Guang Li +2Expert Routing Mixture-of-Experts Models
Hokkaido University
HiMoE-VLA: Hierarchical Mixture-of-Experts for Generalist Vision-Language-Action Policies
Dec 5, 2025Zhiying Du, Bei Liu, Yaobo Liang +7Visuomotor Policy Learning Vision-Language-Action Models
1Fudan University · 2Microsoft Research Asia · 3Xi’an Jiaotong University +1
PuzzleMoE: Efficient Compression of Large Mixture-of-Experts Models via Sparse Expert Merging and Bit-packed inference
Nov 6, 2025Yushu Zhao, Zheng Wang, Minjia ZhangLLM Compression Mixture-of-Experts Models
1Tsinghua University · †Work done while intern at UIUC. · University of Illinois Urbana-Champaign.
Variational Mixture of Graph Neural Experts for Alzheimer's Disease Recognition across Frequency Bands in EEG Brain Networks
Oct 13, 2025Jun-En Ding, Anna Zilverstand, Shihao Yang +2Alzheimer's Disease Electroencephalography
Department of Industrial and Systems Engineering, Rutgers University, Piscataway, NJ, USA · Department of Psychiatry and Behavioral Sciences, University of Minnesota, Minneapolis, MN, USA · Institute of Brain Science, College of Medicine, National Yang-Ming Chiao Tung University, Taipei City, Taiwan
SHMoAReg: Spark Deformable Image Registration via Spatial Heterogeneous Mixture of Experts and Attention Heads
Sep 24, 2025Yuxi Zheng, Jianhui Feng, Tianran Li +2Medical Image Registration Mixture-of-Experts Models
Institute of Science and Technology for Brain-inspired Intelligence Fudan University Shanghai, China · Department of Radiology Leiden University Medical Center Leiden, The Netherlands
Macro Graph of Experts for Billion-Scale Multi-Task Recommendation
Jun 12, 2025Hongyu Yao, Zijin Hong, Hao Chen +6Graph Representation Learning Multi-Task Learning
The Hong Kong Polytechnic University Hong Kong SAR, China · National University of Defense Technology Changsha, China · Beihang University Beijing, China
Multi-Modal Time Series Prediction via Mixture of Modulated Experts
Date pendingLige Zhang, Ali Maatouk, Jialin Chen +3Multivariate Time Series Forecasting Time Series Forecasting
Emory University · Yale University · Amazon
CMoE: Contrastive Mixture of Experts for Motion Control and Terrain Adaptation of Humanoid Robots
Date pendingShihao Ma, Hongjin Chen, Zijun Xu +6Humanoid Robot Locomotion Terrain-Aware Robot Locomotion
College of Intelligent Robotics and Advanced Manufacturing, Fudan University, Shanghai, China, 200433
Are Independently Estimated View Uncertainties Comparable? Unified Routing for Trusted Multi-View Classification
Date pendingYilin Zhang, Cai Xu, Haishun Chen +2Evidential Deep Learning Uncertainty Quantification
Xidian University Xi’an, China
SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers
Date pendingShaowen Wang, Ge Zhang, Kairong Luo +6Language Model Scaling Laws Transformer Attention
Tsinghua University · ByteDance Seed · M-A-P

Apr 21, 2026, cs.CLJ/K move · Enter open · S save

SAMoRA: Semantic-Aware Mixture of LoRA Experts for Task-Adaptive Learning

Boyan Shi, Wei Chen, Shuyuan Zhao, Junfeng Shen+3

School of Computer Science and Technology, Beijing Jiaotong University, China · 2Guangxi Key Lab of Trusted Software, Guilin University of Electronic Technology, China · 3Beijing Key Lab of Traffic Data Mining and Embodied Intelligence, China+2

Multi-Task Learning Low-Rank Adaptation Mixture-of-Experts Models Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

The combination of Mixture-of-Experts (MoE) and Low-Rank Adaptation (LoRA) has shown significant potential for enhancing the multi-task learning capabilities of Large Language Models. However, existing methods face two primary challenges: (1)Imprecise Routing in the current MoE-LoRA method fails to explicitly match input semantics with expert capabilities, leading to weak expert specialization. (2)Uniform weight fusion strategies struggle to provide adaptive update strengths, overlooking the varying complexity of different tasks. To address these limitations, we propose SAMoRA (Semantic-Aware Mixture of LoRA Experts), a novel parameter-efficient fine-tuning framework tailored for task-adaptive learning. Specifically, A Semantic-Aware Router is proposed to explicitly align textual semantics with the most suitable experts for precise routing. A Task-Adaptive Scaling mechanism is designed to regulate expert contributions based on specific task requirements dynamically. In addition, a novel regularization objective is proposed to jointly promote expert specialization and effective scaling. Extensive experiments on multiple multi-task benchmarks demonstrate that SAMoRA significantly outperforms the state-of-the-art methods and holds excellent task generalization capabilities. Code is available at https://github.com/boyan-code/SAMoRA

Momentum

Related topics

Latest papers 297