Mixture of Experts

Also known as MoE

Mixture of Experts models use a gate to assign inputs or tokens to different neural subnetworks, allowing one model to handle varied tasks or data while computing only with selected experts. Common approaches include sparse top-k routing, shared and hierarchical experts, and lightweight expert adapters; they are used in language models, vision, multimodal systems, forecasting, and robotics. The central challenge is encouraging useful specialization without expert collapse, poor load balance, or costly communication and memory demands. Researchers are also examining whether routed experts capture meaningful skills or merely reflect patterns in hidden representations, and how routing can be made more interpretable and reliable.

Latest papers

229 papers26 papers in the last four weeks, down 19% on the four weeks before. 0.2% of all new papers.

Unlocking the Potential of Vision Foundation Models for Multi-Task Dense Prediction via Mixture of Task Adapters
Nov 2, 2026Yunzhi Zhuge, Xinzhuo Yu, Sitong Gong +3Vision Foundation Model Adaptation Multi-Task Learning
the School of Information and Communication Engineering, Dalian University of Technology, Dalian 116081, China · the School of Computer Science and Technology, Dalian University of Technology, Dalian 116081, China
MoRE-GS: Motion-Regime Experts for Future Gaussian Splatting
Oct 30, 2026Shima Salehi, Joshua PeeplesDynamic Novel View Synthesis Dynamic Gaussian Splatting
Department of Electrical and Computer Engineering, Texas A&M University, College Station, Texas, USA
Experts Exhibit Structured Redundancy: Structure-Preserving Expert Pruning for Mixture-of-Experts Large Language Models
Oct 30, 2026Tian Wang, Yongding Tao, Yihong Dong +3Mixture-of-Experts Pruning Mixture-of-Experts Language Models
Peking University · Key Laboratory of High Confidence Software Technologies, Peking University, Ministry of Education · Shanghai Jiaotong University +2
Softmax Routing Dynamics in Mixture-of-Experts Models: Collapse and Recovery
Oct 29, 2026Shengze Xu, Maolin Deng, Tieyong ZengMixture of Experts Mixture-of-Experts Models
The Chinese University of Hong Kong · Beijing Normal-Hong Kong Baptist University · Guangzhou Nanfang College
IMRouter: Imbalance-aware Temporal Pattern Routing for LLM-based Time Series Forecasting
Oct 29, 2026Xin Yang, Cai Xu, Haishun Chen +4Time Series Forecasting LLM-Based Time Series Forecasting
Xidian University, Xi’an, Shaanxi, China · Chang’an University, Xi’an, Shaanxi, China
SAGE: Semantic-Augmented Geometric Experts for Point Cloud Registration
Oct 27, 2026Tomoyasu Shimada, Kazuhiko Murasaki, Shoichiro Takeda +4Point Cloud Registration Point Cloud Learning
NTT, Inc.
WaFIR: Wavelet-Fourier Mixture-of-Experts with Subband-Adaptive Routing for Blind All-in-One Image Restoration
Oct 21, 2026Trong Nhan Nguyen, Vu Hoang Minh Doan, Thanh Dat Le +6Image Deraining Image Restoration
Pukyong National University, Busan, Republic of Korea · University of Alabama at Birmingham, Birmingham, AL, USA
LaMER: Language-Conditioned Multi-Expert Routing for Weakly Supervised Referring Expression Comprehension
Oct 19, 2026Yu Yu Chang, Tzu-Hsuan Weng, Youssef Mohamed Amin +4Vision-Language Grounding Mixture of Experts
National Central University, Taoyuan, Taiwan · Pharos University in Alexandria, Alexandria, Egypt
MoWE: Spatially Adaptive Mixture-of-Wavelet-Experts Pooling for Ultrasound Lesion Segmentation
Oct 18, 2026Reaz Hassan Joarder, Sidratul Tanzila Tasmi, Alif Arshad Bakshi +2Ultrasound Image Segmentation Mixture of Experts
Islamic University of Technology, Bangladesh · United International University, Bangladesh
CART: Compatible Routing for Multimodal Continual Instruction Tuning
Oct 15, 2026Ling-Xiang Zhang, Tao Hu, Da-Wei ZhouContinual Learning for VLMs Multimodal Large Language Models
School of Artificial Intelligence, Nanjing University · State Key Laboratory of Novel Software Technology, Nanjing University
Explaining the Roles of Experts in Mixture-of-Experts LLMs for Personalization
Oct 15, 2026Sekh Mainul Islam, Pepa Atanasova, Isabelle AugensteinMixture-of-Experts Expert Specialization Personalized Language Models
University of Copenhagen
MAEM: Mitigating Recency Bias via a Replay-Free, Capacity-Controlled Dynamic Mixture of Experts for Class-Incremental Graph Learning
Oct 14, 2026Junlin Chen, Wenhao Wang, Junchao Xu +5Communication-Efficient Mixture-of-Experts Class-Incremental Learning
Beihang University, Beijing, China · Renmin University of China, Beijing, China · Trine University, Indiana, USA +1
HierarchiRetina: A Diabetic Retinopathy Screening and Grading Cascade with Heterogeneous Mixture-of-Experts Lesion Segmentation
Oct 13, 2026Md Asif Uddin, Niloy Farhan, Rafeed Rahman +1Diabetic Retinopathy Grading Retinal Disease Classification
The authors are with the Department of Computer Science and Engineering, Brac University, Dhaka, Bangladesh
Towards Complementary Keypoint Detection via Mixture of Detectors
Oct 12, 2026Xiaoyong Lu, Guobao Xiao, Dong Liang +1Keypoint Detection Mixture of Experts
Southeast University · Tongji University · Nanjing University of Aeronautics and Astronautics
SpikeOcc: Towards Resilient Multimodal 3D Occupancy Perception via Spiking Experts
Oct 11, 2026Yiding Sun, Zijie Xu, Hongqiang Lin +73D Occupancy Prediction Multimodal Robustness
Peking University · Xi’an Jiaotong University · Zhejiang University +1
BaT-SAM: Introducing Inductive Bias into SAM via a Mixture of Convolutional Experts for Brain Tumor Segmentation
Oct 11, 2026Sumiaya Jannat, Md Mahedi Hasan, Md Shamimul Islam +1Medical Image Segmentation Brain Tumor Segmentation
Mymensingh Medical College Mymensingh, Bangladesh · Department of LCSEE West Virginia University Morgantown, WV, USA · Department of CS Florida Polytechnic University Lakeland, Florida, USA +1
Shaping Latent Mixture-of-Experts: From Expert Geometry to Layerwise Capacity Allocation
Oct 10, 2026Yaoxiang Wang, Kai Zhang, Ruobing Xie +4Mixture of Experts Mixture-of-Experts Models
Xiamen University · Tencent Hunyuan · Soochow University
State-Isolated Vision Mamba via Update-Tendency Routing for Underwater Image Enhancement
Oct 9, 2026Yan Luo, Laibin Chang, Zefan Feng +3Visual SSMs Underwater Image Enhancement
TAME-Nav: Temporal Asymmetric Mixture-of-Experts Distillation for Agile and Versatile Robot Navigation
Oct 9, 2026Lingjie Zhang, Jun Xue, Shanze Wang +4Robot Navigation Robot Policy Learning
Shanghai Jiao Tong University, Shanghai, China. · Eastern Institute of Technology, Ningbo, China. · Department of Aeronautical and Aviation Engineering, The Hong Kong Polytechnic University, Hung Hom, Kowloon, Hong Kong. +1
One Block, Multiple Depths: Recurrent Vision Transformers with Depth-Programmed Experts
Oct 8, 2026Adrian Bulat, Yassine Ouali, Georgios TzimiropoulosEfficient ViTs Mixture of Experts
Samsung AI Cambridge · Technical University of Iasi · Queen Mary University of London
Syn-Omni: Structured Specialization and Progressive Collaboration for Omnimodal Embeddings
Oct 8, 2026Youngtaek Oh, Qiyu Wu, Hiromi Wakaki +2Multimodal Embedding Cross-Modal Representation Learning
KAIST · Sony Group Corporation
RouterInterp: Understanding Superposed Specialisation in Mixture of Experts Routing
Oct 8, 2026Ilya Lasy, Nora Yinuo Cai, Kola AyonrindeMixture-of-Experts Interpretability Mixture of Experts
Faculty of Informatics, TU Wien · Independent · UK AI Security Institute
EvoKnow: Continual Knowledge Evolution for AI-Generated Image Detection
Oct 8, 2026Zhiheng Peng, Wenwei Jin, Yangshi Ge +3AI-Generated Image Detection Mixture of Experts
Xiaohongshu Inc. · Southeast University
Expert Coupling in MoE Pretraining: Reducing All-to-All Overhead with Correlated Placement and Token Shuffling
Oct 7, 2026Radha Gulhane, Quentin Anthony, Beren MillidgeCommunication-Efficient Mixture-of-Experts Mixture-of-Experts Language Models
Zyphra
MoF: Preference-Aware Mixture Modeling for Black-Box LLM Personalization
Oct 6, 2026Hun ParkLLM Personalization Personalized Language Models
Department of Artificial Intelligence Korea University Seoul, Republic of Korea
Anchor-driven Multi-modal Multi-scale Expert Selection for Survival Prediction
Oct 6, 2026Tao Zhou, Ying Hu, Huazhu Fu +3Cancer Survival Prediction Mixture of Experts
School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, China · Institute of Advanced Intelligence and Computing (IAIC), Agency for Science, Technology and Research (A*STAR), Singapore 138632 · School of Computer Science and Engineering, Southeast University, Nanjing 211189, China +2
Stepped MoE: Segment-Level Routing with Configurable Inference Complexity
Oct 5, 2026Arnav Kundu, Zhaoyang Xu, Bairu Hou +3On-Device Language Model Inference Dynamic Neural Networks
Apple
LoopMoEVR: Loop-Based Degradation-Aware Mixture-of-Experts for Unified UHD Video Restoration
Oct 4, 2026Yucheng Xin, Runci Bai, Yongcong Wang +5Mixture of Experts Image Restoration
Shandong Normal University · China Academy of Information and Communications Technology · Central South University +4
Harnessing Domain Specialists in Multimodal Mixture-of-Experts for Efficient Adaptation
Oct 1, 2026Damiano Marsili, Raphi Kang, Aditya Mehta +2Mixture of Experts Mixture-of-Experts Models
California Institute of Technology
Redundancy Meets Synergy: Dependency-aware Expert Selection for MoE via Submodular Optimization
Sep 30, 2026Zheng Lin, Shaoke Fang, Yuxin Zhang +6Submodular Optimization Mixture-of-Experts Pruning
Interdisciplinary Centre for Security, Reliability and Trust, University of Luxembourg, Luxembourg · Department of Computer Science, Peking University, China · Institute of Space Internet, Fudan University, China +4

Nov 2, 2026J/K move · Enter open · S save

Unlocking the Potential of Vision Foundation Models for Multi-Task Dense Prediction via Mixture of Task Adapters

Yunzhi Zhuge, Xinzhuo Yu, Sitong Gong, Lu Zhang+2

the School of Information and Communication Engineering, Dalian University of Technology, Dalian 116081, China · the School of Computer Science and Technology, Dalian University of Technology, Dalian 116081, China

Vision Foundation Model Adaptation Multi-Task Learning Mixture of Experts Low-Rank Adaptation

Open paper AskPDF arXiv

Multi-task dense prediction seeks a single, coherent representation that supports simultaneously discriminative semantic recognition and regularized geometric/structural inference. For vision foundation models, pretrained semantic invariances conflict with the frequency and smoothness biases of dense objectives; joint fine-tuning and single-route adapters therefore compress competing gradients into one update channel, causing negative transfer and unstable cross-task agreement. We frame this as two-level adaptation: task-generic alignment that preserves priors, and task-aware specialization that resolves late-stage conflicts. Based on this principle, we propose D2MoE , a Mixture of Task Adapters that explicitly factorizes these roles within a frozen DINOv3 backbone. D2MoE introduces Task-Generic Low-Rank Adaptation (TG-LoRA) in attention projections throughout the network to provide a stable, shared adaptation channel, and a Task-Adaptive Mixture of Experts (TA-MoE) injected only at the final attention block of each stage to enact sparse, task-conditioned perturbations of token mixing in a single forward pass. A Semantic Prior Injector further couples multi-level task features with frozen foundation priors to reinforce semantic coherence while preserving boundary and structural fidelity. Experiments across diverse benchmarks substantiate that this structured factorization yields robust multi-task improvements with a small trainable footprint, establishing a principled and scalable route to prior-preserving, foundation-centric multi-task dense perception.