Sparse Mixture-of-Experts

Follow this topic Ask about this topic

Momentum

11 papers in the last four weeks, up 57% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 80

All topics

Dual-Adaptive SAM3: Hierarchical Routing over Low-Rank Expert Layers for Parameter-Efficient Medical Image Segmentation
Jun 30, 2026Ying Chen, Jinyue Li, Kun Wang +2VLM Adaptation Text-Guided Image Segmentation
Shenzhen Research Institute, The Chinese University of Hong Kong · University of Science and Technology of China · Nanyang Technological University +1
TF-MoE: Time-Frequency Mixture-of-Experts for Efficient Speech Separation
Jun 28, 2026Qinzhe Hu, Chenda Li, Wangyou Zhang +3Sparse Mixture-of-Experts Speech Processing
Auditory Cognition and Computational Acoustics Lab Shanghai Jiao Tong University, Shanghai, China · Microsoft Research Asia, China
CLEAR-MoE: Shared-Basis Expert Extraction from Frozen Vision Transformers via Calibration-Driven Layer Selection
Jun 26, 2026Md Irtiza Hossain, Humaira Ayesha, Junaid Ahmed SifatVision Transformer Low-Rank Matrix Decomposition
Brac University
SARA: Unlocking Multilingual Knowledge in Mixture-of-Experts via Semantically Anchored Routing Alignment
Jun 24, 2026Tianyu Dong, Yangyang Liu, Jiang Zhou +9Multilingual Language Models Sparse Mixture-of-Experts
TJUNLP Lab, School of Computer Science and Technology, Tianjin University, China · Alibaba Group, China
How Modular Is a Frontier Mixture-of-Experts? A Pre-registered Causal Test in Which Apparent Expert Modularity Mostly Dissolves
Jun 23, 2026Tony Salomone, Deep Gandhi, Ali AsariaSparse Mixture-of-Experts Mixture-of-Experts Models
Transformer Lab
Geometric and Stochastic Analysis of Discontinuities in Sparse Mixture-of-Experts
Jun 17, 2026Tho Tran Huu, Huu-Tuan Nguyen, Thien-Hai Nguyen +4Mixture of Experts Sparse Mixture-of-Experts
Department of Mathematics, National University of Singapore, Singapore · Faculty of Computer Science and Engineering, Ho Chi Minh City University of Technology (HCMUT), VNU-HCM, Ho Chi Minh City, Vietnam
SoftMoE: Soft Differentiable Routing for Mixture-of-Experts in LLMs
Jun 16, 2026Mikołaj Zasada, Łukasz Struski, Jacek Tabor +1Mixture-of-Experts Language Models Sparse Mixture-of-Experts
AGH University of Krakow, Poland · Faculty of Mathematics and Computer Science, Jagiellonian University, Poland · Jagiellonian University, Poland +1
SPRI: SVD-Partitioned Residual Initialization for Data-Constrained MoE Upcycling
Jun 15, 2026Weiqiao Shan, Ruixiang Mao, Yuang Li +10Sparse Mixture-of-Experts
Northeastern University, China · Huawei TSC, China · CUHK-Shenzhen, China +4
FAME: Forecastability-Aware Mixture of Experts for Heterogeneous Time Series Forecasting
Jun 8, 2026Qianyang Li, Xingjun Zhang, Shaoxun Wang +2Sparse Mixture-of-Experts Time Series Forecasting
School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an, China · Department of Research and Development, Shandong New Beiyang Information Technology Co., Ltd., WeiHai 264200, China · Department of Computer Science and Technology, Tsinghua University, Beijing 100084
STAR: Rethinking MoE Routing as Structure-Aware Subspace Learning
Jun 7, 2026Sumin Park, Noseong ParkSparse Mixture-of-Experts Expert Routing
Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Republic of Korea.
Sparsely gated tiny linear experts
Jun 5, 2026Simon SchugTransformer FFNs Mixture-of-Experts Language Models
Department of Computer Science Princeton University
SHAPE: Coalition-Aware Expert Pruning for Sparse Mixture-of-Experts LLMs
Jun 3, 2026Yuhao ZhangMixture-of-Experts Pruning LLM Pruning
Beihang University
Sparse Mixture-of-Experts Reward Models Learn Interpretable and Specialized Experts for Personalized Preference Modeling
Jun 2, 2026Yifan Wang, Jinyi Mu, Mayank Jobanputra +5Pairwise Preference Learning Reward Modeling
Saarland University · Independent Researcher · Bielefeld University +2
Expert-Aware Causal Tracing of Factual Recall in Sparse MoE Language Models
Jun 2, 2026Yuetian Lu, Ali Modarressi, Yihong Liu +1Mixture-of-Experts Language Models Sparse Mixture-of-Experts
Center for Information and Language Processing (CIS), LMU Munich
PRISM: Synergizing Vision Foundation Models via Self-organized Expert Specialization
Jun 2, 2026Ying Tang, Dong Li, Youjia Zhang +3Vision Foundation Models Sparse Mixture-of-Experts
School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, China.
Hierarchically Decoupled Mixture-of-Experts for Robust Traffic Sign Recognition in Complex Driving Scenarios
Jun 1, 2026Mingxiao Wang, Xiaozhen Qu, Bolin Gao +2Sparse Mixture-of-Experts Autonomous Driving Perception
School of Automotive and Traﬀic Engineering, Liaoning University of Technology, Jinzhou, Liaoning 121001, China. · State Key Laboratory of Intelligent Green Vehicles and Mobility, School of Vehicle and Mobility, Tsinghua University, Beijing 100084, China.
ProbMoE: Differentiable Probabilistic Routing for Mixture-of-Experts
Jun 1, 2026Heng Zhao, Zilei Shao, Guy Van den Broeck +1Sparse Mixture-of-Experts Expert Routing
Department of Computer Science, University of Virginia, Charlottesville, USA · Department of Computer Science, University of California, Los Angeles, Los Angeles, USA
Reusing Fusion-Time Spectral Reliability for Adaptive Fusion and Expert Routing in RGB-Infrared Object Detection
May 31, 2026Yefeng WuMultimodal Robustness Adaptive Model Routing
DAG-MoE: From Simple Mixture to Structural Aggregation in Mixture-of-Experts
May 31, 2026Jiarui Feng, Hanqing Zeng, Karish Grover +11Mixture-of-Experts Language Models Sparse Mixture-of-Experts
Meta MRS · Washington University in St. Louis · Carnegie Mellon University +1
Eigenvectors of Experts are Training-free Non-collapsing Routers
May 29, 2026Giang Do, Hung Le, Truyen TranParameter-Free Mixture-of-Experts Routing Mixture of Experts
Applied Artificial Intelligence Intiative (A2I2), Deakin University, Victoria, Australia.
AME-TS: Anchored Mixture-of-Experts for Time Series Forecasting
May 24, 2026Rui Wang, Renhao Xue, Ray Razi +2Sparse Mixture-of-Experts Time Series Forecasting
Amazon Web Services
Beyond Routing: Characterising Expert Tuning and Representation in Vision Mixture-of-Experts
May 20, 2026Gene Tangtartharakul, Katherine R. StorrsVisual Representation Learning Mixture of Experts
School of Psychology · University of Auckland · Auckland, New Zealand
UB-SMoE: Universally Balanced Sparse Mixture-of-Experts for Resource-adaptive Federated Fine-tuning of Foundation Models
May 15, 2026Van-Tuan Tran, Hong-Hanh Nguyen-Le, Marco Ruffini +1Expert Load Balancing Sparse Mixture-of-Experts
School of Computer Science and Statistics, Trinity College Dublin, Ireland · School of Computer Science, University College Dublin, Ireland
When Does Sparse MoE Help in Vision? The Role of Backbone Compute Leverage in Sparse Routing
May 15, 2026Libo Sun, Po-wei Harn, Peixiong He +1Sparse Mixture-of-Experts Expert Routing
Department of Computer Science and Software Engineering Auburn University, Auburn, AL 36849, USA · Department of Information Management National Central University, Taoyuan 320317, Taiwan · Department of Computer Science and Software Engineering Auburn University
Eradicating Negative Transfer in Multi-Physics Foundation Models via Sparse Mixture-of-Experts Routing
May 14, 2026Ellwil Sharma, Arastu SharmaSparse Mixture-of-Experts PDE Operator Learning
Shodh AI
BEAM: Binary Expert Activation Masking for Dynamic Routing in MoE
May 14, 2026Juntong Wu, Jialiang Cheng, Qishen Yin +5Sparse Mixture-of-Experts Mixture-of-Experts Inference
Taobao & Tmall Group of Alibaba · Shenzhen Graduate School, Peking University
HodgeCover: Higher-Order Topological Coverage Drives Compression of Sparse Mixture-of-Experts
May 13, 2026Tao Zhong, Dongzhe Zheng, Christine Allen-BlanchetteMixture-of-Experts Pruning Hodge Decomposition
Princeton University
Dense vs Sparse Pretraining at Tiny Scale: Active-Parameter vs Total-Parameter Matching
May 13, 2026Abdalrahman WaelLanguage Model Pretraining Mixture-of-Experts Language Models
Independent Researcher
EMO: Frustratingly Easy Progressive Training of Extendable MoE
May 13, 2026Linghao Jin, Chufan Shi, Huijuan Wang +4Sparse Mixture-of-Experts Efficient Language Model Training
USC-ISI · MBZUAI-IFM
Sparse Mixture-of-Experts Routing in Visual Diffusion Transformers:Diagnosis, Boundary Calibration and Evolutionary Roadmap from Routing Collapse to Selective Deadlock
May 12, 2026Haiying ShaVideo Diffusion Models Diffusion Transformer
Independent Researcher

Jun 30, 2026, cs.CVJ/K move · Enter open · S save

Dual-Adaptive SAM3: Hierarchical Routing over Low-Rank Expert Layers for Parameter-Efficient Medical Image Segmentation

Ying Chen, Jinyue Li, Kun Wang, Qiankun Li+1

Shenzhen Research Institute, The Chinese University of Hong Kong · University of Science and Technology of China · Nanyang Technological University+1

VLM Adaptation Text-Guided Image Segmentation Sparse Mixture-of-Experts Low-Rank Adaptation Medical Image Segmentation Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

The Segment Anything Model with Concepts (SAM3) heralds a new paradigm for open-vocabulary segmentation through natural language interaction, offering significant potential for medical image analysis. However, effectively adapting such a powerful vision-language model to the diverse and nuanced domain of medical imaging remains a key challenge. Naive fine-tuning is parameter-inefficient, while standard Mixture-of-Experts (MoE) methods introduce prohibitive computational overhead, limiting their clinical applicability. To address this, we propose Dual-Adaptive SAM3 (DA-SAM3), a novel framework that achieves both high segmentation accuracy and extreme parameter efficiency via a dual-adaptive specialization mechanism. Our first adaptation is task-aware: a Dynamic Expert Router (DER) that sparsely activates the most relevant experts by jointly reasoning about the visual input and the textual concept prompt, mimicking a clinical consultation process. Our second adaptation is parameter-aware: a Decomposed Parameterized Experts (DPE) design that represents each expert as a shared frozen base (inherited from the pretrained SAM3) and a lightweight trainable low-rank delta, reducing MoE parameter overhead by over 80%. Extensive experiments on multiple public medical segmentation benchmarks demonstrate that Dual-Adaptive SAM3 not only matches or exceeds the accuracy of fully fine-tuned SAM3 and standard MoE baselines, but also achieves a notable 5% gain over current state-of-the-art methods, with interpretable results validating its effectiveness. The code is available at: https://github.com/Reconsider80/DA-SAM3.

Momentum

Related topics

Latest papers 80