Expert Offloading

Follow this topic Ask about this topic

Momentum

5 papers in the last four weeks, with none the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 15

All topics

MaskCoFT: Masked Co-Adaptive Fine-Tuning for Memory-Efficient MoE Inference
Sep 28, 2026Junfeng Wu, Zehao Fan, Hadjer Benmeziane +3Expert Offloading Mixture-of-Experts Language Models
Department of Industrial and Systems Engineering, Rensselaer Polytechnic Institute · Department of Electrical, Computer, and Systems Engineering, Rensselaer Polytechnic Institute · IBM Research
OLED-MoE: Accelerating MoE-Based dLLM Inference via Inter-Iteration Locality-Aware Expert Offloading
Sep 27, 2026Jingyuan Xiao, Jiayue Wang, Yitao Hu +7Expert Offloading Mixture of Experts
Tianjin University, Tianjin, China · Tianjin University Tianjin, China
Paging the Experts: A Reproducible Characterization of Flash-Backed MoE Inference on iPhone
Sep 24, 2026Musa ShamsExpert Offloading On-Device Language Model Inference
Independent Researcher
The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction
Sep 16, 2026Yu Lin, Yiming Wang, Runyuan Cai +2Expert Offloading Mixture-of-Experts Inference
AutoArk
SeqMoE: Toward Full-Load Performance via Predictive and Graph-Compatible MoE Offloading
Sep 14, 2026Zihan Wang, Yuqi Wang, Lei Gong +5Expert Offloading LLM Inference Scheduling
University of Science and Technology of China Hefei, China · Suzhou Institute for Advanced Research, University of Science and Technology of China Suzhou, China
RotaryQuant: Fitting 120B MoE Models on Consumer Hardware via Fused Compressed-Space Attention
Aug 8, 2026Anthony. Lui, Mohamed. Elsaied, N. P. SavaniExpert Offloading Mixture-of-Experts Language Models
Cognizant AI & Analytics, London, UB8 3PH, UK · Southern Methodist University (SMU), Engineering Management, Lyle School of Engineering, Dallas, TX, USA · Cognizant AI Lab, Responsible AI Office, San Francisco, USA +1
When Does Trace-Driven Evaluation Mislead MoE Expert Caching? Replay Semantics, Workload Contamination, and Operating Regimes
Aug 8, 2026Yu ZhangExpert Offloading Mixture of Experts
China National Chemical Equipment Co. Ltd.
DraftExpert: Expansion-Aware Self-Speculative Decoding for End-Device MoE Inference
Jul 27, 2026Dengke HanExpert Offloading On-Device Language Model Inference
WiSP: A Working-Set View of Mixture-of-Experts Serving on Extremely Low-Resource Hardware
Jun 20, 2026Jiamu Zhang, Liang Wu, Mayank Darbari +1Expert Offloading Mixture-of-Experts Inference
Nokia, Sunnyvale, CA 94085, USA
MawForge: Memory-Bounded Expert Materialization for Local Mixture-of-Experts Inference
Jun 17, 2026Craig OpieExpert Offloading On-Device Language Model Inference
Holocron Security, Inc.
ReMoE: Boosting Expert Reuse through Router Fine-Tuning in Memory-Constrained MoE LLM Inference
May 26, 2026Xiongwei Zhu, Xiaojian Liao, Tianyang Jiang +3LLM Inference Efficiency Expert Offloading
School of Computer Science and Engineering, Beihang University, Beijing 100191, China · Huawei Technologies Ltd.
TIDE: Efficient and Lossless MoE Diffusion LLM Inference with I/O-aware Expert Offload
May 19, 2026Zhiben Chen, Youpeng Zhao, Yang Sui +2Expert Offloading LLM Inference Scheduling
University of Central Florida · Rice University
CoX-MoE: Coalesced Expert Execution for High-Throughput MoE Inference with AMX-Enabled CPU-GPU Co-Execution
May 18, 2026Muyoung Son, Yi Chen, Seungjae Yoo +2Expert Offloading GPU Acceleration
KAIST · Daejeon, Republic of Korea
VisMMOE: Exploiting Visual-Expert Affinity for Efficient Visual-Language MoE Offloading
May 7, 2026Cheng Xu, Xiaofeng Hou, Jiacheng Liu +1Expert Offloading Vision-Language Models
Shanghai Jiao Tong University Minhang Qu, Shanghai, China
FluxMoE: Decoupling Expert Residency for High-Performance MoE Serving
Date pendingQingxiu Liu, Yongchao He, Runhan Jiang +4Expert Offloading GPU Acceleration
CUHK · Unaffiliated · SCITIX +1

Sep 28, 2026, cs.LGJ/K move · Enter open · S save

MaskCoFT: Masked Co-Adaptive Fine-Tuning for Memory-Efficient MoE Inference

Junfeng Wu, Zehao Fan, Hadjer Benmeziane, Kaoutar El Maghraoui+2

Department of Industrial and Systems Engineering, Rensselaer Polytechnic Institute · Department of Electrical, Computer, and Systems Engineering, Rensselaer Polytechnic Institute · IBM Research

Expert Offloading Mixture-of-Experts Language Models Fine-Tuning Mixture-of-Experts Inference Memory-Efficient Inference LLM Routing

Open paper AskPDF arXiv

Mixture-of-experts (MoE) language models often exceed the memory of a single GPU. Expert offloading keeps most experts in host memory and loads them on demand, so decoding speed depends on how many experts each token must fetch. Caching and prefetching reduce this cost only as far as the routing allows. Router-only fine-tuning can reshape the routing to reuse experts, but it keeps the experts frozen, so they cannot adapt to the tokens the new routing sends them. We propose MaskCoFT, a masked co-adaptive fine-tuning method that trains routers and experts together with the cross-entropy loss alone. During fine-tuning, a learnable binary mask restricts the Top-K routing of each layer to a subset of experts, and the experts adapt to the tokens redirected to them. At inference, the learned mask becomes a soft prior that re-ranks experts, so every expert remains selectable. We simulate a GPU cache of 4 experts per layer for Mixtral-8x7B and 12 for DeepSeek-V2-Lite. MaskCoFT cuts expert fetches per token by 23.7% and 10.1% relative to the base model. In real offloading system serving, it lowers the time per output token by up to 16.4% and 5.5%, respectively. Its average accuracy over nine benchmarks stays above the base model by 0.92 and 0.53 points.

Momentum

Related topics

Latest papers 15