Expert Offloading

Ask about this topic

Overview

Expert offloading studies how to serve mixture-of-experts models when their full set of expert weights cannot fit in accelerator memory. Systems page or cache experts across GPU memory, CPU memory, and storage, using routing-based prediction, prefetching, and pipeline scheduling; some also co-execute work across CPUs and GPUs. Other approaches improve reuse by adapting routers or use speculative decoding to overlap expert loading with generation. The central tension is balancing memory for resident experts and the KV cache against transfer costs and latency, while preserving output quality across workloads and hardware.

Latest papers

CardsList

20 papers4 papers in the last four weeks, down 20% on the four weeks before. 0.0% of all new papers.

  1. COPE: Composable Performance Emulation for Mixture-of-Experts Offloading Policies

    Oct 24, 2026Hanming Zhang, Erhu FengExpert OffloadingMixture-of-Experts Inference

    Institute of Parallel and Distributed Systems, Shanghai Jiao Tong University

  2. Efficient CPU-GPU collaborative MoE inference via Adaptive Expert Assignment

    Oct 16, 2026Wenhui Lv, Yunming Liao, Hongli Xu +2Mixture-of-Experts InferenceLLM Inference Acceleration

  3. SpecFit: Not All Draft Tokens Cost the Same in Offloaded MoE Inference

    Oct 11, 2026Hongliang Cao, Xiuyu Li, Siqi Wang +3LLM Inference AccelerationSpeculative Decoding

    Beihang University · Monash University · The Hong Kong University of Science and Technology

  4. MaskCoFT: Masked Co-Adaptive Fine-Tuning for Memory-Efficient MoE Inference

    Sep 28, 2026Junfeng Wu, Zehao Fan, Hadjer Benmeziane +3Expert OffloadingMixture-of-Experts Language Models

    Department of Industrial and Systems Engineering, Rensselaer Polytechnic Institute · Department of Electrical, Computer, and Systems Engineering, Rensselaer Polytechnic Institute · IBM Research

  5. OLED-MoE: Accelerating MoE-Based dLLM Inference via Inter-Iteration Locality-Aware Expert Offloading

    Sep 27, 2026Jingyuan Xiao, Jiayue Wang, Yitao Hu +7Expert OffloadingMixture of Experts

    Tianjin University, Tianjin, China

  6. Paging the Experts: A Reproducible Characterization of Flash-Backed MoE Inference on iPhone

    Sep 24, 2026Musa ShamsExpert OffloadingOn-Device Language Model Inference

    Independent Researcher

  7. The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction

    Sep 16, 2026Yu Lin, Yiming Wang, Runyuan Cai +2Expert OffloadingMixture-of-Experts Inference

    AutoArk

  8. SeqMoE: Toward Full-Load Performance via Predictive and Graph-Compatible MoE Offloading

    Sep 11, 2026Zihan Wang, Yuqi Wang, Lei Gong +5Expert OffloadingLLM Inference Scheduling

    University of Science and Technology of China Hefei, China · Suzhou Institute for Advanced Research, University of Science and Technology of China Suzhou, China

  9. EStream: Fast and Memory-Efficient MoE Prefill through Expert Virtualization on Mobile NPUs

    Sep 6, 2026Junming Zhang, Zhenzhe Zheng, Fan Wu +2Mixture-of-Experts InferenceOn-Device Language Model Inference

    Shanghai Jiao Tong University Shanghai, China · Cloud Computing Research Institute, China Telecom China · Temple University Philadelphia, Pennsylvania, USA

  10. RotaryQuant: Fitting 120B MoE Models on Consumer Hardware via Fused Compressed-Space Attention

    Aug 8, 2026Anthony. Lui, Mohamed. Elsaied, N. P. SavaniExpert OffloadingMixture-of-Experts Language Models

    Cognizant AI & Analytics, London, UB8 3PH, UK · Southern Methodist University (SMU), Engineering Management, Lyle School of Engineering, Dallas, TX, USA · Cognizant AI Lab, Responsible AI Office, San Francisco, USA +1

  11. When Does Trace-Driven Evaluation Mislead MoE Expert Caching? Replay Semantics, Workload Contamination, and Operating Regimes

    Aug 8, 2026Yu ZhangExpert OffloadingMixture of Experts

    China National Chemical Equipment Co. Ltd.

  12. DraftExpert: Expansion-Aware Self-Speculative Decoding for End-Device MoE Inference

    Jul 27, 2026Dengke HanExpert OffloadingOn-Device Language Model Inference

  13. WiSP: A Working-Set View of Mixture-of-Experts Serving on Extremely Low-Resource Hardware

    Jun 20, 2026Jiamu Zhang, Liang Wu, Mayank Darbari +1Expert OffloadingMixture-of-Experts Inference

    Nokia, Sunnyvale, CA 94085, USA

  14. MawForge: Memory-Bounded Expert Materialization for Local Mixture-of-Experts Inference

    Jun 17, 2026Craig OpieExpert OffloadingOn-Device Language Model Inference

    Holocron Security, Inc.

  15. ReMoE: Boosting Expert Reuse through Router Fine-Tuning in Memory-Constrained MoE LLM Inference

    May 26, 2026Xiongwei Zhu, Xiaojian Liao, Tianyang Jiang +3LLM Inference EfficiencyExpert Offloading

    School of Computer Science and Engineering, Beihang University, Beijing 100191, China · Huawei Technologies Ltd.

  16. TIDE: Efficient and Lossless MoE Diffusion LLM Inference with I/O-aware Expert Offload

    May 19, 2026Zhiben Chen, Youpeng Zhao, Yang Sui +2Expert OffloadingLLM Inference Scheduling

    University of Central Florida · Rice University

  17. CoX-MoE: Coalesced Expert Execution for High-Throughput MoE Inference with AMX-Enabled CPU-GPU Co-Execution

    May 18, 2026Muyoung Son, Yi Chen, Seungjae Yoo +2Expert OffloadingGPU Acceleration

    KAIST · Daejeon, Republic of Korea

  18. VisMMOE: Exploiting Visual-Expert Affinity for Efficient Visual-Language MoE Offloading

    May 7, 2026Cheng Xu, Xiaofeng Hou, Jiacheng Liu +1Expert OffloadingVision-Language Models

    Shanghai Jiao Tong University Minhang Qu, Shanghai, China

  19. FluxMoE: Decoupling Expert Residency for High-Performance MoE Serving

    Apr 3, 2026Qingxiu Liu, Yongchao He, Runhan Jiang +4Expert OffloadingGPU Acceleration

    CUHK · Unaffiliated · SCITIX +1

  20. Specter: Fast MoE Inference via Quantization-Enabled Multi-Step Speculative Offloading

    Nov 18, 2025Wenfeng Wang, Xiaofeng Hou, Xinfeng Xia +5Mixture-of-Experts InferenceLLM Inference Acceleration

    Shanghai Jiao Tong University, Shanghai, China · Shanghai Jiao Tong University, China · Hong Kong University of Science and Technology, Hongkong, China