cs.LGNov 2, 2026

Sonar-MoE: In-Backward Expert Updates for Memory-Efficient Mixture-of-Experts Training

Authors: Avni Mittal, Avinash Anand, Ashutosh Kumar, Dikshant Kukreja, Kritarth Prasad, Chaitanya Lakhchaura, Rishitej Reddy Vyalla, Erik Cambria, 

Organizations: Microsoft, India · Singapore Institute of Technology, Singapore · MIDAS Lab, Indian Institute of Technology Kanpur, Kanpur, India · TurboML · 

Abstract

Most parameters of a Mixture-of-Experts (MoE) model belong to its experts, and standard training keeps every expert gradient in GPU memory until the optimizer step. This memory grows with the number and size of the experts, so it rises as MoE models scale. We introduce Sonar-MoE, which updates each expert tensor during the backward pass and frees its gradient right away. It brings optimizer-in-backward from dense models to batched MoE expert tensors, so the GPU never holds more than one layer of expert gradients. When we fine-tune OLMoE-1B-7B and Granite-3.1-3B-A800M in bf16 with two expert-kernel backends, Sonar-MoE lowers peak memory by 19% to 22%. Its updates are bitwise identical to standard training without expert gradient clipping, and held-out loss stays within 0.003 of all baselines. Each step also takes 17% to 23% less time than standard training with global-norm clipping. Updating whole batched tensors rather than individual experts is what keeps Sonar-MoE fast, as it avoids a 20% to 72% slowdown while using the same memory. Sonar-MoE matches optimizer-in-backward over all parameters in memory and speed while still clipping the dense parameters.

Figures & tables

Explore similar work

CardsList
  1. Memory-Efficient Expert Routing for Distributed MoE Training

    Oct 5, 2026Arnab Kanti Tarafder, Jaume Guasch-Martí, Gokcen Kestor +1Expert RoutingMixture-of-Experts Models

  2. Supercharging Olmo-core for Efficient and Scalable MoE Training

    Nov 13, 2026Tianhua Tao, Akshita Bhagia, Dirk Groeneveld +8Mixture-of-Experts ModelsCommunication-Efficient Mixture-of-Experts