cs.LGMay 14, 2026

φφ-Balancing for Mixture-of-Experts Training

Authors: Lizhang ChenJonathan LiQi WangRunlong LiaoShuozhe LiChen LiangNi LaoQiang Liu

Organizations: University of Texas at Austin · 2Northwestern University.

Abstract

Mixture-of-Experts (MoE) models rely on balanced expert utilization to fully realize their scalability. However, existing load-balancing methods are largely heuristic and operate on noisy mini-batch assignment statistics, introducing bias relative to population-level objectives. We propose φφ-balancing, a principled framework that directly targets population-level expert balance by minimizing a strictly convex, symmetric, and differentiable potential of the expected routing distribution. Using convex duality, we derive an equivalent min-max formulation and obtain a simple online algorithm via mirror descent, yielding an efficient EMA-based routing adjustment with negligible overhead. Across large-scale pretraining and downstream fine-tuning, φφ-balancing consistently outperforms prior Switch-style and loss-free baselines, demonstrating more stable and effective expert utilization.

Explore similar work

CardsList
  1. Hierarchical Mixture-of-Experts with Two-Stage Optimization

    May 8, 2026Gleb Molodtsov, Alexander Miasnikov, Aleksandr BeznosikovMixture-of-Experts ModelsLoad Balancing