cs.CLNov 19, 2026

Understanding MoE Decisions Beyond Gradients

Authors: Zhen Xu, Zitao Zhang, Kevin Zhao, Xuefeng Liu

Organizations: University of Chicago · eBay · University of Florida

Abstract

Understanding a language model's decision requires knowing how its internal computations affect its preference between answers. Gradient-based methods estimate this influence by predicting the effect of changing those computations. In mixture-of-experts (MoE) models, replacing one expert can change which experts later layers select. Those switches can alter the answer scores in ways that derivatives do not capture. We introduce a predictor that uses the model's own expert outputs and selection rules to estimate these extra effects, then adds them to a second-order prediction. Across questions from three benchmarks, our method reduces the error in predicting answer-score changes by 60% on Qwen3.6 and 47% on Gemma4, relative to a multistep baseline using gradients and second derivatives.

Figures & tables

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Reconstruction-Optimized Expert Pruning for Mixture-of-Experts Language Models

    Oct 19, 2026Jelena Markovic-Voronov, Petros Prastakos, Kayhan Behdin +7Mixture-of-Experts PruningMixture-of-Experts Language Models

  2. SoftMoE: Soft Differentiable Routing for Mixture-of-Experts in LLMs

    Jun 16, 2026Mikołaj Zasada, Łukasz Struski, Jacek Tabor +1Mixture-of-Experts Language ModelsSparse Mixture-of-Experts

  3. Speculating Experts Accelerates Inference for Mixture-of-Experts

    Mar 9, 2026Vivan Madan, Prajwal Singhania, Abhinav Bhatele +2Mixture-of-Experts InferenceLLM Inference Acceleration