cs.LGNov 13, 2026

From Local Prediction Error to Final Outputs: A Theory of Expert Sharing in Multi-Step Reasoning

Authors: Tiantong Wang, Zhiren Gong, Yixin Wang, Tiantong Wu, Wei Yang Bryan Lim

Organizations: Nanyang Technological University · KTH Royal Institute of Technology · Beijing Institute of Technology

Abstract

Mixture-of-experts (MoE) models can share experts across layers or computational steps, pooling training information from different contexts. This can reduce local prediction error, but multi-step systems are ultimately evaluated by their final outputs. We study when these local gains survive composition. In specified linear regression models, we derive exact sharing and merging criteria by balancing variance savings against bias from sharing across roles with different target behavior. Under a common fixed linear propagation map, we characterize expected squared final-output risk, retaining error directions and correlations. We construct cases where sharing lowers average local error yet increases final-output error. Within this setting, we also identify when orthogonal full pooling with isotropic noise preserves every local improvement at the final output. For general routed computations, we derive conditional end-to-end error bounds that account for distribution shifts, errors in surrounding components, and changes in expert selection. Separately, we show that correct predictions and route labels alone do not establish that individual experts implement the attributed operations. Controlled experiments realize a strict local-to-final risk reversal. In low-data symbolic settings, trained Transformers exhibit statistically supported teacher-forced one-step gains without statistically resolved free-running final-output gains. Separate intervention tests distinguish successful multi-layer actions from single-expert attribution and generalization to unseen transitions. Together, these results distinguish when sharing improves local estimation, when those gains survive to the final output, and what evidence is needed to attribute reusable operations to individual experts.

Figures & tables

Appendix figures & tables34 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Towards a Statistical Understanding of Mixture-of-Experts

    Sep 3, 2026Siyuan He, Bokai Yang, Jie Hu +2Mixture of ExpertsSparse Mixture-of-Experts

  2. Meta Expert: Cross-Layer Expert Sharing for Storage- and Retrieval-Efficient MoE

    Oct 20, 2026Ermo Hua, Xiangyu Hong, Baiting Wu +11Sparse Mixture-of-ExpertsMixture-of-Experts Models

  3. A theoretical model for task routing in mixture-of-expert transformers

    Jun 12, 2026Vinoth Nandakumar, Yongli Xiang, Yunzhi Yao +2Transformer ExpressivityMixture of Experts