cs.LGNov 2, 2026

M3Former: Fine-Grained Cross-Layer Mixing for Transformers

Authors: Jiasong Han, Yunkai Chai, Tong Zhu, Hengyu Zhang, Wei Fang, Jintao Wang, Guoqi Li, Yu Cheng

Organizations: Tsinghua University, Beijing, China · Shanghai Jiao Tong University, Shanghai, China · Shanghai AI Laboratory, Shanghai, China · Institute of Automation, Chinese Academy of Sciences, Beijing, China · 

Abstract

Residual connections give each Transformer block access to all earlier layers only through one fixed sum, broadcast to all of the block's inputs. This broadcast bottleneck imposes three constraints: the query, key, value and residual inputs of a block, which we call its roles, read the same vector; all hidden dimensions within a role share one set of weights over earlier layers; and every earlier layer has the same fixed weight for every token. We propose the Multi-Role Multi-Group Mixing Residual (M3), which removes all three: every role, and every group of hidden dimensions within a role, reads its own token-dependent mixture of all earlier layers. M3 starts as an exact residual connection and adds as little as 1.4% parameters and 1.8% FLOPs. From 100M to 1.4B parameters, M3Former outperforms the Transformer and coarse-grained cross-layer connections at every size, matching a Transformer 1.19-1.34x the size of the one it modifies, trained with 1.4-1.8x the compute. Relaxing only some of the constraints recovers only part of this gain. M3Former also improves sparse mixture-of-experts models of up to 4.6B parameters and trains more stably in deeper networks. Code: https://github.com/Kingchet10/m3former

Figures & tables

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Mixture of Layers with Hybrid Attention

    May 10, 2026Ivan Ternovtsii, Yurii BilakMixture-of-Experts ModelsHybrid Attention

  2. Hierarchical Residuals: A Minimalist Decoupling of Residual Width from Compute Width

    Oct 30, 2026Ziteng Wang, Jianqiao Lu, Xinyu Ma +4TransformerResidual Learning