Organizations: Tsinghua University, Beijing, China · Shanghai Jiao Tong University, Shanghai, China · Shanghai AI Laboratory, Shanghai, China · Institute of Automation, Chinese Academy of Sciences, Beijing, China · Nanyang Technological University, Singapore
Residual connections give each Transformer block access to all earlier layers only through one fixed sum, broadcast to all of the block's inputs. This broadcast bottleneck imposes three constraints: the query, key, value and residual inputs of a block, which we call its roles, read the same vector; all hidden dimensions within a role share one set of weights over earlier layers; and every earlier layer has the same fixed weight for every token. We propose the Multi-Role Multi-Group Mixing Residual (M3), which removes all three: every role, and every group of hidden dimensions within a role, reads its own token-dependent mixture of all earlier layers. M3 starts as an exact residual connection and adds as little as 1.4% parameters and 1.8% FLOPs. From 100M to 1.4B parameters, M3Former outperforms the Transformer and coarse-grained cross-layer connections at every size, matching a Transformer 1.19-1.34x the size of the one it modifies, trained with 1.4-1.8x the compute. Relaxing only some of the constraints recovers only part of this gain. M3Former also improves sparse mixture-of-experts models of up to 4.6B parameters and trains more stably in deeper networks. Code: https://github.com/Kingchet10/m3former
Figures & tables
Figure 1: Validation loss at 20 training tokens per parameter (active parameters for mixture-of-experts models). In the bars: losses; red: change of M3Former relative to the Transformer; hatched: diverged run; white slash: bar cut off at the top. Each panel spans 0.12 in loss, the last 0.06.
Figure 2: M3: multi-role multi-group mixing. (a) The three constraints: a residual connection gives every role and group the same fixed sum hℓ−1 , whereas M3 gives each of the 4G role–group pairs its own token-dependent mixture. (b) Block ℓ assembles each of its four role inputs from G group-wise mixtures of the history H<ℓ . (c) A role–group mixer weights group g of every earlier state by aℓ,r,g,j(t) and sums them. (d) A generator computes all coefficients from hℓ−1(t) ; its initialization reproduces the standard Transformer. G=4 is shown; coefficient profiles are schematic.
(1)
(2)
(3)
Method
Roles
Groups
Weights
Sources
Residual
–
–
fixed
hℓ−1
DLCL, DenseFormer
–
–
static
all
Hyper-conn., mHC
–
–
dynamic
streams
Frac-connections
–
✓
dynamic
hℓ−1
AttnRes
–
–
dynamic
blocks
Table 1: Cross-layer connections and the three constraints. Static weights are learned but the same for every token; dynamic weights depend on the token.
Model
L×d
FFN
N
100M
16×640
1792
81M
300M
24×1024
2816
308M
500M
28×1152
3200
458M
700M
32×1280
3520
642M
1.4B
40×1664
4608
1363M
1.8B-A0.38B
24×1024
64×352
1.76B/0.31B
Table 2: Models. N is the non-embedding parameter count of the Transformer, total/active for the mixture-of-experts models, whose FFN column gives the number and width of experts. All heads have width 64.
Figure 3: Equivalent size k=Neq/N (§ 3 ) at 20 tokens per parameter, against the size N of the Transformer that each model modifies.
Figure 4: Validation loss against training tokens, at 5, 10 and 20 tokens per parameter for every size. M3 uses the best G of each size.
Size
Model
PPL ↓
LAMBADA
PIQA
WinoGrande
ARC-E
ARC-C
SciQ
LogiQA
BoolQ
HellaSwag
RACE-M
Avg
700M
Transformer
9.72
30.60
68.61
54.30
60.31
29.78
88.30
23.35
59.94
45.55
39.42
50.02
AttnRes
9.67
31.22
68.44
51.54
58.59
32.34
87.50
23.35
61.41
45.81
40.95
50.11
MUDDFormer
9.38
33.38
68.61
53.91
57.70
32.34
89.90
23.96
60.58
47.05
40.53
50.80
M3
9.31
34.29
68.66
55.25
59.93
31.57
89.60
22.27
61.25
47.62
41.99
51.24
1.4B
Transformer
8.33
38.29
71.38
58.64
64.65
35.49
90.40
20.74
61.04
54.56
42.97
53.82
AttnRes
8.21
39.39
71.65
59.04
62.71
37.63
91.00
24.42
62.42
54.71
42.27
54.52
Table 3: Zero-shot downstream accuracy (%) at 20 training tokens per parameter. PIQA, ARC and HellaSwag use length-normalized accuracy. PPL is the perplexity on the held-out FineWeb-Edu tokens. Bold: best at each size.
Multi-
Multi-
Residual
AttnRes a
group
role b
M3
(1) Roles
✓
✓
(2) Groups
✓
✓
(3) Weights
✓
✓
✓
✓
100M
2.798
2.751
2.739
2.717
2.713
300M
2.437
2.417
2.405
2.394
2.383
Table 4: Ablation of the three constraints: each column relaxes the checked ones. Validation loss at 20 tokens per parameter. a Block attention residuals ( Kimi Team, 2026 ) , which use a softmax over blocks of layers and, unlike the other columns, are not in the family of § 2.2 . b MUDDFormer ( Xiao et al., 2025 ) , which diverged (div.) at 1.4B.
Figure 5: Validation loss with G groups per role, where H is the number of attention heads; G=1 is MUDDFormer.
Figure 6: Coefficients of separately trained 100M and 300M M3 models ( G=16 ), averaged over validation tokens; colors scale as sign(a)∣a∣0.65 . (a), (b): 300M. (c): increase of the validation loss when the coefficients of one block are averaged over roles, groups or tokens; one point per block, bars: medians.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Size
G
Params
MACs
Fit
Observed
100M
H=10
1.44%
1.85%
0.0043
0.0850
300M
H=16
1.40%
1.78%
0.0033
0.0542
500M
H=18
1.40%
1.80%
0.0031
0.0412
700M
4H=80
3.86%
4.25%
0.0081
0.0438
1.4B
H=26
1.52%
1.91%
0.0029
0.0308
Appendix
Table 5: Overhead of M3 at the reported G , relative to the non-embedding parameters and per-token weight multiply–adds (MACs) of the Transformer, and loss reduction at 20N : from the extra parameters by Eq. ( 7 ) (Fit) and observed over the Transformer.
WSD; linear warmup over 1% of the 20N budget; 1−⋅ decay to 10% of the peak over the last 20% of each budget
Appendix
Table 6: Training hyperparameters of the dense models, shared by all methods.
Small
Large
Blocks L
24
28
Width d
1024
1536
Attention heads (MHA)
16
24
Head dimension
64
64
Experts per layer
64
64
Experts per token
8
8
Appendix
Table 7: Mixture-of-experts architectures. Every feed-forward layer is a mixture-of-experts layer, and experts are not shared across layers.
Size
Model
LAMBADA
PIQA
WinoGrande
ARC-E
ARC-C
SciQ
LogiQA
BoolQ
HellaSwag
RACE-M
Avg
700M
Transformer
29.90
68.72
55.33
65.70
32.85
90.20
22.43
62.26
45.71
41.50
51.46
AttnRes
30.04
67.41
52.80
66.37
34.04
90.60
20.58
56.73
45.37
41.57
50.55
MUDDFormer
30.56
68.66
54.06
68.48
34.47
90.40
25.04
58.65
47.06
42.20
51.96
M3
33.71
69.04
55.96
68.69
34.90
91.80
23.35
53.15
47.84
43.38
52.18
1.4B
Transformer
36.25
72.25
59.35
72.39
39.16
93.90
22.43
62.78
54.67
45.26
55.85
AttnRes
40.40
71.55
58.64
72.35
40.36
93.40
23.50
62.97
54.72
44.85
56.27
Appendix
Table 8: Five-shot downstream accuracy (%) at 20 training tokens per parameter, with the metrics of Table 3 . Bold: best at each size.
Size
D
Tokens
Transf.
AttnRes
MUDDFormer
M3
100M
5N
0.41B
3.0899
3.0447
2.9780
2.9811
10N
0.81B
2.9209
2.8743
2.8289
2.8273
20N
1.63B
2.7976
2.7507
2.7172
2.7126
300M
5N
1.54B
2.6395
2.6183
2.5836
2.5724
10N
3.08B
2.5243
2.5043
2.4774
2.4669
20N
6.17B
2.4367
2.4168
2.3937
2.3825
Appendix
Table 9: Validation loss of all runs in the main setup. M3 uses the best G at each size (Table 5 ). Attention residuals use blocks of four layers. Multi-group mixing (Table 4 ) is trained at 20N only, except at 700M (Table 10 ).
700M
5N
10N
20N
Transformer
2.4566
2.3534
2.2747
Attention residuals
2.4455
2.3469
2.2692
Multi-group, G=4H
2.4349
2.3316
2.2533
MUDDFormer
2.4139
2.3159
2.2385
M3, G=H
2.4259
2.3108
2.2332
M3, G=2H
2.4032
2.3099
2.2315
Appendix
Table 10: All 700M runs of the main setup ( H=20 ).
Figure 8: Coefficients of every role on every earlier state in every block, averaged over groups and validation tokens, with the colors of Figure 6 . Top: 300M; bottom: 100M.
Figure 9: Coefficients of the 16 groups of every role in the last block of the 300M model, averaged over validation tokens, with the colors of Figure 6 .
Figure 10: Weight of every role on the latest state (a) and on the embedding (b), averaged over groups and validation tokens, against relative depth.
Figure 11: Interventions of Figure 6 c. (a) Loss increase against relative depth; solid: 300M, dashed: 100M. (b), (c) Median over blocks of the loss increase when only one role is averaged over tokens (b) or over its groups (c). (d) Loss increase from averaging over tokens, summed over blocks, at positions whose next token continues the current word (in word) and at other positions. Dark: 300M; light: 100M.
Standard Mixture-of-Experts (MoE) transformers route tokens to expert subnetworks within each layer, but the layer structure itself remains monolithic. We introduce Mixture of Layers (MoL), which replaces full-width transformer blocks (d_model) with K parallel thin blocks at reduced dimensionality (d_thin << d_model), connected via learned down/up projections and composed via top-k block routing. Scaling sparse block routing to many blocks creates an attention coverage problem, as each block sees fewer tokens. We address this by introducing hybrid attention, which pairs one shared softmax block for global context with Gated DeltaNet linear attention in routed blocks.
Ivan Ternovtsii, Yurii Bilak
Department of Software Systems, Uzhhorod National University Narodna sq. 3, Uzhhorod, Ukraine, 88000
When generating text, a Transformer produces representations of past tokens at every layer, but each layer can normally use only representations from the same depth. This restriction prevents the model from fully reusing information it has already computed. We introduce WhiteMatter, which allows every layer to draw on past-token representations from any depth. A learned mixer selects the most useful depths for the current context and combines their representations into shared key-value (KV) cache channels. Sharing these channels across layers can reduce the cache size. Given the same number of training tokens, WhiteMatter with a full-size cache performs comparably to a standard Transformer with 50% more layers. With half the KV cache, WhiteMatter outperforms matched standard Transformers at two model scales, up to 1.3B parameters. Cross-layer connections, however, introduce dependencies that slow training and prompt processing. We address this problem with cyclic iteration, which updates interleaved groups of tokens in turn while processing the tokens within each group in parallel. On a reference model trained with exact autoregressive execution, cyclic iteration converges 12.5x faster than standard Jacobi iteration.
Residual connections underpin Transformers, but they implicitly tie the residual width carried through depth to the compute width used inside each layer. We introduce Hierarchical Residuals (HiRes), a minimalist architectural change that decouples these dimensions, widening the residual state while keeping per-layer compute width fixed. HiRes groups layers into blocks: standard d-dimensional PreNorm layers operate inside each block, while a wider D-dimensional PostNorm outer residual connects blocks. Learned read/write projections bridge the two spaces at block boundaries, leaving individual Transformer layers unchanged. Across different model sizes and depths, HiRes consistently outperforms vanilla residual addition, with fitted scaling laws indicating a 1.30-fold improvement in FLOPs efficiency. On OLMoE-1B-7B pretraining for 500B tokens, HiRes achieves 0.0208 lower training loss and 1.14-point higher downstream accuracy. HiRes is also competitive with other residual expansion methods such as VWN, mHC and AttnRes with less activation memory and I/O costs. Built from only dense projections and normalizations, it requires neither specialized kernels nor infrastructure-level optimizations. HiRes thus exposes residual width as a practical scaling variable using only standard operators.