Transformer feed-forward blocks combine several design choices: squared activations, gating, and tied versus untied parameter sharing, whose separate contributions to expressivity remain poorly understood. We introduce CP-MLPs, a tensor framework in which each multiplicative unit contributes a rank-1 interaction with three roles: detecting an input feature, reading a value, and writing to the output. Tied gated blocks such as ReLU2 force the detector and value roles to share the same direction; untied gated blocks such as ReGLU and SwiGLU let them differ. We show that this distinction yields exact, degree-wise tensor-rank separations: untied detector-value units are strictly more width-efficient for indefinite quadratic interactions and role-asymmetric monomials, while tied units are structurally matched to symmetric pure-power targets. The width advantage of untying therefore depends on the target structure. Experiments show loss plateaus below the exact-representation width, but these plateaus alone do not establish a positive approximation gap: arbitrarily accurate approximation is governed by border rank. In trained gated MLPs, untied blocks retain an advantage in selected comparisons at matched input-projection budget. Code is available at https://github.com/rarefin/CP-MLPs.
Figures & tables
Figure 1: Leg diagram of a single CP-MLP channel. The channel computes ha(x)=g(ua⊤x)(va⊤x) and writes the result through wa . At each active polynomial degree, the channel contributes one rank-1 CP atom with output, detector, and value legs.
Block
Gate g(z)
Tying
Signature S(g)
Effective atom
Bilinear ( Pearce et al., 2025 )
z
untied
{2}
w⊗\Sym(u⊗v)
ReGLU ( Shazeer, 2020 )
ReLU(z)
untied
{2} †
region-wise w⊗\Sym(u⊗v)
SwiGLU ( Shazeer, 2020 )
SiLU(z)=zσ(z)
untied
{2,3,5,7,…}
w⊗\Sym(u⊗(k−1)⊗v)
ReLU 2 ( So et al., 2021 )
ReLU(z)
tied
{2} †
region-wise w⊗u⊗2
Tied SwiGLU
SiLU(z)=zσ(z)
tied
{2,3,5,7,…}
w⊗u⊗k
Table 1: Feed-forward blocks as CP-MLPs. The gate controls which degrees are active through its polynomial signature; the tying pattern controls whether detector and value directions are shared.
Figure 2: Quadratic targets with (n+,n−) positive and negative directions. Left: required widths match the predictions n++n− (tied) and max{n+,n−} (untied). Right: loss stays near the target variance below the predicted width and reaches machine precision at the predicted width.
Figure 3: Coefficient-space width sweep for Fk=x1k−1x2 , k∈{2,3,4,5} . Untied atoms fit at width m=1 . Tied atoms keep a positive loss at m=1 ; for 2≤m<k their losses are small but nonzero, as expected from border rank two, and they reach machine precision at m=k for k≤3 . Median over two seeds; bands show min–max.
Figure 4: Target structure changes the low-width winner. At m=1 , tied atoms fit pure powers x1k , while untied atoms fit role-asymmetric monomials x1k−1x2 . Bands show min and max over seeds.
Figure 5: Width versus loss for actual gates on x1k−1x2 . Left: for k=2 , untied ReGLU and SwiGLU reach near-machine precision at smaller width than tied ReLU2 and tied SwiGLU. Right: for k=3 , the exact rank threshold becomes a qualitative ordering; untied SwiGLU is best, and each untied block stays below its same-gate tied counterpart for m≥2 . Median over three seeds; bands show min–max.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Experiment
Section
Input dim d
Sample size
Seeds
Restarts
Indefinite quadratic targets
§ 5.1
8
256
3
8
Role-asymmetric monomial targets
§ 5.2
2
coef. space
2
5
Pure-vs-mixed reversal
§ 5.3
4
256
10
5
Gated MLPs
§ 5.4
4
200
3
3
Matched-parameter control (Gated MLPs)
§ 5.4
4
200
3
3
Scope boundary
§ D.7
3
coef. space
2
4 ∗
Appendix
Table 2: Hyperparameters for each experiment.
Figure 6: Scope boundary at k=3 . Untied atoms are rank one on the role-asymmetric witness Fdv=x1k−1x2 but not on the square-free product Fsf=x1x2⋯xk ; the multi-leg oracle is rank one on both.
Target
Architecture
m=1
m=2
m=3
m=4
x1k−1x2 (role-asym.)
tied
4.0×10−1
2.0×10−8
2.3×10−10
1.2×10−13
untied (det.-val.)
<10−15
6.3×10−15
2.0×10−15
<10−15
multi-leg oracle
1.2×10−13
6.4×10−14
<10−15
<10−15
x1x2⋯xk (square-free)
tied
6.1×10−1
3.3×10−1
1.4×10−1
1.2×10−15
untied (det.-val.)
3.3×10−1
<10−15
<10−15
<10−15
multi-leg oracle
<10−15
<10−15
<10−15
<10−15
Appendix
Table 3: Scope boundary at k=3 . Median best-of-restarts MSE over two seeds.
Figure 7: Matched input-projection control on x1k−1x2 : untied width m (dashed) versus tied width 2m (solid). Tied SwiGLU uses SiLU(z)z . Median over three seeds; bands show min–max. Total parameter counts are m(2d+1) and 2m(d+1) , respectively.
Family
m
Regime
Best loss
Best Amax
Near-best Amax
κ
Untied
1
Exact control
1.6×10−29
1.00
1.00
1.7
Tied
1
Positive infimum
0.5556
0.385
0.385
1.5
Tied
2
Zero, unattained
5.4×10−12
83.17
83.17
4.4×107
Tied
3
Exact width
1.5×10−28
0.471
0.471
n.i.
Appendix
Table 4: Coefficient-size diagnostics for F3=x12x2 . Exact and border ranks determine the regimes; losses and sizes describe the candidate fits.
Family
m
Regime
Best loss
Best Amax
Near-best Amax
κ
Untied
2
Exact control
6.4×10−19
1.00
1.00
1.7
Tied
2
Positive infimum
0.5000
46.95
2.27
8.6×102
Tied
3
Positive infimum
0.2778
66.30
2.27
8.6×102
Tied
4
Zero, unattained
1.1×10−10
47.74
47.736
8.3×106
Tied
5
Zero, unattained
9.3×10−11
34.40
34.405
1.9×1013
Tied
6
Exact width
8.3×10−16
3.72
3.72
n.i.
Appendix
Table 5: The same diagnostic for F3,2=x12x2+x32x4 . Its tied border width is four and tied exact width is six.
Tied width
Untied width
Best untied loss
k
3
4
5
3
4
5
at width 3
3
0.33
0.32
0.70
0.80
0.92
1.00
1.2×10−20
4
0.42
0.67
0.75
0.17
0.65
0.88
2.4×10−18
5
0.77
0.80
0.85
0.00
0.02
0.00
1.7×10−5
6
0.08
0.20
0.22
0.00
0.00
0.00
8.1×10−9
Appendix
Table 6: Recovery fractions for configurations with a width-three generating witness. Witness losses are approximately 10−32 . Widths four and five pad the representation with additional channels.
d
k
Target
N
Tied obs./pred.
Untied obs./pred.
Obs. ratio
Above prediction
2
5
0
6
3/3
3/2
1.00
Untied
2
5
1
6
4/3
2/2
2.00
Tied
2
6
0
7
4/4
3/3
1.33
—
2
6
1
7
5/4
3/3
1.67
Tied
3
3
0
10
4/4
3/3
1.33
—
3
3
1
10
4/4
3/3
1.33
—
Appendix
Table 7: Dense random targets: observed recovery widths and complex-generic predictions. “Above” marks recovery only above the prediction and does not certify a larger real rank.
d
K
Ntot
Tied
Untied
2
{2,3,4,5}
18
5(4),4(4)
4(3),3(3)
3
{2,3}
16
5(4),4(4)
4(3),3(3)
3
{2,3,4}
31
7(7)
5(5)
4
{2,3}
30
7(6),7(6)
4(4),5(4)
Appendix
Table 8: Exploratory degree-mixture fits. Each entry is recovered width (dimension-count benchmark). Paired entries correspond to two independently sampled targets.
Teacher
m
Untied m
Tied m
Ratio
Tied 1.5m
Ratio
Untied
4
4.02×10−1
3.96×10−1
0.99
2.29×10−1
0.57
Untied
8
1.03×10−1
1.16×10−1
1.13
8.70×10−4
0.01
Untied
10
1.37×10−4
7.51×10−2
549
1.09×10−3
7.94
Untied
12
8.11×10−6
8.70×10−4
107
6.11×10−4
75.3
Tied
8
2.82×10−1
2.99×10−1
1.06
1.64×10−1
0.58
Tied
12
9.30×10−2
1.64×10−1
1.76
6.47×10−2
0.70
Appendix
Table 9: Trained-block distillation. Ratios are tied/untied errors; values above one favor untied. Ratios are retained at their reported precision and can differ slightly from ratios of rounded losses.
Parameters
Asymmetric teacher
Tied teacher
Untied/tied
Untied
Tied
Untied
Tied
17/18
0.3162
0.2580
0.3808
0.1899
34/36
0.0843
0.1049
0.0178
0.0014
51/54
0.0322
0.0387
0.0045
0.0013
68/72
0.0188
0.0370
0.0010
0.0006
102/108
0.0088
0.0052
0.0009
0.0006
Appendix
Table 10: Adam-trained scalar regression with approximately matched total parameters. Counts are untied/tied; the tied models have slightly larger budgets. Bold marks the smaller error within each teacher and row.
We show that the conventional gated MLP can be viewed as a rank-1 approximation to a bilinear attention mechanism with two distinct factors corresponding to the query and the key. We further show that moving the nonlinearity onto one factor breaks the exchange symmetry between the two factors and, for non-homogeneous activations, the inverse-scaling symmetry as well. This perspective may help explain why gated MLPs are effective in practice and inform the design of future architectures.
A transformer MLP gives each hidden channel two roles: a reader that decides when the channel responds, and a writer that decides what it adds back to the residual stream. Read as a key--value memory, the layer can be measured one factor set at a time: the pairing of readers to writers is carried only by their shared index, invisible to any quantity computed from one factor set alone. Yet training couples the two factors through their joint update, so we ask what reading them together shows. The block supplies the joint instrument: the MLP writes back into the very stream it reads, so the exact Jacobian of its update is square, and its agreement with its own transpose resolves into reader--writer terms whose inner products give each channel, read as a couple, a signed pairing coordinate. Ranked by this coordinate, each block we measure has a small core of strongly coupled channels; where initialization gives either sign equally, the core is nearly all opposed: each writer points against the direction its reader responds to. This opposed core recurs from the smallest models trained from scratch to released checkpoints; its sign is in place within a few percent of training and holds while the coupling grows. Read as couples, most channels return their contribution against the reader that fired it, and the readers of later blocks take it up; few show in the logits. Training holds the couples in place: flipping the opposed writers moves the output toward common tokens, and a writer permuted onto another reader regrows to fit that reader while the reader stays.
A widely cited result by Dong et al. (2021) showed that Transformers built from self-attention alone, without skip connections or feed-forward layers, suffer from rapid rank collapse: all token representations converge to a single direction. The proposed remedy was the MLP. We show that this picture, while correct in the regime studied by Dong, is incomplete in ways that matter for architectural understanding. Three results are established. First, layer normalisation is precisely affine-rank-neutral: it preserves the affine rank of the token representation set exactly. The widespread claim that LN "plays no role" is imprecise; the correct statement is sharper. Second, residual connections generically obstruct rank collapse in real Transformers such as BERT-base, in a measure-theoretic sense, without contribution from the MLP. The MLP's irreplaceable function is different: generating feature directions outside the linear span of the original token embeddings, which no stack of attention layers can produce. Third, a phenomenon distinct from rank collapse is identified: head-channel non-identifiability. After multi-head attention sums per-head outputs through the output projection, individual contributions cannot be canonically attributed to a specific head; n(H-1)d_k degrees of freedom per layer remain ambiguous when recovering a single head from the mixed signal. The MLP cannot remedy this because it acts on the post-summation signal. A constructive partial remedy is proposed: a position-gated output projection (PG-OP) at parameter overhead below 1.6% of the standard output projection. The four collapse phenomena identified in the literature -- rank collapse in depth, in width, head-channel non-identifiability, and entropy collapse -- are unified under a symmetry-breaking framework, each corresponding to a distinct symmetry of the Transformer's forward pass.
Giansalvo Cirrincione
aLaboratoire LTI, Université de Picardie Jules Verne, Chemin du Thil, 80025 Amiens, France