Transformer feed-forward blocks combine several design choices: squared activations, gating, and tied versus untied parameter sharing, whose separate contributions to expressivity remain poorly understood. We introduce CP-MLPs, a tensor framework in which each multiplicative unit contributes a rank-1 interaction with three roles: detecting an input feature, reading a value, and writing to the output. Tied gated blocks such as ReLU2 force the detector and value roles to share the same direction; untied gated blocks such as ReGLU and SwiGLU let them differ. We show that this distinction yields exact, degree-wise tensor-rank separations: untied detector-value units are strictly more width-efficient for indefinite quadratic interactions and role-asymmetric monomials, while tied units are structurally matched to symmetric pure-power targets. The width advantage of untying therefore depends on the target structure. Experiments show loss plateaus below the exact-representation width, but these plateaus alone do not establish a positive approximation gap: arbitrarily accurate approximation is governed by border rank. In trained gated MLPs, untied blocks retain an advantage in selected comparisons at matched input-projection budget. Code is available at https://github.com/rarefin/CP-MLPs.
Figures & tables
Figure 1: Leg diagram of a single CP-MLP channel. The channel computes ha(x)=g(ua⊤x)(va⊤x) and writes the result through wa . At each active polynomial degree, the channel contributes one rank-1 CP atom with output, detector, and value legs.
Block
Gate g(z)
Tying
Signature S(g)
Effective atom
Bilinear ( Pearce et al., 2025 )
z
untied
{2}
w⊗\Sym(u⊗v)
ReGLU ( Shazeer, 2020 )
ReLU(z)
untied
{2} †
region-wise w⊗\Sym(u⊗v)
SwiGLU ( Shazeer, 2020 )
SiLU(z)=zσ(z)
untied
{2,3,5,7,…}
w⊗\Sym(u⊗(k−1)⊗v)
ReLU 2 ( So et al., 2021 )
ReLU(z)
tied
{2} †
region-wise w⊗u⊗2
Tied SwiGLU
SiLU(z)=zσ(z)
tied
{2,3,5,7,…}
w⊗u⊗k
Table 1: Feed-forward blocks as CP-MLPs. The gate controls which degrees are active through its polynomial signature; the tying pattern controls whether detector and value directions are shared.
Figure 2: Quadratic targets with (n+,n−) positive and negative directions. Left: required widths match the predictions n++n− (tied) and max{n+,n−} (untied). Right: loss stays near the target variance below the predicted width and reaches machine precision at the predicted width.
Figure 3: Coefficient-space width sweep for Fk=x1k−1x2 , k∈{2,3,4,5} . Untied atoms fit at width m=1 . Tied atoms keep a positive loss at m=1 ; for 2≤m<k their losses are small but nonzero, as expected from border rank two, and they reach machine precision at m=k for k≤3 . Median over two seeds; bands show min–max.
Figure 4: Target structure changes the low-width winner. At m=1 , tied atoms fit pure powers x1k , while untied atoms fit role-asymmetric monomials x1k−1x2 . Bands show min and max over seeds.
Figure 5: Width versus loss for actual gates on x1k−1x2 . Left: for k=2 , untied ReGLU and SwiGLU reach near-machine precision at smaller width than tied ReLU2 and tied SwiGLU. Right: for k=3 , the exact rank threshold becomes a qualitative ordering; untied SwiGLU is best, and each untied block stays below its same-gate tied counterpart for m≥2 . Median over three seeds; bands show min–max.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Experiment
Section
Input dim d
Sample size
Seeds
Restarts
Indefinite quadratic targets
§ 5.1
8
256
3
8
Role-asymmetric monomial targets
§ 5.2
2
coef. space
2
5
Pure-vs-mixed reversal
§ 5.3
4
256
10
5
Gated MLPs
§ 5.4
4
200
3
3
Matched-parameter control (Gated MLPs)
§ 5.4
4
200
3
3
Scope boundary
§ D.7
3
coef. space
2
4 ∗
Appendix
Table 2: Hyperparameters for each experiment.
Figure 6: Scope boundary at k=3 . Untied atoms are rank one on the role-asymmetric witness Fdv=x1k−1x2 but not on the square-free product Fsf=x1x2⋯xk ; the multi-leg oracle is rank one on both.
Target
Architecture
m=1
m=2
m=3
m=4
x1k−1x2 (role-asym.)
tied
4.0×10−1
2.0×10−8
2.3×10−10
1.2×10−13
untied (det.-val.)
<10−15
6.3×10−15
2.0×10−15
<10−15
multi-leg oracle
1.2×10−13
6.4×10−14
<10−15
<10−15
x1x2⋯xk (square-free)
tied
6.1×10−1
3.3×10−1
1.4×10−1
1.2×10−15
untied (det.-val.)
3.3×10−1
<10−15
<10−15
<10−15
multi-leg oracle
<10−15
<10−15
<10−15
<10−15
Appendix
Table 3: Scope boundary at k=3 . Median best-of-restarts MSE over two seeds.
Figure 7: Matched input-projection control on x1k−1x2 : untied width m (dashed) versus tied width 2m (solid). Tied SwiGLU uses SiLU(z)z . Median over three seeds; bands show min–max. Total parameter counts are m(2d+1) and 2m(d+1) , respectively.
Family
m
Regime
Best loss
Best Amax
Near-best Amax
κ
Untied
1
Exact control
1.6×10−29
1.00
1.00
1.7
Tied
1
Positive infimum
0.5556
0.385
0.385
1.5
Tied
2
Zero, unattained
5.4×10−12
83.17
83.17
4.4×107
Tied
3
Exact width
1.5×10−28
0.471
0.471
n.i.
Appendix
Table 4: Coefficient-size diagnostics for F3=x12x2 . Exact and border ranks determine the regimes; losses and sizes describe the candidate fits.
Family
m
Regime
Best loss
Best Amax
Near-best Amax
κ
Untied
2
Exact control
6.4×10−19
1.00
1.00
1.7
Tied
2
Positive infimum
0.5000
46.95
2.27
8.6×102
Tied
3
Positive infimum
0.2778
66.30
2.27
8.6×102
Tied
4
Zero, unattained
1.1×10−10
47.74
47.736
8.3×106
Tied
5
Zero, unattained
9.3×10−11
34.40
34.405
1.9×1013
Tied
6
Exact width
8.3×10−16
3.72
3.72
n.i.
Appendix
Table 5: The same diagnostic for F3,2=x12x2+x32x4 . Its tied border width is four and tied exact width is six.
Tied width
Untied width
Best untied loss
k
3
4
5
3
4
5
at width 3
3
0.33
0.32
0.70
0.80
0.92
1.00
1.2×10−20
4
0.42
0.67
0.75
0.17
0.65
0.88
2.4×10−18
5
0.77
0.80
0.85
0.00
0.02
0.00
1.7×10−5
6
0.08
0.20
0.22
0.00
0.00
0.00
8.1×10−9
Appendix
Table 6: Recovery fractions for configurations with a width-three generating witness. Witness losses are approximately 10−32 . Widths four and five pad the representation with additional channels.
d
k
Target
N
Tied obs./pred.
Untied obs./pred.
Obs. ratio
Above prediction
2
5
0
6
3/3
3/2
1.00
Untied
2
5
1
6
4/3
2/2
2.00
Tied
2
6
0
7
4/4
3/3
1.33
—
2
6
1
7
5/4
3/3
1.67
Tied
3
3
0
10
4/4
3/3
1.33
—
3
3
1
10
4/4
3/3
1.33
—
Appendix
Table 7: Dense random targets: observed recovery widths and complex-generic predictions. “Above” marks recovery only above the prediction and does not certify a larger real rank.
d
K
Ntot
Tied
Untied
2
{2,3,4,5}
18
5(4),4(4)
4(3),3(3)
3
{2,3}
16
5(4),4(4)
4(3),3(3)
3
{2,3,4}
31
7(7)
5(5)
4
{2,3}
30
7(6),7(6)
4(4),5(4)
Appendix
Table 8: Exploratory degree-mixture fits. Each entry is recovered width (dimension-count benchmark). Paired entries correspond to two independently sampled targets.
Teacher
m
Untied m
Tied m
Ratio
Tied 1.5m
Ratio
Untied
4
4.02×10−1
3.96×10−1
0.99
2.29×10−1
0.57
Untied
8
1.03×10−1
1.16×10−1
1.13
8.70×10−4
0.01
Untied
10
1.37×10−4
7.51×10−2
549
1.09×10−3
7.94
Untied
12
8.11×10−6
8.70×10−4
107
6.11×10−4
75.3
Tied
8
2.82×10−1
2.99×10−1
1.06
1.64×10−1
0.58
Tied
12
9.30×10−2
1.64×10−1
1.76
6.47×10−2
0.70
Appendix
Table 9: Trained-block distillation. Ratios are tied/untied errors; values above one favor untied. Ratios are retained at their reported precision and can differ slightly from ratios of rounded losses.
Parameters
Asymmetric teacher
Tied teacher
Untied/tied
Untied
Tied
Untied
Tied
17/18
0.3162
0.2580
0.3808
0.1899
34/36
0.0843
0.1049
0.0178
0.0014
51/54
0.0322
0.0387
0.0045
0.0013
68/72
0.0188
0.0370
0.0010
0.0006
102/108
0.0088
0.0052
0.0009
0.0006
Appendix
Table 10: Adam-trained scalar regression with approximately matched total parameters. Counts are untied/tied; the tied models have slightly larger budgets. Bold marks the smaller error within each teacher and row.