Transformer MLP Gate Thresholds Are Couplings to a Carried Reference Direction

Olli Tuomi

cs.CLSep 25, 2026

Transformer MLP Gate Thresholds Are Couplings to a Carried Reference Direction

Authors: Olli Tuomi

Organizations: Evident Solutions Oy

Abstract

The corpus-mean direction of a transformer's residual stream is a component shared across all inputs, and is commonly removed by mean-centering before representational analysis. We present evidence that it is a functional component: the reference against which the MLP gate population sets its operating point. In Phi-2, an exact decomposition of resting gate pre-activations shows that at mid-stack layers the resting inhibition of >99.9% of gates is carried by the coupling $w \cdot b_L$ to the carried mean direction, at 48-56 $\times$ the explicit bias parameter, and the coupling is direction-specific: a random direction at matched norm orders the population's firing rates at Spearman $ρ\leq 0.09$ where the reference reaches 0.95. (That 0.95 is near-tautological on its own; the paper derives its null.) Causally, removing the stream's projection on the reference multiplies above-threshold firing by about 9 $\times$ , dose-monotonically, at 23-59 $\times$ a norm-matched control; a random-initialised twin is flat, and replacement tests show that the direction carries the function and the magnitude does not. A direction-matched control makes the same point: noise injected along the reference costs 8-42 $\times$ the same energy along a random direction. The decomposition replicates on four further families spanning both gate types (GELU with an explicit gate bias, bias-free SwiGLU), and the dose-response on two of them. Across an eight-model scan the mechanism is present in every GELU and SiLU family and absent only in OPT, where an opposing LayerNorm bias cancels the carried reference. Gate thresholds are implemented as couplings to a constant the network builds for the distribution it is reading, with the bias parameters contributing little; how much of that constant is carried in the stream and how much in parameters depends on the architecture.

Figures & tables

$\alpha$	$\Delta$ frac(act $>$ 0.5)			KL vs. clean
	L6	L10	L14	L6	L10	L14
0.0 (removed)	$+0.103$	$+0.086$	$+0.075$	1.427	0.519	0.274
0.5	$+0.023$	$+0.023$	$+0.021$	0.047	0.040	0.035
1.0 (identity)	0.000	0.000	0.000	0.000	0.000	0.000
1.5	$-0.006$	$-0.007$	$-0.007$	0.014	0.018	0.020
2.0 (doubled)	$-0.008$	$-0.009$	$-0.009$	0.055	0.060	0.110

Table 1: Dose-response of gate firing and output damage to scaling the stream’s projection on

\hat{m}_{L}

at the manipulated layer (Phi-2, trained). The twin is flat at every

\alpha

(

|\Delta\text{frac}|\leq 0.003

, no ordering). Point estimates are document medians over 39 documents; 95% document-bootstrap CIs (2000 resamples) on the removal row:

\Delta

frac

[+0.096,+0.112]/[+0.079,+0.098]/[+0.068,+0.088]

and KL

[1.14,2.03]/[0.36,0.75]/[0.15,0.40]

(L6/L10/L14), each disjoint from the norm-matched control’s KL CI (

\leq[0.010,0.032]

). The document-sampling spread on KL dwarfs the run-to-run fp16 jitter (

\sim

1%), so only the leading two digits are informative.

replacement $\hat{c}$	$\cos(\hat{c},\hat{m}_{10})$	$\Delta$ frac(act $>$ 0.5)	KL
none (pure removal)	—	$+0.0845$	0.516
coordinate-shuffled $\hat{m}_{10}$	$-0.003$	$+0.0840$	0.950
random direction	$+0.011$	$+0.0778$	0.484
function-word $W_{U}$ sum	$+0.096$	$+0.0740$	0.395
layer 18’s $\hat{m}_{18}$	$+0.940$	$+0.0027$	0.004
identity	1.000	0.000	0.000

Table 2: Replacement panel at L10 (Phi-2, 23 documents). Preservation of the baseline firing level tracks

\cos(\hat{c},\hat{m}_{10})

monotonically.

layer	$w\cdot b_{L}$ (median)	$b_{n}$ (median)	resting	ref-dominant	$\rho$ (resting, duty)	$\rho$ (random dir)
6	$-1.535$	$-0.027$	$-1.568$	99.98%	0.943	0.058
10	$-1.226$	$-0.022$	$-1.250$	99.99%	0.959	0.072
14	$-1.091$	$-0.021$	$-1.115$	99.96%	0.960	0.085
22	$-1.130$	$-0.018$	$-1.149$	99.94%	0.954	0.073

Table 3: Exact resting decomposition of gate pre-activations (Phi-2, 19 documents). “Ref-dominant” is the fraction of gates whose resting inhibition is carried mostly by

w\cdot b_{L}

(

|w\cdot b_{L}|>|b_{n}|

). The last column repeats the rank correlation with a random direction substituted for

b_{L}

at matched norm (median of 50 draws,

p_{95}\leq 0.192

, maximum

0.223

; from a 39-document run, which reproduces the

\rho

(resting, duty) column at

0.946

–

0.964

). The

\rho

(resting, duty) column is near-tautological read alone; it is the contrast between the last two columns that carries direction specificity. 95% document-bootstrap CIs (2000 resamples): ref-dominant stays

\geq 99.89\%

at every layer (widest

[99.89,99.98]

at L22), and

\rho

[0.936,0.945]/[0.951,0.959]/[0.953,0.959]/[0.945,0.952]

(L6/L10/L14/L22; percentile intervals, and resampling biases

\rho

slightly low, so a point estimate can sit just above its interval). The

48

–

56\times

quoted in the text is the ratio of median magnitudes,

|w\cdot b_{L}|

against

|b_{n}|

(the latter

0.027/0.023/0.023/0.021

), which differ slightly from the signed medians shown. The resting column is a per-gate median and does not equal the sum of the

w\cdot b_{L}

and

b_{n}

medians, since a median does not distribute over addition across a heterogeneous gate population.

model	gate (bias?)	resting med.	ref-dom.	$\rho$ (rest, duty)
GPT-2-medium	c_fc (yes)	$-1.52$ to $-0.99$	$99.4$ – $99.7\%$	$0.91$ – $0.97$
Pythia-1.4B	h_to_4h (yes)	$-1.08$ to $-0.74$	$>99.9\%$	$0.96$ – $0.98$
Qwen2.5-1.5B	gate_proj (no)	$-4.14$ to $-0.37$	—	$0.62$ – $0.99$
TinyLlama-1.1B	gate_proj (no)	$-0.21$ to $-0.07$	—	$0.97$ – $0.99$

Table 4: Cross-model resting decomposition (30 documents, four mid-stack layers per model at depth fractions

0.2

–

0.65

; ranges span those layers). “Ref-dom.” is the fraction of gates whose resting inhibition is carried mostly by

w\cdot b_{L}

rather than the bias parameter (vacuous for the bias-free SwiGLU gates; for these the informative statistic is the negative resting coupling and its duty-prediction). Where a bias exists,

|w\cdot b_{L}|/|b_{n}|

15

–

18\times

(GPT-2) and

38

–

43\times

(Pythia). Every model’s random-init twin is floored:

|\text{resting median}|\leq 0.01

, duty

\approx 0.5

Transformer Interpretability

Transformer interpretability studies how attention, feed-forward layers, and hidden representations support model predictions and behavior.

See all 340 papers

Oct 28, 2026

Closed Gates Leak: Decomposing Adjacent-Layer Cancellation in Gated Language Models

In transformer language models with gated multilayer perceptrons (MLPs), adjacent layers write partly against each other into the residual stream. Previous work interpreted this cancellation as layers proposing features and the next layer selectively rejecting the inappropriate ones. We decompose the cancellation by removing the previous layer's write from the input of the next MLP: what the next layer writes along that write splits into what it writes anyway and how much less its MLP writes because the write is present, which we call the damping, and we trace the damping to the neurons that carry it. In gated MLPs the damping is carried by neurons whose gates are closed. SiLU and GELU are slightly negative below zero, so a closed gate still passes a small negative multiple of what the neuron reads, and a neuron whose output weight points along its input weight then subtracts a fraction of the direction it read. Summed over such neurons, closed gates contract the average direction a gated MLP reads, and the previous layer's write is not singled out: the same layer's write at another token is damped as much or more, and the eigenvalue signature from which that work concluded that the correction is selective appears for another token's write as well. Over training the damping grows while what the next layer writes anyway stays flat, and cancellation begins when the damping overtakes it; annealing the negative side of the gate activation away in continued pretraining removes the damping and the cancellation. Adjacent-layer cancellation is therefore weak evidence about which features a model rejects: it is produced by a contraction that gated MLPs apply, on average over directions, to whatever they read.

Harvey Dam, Ganesh Gopalakrishnan

Kahlert School of Computing, University of Utah, Salt Lake City, USA.

Transformer FFNs Multilayer Perceptrons Transformer Training Dynamics Causal Interventions in Language Models

Ask PDF

Jun 17, 2026

Algebraic Dead Directions in LayerNorm Transformers: A Forward-Pass-Only Diagnostic at LLM Scale

Pretrained transformers sit near singular minima of the loss, where the Fisher information metric degenerates along dead directions: directions in parameter space along which the directional Fisher vanishes. Locating such a direction normally needs a forward pass and an eigendecomposition of activations, or a sampling-based complexity estimate; none returns a direction computable from the network's parameters alone. We give one, for LayerNorm transformers. The inverse-scale direction

γ^{-1}/\|γ^{-1}\|

of the LayerNorm affine is an exact algebraic kernel of the post-final-norm centred activation covariance, for any input distribution, and induces a corresponding dead direction in parameter space. It is read from the LN scale parameter alone, with no forward or backward pass and no eigensolve: the cheapest dead-direction read, specific to LayerNorm. We test it on

14

pretrained transformers (

9

LayerNorm,

5

RMSNorm;

160

M-

35

B; language and vision objectives). At random initialisation the predicted direction matches the measured bottom singular direction (one forward pass, direct SVD) to four decimal places on

9/9

LayerNorm models, and is correctly absent on

5/5

RMSNorm models, which lack the mean-subtraction projector that creates it. On the trained checkpoint the covariance eigenvalue along this direction deepens by

{\sim}10^3\times

and further dead directions open; the random-init-to-trained gap is a one-forward-pass, per-checkpoint readout of singular structure along the predicted coordinate. Two consequences follow in closed form: the residual stream's smallest singular value is preserved block-to-block on

13/14

transformers measured on their own input distribution, the one exception (Gemma

4

31

B) a genuine dead direction the same read pinpoints; and the kernel direction's presence classifies a transformer's normalisation from the parameters alone.

Tejas Pradeep Shirodkar, P. J. Narayanan

IIIT, Hyderabad

Information Geometry Transformer Neural Network Optimization Layer Normalization

Ask PDF

Mar 3, 2026

Half the Nonlinearity Is Wasted: Measuring and Reallocating the Transformer's MLP Budget

We investigate when transformer MLP nonlinearity is actually necessary. A gate with

d+1

parameters decides when to replace the full MLP with a linear surrogate. Through systematic investigation across six models (162M-2.8B parameters), two architectures, and three corpora, we establish that nonlinearity need cannot be predicted from token identity: cross-corpus correlation is zero (

r < 0.05

). The routing decision is fully contextual. Despite weak per-instance predictability, the gate exploits a heavily skewed distribution where most MLP computations are near-linear, achieving 25-56% linear routing at <1% perplexity cost in GPT-2. In GPT-2 Large, 11 of 36 layers beat baseline with gating and no layer exceeds 3.7% all-linear cost. This success is architecture-dependent: Pythia models show higher costs, though Pythia-2.8B's full 32-layer sweep reveals one layer that narrowly beats baseline. As a proof of concept, we progressively replace middle-layer MLPs with frozen linear matrices: 5 of 24 layers linearize at zero cost. With a full training budget, 4 linearized layers yield a 10.2% perplexity improvement -- and a two-phase gated approach pushes this to 17.3%, beating a vanilla fine-tuning control and confirming that the nonlinear MLPs at these layers were actively harmful.

Peter Balogh

Transformer FFNs Conditional Computation

Ask PDF

Abstract

Transformer Interpretability

Explore similar work

Closed Gates Leak: Decomposing Adjacent-Layer Cancellation in Gated Language Models

Algebraic Dead Directions in LayerNorm Transformers: A Forward-Pass-Only Diagnostic at LLM Scale

Half the Nonlinearity Is Wasted: Measuring and Reallocating the Transformer's MLP Budget