The Discrete Charm of the MLP: Binary Routing of Continuous Signals in Transformer Feed-Forward Layers

Peter Balogh

cs.LGMar 11, 2026

The Discrete Charm of the MLP: Binary Routing of Continuous Signals in Transformer Feed-Forward Layers

Authors: Peter Balogh

Abstract

We show that MLP layers in transformer language models perform binary routing of continuous signals: the decision of whether a token needs nonlinear processing is well-captured by binary neuron activations, even though the signals being routed are continuous. In GPT-2 Small (124M parameters), we find that specific neurons implement a consensus architecture -- seven "default-ON" neurons and one exception handler (N2123 in Layer 11) that are 93-98% mutually exclusive -- creating a binary routing switch. A cross-layer analysis reveals a developmental arc: early layers (L1-3) use single gateway neurons to route exceptions without consensus quorums; middle layers (L4-6) show diffuse processing with neither gateway nor consensus; and late layers (L7-11) crystallize full consensus/exception architectures with increasing quorum size (1 to 3 to 7 consensus neurons). Causal validation confirms the routing is functional: removing the MLP at consensus breakdown costs 43.3% perplexity, while at full consensus removing it costs only 10.1% -- exceeding a 4x difference. Comparing binary vs. continuous features for the routing decision confirms that binarization loses essentially no information (79.2% vs. 78.8% accuracy), while continuous activations carry additional magnitude information (R^2 = 0.36 vs. 0.22). This binary routing structure explains why smooth polynomial approximation fails: cross-validated polynomial fits (degrees 2-7) never exceed R^2 = 0.06 for highly nonlinear layers. We propose that the well-established piecewise-affine characterization of deep networks can be complemented by a routing characterization: along the natural data manifold, the piecewise boundaries implement binary decisions about which tokens need nonlinear processing, routing continuous signals through qualitatively different computational paths.

Figures & tables

Layer	Deg 2	Deg 3	Deg 4	Deg 5	Deg 6	Deg 7
L9	0.062	0.041	0.041	0.034	0.022	0.022
L11	0.170	0.248	0.260	0.262	0.202	0.208

Table 1: Cross-validated polynomial

R^{2}

on high-

\|\delta\|

tokens (top 10%), 100K tokens.

Clustering Method	Avg Val $R^{2}$	Best Cluster $R^{2}$
KMeans (input)	$-$ 0.068	0.021
KMeans (delta dir.)	$-$ 0.023	0.008
KMeans (joint)	$-$ 0.061	0.020
Spectral (delta dir.)	$-$ 2.182	0.013
UMAP + KMeans	$-$ 0.030	$-$ 0.021

Table 2: Branch detection: polynomial

R^{2}

after clustering Layer 9 high-

\|\delta\|

tokens. Five methods, all cross-validated. No method finds polynomial-tractable subsets.

Neuron	Linear %	High-NL %	$\Delta$	Semantic Role
2123	0.4%	80.7%	+80.3	Exception handler
2	99.1%	26.1%	$-$ 73.0	General punctuation
2361	93.2%	22.6%	$-$ 70.6	Subordinate clauses
2460	86.1%	19.8%	$-$ 66.3	Content words
2928	80.4%	16.2%	$-$ 64.2	Default-ON
1831	79.9%	17.7%	$-$ 62.2	Default-ON

Table 3: Layer 11 neurons with largest firing-rate shift between linear-default (bottom 25% by

\|\delta\|

) and highly-nonlinear (top 5%) tokens. 500K WikiText-103 tokens. Semantic roles are descriptive labels based on inspecting the top-firing tokens for each neuron; they are not independently validated (see § 5.8 for discussion).

Neuron Pair	Both Fire	Union Fire	Exclusivity
2123 vs N2	18,259	429,427	95.7%
2123 vs N2361	21,792	394,781	94.5%
2123 vs N2460	14,199	436,944	96.8%
2123 vs N2928	18,846	473,231	96.0%
2123 vs N1831	8,522	394,259	97.8%
2123 vs N1245	12,471	439,196	97.2%

Table 4: Mutual exclusivity between N2123 and the seven default-ON neurons. “Both fire” counts tokens where both GELU outputs exceed 0.1. 500K tokens.

Threshold	N2123 FR	Def-ON FR	Exclusivity	Monotonic	Range
0.01	13.1%	79.2%	94.1%	✓	96.4pp
0.05	12.2%	77.2%	95.0%	✓	95.9pp
0.10	11.3%	74.5%	95.8%	✓	94.3pp
0.50	7.9%	50.9%	98.5%	✓	66.3pp
1.00	6.2%	23.4%	99.3%	✓	23.0pp

Table 5: Threshold sensitivity. Consensus structure at five GELU binarization thresholds. Layer 11, 500K tokens, forward hooks. All thresholds produce monotonic gradients.

Figure 1: Two Regimes. N2123 fire rate (red, left axis) and MLP output norm (blue, right axis) as a function of default-ON consensus neuron count. The gradient is perfectly monotonic: consensus breakdown triggers the exception handler and 2.8

\times

output norm. 500K WikiText-103 tokens, GPT-2 Small Layer 11.

Figure 2: Exception handler architecture emerging from learned weights in Layer 11. Seven default-ON consensus neurons and N2123 are 93–98% mutually exclusive. When consensus holds, the MLP operates near-linearly (norm

\approx

70); when it breaks down, N2123 fires and triggers full nonlinear computation (norm

\approx

194).

Layer	Phase	Mean $\Delta$	Exception	Excl.	#Cons	Gateway	Monotonic
0	Scaffold	10.9	N2053 (9%)	94%	1	—	✓
1	Scaffold	8.2	—	—	—	N2882 (19/20)	$\times$
2	Scaffold	11.4	—	—	—	N2380 (20/20)	$\times$
3	Scaffold	11.7	—	—	—	N746 (17/20)	$\times$
4	Diffuse	11.9	—	—	—	—	$\times$
5	Diffuse	13.9	—	—	—	—	$\times$

Table 6: Three-phase developmental arc across GPT-2 Small. Consensus/exception architecture emerges through depth. Mean

\Delta

: average nonlinearity delta (higher = more nonlinear computation). #Cons: number of consensus neurons identified. Gateway: neuron firing in most top-enriched binary patterns. 500K WikiText-103 tokens.

Consensus	N tokens	Base PPL	No-MLP PPL	$\Delta$ %
0 / 7	11,351	5.4	7.7	+43.3%
1 / 7	19,110	7.3	9.7	+32.6%
2 / 7	19,121	12.2	16.1	+31.6%
3 / 7	26,169	20.3	25.7	+26.7%
4 / 7	52,994	19.5	23.3	+19.1%
5 / 7	95,194	39.1	46.0	+17.6%

Table 7: Causal validation. Perplexity impact of removing the Layer 11 MLP for tokens at each consensus level. Losses at position

t

(predicting token

t{+}1

) are grouped by the consensus level at position

t

. 500K WikiText-103 tokens, GPT-2 Small. Token counts are slightly lower than Table 9 (499,224 vs. 499,712) because the last token of each 1024-token sequence has no next-token loss.

Consensus	N	KL div	Boost	$\Delta$ Rank
0 / 7	11,351	0.414	1.15 $\times$	$-$ 33.9
1 / 7	19,110	0.338	1.08 $\times$	$-$ 45.1
2 / 7	19,121	0.315	1.05 $\times$	$-$ 22.4
3 / 7	26,169	0.290	1.03 $\times$	$-$ 12.3
4 / 7	52,994	0.238	0.97 $\times$	$-$ 8.4
5 / 7	95,194	0.222	0.94 $\times$	$-$ 9.9

Table 8: MLP mechanism by consensus level. KL divergence between full and no-MLP logits, correct-token probability with and without MLP, and rank change. 500K WikiText-103 tokens, GPT-2 Small Layer 11.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Consensus	Count	%	N2123 Fire Rate	Avg $\\|\text{MLP output}\\|$
0 / 7	11,358	2.3%	94.7%	194.1
1 / 7	19,131	3.8%	83.6%	155.5
2 / 7	19,142	3.8%	56.5%	126.1
3 / 7	26,189	5.2%	24.4%	102.4
4 / 7	53,042	10.6%	10.3%	88.6
5 / 7	95,267	19.1%	4.5%	82.3

Appendix

Table 9: Two Regimes (Layer 11). N2123 firing rate and MLP output norm as a function of consensus neuron count (N2, N2361, N2460, N2928, N1831, N1245, N2600). 500K WikiText-103 tokens, GPT-2 Small.

Consensus	Count	%	N2123 Fire Rate	Avg Norm
0 / 7	116,024	58.0%	1.7%	25.6
1 / 7	70,100	35.1%	1.9%	25.3
2 / 7	12,312	6.2%	2.7%	26.9
3 / 7	1,408	0.7%	2.2%	27.7

Appendix

Table 10: Two Regimes consensus table for GPT-2 Medium Layer 11 (4096 hidden neurons, 200K tokens). The monotonic gradient is absent; neuron 2123 fire rate is uniformly low (

\sim

2%).

Consensus	Count	%	N2123 Fire Rate	Avg Norm
0 / 7	115,182	57.6%	1.6%	40.8
1 / 7	67,693	33.9%	3.0%	43.7
2 / 7	14,941	7.5%	5.4%	46.6
3 / 7	1,945	1.0%	10.1%	47.0
4 / 7	168	0.1%	8.3%	48.8

Appendix

Table 11: Two Regimes for GPT-2 Large Layer 35 (5120 hidden, 200K tokens). A mild gradient exists but is far weaker than GPT-2 Small.

PCA dims	$\alpha=0.1$	$\alpha=1$	$\alpha=10$	$\alpha=100$
10	0.038	0.039	0.037	0.029
20	0.042	0.043	0.040	0.031
50	0.041	0.041	0.038	0.030
100	0.039	0.038	0.035	0.028

Appendix

Table 12: Polynomial

R^{2}

sensitivity to PCA dimensions and Ridge

\alpha

(Layer 9, degree 3, 100K tokens, cross-validated). The result is stable: no configuration exceeds

R^{2}=0.07

Explore similar work

Mar 3, 2026

Half the Nonlinearity Is Wasted: Measuring and Reallocating the Transformer's MLP Budget

We investigate when transformer MLP nonlinearity is actually necessary. A gate with

d+1

parameters decides when to replace the full MLP with a linear surrogate. Through systematic investigation across six models (162M-2.8B parameters), two architectures, and three corpora, we establish that nonlinearity need cannot be predicted from token identity: cross-corpus correlation is zero (

r < 0.05

). The routing decision is fully contextual. Despite weak per-instance predictability, the gate exploits a heavily skewed distribution where most MLP computations are near-linear, achieving 25-56% linear routing at <1% perplexity cost in GPT-2. In GPT-2 Large, 11 of 36 layers beat baseline with gating and no layer exceeds 3.7% all-linear cost. This success is architecture-dependent: Pythia models show higher costs, though Pythia-2.8B's full 32-layer sweep reveals one layer that narrowly beats baseline. As a proof of concept, we progressively replace middle-layer MLPs with frozen linear matrices: 5 of 24 layers linearize at zero cost. With a full training budget, 4 linearized layers yield a 10.2% perplexity improvement -- and a two-phase gated approach pushes this to 17.3%, beating a vanilla fine-tuning control and confirming that the nonlinear MLPs at these layers were actively harmful.

Peter Balogh

Transformer FFNs Conditional Computation

Ask PDF

Apr 6, 2026

Darkness Visible: Reading the Exception Handler of a Language Model

The final MLP of GPT-2 Small exhibits a fully legible routing program -- 27 named neurons organized into a three-tier exception handler -- while the knowledge it routes remains entangled across ~3,040 residual neurons. We decompose all 3,072 neurons (to numerical precision) into: 5 fused Core neurons that reset vocabulary toward function words, 10 Differentiators that suppress wrong candidates, 5 Specialists that detect structural boundaries, and 7 Consensus neurons that each monitor a distinct linguistic dimension. The consensus-exception crossover -- where MLP intervention shifts from helpful to harmful -- is statistically sharp (bootstrap 95% CIs exclude zero at all consensus levels; crossover between 4/7 and 5/7). Three experiments show that "knowledge neurons" (Dai et al., 2022), at L11 of this model, function as routing infrastructure rather than fact storage: the MLP amplifies or suppresses signals already present in the residual stream from attention, scaling with contextual constraint. A garden-path experiment reveals a reversed garden-path effect -- GPT-2 uses verb subcategorization immediately, consistent with the exception handler operating at token-level predictability rather than syntactic structure. This architecture crystallizes only at the terminal layer -- in deeper models, we predict equivalent structure at the final layer, not at layer 11. Code and data: https://github.com/pbalogh/transparent-gpt2

Peter Balogh

LLM Interpretability Circuits in Language Models

Ask PDF

Mar 7, 2026

PolyGLU: State-Conditional Activation Routing in Transformer Feed-Forward Networks

Biological neural systems employ diverse neurotransmitters -- glutamate, GABA, dopamine, acetylcholine -- to implement distinct signal-processing modalities within shared neural circuits. In contrast, modern transformers apply a single fixed activation function across all feed-forward neurons. We introduce PolyGLU (Polychromatic Gated Linear Unit), a drop-in replacement for SwiGLU that enables each FFN neuron to dynamically route among K=4 activation functions via a differentiable mechanism combining learned static preferences with input-conditioned gating, trained end-to-end with Gumbel-Softmax. We train PolychromaticLM, a 597M-parameter transformer, on ~10B tokens using a single NVIDIA A100 GPU. Our key finding is emergent routing behavior: without any explicit sparsity loss or entropy regularization, the routing mechanism converges to near-deterministic activation selections (mean dynamic entropy = 0.030% of maximum), with a striking depth-dependent specialization pattern -- early layers prefer GELU while deep layers strongly favor Tanh. Three layers maintain elevated routing entropy, suggesting computational flexibility points. The routing architecture adds only 0.23% parameter overhead (~1.4M parameters) and proves fully robust to supervised fine-tuning: routing entropy remains constant at ln(4) throughout 13,067 SFT steps. On standard benchmarks, PolychromaticLM achieves 62-89% of Qwen3-0.6B-Base performance despite training on 3,600x fewer tokens. All code, weights, and training infrastructure are released under Apache 2.0.

Daniel Nobrega Medeiros

Independent researcher.

Transformer FFNs Neural Network Activation Functions Conditional Computation

Ask PDF