Darkness Visible: Reading the Exception Handler of a Language Model

Peter Balogh

cs.LGApr 6, 2026

Darkness Visible: Reading the Exception Handler of a Language Model

Authors: Peter Balogh

Abstract

The final MLP of GPT-2 Small exhibits a fully legible routing program -- 27 named neurons organized into a three-tier exception handler -- while the knowledge it routes remains entangled across ~3,040 residual neurons. We decompose all 3,072 neurons (to numerical precision) into: 5 fused Core neurons that reset vocabulary toward function words, 10 Differentiators that suppress wrong candidates, 5 Specialists that detect structural boundaries, and 7 Consensus neurons that each monitor a distinct linguistic dimension. The consensus-exception crossover -- where MLP intervention shifts from helpful to harmful -- is statistically sharp (bootstrap 95% CIs exclude zero at all consensus levels; crossover between 4/7 and 5/7). Three experiments show that "knowledge neurons" (Dai et al., 2022), at L11 of this model, function as routing infrastructure rather than fact storage: the MLP amplifies or suppresses signals already present in the residual stream from attention, scaling with contextual constraint. A garden-path experiment reveals a reversed garden-path effect -- GPT-2 uses verb subcategorization immediately, consistent with the exception handler operating at token-level predictability rather than syntactic structure. This architecture crystallizes only at the terminal layer -- in deeper models, we predict equivalent structure at the final layer, not at layer 11. Code and data: https://github.com/pbalogh/transparent-gpt2

Figures & tables

Tier	Neurons	Count	Fire Rate	Jaccard	Function
Core	N2123, N2910, N740,	5	90–100%	$\geq$ 0.91	Vocabulary reset
	N1611, N2044
Diff	N2462, N2173, N1602,	10	35–88%	0.15–0.89	Candidate suppression,
	N1800, N2379, N1715,				subword repair
	N611, N3066, N584, N2378
Spec	N2921, N2709, N971,	5	14–37%	$<$ 0.15	Boundary detection

Table 1: Three-tier exception handler. Fire rates conditional on N2123 firing.

Level	Tokens	Mean $\Delta P$	95% CI
0/7	206	$+$ 0.187	[ $+$ 0.145, $+$ 0.231]	$\blacktriangle$
1/7	913	$+$ 0.094	[ $+$ 0.079, $+$ 0.109]	$\blacktriangle$
2/7	3,051	$+$ 0.045	[ $+$ 0.039, $+$ 0.051]	$\blacktriangle$
3/7	6,283	$+$ 0.030	[ $+$ 0.026, $+$ 0.033]	$\blacktriangle$
4/7	12,617	$+$ 0.014	[ $+$ 0.011, $+$ 0.016]	$\blacktriangle$
5/7	34,155	$-$ 0.004	[ $-$ 0.006, $-$ 0.003]	$\blacktriangledown$

Table 2: L11 MLP effect on next-token probability by consensus level (204,800 tokens, sequence-level bootstrap; §3). All CIs exclude zero; crossover at 4–5/7.

Figure 2: L11 MLP effect by consensus level. Blue: MLP helps. Red: MLP hurts. Crossover between 4/7 and 5/7 matches the causal analysis of Balogh (2026) .

Tier	Neurons	PPL	$\Delta$	95% CI
Baseline	—	31.79	—	—
Core	5	31.86	$+$ 0.2%	[ $+$ 0.1%, $+$ 0.4%]
Differentiators	10	32.20	$+$ 1.3%	[ $+$ 0.9%, $+$ 1.7%]
Specialists	5	31.69	$-$ 0.3%	[ $-$ 0.6%, $-$ 0.1%]
All exception	20	32.44	$+$ 2.1%	[ $+$ 1.6%, $+$ 2.5%]

Table 3: PPL impact of zeroing exception handler tiers (204,800 tokens, sequence-level bootstrap CIs).

Figure 3: The DC offset paradox. Left: Exception-path output norm. Right: PPL impact. The Core dominates at 54% of norm but contributes only

+

0.2% PPL—a DC offset the residual stream already provides.

	L0–L3	L4–L6	L7–L9	L10	L11 (auto)	L11 (known)
Exc. fire rate	6–15%	12–15%	25–36%	38.5%	10.9%	11.3%
Max Jaccard	0.06–0.16	0.13–0.18	0.26–0.36	0.384	0.123	0.998
Enrichment	1.06–1.17	1.11–1.21	1.06–1.09	1.05	1.15	2.0
Hi-Jaccard ( $>$ 0.5)	0	0	0	0	0	4

Table 4: Exception handler structure across layers (102,400 tokens). Only L11’s known architecture shows tight co-firing.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Neuron	Dimension	Rate	Key Evidence
N2	Clausal continuation	88.4%	Fires on and , but , also mid-clause; silent at clause boundaries
N2361	Syntactic elaboration	84.1%	Fires on that , while , neither , fully ; depleted on There , United
N2460	Relational embedding	86.0%	Fires on an , into , per , other in prep. phrases; depletes According to 3.3%
N2928	Sequential structure	91.4%	Fires on ordinals, lists, parallel structure; highest mean activation (0.79)
N1831	Discourse coherence	81.0%	Fires on topic-continuing phrases; 77K disagreement tokens
N1245	Argument structure	85.5%	Fires on leadership , validity , one of the ; marks semantic roles

Appendix

Table 5: Consensus neuron specializations (512K tokens).

Prompt (abbreviated)	Target	Base	Static	Context
Retrievable (context rank $\leq$ 10):
…300,000 km per	second	1	244	1
…100 degrees	C	3	68	1
…created by Linus	Tor	1	2,551	1
…London occurred in	16	4	372	2
…begins with four	notes	2	3,159	2

Appendix

Table 6: Knowledge retrieval across 160 factual prompts.

	In top-20	Expected by chance
Consensus neurons (7)	2.6 avg	0.05
All routing neurons (27)	7.3 avg	0.18
Enrichment	36.5 $\times$	—

Appendix

Table 7: Integrated gradients overlap with routing neurons (20 prompts).

Layer	Logit Top-1	Tuned Top-1	$\Delta$	First lock-in (Tuned)
Emb	0.7%	3.5%	$+$ 2.8	3.5%
L0	3.0%	9.0%	$+$ 6.0	6.2%
L1	3.9%	9.7%	$+$ 5.8	1.2%
L2	3.9%	10.7%	$+$ 6.7	1.4%
L3	5.0%	11.7%	$+$ 6.6	1.4%
L4	5.9%	12.6%	$+$ 6.7	1.3%

Appendix

Table 8: Top-1 accuracy via logit and tuned lens [ Belrose et al., 2023 ] (204,800 tokens). “First lock-in (Tuned)” = percentage of tokens that first reach top-1 accuracy at this layer under the tuned lens.

Figure 4: Top-1 accuracy across layers. The developmental arc holds under both logit lens (dashed) and tuned lens (solid).

#	Intrans. verb	Trans. verb	Disambig.	$S_{\text{int}}$	$S_{\text{trans}}$
1	struggled	scratched	took	5.4	11.6
2	sneezed	visited	prescribed	6.8	16.2
3	dozed	watched	next	8.9	11.5
4	escaped	attacked	searched	17.0	19.4
5	slept	ignored	standing	13.2	11.3
6	cried	woke	in	9.0	6.9

Appendix

Table 9: All 15 garden-path minimal pairs with surprisal at disambiguation. Pairs 1–5 follow Van Gompel and Pickering [2001] ; pairs 6–15 extend the set.

Neuron	Tier	Base rate	Exc rate	Enrichment	$p$ -value
N2123	Core	70.9%	100.0%	1.41 $\times$	$<10^{-300}$
N584	Differentiator	6.2%	8.3%	1.33 $\times$	$<10^{-300}$
N2378	Differentiator	5.7%	7.4%	1.30 $\times$	$<10^{-300}$
N737	Specialist	4.0%	4.9%	1.22 $\times$	$<10^{-300}$
N1602	Differentiator	22.7%	24.4%	1.08 $\times$	$<10^{-300}$
N611	Differentiator	67.2%	71.8%	1.07 $\times$	$<10^{-300}$

Appendix

Table 10: Enrichment of routing neurons under exception firing (Fisher’s exact, Bonferroni-corrected).

Explore similar work

The Discrete Charm of the MLP: Binary Routing of Continuous Signals in Transformer Feed-Forward Layers
Mar 11, 2026Peter BaloghTransformer FFNs Transformer Interpretability
Half the Nonlinearity Is Wasted: Measuring and Reallocating the Transformer's MLP Budget
Mar 3, 2026Peter BaloghTransformer FFNs Conditional Computation
Peter Balogh
A Mechanistic Account of Attention Sinks in GPT-2: One Circuit, Broader Implications for Mitigation
Apr 16, 2026Yuval Ran-Milo, Hila Ofek, Shahar MendelSelf-Attention Decoder-Only Language Models
Tel Aviv University

Mar 11, 2026J/K move · Enter open · S save

The Discrete Charm of the MLP: Binary Routing of Continuous Signals in Transformer Feed-Forward Layers

Peter Balogh

Transformer FFNs Transformer Interpretability Circuits in Language Models

Open paper AskPDF arXiv

We show that MLP layers in transformer language models perform binary routing of continuous signals: the decision of whether a token needs nonlinear processing is well-captured by binary neuron activations, even though the signals being routed are continuous. In GPT-2 Small (124M parameters), we find that specific neurons implement a consensus architecture -- seven "default-ON" neurons and one exception handler (N2123 in Layer 11) that are 93-98% mutually exclusive -- creating a binary routing switch. A cross-layer analysis reveals a developmental arc: early layers (L1-3) use single gateway neurons to route exceptions without consensus quorums; middle layers (L4-6) show diffuse processing with neither gateway nor consensus; and late layers (L7-11) crystallize full consensus/exception architectures with increasing quorum size (1 to 3 to 7 consensus neurons). Causal validation confirms the routing is functional: removing the MLP at consensus breakdown costs 43.3% perplexity, while at full consensus removing it costs only 10.1% -- exceeding a 4x difference. Comparing binary vs. continuous features for the routing decision confirms that binarization loses essentially no information (79.2% vs. 78.8% accuracy), while continuous activations carry additional magnitude information (R^2 = 0.36 vs. 0.22). This binary routing structure explains why smooth polynomial approximation fails: cross-validated polynomial fits (degrees 2-7) never exceed R^2 = 0.06 for highly nonlinear layers. We propose that the well-established piecewise-affine characterization of deep networks can be complemented by a routing characterization: along the natural data manifold, the piecewise boundaries implement binary decisions about which tokens need nonlinear processing, routing continuous signals through qualitatively different computational paths.

Abstract

Explore similar work

The Discrete Charm of the MLP: Binary Routing of Continuous Signals in Transformer Feed-Forward Layers

Half the Nonlinearity Is Wasted: Measuring and Reallocating the Transformer's MLP Budget

A Mechanistic Account of Attention Sinks in GPT-2: One Circuit, Broader Implications for Mitigation