cs.LGOct 21, 2026

Reader--Writer Coupling in Transformer MLPs

Authors: Giyeong Oh, Sunghee Ahn, Youngjae Yu

Organizations: Seoul National University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

A transformer MLP gives each hidden channel two roles: a reader that decides when the channel responds, and a writer that decides what it adds back to the residual stream. Read as a key--value memory, the layer can be measured one factor set at a time: the pairing of readers to writers is carried only by their shared index, invisible to any quantity computed from one factor set alone. Yet training couples the two factors through their joint update, so we ask what reading them together shows. The block supplies the joint instrument: the MLP writes back into the very stream it reads, so the exact Jacobian of its update is square, and its agreement with its own transpose resolves into reader--writer terms whose inner products give each channel, read as a couple, a signed pairing coordinate. Ranked by this coordinate, each block we measure has a small core of strongly coupled channels; where initialization gives either sign equally, the core is nearly all opposed: each writer points against the direction its reader responds to. This opposed core recurs from the smallest models trained from scratch to released checkpoints; its sign is in place within a few percent of training and holds while the coupling grows. Read as couples, most channels return their contribution against the reader that fired it, and the readers of later blocks take it up; few show in the logits. Training holds the couples in place: flipping the opposed writers moves the output toward common tokens, and a writer permuted onto another reader regrows to fit that reader while the reader stays.

Figures & tables

Figure 1: Paired channel geometry. (a) Read one factor at a time, an MLP’s readers uˉh\bar{u}_{h} and writers vhv_{h} are two factor sets that look patternless on their own. (b) Read as couples, most reader and writer couples roughly line up, pointing opposite ways. (c) What couples show is a signed axis, the pairing coordinate mh=⟨uˉh,vh⟩m_{h}=\langle\bar{u}_{h},v_{h}\rangle , whose sign the norm product lacks. (d) Measured, in the strongest couples the writers point against their readers through the stack, most strongly mid-stack; the models trained here release toward both ends, OLMo-1B only at the output (mean cosine of the sixteen strongest, small at context 256∗256^{\ast} and 1,024†1{,}024^{\dagger} ).
Figure 2: The sign across the population, and four core channels read as couples. Left: every channel of three medium seeds at its reader–writer angle θh\theta_{h} and, outward, its norm product; the 90∘90^{\circ} line separates opposed (above) from amplifying (below); colour the block, cores outlined, A–D marked; above the line lie 83.2%83.2\% of channels, 98.5%98.5\% of cores, half at initialization. Right: four core channels; each cell names its kind, logit or stream couple, each row whether the norm product’s top sixteen holds it, the logits moved, later readers, and the return on its reader. # marks a leading space.
Figure 3: Learned pairing and update geometry. (a) A re-pairing keeps both weights and changes only who pairs with whom. (b) Trained updates are symmetric far beyond any re-pairing. (c) Negative eigenvalues carry most of the symmetric part’s energy, where re-pairing gives one half; only OLMo-1B’s last block, whose couples flip, and the small model’s first expand (tiny, small and medium: blockwise medians over three seeds). (d) Most of the symmetry is mutual: channels reading each other’s writer contribute more than the couples on their own.
Figure 4: Writers grow back, readers stay.
Figure 5: Weight-level pairing along the stack. (a) Web text, eight blocks: the raw-row coordinate mw,h=⟨kh,vh⟩m_{w,h}=\langle k_{h},v_{h}\rangle over all 1,0241{,}024 channels of each block, three seeds pooled, split at zero into opposed (blue) and amplifying (red) halves; dots mark one seed’s sixteen core channels. (b) Share of channels with mw,h>0m_{w,h}>0 per block at {4,6,8,12} blocks on relative depth, seed means.
Figure 6: The effective core and the corpus. (a) Each block’s final top- 1616 by ∣mh∣|m_{h}| , negative minus positive members over sixteen, at every saved state of medium and at twenty-four public checkpoints of OLMo; tokens on both axes, OLMo’s on a log scale. (b) Tiny models, one marker and colour per depth (legend): the pairing coordinate summed over each block’s core at the end of training against relative depth; natural streams as a band with their median, streams with the unigram frequencies kept and the context destroyed as dashed lines.
Figure 7: The couple census. Three medium seeds. (a) Every channel by family, area by share, classes inside. (b) The same families block by block, each bar one block’s channels; (a) is the average of the bars.
Appendix figures & tables38 assets

Supplementary material from the paper’s appendix.

Appendix

 TinyTiny ladderTiny gridSmallMedium
Blocks ×\times width4×2564\times 256{4,6,8,12}×256\{4,6,8,12\}\times 2564×2564\times 25612×76812\times 76824×1,02424\times 1{,}024
Heads88888812121616
Parameters28.928.9 M28.928.9 – 35.235.2 M28.928.9 – 29.529.5 M123.6123.6 M405.3405.3 M
of which blocks3.13.1 M3.13.1 – 9.49.4 M3.13.1 – 3.73.7 M85.085.0 M302.2302.2 M
Seeds9933 per depth11 per cell33 per cell33
Context2562562562562562561,0241{,}024 / 256256256256
Appendix
Table 1: Training hyperparameters. One column per cohort; the small column reads context 1,0241{,}024 / 256256 where its two cells differ. The context- 256256 SwiGLU family of Figure 3(b) differs from that cell only in its branch (gated, hidden width 2,0462{,}046 for parameter matching, 123.5123.5 M parameters). Extended = GPT-2 plus reserved mask and sentinel identifiers.
SetSourceExtentUsed in
Controlled trainingFineWeb-EDU10810^{8} tokensSection 5
Controlled validationvalidation split5×1065\times 10^{6} tokensSections 4.2 and D
Grid trainingFineWeb-EDU0.330.33 B of 2.62.6 BAppendix D
Grid validationvalidation split10710^{7} tokensAppendix D
Medium trainingFineWeb-EDU 100100 B sample8.118.11 B of 9.849.84 B per seed, disjoint slicesSection 5
Medium validationkey-disjoint split, per seed10710^{7} tokensSection 5
Appendix
Table 2: Streams, panels, and validation sets. Offsets and intervals index tokens. Set B contains 65,28065{,}280 scored transitions; the released-label panel gives 128128 observations per block. The medium frequency reference is the entire evaluation pool.
QuantityValue
Jacobians per block
Assignment, trajectory, pair-break protocols88
Band endpoints and bare-branch studies3232
Draws
Writer permutations999999
Trajectory prefix199199
Appendix
Table 3: Measurement, draw, and selection budgets. Protocols of the controlled cohorts. Jacobian counts per block are inputs ×{\times} positions, 8=2×48=2\times 4 and 32=4×832=4\times 8 . Permutation draws are per seed and block; the doses move 5151 or 256256 of 1,0241{,}024 channels ( 5%5\% and 25%25\% ); the selection thresholds are the prespecified evaluability conditions.
Figure 8: Endpoint gap along the stack. ΓC\Gamma_{C} by relative depth for the tiny depth ladder, the two small contexts, and the medium; lines are seed means, bands the seed range. Every cohort peaks inside the stack, the tiny ladder at blocks 11 , 33 , 55 and 66 , the small at blocks 44 – 66 , the medium at block 1111 ; the small and medium start near zero at their input block.
Figure 9: The minimum eigenvalue and its scale dependence. Smallest eigenvalue of S=sym(Jˉ)S=\mathrm{sym}(\bar{J}) per block, three medium seeds against their 6060 same-factor re-pairings each. (a) In raw units the trained operator’s most contracting direction is more negative than every re-pairing in every block. (b) Once each operator is divided by its own RMS eigenvalue ∥S∥F/d\|S\|_{F}/\sqrt{d} , the trained curve crosses the re-paired band, less extreme than its re-pairings through the middle of the stack and more extreme only in the first five and last two blocks: the raw separation is mostly the trained operator being larger.
Scale cellsMergesGap retained
GELU000.990.99
SwiGLU000.990.99
Appendix
Table 4: Scale constraints and input exchange. Every trained assignment exceeds all 999999 stored draws. Left: the two small context- 256256 families, three seeds each (median retention, no cell merged). Right: one seed per cohort and eight new panels; departures are relative to its original panel’s gap.
Figure 10: Symmetric and antisymmetric energy. (a) Squared norms of the symmetric SS (Sym) and antisymmetric KK (Antisym) parts of the panel-mean Jacobian, relative to their own re-pairing means, three medium seeds; gray the mean and full 6060 -draw range. (b) Mean eigenvalue of SS divided by its RMS eigenvalue.
Figure 11: Weight-level pairing, gated against elementwise. OLMo-1B (a) and GPT-2 small (b) at their final public checkpoints. Left: the distribution of mw,hm_{w,h} over every channel of each block, split at zero into opposed (blue) and amplifying (red) halves; dots are the block’s sixteen core channels of the effective coordinate at their weight-level value, and the axis covers every dot. Right: share of channels below zero per block, effective against weight-level. On the gated model the input blocks’ effective cores sit in the amplifying half of the weights; on the elementwise model the cores sit on the opposed side at every depth and the two shares run together.
Figure 12: The gate separates the two readings. (a) Share of channels whose weight-level and effective coordinates agree in sign, by relative depth; the band is the range of the elementwise families trained here. (b) The same agreement before (open markers) and after (filled) removing the reader contributions at negative-gate positions, with the small GELU control. (c) Opposed share as the negative gate is scaled toward zero, input block against middle band; crosses at zero are random deletions of matched size.
Figure 13: Formation and recovery controls. (a) Three tiny-seed gap trajectories; gray shows one step-zero null SD, maximized across seeds. (b) Mean block signed-sum zz relative to its uninterrupted control. (c) Reader permutation: writer following (solid), reader returning (dashed). In (b,c), lines are seed means, bands their ranges, and colors identify break steps; resumed training ends at step 2020 k.
Figure 14: How the final core develops, composition and magnitude. Each row tracks a block’s final top- 1616 by ∣mh∣|m_{h}| , holding the channel indices fixed across training. Columns enumerate saved checkpoints. (a,c) Positive counts fill cells from above and negative counts from below; gray marks near-orthogonality. (b,d) Separate positive and negative magnitudes include every selected channel, using a shared logarithmic length scale within each model.
Figure 15: Backtracking every endpoint core. Each row is one block’s sixteen final core channels, held fixed across the ten saved OLMo checkpoints. Counts are not averaged across blocks.
Figure 16: The same final-block channels through training. Rows follow endpoint rank. Left: each channel’s pairing coordinate divided by its maximum absolute value over the saved states; outlined cells are in that state’s top- 1616 . Middle: reader–writer cosine. Right: absolute-coordinate rank among all 8,1928{,}192 channels. Color magnitudes in the left panel are comparable within a row only.
ID / tokensReader (native)Writer (native)Writer (final head)
7325 / 4B[hy] [ights][ave] [IE][ASE] [endo]
7325 / 16B[ert] [hy][Sock] [pkg][sar] [deck]
7325 / 41B[ert] [stan][Sock] [house][WORK] [RET]
7325 / 3.095T[ted] [ting][ted] [sm][ted] [sm]
7749 / 4B[items] [ought][iments] [s][#Prote] [#Had]
7749 / 16B[items] [ium][vent] [op][#Damage] [#Did]
Appendix
Table 5: Vocabulary projections of the same channel IDs through training. Two fixed endpoint-core IDs chosen for readable endpoint vocabulary lists. Columns show the top two entries under each checkpoint’s native unembedding, with a fixed final-head writer readout alongside it. These are projected directions, not observed triggers or established semantic roles. Leading spaces use # .
Figure 17: The conditional comparison across released labels. (a) Blockwise-median ΓC\Gamma_{C} per size at the released labels, equally spaced on the axis; the open marker is the 2.8B label 6565 k standing in for the duplicated 6464 k bytes. The gap departs its ensemble mean at labels 512512 – 11 k, grows through the middle of training, and plateaus. (b) Per-block ΓC\Gamma_{C} at the final label against relative depth: one hump at every size, near the mean at the first block and closing toward the output.
Figure 18: Symmetry along Pythia training. Top: negative-energy share of the symmetric part minus one half; bottom: the fixed-denominator gap GG of Equation 27 , which the axis writes as C(Jˉ)−EπCC(\bar{J})-\mathbb{E}_{\pi}C with the trained denominator; one line per block, coloured by relative depth, against training tokens on a log axis with label 00 at the left edge. Ticks above each panel mark the labels of the sparse replay. The 2.8B column uses those labels only.
ModelSign settled (step)∑∣mh∣\sum|m_{h}| at 14\tfrac{1}{4}at 12\tfrac{1}{2}Core ∣mh∣|m_{h}| at sign step
GPT-2 small, five seeds200 (90–350)5,400 (4,425–6,600)20k (15k–23k)1%
Pythia-160M128 (128–100k)4,000 (2,250–4,000)11k (8,000–15k)9%
Pythia-410M1,000 (512–1,000)4,000 (3,000–4,000)14k (7,250–16k)3%
Pythia-1B1,000 (1,000–1,000)5,000 (3,250–6,000)20k (15k–26k)2%
Pythia-1.4B1,000 (1,000–1,000)6,000 (5,000–8,000)23k (15k–27k)2%
Appendix
Table 6: The sign is in place before the coupling grows. Over the interior blocks of each model, median and interquartile range of the step at which the block’s core stays majority opposed, of the step at which the block’s summed ∣mh∣|m_{h}| stays above a quarter and a half of its final value, and the core’s ∣mh∣|m_{h}| at the sign step as a share of its final value. GPT-2 small pools five seeds; one Pythia block turns late and widens its range.
Figure 19: Core formation across GPT-2 seeds. Five GPT-2 small replications ( Karamcheti et al., 2021 ) . Top: for every block, the balance of the block’s sixteen-channel core, opposed members minus amplifying over sixteen, against training tokens, one line per seed; the dotted line marks twenty tokens per parameter. Bottom left: the tokens at which each block’s core turns majority opposed, one marker per seed, blocks in depth order. Bottom right: the opposed share of all channels by depth band. Step 00 sits at the left edge of each axis.
SizeBlocksAll beyond fromFinal ΓC\Gamma_{C} , median (SD)
70M663232 k0.2240.224 ( 0.1610.161 )
160M121211 k0.2190.219 ( 0.1550.155 )
410M242422 k0.2540.254 ( 0.1480.148 )
1B16165125120.1870.187 ( 0.1430.143 )
1.4B242422 k0.1890.189 ( 0.1430.143 )
2.8B32325125120.1390.139 ( 0.1310.131 )
Appendix
Table 7: Persistence across released labels. First label after which every block stays above its re-pairing mean, and the final median gap with across-block SD. All endpoint blocks are above their means; earlier crossings can reverse. The 6.96.9 B model uses 199199 draws per block, the others 999999 .
Figure 20: Core channel signs across the full corpus grid. The effective coordinate mhm_{h} of every endpoint core channel, sixteen per block, for six corpora, four depths (four, six, eight and twelve blocks) and three seeds (shape and offset) on the exact-reader panels of Section F.3 . Every natural corpus forms opposed cores with a few amplifying members in its last blocks; at twelve blocks the input block adds small positive members; the two order-free controls stay at zero.
Figure 21: Every channel, with and without order. Left: the effective pairing coordinate mhm_{h} of all 1,0241{,}024 channels of each block over training (gray), one eight-block run of the corpus recipe on the web stream (top) and on the same stream with its token order removed (bottom); the sixteen channels that end as the block’s core are coloured, blue if opposed and orange if amplifying. Without order no channel leaves the bulk. Right: the stream cut into chunks of ℓ\ell tokens with only the chunk order shuffled; the core sum (top) and the number of amplifying core channels (bottom) at the end, three seeds (band). Order within a few dozen tokens is enough to form the opposed cores. On these GELU models the weight-level coordinate agrees in sign but selects a different top sixteen (Appendix E.2 ).
Figure 22: Corpus composition readings. Rounded percentages in five surface categories, read from raw-lens lists under the corpus taxonomy. Each row contains 384384 channels, except the eligible amplifying sets for permuted web ( 180180 ). Freq. denotes common-token lists; Morph. denotes morphemes; Norm is the writer-norm-matched set. Colors share one scale across streams for these five qualitative categories.
StreamReaderWriter projectionRecurrence
WebBiochemical fragmentsPromotes civic nouns: government , policyBlock 6; three seeds
BooksFemale names and titlesSuppresses female pronouns: her , sheBlock 6; three seeds
StoriesEating vocabularySuppresses eat , ate , hungry , foodBlock 6; two seeds
StoriesNot specifiedAmplifying writer suppresses female pronounsBlock 7; two seeds
NewsUnrelated readersModal verbs and sports vocabularyBlocks 5–7; writer only
Appendix
Table 8: Corpus-specific couple readings. Selected recurring patterns from the full core reading, not prevalence estimates or realized next-token effects. Recurrence refers to the same block and kind of reading, not an identical channel index.
Figure 23: Angle, scale, and the reader-size dependence. Top: every channel of one medium seed at an early, a middle, and a late block, as in Figure 2 : distance from the origin is the reader–writer norm product (square-root scale), the angle is the angle between the two, so opposed channels ( mh<0m_{h}<0 ) lie above the line. Filled dots are the sixteen core channels by ∣mh∣|m_{h}| ; open squares and diamonds are the sixteen largest readers and writers. The core sits at the large-reader, most-opposed corner and overlaps the largest readers (eight, six and four of sixteen), while the largest writers scatter across the angle: opposition is organized by the reader, not by the writer. Bottom: opposed share by reader-norm quartile on the large sign panel, one line per seed, averaged over each eight-block band. Amplifying channels sit mostly in the smallest-reader quartile.
Observed effectMedium ( 381381 channels)OLMo ( 243243 channels)
Post-MLP residual norm decreases373162
Centered logit norm increases151144
Centered target logit increases167177
Target increase: interval above zero4085
Target decrease: interval below zero6815
Appendix
Table 9: Negative feedback and output effects differ. Counts among endpoint-core channels with mh<0m_{h}<0 . The first three rows use mean effects; the last two require a pointwise 95%95\% interval excluding zero. Output effects include later-block responses.
ChannelKindReader normWriter normNorm productPush
Alogit couple12121111110.850.85
Bstream couple66626626550.070.07
Clogit couple6576573,8623{,}8629769762.22.2
Dstream couple54541123230.050.05
Appendix
Table 10: The four core channels of Figure 2 . Ranks descend within the channel’s own block; push is the direct movement of the next-token logits, as a share of the gap between the two largest, at the strongest firings.
ModelUnitsInside band after breakRrecR_{\rm rec}
GPT-2 (pre)12 blocks12—
Pythia-70M (pre)6 blocks6—
SmolLM2-135M (pre)30 blocks—median 99.8%99.8\%
Tiny GELU, controlled12 seed–blocks12—
Appendix
Table 11: Outcomes of the one-sided pair break. Units returned inside their matched band, and continuous recovery RrecR_{\rm rec} where it was summarized (a dash marks the readout not summarized for that model).
GroupBlocksWithin marginDbreakD_{\rm break}DreinitD_{\rm reinit}Dbreak/DreinitD_{\rm break}/D_{\rm reinit}
Controlled (3 seeds)1240.065±0.0040.065\pm 0.0040.284±0.0060.284\pm 0.0060.22±0.010.22\pm 0.01
GPT-21200.106±0.0530.106\pm 0.0530.356±0.0790.356\pm 0.0790.30±0.170.30\pm 0.17
SmolLM23040.070±0.0210.070\pm 0.0210.216±0.0780.216\pm 0.0780.34±0.120.34\pm 0.12
Appendix
Table 12: Spectral-shape displacement under the one-sided break. Mean ±\pm spread of the blockwise displacement: over the three seeds for the controlled family, over blocks for the two pretrained models. Within-margin blocks are counted against the blocks measured.
Seedθs\theta_{s} (millinats)Positive strata (of 8)Raw ppHolm pp
#15.1580.01070.0107
#24.7470.00010.0006
#36.0860.00010.0006
#48.7280.00010.0006
#54.1660.00080.0024
#66.2060.00080.0024
Appendix
Table 13: Loss difference at equal disturbance, per seed. Among re-pairings matched on weight, Jacobian, and output disturbance, the ones that damage symmetry more cost more loss: θs\theta_{s} is the equal-stratum loss difference in millinats, positive in every seed and in most strata, small next to the loss itself, and significant after correction in every seed. Tiny four-block GELU models at context 256256 ; the six seeds are disjoint from the three on which the comparison was first run, and all 4848 dose–block strata met the matching requirements.
Figure 24: Deletion costs separate along training. All blocks are edited together and the full network is reevaluated; each deletion is compared with a random direction of matched size (dotted). Early in training the network depends on the perpendicular component alone; the parallel component, the one the coupling measures, becomes a cost of its own later and dominates at the endpoint, in the medium and in OLMo alike. In (b) filled markers are the deletions and hollow markers their random comparators, shape identifying the seed. These costs are network reevaluations, neither additive over blocks nor direct-logit attributions.
Figure 25: Flipping opposed writers favors common tokens. Selected writer vectors are negated in every medium block: most negative mhm_{h} (opposed) or largest positive mhm_{h} (amplifying). Both panels show selected minus activity- and size-matched flips. Thin curves show three seeds and thick curves their mean; shading marks 1616 of 4,0964{,}096 writers per block. Frequency error bars at 1616 are 95%95\% paired-window bootstrap intervals, conditional on the three endpoints.
Figure 26: From profiles to classes. Stored channel profiles (activation contexts, replayed forward) yield three registers per channel: the input that fires it, the contribution’s first-order response on later readers, and its direct push on the next-token logits, at the strongest firings and at typical activations. Grey stages are measured; orange stages are language-model annotation, with the engine of each stage under its column.
ClassSeed 1Seed 2Seed 3
stream couples71.071.071.971.971.271.2
Opaque-trigger stream writer28.628.629.329.328.928.9
Cross-family stream handoff11.911.912.712.712.212.2
Silent morphology relay9.69.69.39.39.59.5
Silent entity name relay8.48.48.28.28.48.4
Silent function-word relay7.37.37.37.36.96.9
Appendix
Table 14: Couple census by class and family. Shares (%) of the 98,30498{,}304 channels of each medium seed; classes below 0.05%0.05\% in every seed are omitted from the listing but counted in their family; dispositions are rows of the same table, and shares are rounded independently.
SeedClassFamily
Seed 177.0±1.377.0\pm 1.391.7±0.891.7\pm 0.8
Seed 276.7±1.376.7\pm 1.391.5±0.991.5\pm 0.9
Seed 374.8±1.474.8\pm 1.491.3±0.991.3\pm 0.9
Appendix
Table 15: Agreement with the reference labels. Share (%) of 1,0001{,}000 reference-labelled channels per seed on which the population label equals the reference label, at the class level and at the family level, mean ±\pm bootstrap standard error.
Figure 27: What the core writes. (a) Median promoted-token frequency rank across lens depths; more frequent is up. Removing leading unembedding directions narrows the medium core’s separation from its controls. (b) Common-token, morpheme, and semantic-field readings of the same four sets, with colors and markers shared across panels. Each set contains 1,1521{,}152 medium channels; the full taxonomy and raw-rank comparisons are in Tables 18 and 16 .
FamilyCoreAmplifyingEqual-normRandom
Medium9,2499{,}24912,15912{,}15911,03311{,}03311,28611{,}286
Small, context 1,0241{,}0247,8437{,}84312,27512{,}27510,70810{,}7089,9729{,}972
Small, context 2562569,4509{,}45013,15813{,}15811,24811{,}24810,85210{,}852
Tiny1,3411{,}3411,0631{,}0638918912,3592{,}359
Appendix
Table 16: Raw-lens frequency ranks across cohorts. Median promoted-token rank for the four selected sets; lower is more frequent. Medium spans 7272 seed–blocks, each small context 3636 , with frequency references defined within each family. The medium lens-depth curves are in Figure 27 .
Seed / blockPromoted (top 3)Suppressed (top 3)med. rank P/S
Common-token pattern
#1 / block 19 ∗#them , #the , #aolver , #feel , #eat76 / 21,06876\,/\,21{,}068
#3 / block 18 ∗#and , , , -/ ⟨\langle zwsp ⟩\rangle , adata , kees15 / 32,703.515\,/\,32{,}703.5
Appendix
Table 17: Every rule-qualifying channel. All medium core channels passing the fixed surface rules at the raw vocabulary lens ( 22 common-token, none repeated-separator, of 1,1521{,}152 scanned). # marks a leading space; median ranks cover each channel’s sixteen-token lists (rank 11 = most frequent). Starred rows are each seed’s common-token maximum; the second seed has none. ⟨\langle zwsp ⟩\rangle is a zero-width space.
Channel setnncommonmirrorfieldmorphemeseparatornone
core, all blocks1,1521{,}1524422664545003636
core, blocks 00 – 773843841100226868002929
core, blocks 88 – 15153843841111224848004646
core, blocks 1616 – 2323384384993314141818003434
amplifying sixteen, all blocks1,1521{,}152221114144747003232
amplifying sixteen, blocks 1616 – 2323384384553336361515003030
Appendix
Table 18: Surface census of the four sets. Channel shares (%) from language-model readings of raw-lens lists on the medium; residual classes are omitted. The core is opposed in 1,1351{,}135 of 1,1521{,}152 channels, leaving no comparable within-core sign split.
More in this topic

Transformer FFNs

Transformer feed-forward networks (FFNs) are the within-token MLP blocks whose activations, structure, efficiency, and role in model computation are studied.

See all 42 papers

Explore similar work

CardsList
Aug 22, 2026

The Communication Map of a Transformer

The components of a transformer communicate by writing to and reading from a shared residual stream, and the mechanistic interpretability literature has mapped these connections by hand, one circuit at a time. We present the communication map, which charts every potential communication channel from the geometry of the model's weights alone, generalizing the composition score of Elhage et al. (2021) into a single coupling coefficient covering all 18 connection classes, from head-to-head to neuron-to-neuron and everything in between. We provide an account of the properties of the coupling coefficient, including its geometric interpretation and its exact chance level. The census finds that 70-89% of head pairs are oriented far from chance, some coupled strongly and others actively avoiding each other. We demonstrate the communication map in two novel applications. In Application 1, we recover the known induction circuits blind from the strongest head-to-head couplings and group the heads into communities, and ablating one such community destroys the model's in-context copying. In Application 2, we pool the coupling coefficients of every head to identify a distinct two-dimensional residual stream subspace, whose deletion abolishes the induction capability in six models up to Pythia-6.9B. We show that this subspace is different from those identified by either activation PCA or outlier dimensions. We release the map, the statistical machinery, and the intervention suite.

Richard Zhe Wang

St. John Fisher University

Transformer InterpretabilityCircuits in Language Models
AskPDF
Sep 9, 2026

Contrastive Projection: Reading Transformer Internals by Differencing Logit Lenses

Reading a transformer's internal states in token space is easy to do and hard to trust: a logit lens on a single hidden state is dominated, at intermediate layers, by the generic tokens the model would predict for almost any input. We read the difference instead. Subtracting two closely matched prompts' hidden states and projecting through the unembedding cancels the shared component and surfaces what separates them, an operation equivalent to reading a RepE/ActAdd steering vector through a logit lens. Built into a training-free tracer that reads at every position, sub-layer, and head and averages over designed baselines, it traces a compound- noun MLP->attention chain in Phi-2, confirmed there by activation patching, with the same distinction recovered across three architectures by readout and probe rather than by patching; it reads what retrieval surfaces for real versus fictional entities, and reads metaphor as a set of domain-to-domain mappings rather than a single figurativity feature. A cross-seed control marks the boundary: across five networks differing only in initialization, the same distinction surfaces as almost entirely different tokens (top-10 overlap 0.08). What a computation looks like in token space is network-specific; the distinction it draws is not

Olli Tuomi

Evident Solutions Oy

Transformer InterpretabilityLanguage Model IntrospectionTransformer Attention
AskPDF
Sep 25, 2026

Transformer MLP Gate Thresholds Are Couplings to a Carried Reference Direction

The corpus-mean direction of a transformer's residual stream is a component shared across all inputs, and is commonly removed by mean-centering before representational analysis. We present evidence that it is a functional component: the reference against which the MLP gate population sets its operating point. In Phi-2, an exact decomposition of resting gate pre-activations shows that at mid-stack layers the resting inhibition of >99.9% of gates is carried by the coupling w⋅bLw \cdot b_L to the carried mean direction, at 48-56×\times the explicit bias parameter, and the coupling is direction-specific: a random direction at matched norm orders the population's firing rates at Spearman ρ≤0.09ρ\leq 0.09 where the reference reaches 0.95. (That 0.95 is near-tautological on its own; the paper derives its null.) Causally, removing the stream's projection on the reference multiplies above-threshold firing by about 9×\times, dose-monotonically, at 23-59×\times a norm-matched control; a random-initialised twin is flat, and replacement tests show that the direction carries the function and the magnitude does not. A direction-matched control makes the same point: noise injected along the reference costs 8-42×\times the same energy along a random direction. The decomposition replicates on four further families spanning both gate types (GELU with an explicit gate bias, bias-free SwiGLU), and the dose-response on two of them. Across an eight-model scan the mechanism is present in every GELU and SiLU family and absent only in OPT, where an opposing LayerNorm bias cancels the carried reference. Gate thresholds are implemented as couplings to a constant the network builds for the distribution it is reading, with the bias parameters contributing little; how much of that constant is carried in the stream and how much in parameters depends on the architecture.

Olli Tuomi

Evident Solutions Oy

Transformer InterpretabilityTransformer FFNs
AskPDF