Gathered, Not Admitted: How Attention Brings a Latent Variable into Verbalizable Form

Parsa Mazaheri

cs.AIAug 15, 2026

Gathered, Not Admitted: How Attention Brings a Latent Variable into Verbalizable Form

Authors: Parsa Mazaheri

Organizations: University of California, Santa Cruz

Abstract

Language models hold latent quantities in a form they can report on, and more of a quantity is present in that form when the task requires reusing it flexibly. What causes a representation to enter that form is open, and the word workspace invites an admission story: a gate that decides what gets in. Testing it on open-weight models with Jacobian lenses, over a benchmark whose five arms share an identical context, we find no gate where it predicts one. Demand raises a concept's lens visibility beyond what applying an operator to a supplied value produces: +0.050 [+0.045, +0.057] in percentile rank on our primary checkpoint, positive on all four we measure, though that arm answers at ceiling and the accuracymatched contrast is stronger under that readout. At the same time one shared linear map decodes the variable from every arm, the control included, at 6.4-9.0x its selection-corrected floor. What produces the later readable form at the queried position is attention-mediated gathering inside a mid-depth window: separating patch depth from readout depth puts transport there at least 17x above anywhere shallower under non-saturating readouts, with no tested MLP output contributing positively inside it. Under the saturating percentile rank the same grid does not localise the window, which is a fact about that measure. An arm that needs the variable for nothing concentrates sevenfold less, so the window is demand-specific. That window has two measured edges, a survival failure below and destruction above, and it falls at the same fractional depth in a 64-layer hybrid and a 62-layer dense model from another family. We localise where the variable is installed and read, not the route from the passage, which transports nothing. But the readout is not a calibrated measure of use: three components move it to within 12% of one another and differ 7.4x in what they do to the answer.

Figures & tables

Figure 1: Two accounts of what changes when a task demands flexible reuse of a latent variable $z$ . (a) Admission, as the workspace vocabulary invites one to read it: $z$ is at the query position at every depth and a gate sets whether it enters. This is our formalisation of the hypothesis, not a claim Gurnee et al. (2026) make. (b) What we measure: attention gathers $z$ into the query position within a mid-depth window, a value installed below it does not survive to the readout, and the answer depends far more on the consolidated stream than on the gather. The passage is the plausible source but we do not localise it. (b) displaces (a) at the query position; it does not rule out a gate acting on the gather itself. The re-derivation labelled in (b) is our favoured reading of the lower edge, not a measurement (§ 5 ).

contrast	$\Delta R_{z}$	$\Delta M_{z}$	$\Delta$ accuracy
flexible $-$ control	$+0.0891\,[+0.0799,+0.0983]$	$+2.79\,[+2.57,+3.01]$	$+0.000\,[-0.045,+0.045]$
report $-$ control	$+0.0991\,[+0.0891,+0.1092]$	$+3.52\,[+3.31,+3.73]$	$+0.040\,[+0.005,+0.075]$
flexible $-$ supplied	$+0.0504\,[+0.0445,+0.0567]$	$+4.05\,[+3.81,+4.28]$	$-0.060\,[-0.095,-0.030]$
supplied $-$ control	$+0.0387\,[+0.0310,+0.0466]$	$-1.26\,[-1.37,-1.15]$	$+0.060\,[+0.030,+0.095]$

Table 1: Lens visibility by condition pair on Qwen3.6-27B, under the saturating percentile rank and the non-saturating margin, both averaged over the workspace band. Paired bootstrap over semantic instances (

10{,}000

resamples,

n=200

), the instance being the resampling unit because all five arms share a passage. Arm accuracies are control

0.940

, report

0.980

, flexible

0.940

, supplied

1.000

; control is a yes/no question where the others offer four candidates, so measured above its own chance floor it reads

0.880

against flexible ’s

0.920

. The first row is matched on prompt format and on raw accuracy; the last two separate the operator from the latent variable, and the operator’s own contribution is the one quantity here that changes sign with the measure.

Figure 2: The causal chain, measured three ways on the language family; all intervals are paired bootstrap intervals over instances. (a) Where a donor’s value can be installed , read three to five layers above the patch so every depth is measured at matched distance, in the non-saturating $L_{z}$ at $n=120$ , averaged over four donor pairings; per-cell means, and how many pairings each clears, are in Table 4 . (b) The behavioural counterpart at $n=150$ : donor-symbol rate minus the matched per-distractor rate. The shaded band is where the patch still leaves the task solvable; the effect is significant above it too, at accuracies of $0.17$ – $0.21$ . (c) Specificity of the mediating direction at $n=150$ , against a null of one point per wrong concept.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: The second task family, in

R_{z}

n=140

, the readout the four-pairing table below reports, named in the panel title because the language panels use

L_{z}

. (a) The branch asymmetry replicates: the stream and the attention output transport and the MLP at L39 opposes, but the attention effect is shared between L39 and L48 rather than concentrated at one layer as it is on language; L48 is a linear-attention layer. (b) The 24 heads of L39 in both families, one donor pairing each; the language series is

n=60

. On language one head matches both the block and the whole stream to three decimal places; on tracking the same head participates at about a fifth of its own family’s stream, which four pairings confirm is nonzero (table below). Language depth profile: Figure 2 a.

Figure 4: What travels across checkpoints. (a, b) The distractor-controlled counterfactual against fractional patch depth at $n=150$ pairs per pairing, on the same axes, with all three donor pairings drawn (the thick line carries the bootstrap band) and task accuracy dashed. Where accuracy falls below $0.5$ the patch has replaced the late computation, and the shading marks that a donor-symbol rate there is not evidence of transport. (c) The entry effect with the operator held fixed ( flexible $-$ supplied ) at $n=200$ instances per checkpoint is positive on all four; the operator’s own contribution ( supplied $-$ control ) is not stable in sign.

Figure 5: A readout shift is not a calibrated measure of use, on the same $80$ trials throughout, one donor pairing. (a) Three components at L39 shift the donor concept’s readout to within $12\%$ of one another while their effect on behaviour differs $7.4\times$ . (b) Per trial the two are related but weakly: uncalibrated rather than uninformative.

Figure 6: The behavioural window and both of its edges. (a) Rates against patch depth at $n=150$ , one pairing: the donor’s predicted symbol, the matched per-distractor rate, and task accuracy. From L48 the distractor rate approaches chance while accuracy is below $0.2$ , which is destruction rather than transport. (b) The distractor-controlled effect; the shaded band marks the cells where accuracy still permits a reading. (c) The lower edge is a survival failure, at $n=100$ and span 1: a value installed at L33 is gone by L46, and the target’s own value is progressively suppressed as the donor’s is installed.

Figure 7: (a) Necessity by leave-one-out mean ablation, $50$ instances per arm. A component is selectively necessary when it costs the arms that need $z$ and not the format-matched control; only resid.L39 clears zero under this ablation, and mlp.L39 is the one cell pointing the other way. That the control arm is measurable at all is shown by the harsher zero ablation in Table 6 , not by this panel. (b) Mediation at $n=150$ , one pairing: the loss in counterfactual answer rate from projecting out the concept’s static lens vector, and from the same vector orthogonalised against the other nineteen concepts, against a null drawn one point per wrong concept. Paired per instance throughout, so a rival that happened to help would appear left of zero. The null is plotted as one estimate per concept identity rather than as one interval, because a single random control cannot be told from a lucky draw.

Figure 8: Visibility dissociates from availability, $n=200$ semantic instances throughout. (a) Mean $R_{z}$ of the gold concept at the query position by layer and condition. The L33 dip appears in all four arms but is about three times deeper in flexible and control than in report and automatic, so it cancels within either pair and not across them. (b) Each contrast’s visibility effect against the accuracy difference it must not be; the primary contrast sits at $\Delta$ accuracy $=0$ , and shading marks where difficulty differs enough that the effect is not interpretable. (c) A single linear probe trained jointly across all four arms decodes $z$ in every one. This panel plots the layer-selected form , whose diagonal runs $0.64$ (control) to $0.84$ (report); the nested figures quoted in § 3 , $0.575$ to $0.808$ , choose the layer on an inner split of the training instances only and are the ones the claim rests on. Bars are the mean over five instance-disjoint splits and whiskers their standard deviation; because the layer is selected, the reference line is the $0.090$ reached by permuted labels under the same selection rule, not the nominal $0.05$ .

role	artifact	$n$	unit
selection: small by design, frozen before everything below
coarse screen (chose L39)	coarse20	20	pairs
head decomposition (chose H15 )	heads_full	25	pairs
bidirectional screen	coarse60	60	pairs
entry and availability
Stage 1, four arms	s1v2	200	instances

Appendix

Table 2: Every run in the paper with the sample it used, read from the args block of the artifact it names so the table cannot drift from what was run. Artifact names are given without their directory (one per experiment family under the data root) and without the model slug each file carries as a suffix. Instance selection is deterministic (sorted by identifier, then truncated), so a rerun sees the same instances. Where a range over donor pairings is reported the seeds are

0

–

3

; otherwise seed

0

. The probe averages five split seeds.

train $\rightarrow$ test	raw	per-arm centred
report $\rightarrow$ flexible	$0.625$	$\mathbf{0.825}$
flexible $\rightarrow$ report	$0.525$	$\mathbf{0.750}$
control $\rightarrow$ automatic	$0.213$	$\mathbf{0.675}$
control $\rightarrow$ flexible	$0.175$	$\mathbf{0.625}$
control $\rightarrow$ report	$0.163$	$\mathbf{0.613}$
report $\rightarrow$ automatic	$0.175$	$\mathbf{0.600}$

Appendix

Table 3: Cross-arm probe transfer,

n=200

instances, 20 classes, chance

0.05

, split by semantic instance. Raw transfer is weak because the arms differ in prompt format and occupy different regions of activation space; removing each arm’s own mean lifts transfer to the level of the within-arm diagonal (

0.64

–

0.84

; Table 3 ). Both columns are reported because the centring was added after seeing the raw numbers.

	$\Delta R_{z}$			$\Delta L_{z}$
patch layer	stream	attention	MLP	stream	attention	MLP
L12	$+0.0028$	$-0.0003$	$+0.0004$	$+0.020^{1}$	$+0.004$	$+0.001$
L15	$+0.0071^{1}$	$+0.0009^{1}$	$+0.0020$	$+0.023^{1}$	$+0.001$	$+0.006$
L18	$+0.0051^{1}$	$-0.0003$	$+0.0001$	$\mathbf{+0.051}^{4}$	$+0.002$	$+0.011^{1}$
L21	$+0.0031^{2}$	$-0.0001$	$-0.0002$	$\mathbf{+0.063}^{4}$	$+0.002$	$-0.001$
L24	$+0.0003$	$+0.0002^{1}$	$+0.0001$	$+0.030^{1}$	$+0.004^{1}$	$+0.002$

Appendix

Table 4: Language family, transport at matched readout distance (3–5 layers above the patch),

n=120

pairs, four donor pairings . Each cell is the mean over the four, and the superscript counts how many of them have an interval excluding zero; bold is all four. Both rank readouts are shown because they rank the layers differently:

R_{z}

makes L48 the largest cell, while

L_{z}

, which does not compress the top of the vocabulary, puts the peak at L39 and reduces attn.L48 to a twenty-fifth of attn.L39 . The window is where stability lives: every stream cell from L36 to L48 survives all four pairings, while below it only resid.L18 and resid.L21 do, and under

R_{z}

no shallow cell does at all. Two cells at L33 hold in only one or two pairings of four: a significantly negative mlp.L33 and a positive attn.L33 . Distances 3, 4 and 5 are averaged per instance before pooling. Benjamini–Hochberg at

q=0.05

is run inside each pairing over this sweep’s 39 cells and changes no cell here. Plotted in Figure 2 a (

L_{z}

patched group	donor $-$ distractor	accuracy after
resid.L42	$+0.4250\,[+0.2833,+0.5667]$	$0.450$
attention, L36 $+$ L39 $+$ L42	$-0.0333\,[-0.0833,+0.0167]$	$0.883$
MLP, L36 $+$ L39 $+$ L42	$-0.0250\,[-0.0583,+0.0000]$	$0.950$
attention $+$ MLP, L36–L42, all 14 components	$+0.0250\,[-0.0167,+0.0833]$	$0.950$

Appendix

Table 5: Joint component groups do not reproduce the stream’s behavioural effect. Clean accuracy

0.967

;

n=60

pairs, twelve query positions, one run and one donor pairing, so that the stream reference and the groups are measured on the same instances.

ablation	component	report	flexible	control
leave-one-out mean	resid.L39	$\mathbf{-0.600\,[-0.740,-0.460]}$	$\mathbf{-0.120\,[-0.220,-0.040]}$	$\mathbf{+0.080\,[+0.020,+0.160]}$
	attn.L39	$-0.020\,[-0.060,+0.000]$	$-0.020\,[-0.060,+0.000]$	$+0.020\,[-0.040,+0.080]$
	attn.L39.H15	$+0.000\,[+0.000,+0.000]$	$-0.060\,[-0.140,+0.000]$	$+0.020\,[+0.000,+0.060]$
	attn.L43	$-0.020\,[-0.060,+0.000]$	$-0.020\,[-0.060,+0.000]$	$+0.000\,[-0.080,+0.080]$
	mlp.L39	$+0.000\,[+0.000,+0.000]$	$+0.000\,[+0.000,+0.000]$	$-0.020\,[-0.060,+0.000]$
zero	resid.L39	$-0.440\,[-0.580,-0.300]$	$-0.640\,[-0.760,-0.500]$	$-0.500\,[-0.660,-0.340]$

Appendix

Table 6: Ablation cost in accuracy by arm:

\Delta

against the unablated run on the same

50

instances per arm, twelve query positions, clean accuracies report

0.980

, flexible

0.960

, control

0.920

. Under the leave-one-out mean ablation § 7 uses, resid.L39 is the only component with any interval excluding zero, and it is selective: it costs both arms that need

z

and helps the format-matched control. attn.L39.H15 points the same way in the flexible arm but does not clear zero there. The zero rows are the positive control for those nulls: zeroing mlp.L39 costs control alone, which is what shows the control arm is not sitting at a measurement floor. Zeroing resid.L39 damages every arm and so distinguishes nothing; it is printed to make that visible. Plotted, mean rows only, in Figure 7 a.

Figure 9: One semantic instance, all four conditions, on Qwen3.6-27B. (a) The rank of the gold concept ( Korean ) among the vocabulary at the query position, by layer, on a log scale with top-1 at the top; shading marks the transport window. (b) The top three tokens of the readout at the three layers where the arms diverge, with the gold token bold where it appears. The four arms ask, respectively, whether the passage contains a question mark, for the next sentence, what language the passage is written in, and which symbol corresponds to that language. Ranks in (b) count competing tokens and so start at $0$ , one below the $1$ -indexed rank used everywhere else in the paper. Token strings are shown escaped, so ’\n\n’ is a literal double newline and a leading space is visible.

LLM Interpretability

Large language model (LLM) interpretability studies how internal representations and computations produce outputs, and how they can be explained or controlled.

See all 778 papers

Oct 20, 2026

How Language Models Organize Task Information

Dense linear probes assess task readability from language model (LM) representations, but do not directly reveal how many features suffice for prediction. We measure readout sparsity with

k^*

, the smallest feature count found by a fixed search protocol that recovers 95% of dense-probe AUROC above chance. We evaluate 19 LMs from five labs on 14 core tasks. Ten base models give the nine reliably readable tasks similar feature counts and nearly the same order. This agreement extends to the selected features themselves, which on three tasks show cross-model response correspondence beyond class-mean differences and a label-correlation-matched null. Instruction-tuned deployment selectively reduces

k^*

, with both weights and input format contributing. In controlled synthetic experiments across 20 models, successfully learned targets admit single-feature readouts, whereas readable controls require tens of features. Reference-model

k^*

is associated with label requirements across eight students, following an empirical scaling of

N^*\propto(k^*)^{0.65}

. After continual learning, saved heads can fail while linear readability remains high, and recovery requires fewer labels than initial task learning under the tested procedures. Pruning reduces linear readability, with wider pre-pruning readouts generally associated with larger losses. These findings distinguish task readability from readout sparsity and establish their complementary value for predicting learning requirements and diagnosing LM changes.

Kaiyang Wan, Yuxia Wang

INSAIT, Sofia University “St. Kliment Ohridski” Sofia, Bulgaria

Language Model Probing Language Model Performance Prediction Representation Geometry in Language Models

Ask PDF

Oct 2, 2026

Emergent Structure in the Marginal Attention Space of Language Models

While representation similarity across independently trained language models is well-documented, how internal mechanics such as attention behave across models remains far less characterized. Inspired by this gap, we examine the structure of post-softmax attention weights by marginalizing over query positions, mapping them into a joint token-head "marginal attention space". Evaluating across 60+ diverse LLMs, we find that different properties emerge when reducing this space along its token and head axes. When reduced token-wise, marginal attention yields a text-intrinsic signal robustly conserved across models. To explain this property, we empirically connect marginal attention to the input-output Jacobian of the network, and prove theoretically that under a smoothness assumption, models with similar next-token distributions are guaranteed to have similar input-output Jacobian statistics. When reduced head-wise, it forms a model-private signature conserved across documents. Practically, this provides a natural way to estimate a per-head budget for key-value (KV) cache eviction, effectively decoupling model-specific budget allocation from text-intrinsic token scoring. On standard eviction benchmarks, a per-head budget precomputed offline on pretraining text, combined with a training-free token score, shows competitive performance with methods that recompute the budget on every document or train it per target. Code available at https://github.com/Flegyas/marginal-attention

Valentino Maiorca, Walter Nelson, Francesco Locatello

Institute of Science and Technology Austria (ISTA)

Attention Mechanisms Attention Head Analysis KV-Cache Eviction Language Model Fingerprinting

Ask PDF

Sep 28, 2026

A mechanistic study of language model introspection

Large language models (LLMs) can sometimes report perturbations to their internal activations---even when the input provides no evidence that an intervention occurred. How do models detect and localize such internal changes? We study this question using a controlled task that keeps the input text fixed. We either inject a concept vector into the hidden state at one of ten token positions or apply no intervention. The model is asked to identify the perturbed position or report that no intervention occurred. Across three model families, we identify two small groups of attention heads with distinct roles in introspective reporting. Middle-layer gate heads influence whether the model reports a change, while router heads in a later layer help select the position to report. Interventions on gate heads can suppress position reports even when router heads supply location information. We further examine why reporting accuracy varies across concepts. Concept vectors that are localized more accurately produce stronger attention-score and output responses in gate heads, which is associated with better alignment of the induced key and value changes in their QK and OV computations. Together, these findings identify attention-head mechanisms supporting introspective detection and localization.

Jiahong Zou, Xiangkun Sun, Lingkai Kong +1

Shandong University · Tsinghua University · Northeastern University +1

Attention Head Analysis LLM Interpretability Language Model Introspection Metacognition in Language Models

Ask PDF

Abstract

LLM Interpretability

Explore similar work

How Language Models Organize Task Information

Emergent Structure in the Marginal Attention Space of Language Models

A mechanistic study of language model introspection