Shared relations, specific choices: conditional sensitivity and competing pathways in language models

Kaixiang Shu

cs.CLNov 1, 2026

Shared relations, specific choices: conditional sensitivity and competing pathways in language models

Authors: Kaixiang Shu

Organizations: Independent Researcher

Abstract

How does a relational representation shared across queries drive a query-specific choice? Controlled entity-binding experiments with GPT-2 Small, Qwen2.5-1.5B and Llama-3.2-1B reveal a low-dimensional relational background that is predominantly shared across query roles and transfers to new names. Same-domain interactions persist when another attribute domain is queried. After removing this component, restoring independently trained relation directions brings output contrasts closer to their original values than sign-reversed or equal-norm random controls in all three models. Yet a transferable relational representation need not have a transferable functional sign. In Qwen2.5, fixing model weights, the training-mean relation vector and edit dose yields opposite selection effects across binary and lexical-mapping prompts with different answer vocabularies. A factorial contrast of downstream sensitivities across attribute-query combinations determines the shared direction's local signed selection gain. In a separate eight-condition protocol, mean local gain is positive in Qwen2.5 and negative in Llama3.2. Late-component clamps resolve opposing pathway contributions: in Llama3.2, negative transmission through selected attention-only paths is partially offset by positive transmission through paths traversing both selected attention and MLP sets. Source-coordinate analyses establish distributed factor coverage, hidden-receiver readouts reveal layout-dependent combination, and instance-geometry interventions separate response direction from amplitude. These findings show how conditional sensitivity and competing pathways give shared relational structure its effect on specific choices.

Figures & tables

Material and intervention contrasts
	GPT-2	Qwen2.5	Llama3.2
Blocks / channels	12 / 768	28 / 1,536	16 / 2,048
Query / KV heads	12 / 12	12 / 2	32 / 8
Normalization	LayerNorm	RMSNorm	RMSNorm
Position encoding	Absolute	RoPE	RoPE
Layer anchors	2 / 6 / 10	5 / 14 / 23	3 / 8 / 13

Table 1: Models and controlled comparisons. Layer anchors identify shallow/middle/deep attention outputs. Complete counterfactual groups are the sampling unit for controlled materials. Architecture, tokenization and material details are in Supplementary Table S1 .

Figure 1: Separating shared relations from their conditional use. A , The same facts support different targets when the queried entity changes. B , A token-by-depth schematic depicts internal representations and conditional readout: distributed factors, a middle-query relational background and signed reading at the prediction receiver. Colored routes illustrate factor contributions and opposing downstream gains; their widths and positions are schematic. Source-coordinate observations and instance-geometry tests provide complementary evidence about shared structure and its conditional use, through factor coverage, layout and response direction/amplitude. C , Shallow, middle and deep anchors in the three models. D , Each adjacent inverse edge uses local geometry to construct a reverse signal, then combines frequency-specific state estimates and forward-consistency corrections.

Figure 2: Shared task geometry supports a predominantly role-independent relational background. (A) Binding and syntax interaction fractions across the three measured depths. (B) Paired conditional-

A

energy coverage and centered pure-main-effect basis overlap, each compared with its corresponding natural reference. (C) Composition of

AB+ABR

energy: model-colored segments show

AB

, warm-grey segments show

ABR

, and boundary whiskers show the

AB

fraction interval. The right-hand plot gives coverage of held-out-name

AB

energy by independent training rank-1 and rank-16 bases. (D) Same-domain versus cross-domain interaction, followed by the length-matched lexical comparison. (E) Complete-group-centered three-value relation classification using the original binary interaction axis

I

, and held-out query displacement prediction in a fixed rank-16 basis. Classification and prediction are separate protocols. Intervals in C, D (right) and E are 95% source-group bootstrap intervals; the domain comparison uses 32 groups and the role, lexical and three-value comparisons use 64. Dotted reference lines denote 50% balanced accuracy and NMSE 1. Model colors are consistent throughout. Pale bands mark the middle checkpoint.

Pure-main-effect sharing in binding
Measurement	GPT-2	Qwen2.5	Llama3.2
Basis overlap (%)	58.06	59.22	32.53
Natural-pair overlap (%)	11.01	15.28	10.43
$A$ -on- $B$ excess (pp)	37.04 [33.97, 40.02]	20.74 [16.88, 24.71]	19.75 [17.54, 22.03]
$B$ -on- $A$ excess (pp)	36.08 [31.64, 40.59]	21.70 [18.04, 25.58]	16.37 [13.66, 19.09]
Role-decoupled background and new-name coverage

Table 2: Core geometry and generalization measurements. Brackets are 95% source-group bootstrap intervals. Energy and coverage entries are percentages; excess coverage is in percentage points. Relation classification uses the original binary axis after unlabeled centering within complete counterfactual groups. The three-value classifier and held-out-combination predictor are separate tests. Except for the 32-group domain experiment, confirmation/test sets contain 64 source groups.

Figure 3: Shared relational directions acquire conditional signed effects. (A) Deep query/prediction interaction fractions in four-corner binding materials. (B) Matching-context and common-rotation contrasts in response cosine. (C) Independent training-donor advantages over sign-reversed and equal-norm random refills. (D) Protocol-specific $AB$ derivatives, including fixed final-denominator controls. (E) A fixed training-mean direction across binary and mapped-lexeme materials, together with equal-norm random controls. The output pairs are orange/purple and red/blue, respectively; category and query-role coding are shared. (F) Four tangent-path sums with 95% intervals, classified by traversal of the selected attention ( $\mathcal{A}$ ) and MLP ( $\mathcal{M}$ ) sets; their sum equals native $\beta_{G}$ . (G) Static normalized writes and dynamic net transmission in separate coordinates, including fact-order cancellation at Qwen2.5 Attention 23. Error bars are 95% source-group bootstrap intervals where shown. The derivative, finite-dose and four-corner experiments retain their respective protocols.

Independent training-donor restoration, $n=64$
Quantity and protocol	GPT-2	Qwen2.5	Llama3.2
Donor $-$ reversed	$+0.338$ $[+0.257,+0.427]$	$+7.473$ $[+4.684,+10.356]$	$+12.181$ $[+9.845,+14.388]$
Donor $-$ random	$+0.209$ $[+0.143,+0.281]$	$+5.319$ $[+3.468,+7.316]$	$+9.099$ $[+6.946,+11.399]$
Group-specific projection: derivatives and finite edits
$\beta_{G}$ , analytic, $n=32$	$-0.018$ $[-0.141,+0.105]$	$+6.505$ $[+3.898,+9.111]$	$-7.273$ $[-9.393,-5.086]$
$\Delta G$ , dose $-0.5$ , $n=64$	$+0.021$ $[-0.018,+0.058]$	$-1.887$ $[-2.919,-0.867]$	$+3.678$ $[+2.892,+4.372]$

Table 3: Relational restoration, signed effects and tangent paths by protocol. Entries are estimates [95% source-group bootstrap intervals] in

10^{-3}

units. Donor advantage measures greater reduction in absolute distance from native

G

than the stated refill control. Projection rows use each group’s

AB

coefficient in the frozen training rank-16 basis; material-transfer rows use one fixed training-mean vector, with output pairs orange/purple (binary) and red/blue (mapped). GPT-2 material effects use an unedited baseline recomputed on the same CPU in batches of 16. The 32-group derivative subset belongs to the 64-group projection test set. Path sums use unrounded values; dashes mark comparisons not included in the late-clamp protocol.

Figure 4: Receiving arrangements and fact order organize functional competition. (A) Two position-level operations: shared or separated receivers for matched factor perturbations, and removal of a designated deep post-softmax source-value contribution at the prediction receiver. Source removal preserves the other attention probabilities. (B) Shared-minus-separated interaction RMS in all 18 task–model–depth panels; each cell reports its original value, with a logarithmic color scale. (C) Candidate-removal effects for early and late queried facts and their paired difference. (D) Baseline fact-order preference alongside late attention/MLP static writes to $G$ , in separate coordinates. Error bars indicate 95% source-group bootstrap intervals. The receiving comparison uses route-wise RMS averaging in GPT-2 and pooled-route RMS in Qwen2.5 and Llama3.2. Source removal and static write attribution retain their respective operations and scales.

Figure 5: Source-factor carrying and hidden-receiver complementarity. A , Middle-layer deletion effects for q7’s estimated source coordinates at positions

s>7

, subtracted from the fixed full-query inverse sum; intervals are 95% document-bootstrap intervals. Adjacent factor accuracies use Qwen2.5 prefix-source spans in principal and complementary statistical coordinates. B , Cross-source cosine at the same q14 and input node, before and after subtraction of an independently estimated inverse-state mean; intervals describe fixed mismatched pairs resampled by source document. C , Direct and attribute-composed relation accuracy in 512 matched GPT-2 binding groups. Composition is performed by the evaluator after separately decoding both attributes. D , Combining hidden-receiver features at q12–q15 yields opposite accuracy changes in two Llama3.2 layouts despite equal single-point accuracies. The adjacent fractions summarize 12 P0 panels per model: correction gains require positive 95% intervals for correction adaptation with the new single-point baseline fixed; all-controls success requires positive 95% intervals against the single-point, repeated-point and shuffled-position controls. These criteria are met in 30/36 and 16/36 panels, respectively.

Measurement	Model or view	Estimate [95% interval]
Future-component deletion at the middle layer
$\Delta$ NMSE	GPT-2	$+0.006913$ $[0.006611,0.007233]$
	Qwen2.5	$+0.001873$ $[0.001782,0.001970]$
	Llama3.2	$+0.001470$ $[0.001351,0.001583]$
Direct minus attribute-composed relation accuracy (pp)
GPT-2 binding	Hidden receivers	$+6.01$ $[3.86,8.11]$

Table 4: Source-coordinate observations and hidden-receiver readouts. Source-deletion effects subtract q7’s estimated coordinates at

s>7

from the same source’s fixed full-query inverse sum; changes in input-reference NMSE have 95% document-bootstrap intervals. Relation contrasts compare hidden receivers and inverse source spans, using paired percentage-point differences in 512 source groups. Adaptation counts concern direct hidden-receiver features at q12–q15 and require positive 95% group-bootstrap intervals; each model contributes 12 panels.

Figure 6: Instance geometry separates source direction, response direction and amplitude. A , Original endpoint reinjection: fixed-

J

(open) and joint-

H/J

(filled) cosines, with response-NMSE improvement on a separate axis. This protocol reports source cosines for Qwen2.5/Llama3.2; GPT-2 source cosines were not separately reported. B , Synthetic endpoint updates: syntax module effects and binding value-minus-routing NMSE. C , Llama3.2 layout contrasts and persistence of the aggregate sign before final normalization. D , Syntax prediction error decomposed into norm and projection terms: sand segments show

\Delta\rho^{2}

, sage segments show

-\Delta(2p)

, and diamonds mark the total

\Delta E

. The adjacent panel shows five-fold scalar adaptation with prediction directions fixed. Points are measured means; error bars are 95% source-group bootstrap intervals.

Observed difference	Comparison and evidence scope	Mechanistic interpretation
Relation gain: native $\beta_{G}$ is $+5.193$ in Qwen2.5 and $-7.425$ in Llama3.2 ( $\times 10^{-3}$ ).	Native edits and fixed final denominators; clean clamps of selected late components in 64 eight-condition groups.	Conditional sensitivities and the measured balance of selected tangent paths determine signed transfer.
Candidate removal, pooled $\Delta G$ : $-0.00218$ , $+0.08621$ , $-0.04516$ . The GPT-2 interval spans zero.	Post-softmax removal, paired within 64 groups: late-minus-early $\Delta G$ is $-0.3039$ , $-0.2068$ , $-0.3983$ , with negative intervals.	Candidate contributions support later facts relative to earlier ones across models, despite different pooled signs.
Binding value-minus-routing NMSE: $+0.0708$ , $+0.0101$ , $-0.0167$ .	Synthetic derivative endpoints at natural amplitude in 32 binding groups per model; direction/amplitude decomposition and 16 distinct groups per layout.	Llama3.2’s value advantage is predominantly directional and concentrated in P0; layout strata vary position, spacing and context together.
Syntax local-minus-global $H+J$ NMSE: $-0.0690$ , $-0.0436$ , $+0.0396$ .	Held-out prediction with model-specific fits; norm/projection decomposition and five-fold scalar fitting at fixed prediction directions.	Llama3.2 combines better direction with excess amplitude. Calibrated local-minus-global error is $-0.00175$ [ $-0.00597,+0.00213$ ], leaving the difference unresolved.

Table 5: Cross-model differences and the comparisons that resolve their measured dependencies. Triplets follow GPT-2, Qwen2.5 and Llama3.2 order. Relation derivatives are from the two-model 64-group late-clamp protocol; removal effects and NMSE differences retain their own units. The final interval is a 95% source-group bootstrap interval. Full estimates and intervals are in Tables 3 , , S27 and S28 .

Appendix figures & tables45 assets

Supplementary material from the paper’s appendix.

Appendix

A. Model configuration
Attribute	GPT-2	Qwen2.5	Llama3.2
Blocks / channels	12 / 768	28 / 1,536	16 / 2,048
Query / KV heads	12 / 12	12 / 2	32 / 8
Head dimension	64	128	64
Unique parameters	124,439,808	1,543,714,304	1,235,814,400
Normalization	LayerNorm	RMSNorm	RMSNorm

Appendix

Table S1: Architecture and material alignment. Panel A records the actual model configurations. Panel B records the counterfactual groups used for the principal controlled designs; training, development and test counts refer to complete source groups.

Figure S1: Controlled design and receiver alignment. A , Shallow, middle and deep anchors, positioned by layer index divided by total model depth. B , Related controlled designs cross two attributes (four conditions), attributes and query role (eight conditions), or color, shape, queried entity and queried domain (64 conditions). Each design uses its own complete source groups. C , The query and prediction slots differ between the 16-token and 15-token binding families. D , The role-balanced design uses disjoint training, development and test names; projections and interventions reuse training-frozen bases.

Symbol	Definition
$H_{v},R_{v}\in\mathbb{R}^{L\times C_{v}}$	Parent state and activation-weighted reverse signal.
$J_{v}$	Local derivative from parent state to retained child state.
$X^{\prime}_{v}=\mathcal{F}X_{v}$	Orthonormal transform along token position.
$S_{XY,v}(\omega)$	$\mathbb{E}[X^{\prime}_{v}(\omega)Y^{\prime}_{v}(\omega)^{*}]$ on calibration windows.
$\mathsf{G}_{v}(\omega)$	$S_{HR,v}[S_{RR,v}+\lambda_{G,v}I]^{-1}$ , an independent full-channel map for each bin.
$\widehat{H}_{v}^{(G)}$	$\mathcal{F}^{-1}[\mathsf{G}_{v}R^{\prime}_{v}]$ .

Appendix

Table S2: Source-coordinate operators. Stars denote complex-conjugate transpose in Fourier moments and the adjoint for local derivatives. The moments are uncentered because both regression stages have zero intercept.

	GPT-2	Qwen2.5	Llama3.2
Target depth / edges	2/6/10	5/14/23	3/8/13
Windows per depth	65,536	131,072	131,072
Additional budget	L6: 131,072	—	—
Map seeds	123/321/567	123/321/567	123/321/567
Window length / bins	16 / 9	16 / 9	16 / 9
Relative ridge, $\mathsf{G}/D$	0.01 / 0.01	0.01 / 0.01	0.01 / 0.01

Appendix

Table S3: Calibration configuration. Window counts apply separately at each target depth. The Qwen2.5/Llama3.2 seeds permute the same full calibration pool.

Comparison	Reference	Full adjacent-edge result
Frequency constraint (configuration)	Earlier GPT-2: shared full-channel maps across frequency bins; 4,096 calibration windows	Independent full-channel maps at each of the nine stored calibration frequencies
GPT-2 L6, 65,536 windows	Three retained edges: 0.5863	Six adjacent edges: 0.4718
GPT-2 L10, 65,536 windows	Historical sparse boundaries: 0.6874	Ten adjacent edges: 0.4563
Qwen2.5 L14, 131,072	Historical sparse boundaries: 0.8961	Fourteen adjacent edges: 0.8498
Llama3.2 L3, 131,072	Historical sparse boundaries: 0.8038	Three adjacent edges: 0.8592
Llama3.2 L8, 131,072	Historical sparse boundaries: 0.9013	Eight adjacent edges: 0.8522

Appendix

Table S4: Separating estimator constraints, topology and calibration budget. The quantitative rows compare source-reference NMSE on paired materials. Each row identifies the factor changed; these distances characterize the specified source coordinates.

Comparison	$n$	Effect	95% interval	$p$	Holm $p$
GPT-2: source comparisons
Source-response NMSE: random minus estimated direction	256	$-0.0192163$	$[-0.0257325,-0.0130746]$	$1$	$1$
Syntax nested readout: single minus combined CE	512	$0.0439348$	$[0.0356704,0.0518931]$	$0.0001$	$0.0006$
Syntax principal-direction selectivity minus random	128	$2.628\!\times\!10^{-5}$	$[1.606\!\times\!10^{-5},3.582\!\times\!10^{-5}]$	$0.0001$	$0.0006$
Syntax response prediction: global minus local NMSE	512	$0.0689937$	$[0.06365,0.0743455]$	$0.0001$	$0.0006$
Syntax shared minus split receiver RMS	128	$1.920\!\times\!10^{-6}$	$[1.510\!\times\!10^{-6},2.376\!\times\!10^{-6}]$	$0.0001$	$0.0006$

Appendix

Table S5: Original frozen comparisons. Effect sizes retain their original metrics and comparison order. Confidence intervals are 95% group/document bootstrap intervals. Holm correction is within each of the four named families.

n

counts complete source groups except the two natural-document deletion comparisons. The source-response and direction interventions use dose 0.25; statistical-direction selectivity uses a rank-16 fixed-norm direction.

Figure S2: Calibration and online transport have distinct roles. A , A target at depth

d

retains every adjacent boundary up to its attention output. B , Independent natural windows fit full-channel, frequency-specific first-stage maps and residual corrections. C , The selected feature, local derivative and repaired online child determine each reverse step. D , Calibration budgets and retained edge counts vary by model and depth; repeated map seeds are aggregated within the source sampling unit.

Figure S3: Conditional directions and interaction across the anchored depths. A–C, syntax and binding conditional cosines share one coordinate system within each model; filled circles with solid lines identify factor

A

, and open diamonds with dashed lines identify factor

B

. D–F, interaction-energy fractions. Blue denotes syntax and terracotta denotes binding in both rows. Columns correspond to GPT-2, Qwen2.5 and Llama3.2. Lines join the three measured checkpoints; whiskers show 95% source-group bootstrap intervals (

n=64

). Pale bands mark the middle checkpoint.

Model / task	Layer	Cosine $A$	Cosine $B$	$\eta_{I}$ (%)	$\rho_{I}$
GPT-2 / Syntax	2	0.985 [0.983, 0.987]	0.986 [0.981, 0.989]	0.38 [0.31, 0.46]	0.167 [0.154, 0.180]
GPT-2 / Syntax	6	0.976 [0.972, 0.980]	0.964 [0.960, 0.968]	0.71 [0.64, 0.78]	0.235 [0.223, 0.247]
GPT-2 / Syntax	10	0.852 [0.821, 0.878]	0.860 [0.839, 0.879]	3.49 [3.03, 4.02]	0.509 [0.479, 0.542]
GPT-2 / Binding	2	0.628 [0.589, 0.665]	0.634 [0.589, 0.676]	9.49 [8.68, 10.31]	0.829 [0.794, 0.862]
GPT-2 / Binding	6	-0.379 [-0.416, -0.339]	-0.260 [-0.315, -0.200]	48.91 [46.51, 51.27]	1.617 [1.587, 1.643]
GPT-2 / Binding	10	-0.615 [-0.677, -0.547]	-0.464 [-0.565, -0.362]	59.39 [55.40, 63.27]	1.731 [1.692, 1.772]

Appendix

Table S6: Anchored four-corner geometry at position

q14

. Cosines and

\rho_{I}

are dimensionless;

\eta_{I}

is a percentage of

A+B+I

energy. Brackets are 95% source-group bootstrap intervals (

n=64

Figure S4: Distinct controls for low-dimensional factor sharing. A, the balanced second-moment identity separates main effects from interaction mixing. B, paired conditional-

A

energy coverage in natural-hidden-state bases at ranks 16, 64 and 256 on a logarithmic rank axis; diamonds mark task rank-16 coverage. C, centered pure-main-effect basis overlap versus the mean overlap of independent natural-covariance sketch pairs. D, paired confirmation coverage excess in the opposite-factor task basis over the natural-random basis for factors

A

and

B

. Intervals in D resample 64 confirmation source groups. Coverage, overlap and excess are plotted on their respective axes.

A. Conditional- $A$ energy: task rank 16; natural ranks 16, 64, 256; task–natural rank-16 overlap
Model	Task / factor	Task / opposite	Natural	Rank 256 / excess	Basis overlap
Model	Task r16	Natural r16	Natural r64	Natural r256	Overlap
GPT-2	69.90	15.51	36.23	77.03	16.31
Qwen2.5	66.80	14.56	25.84	62.36	8.76
Llama3.2	63.47	6.14	21.44	53.98	7.02
B. Centered pure-main-effect and independent natural-pair overlaps

Appendix

Table S7: Natural-reference comparisons. Block A uses paired conditional-

A

binding energy and natural-hidden PCA. Block B uses centered pure main effects and natural-covariance sketch pairs. Block C gives confirmation coverage of each factor in the opposite-factor basis and in natural-random bases, with paired excess. Coverage and overlap are percentages; excess is in percentage points. Block C intervals use 2,000 bootstrap resamples of 64 confirmation source groups.

Figure S5: Role-independent interaction across layers, nodes and receivers. A–C, $AB/(AB+ABR)$ energy fraction at residual inputs, attention writes, MLP writes and residual outputs for the entity-A, entity-B, prefix, query and prediction positions in each model. All models use the same linear 0–100% color scale. White cells indicate undefined zero-relation-energy fractions; warm ivory is the zero point for a defined fraction. D, training-frozen rank-1 and rank-16 coverage of held-out-name $AB$ energy at the middle query receiver, with 95% source-group bootstrap intervals ( $n=64$ ). The associated source-data file retains all seven factor-energy shares for the complete layer/node/role grid.

Model	Layer / role	$A$	$B$	$R$	$AB$	$AR$	$BR$	$ABR$
A. Seven-factor energy shares (%)
GPT-2	L2 / query	3.71	3.39	91.12	0.50	0.63	0.59	0.05
GPT-2	L2 / pred	26.46	26.50	44.14	2.72	0.10	0.08	0.00
GPT-2	L6 / query	0.27	0.28	98.79	0.40	0.11	0.12	0.03
GPT-2	L6 / pred	16.68	15.92	19.83	40.15	2.06	2.19	3.17
GPT-2	L10 / query	4.70	4.99	68.97	18.99	0.57	0.57	1.21

Appendix

Table S8: Eight-condition energies and training-frozen transfer. Block A lists all seven non-mean energy shares at the anchored attention outputs; each row is normalized within group before averaging over 64 test groups. Block B uses the separate relation-energy denominator and test-group

AB

energy. Brackets denote 95% source-group intervals.

Figure S6: Domain selectivity, lexical/value transfer and fixed-basis prediction. A, queried and unqueried same-domain and cross-domain interaction under direct answers, domain-specific answer codes and a shared binary code. B, complete-group-centered classification with the original frozen binary relation axis in length-matched different-lexeme and three-value materials. C, query and prediction receiver NMSE for held-out combinations, comparing additive, conditional-linear, local-RBF and development-scaled conditional models. Whiskers are the saved 95% source-group bootstrap intervals. The 50% and NMSE-1 reference lines correspond to chance classification and zero-response prediction, respectively.

Direct attribute answers
Condition / measurement	GPT-2	Qwen2.5	Llama3.2
Queried same-domain $\eta_{I}$	40.82 [39.49, 42.24]	59.48 [58.57, 60.35]	57.64 [57.05, 58.22]
Unqueried same-domain $\eta_{I}$	42.30 [41.08, 43.51]	58.92 [57.92, 59.86]	58.37 [57.61, 59.10]
Cross-domain $\eta_{I}$	0.090 [0.085, 0.095]	1.076 [1.050, 1.104]	0.449 [0.433, 0.468]
Domain-specific answer codes
Queried same-domain $\eta_{I}$	5.97 [5.68, 6.26]	53.60 [52.41, 54.80]	56.10 [54.56, 57.60]

Appendix

Table S9: Middle-query domain, word-mapping and value controls. Energy entries are interaction fractions (%), cosine entries are dimensionless, and accuracy entries are balanced percentages after complete-group unlabeled mean centering. Brackets are 95% source-group intervals. Domain controls use 32 source groups; lexical and three-value controls use 64.

Model / receiver	Predictor / comparison	NMSE / paired difference	Dev. gain
GPT-2 / query	Additive	1.00546 [0.99923, 1.01170]	–
GPT-2 / query	Conditional linear	1.00318 [0.99718, 1.00919]	–
GPT-2 / query	Local RBF	0.99999 [0.99998, 0.99999]	–
GPT-2 / query	Scaled conditional	1.00000 [1.00000, 1.00000]	0.0000
GPT-2 / query	Zero response	1.00000 [1.00000, 1.00000]	–
GPT-2 / query	Scaled $-$ RBF	0.00001 [0.00001, 0.00002]	–

Appendix

Table S10: Response prediction within the common rank-16 basis. NMSE entries and paired contrasts have 95% source-group bootstrap intervals (

n=64

). A development gain is fitted once per model and receiver; it is an amplitude diagnostic rather than a new output basis.

Model	Receiver	$T$ versus $G$	$S_{\rm entity}$ versus $G$	$S_{\rm target}$ versus $G$
GPT-2	L2/pred	$-0.097$ $[-0.347,+0.184]$	$-0.250$ $[-0.459,-0.032]$	—
GPT-2	L6/pred	$-0.490$ $[-0.680,-0.245]$	$-0.219$ $[-0.450,+0.033]$	—
GPT-2	L10/pred	$+0.195$ $[-0.053,+0.436]$	$-0.227$ $[-0.455,+0.046]$	$+0.796$ $[+0.668,+0.875]$
Qwen2.5	L5/pred	$+0.001$ $[-0.271,+0.263]$	$+0.022$ $[-0.202,+0.232]$	—
Qwen2.5	L14/pred	$+0.144$ $[-0.101,+0.363]$	$-0.018$ $[-0.241,+0.225]$	—
Qwen2.5	L23/pred	$-0.437$ $[-0.630,-0.197]$	$-0.057$ $[-0.299,+0.188]$	$+0.011$ $[-0.265,+0.264]$

Appendix

Table S11: Prediction-receiver attention-output Spearman correlations with symmetric role-selection margin

G

. Each row uses 64 source groups; brackets are exploratory 95% source-group bootstrap intervals. Role alignment is shown at the deep anchor.

Context contrast at dose 0.25: estimates [95% CI]
Model/task	Own $-$ other	Own $-$ rotated	$n$
GPT-2/syntax	$-0.0029$ $[-0.0064,+0.0005]$	$+0.1484$ $[+0.0957,+0.2003]$	64
GPT-2/binding	$+0.0994$ $[+0.0254,+0.1735]$	$+0.0075$ $[-0.0515,+0.0672]$	64
Qwen2.5/syntax	$+0.0057$ $[-0.0023,+0.0134]$	$+0.0451$ $[+0.0165,+0.0738]$	64
Qwen2.5/binding	$+0.1071$ $[+0.0450,+0.1687]$	$+0.0581$ $[+0.0176,+0.1000]$	64
Llama3.2/syntax	$+0.0745$ $[+0.0550,+0.0963]$	$+0.1224$ $[+0.0955,+0.1502]$	64

Appendix

Table S12: Context-conditioned factor directions: functional contrasts and dose proportionality. The top block uses response-cosine differences; the bottom block records 64-group means in their stated units.

Figure S7: Receiver energy and downstream reading geometry. (A) Deep prediction-receiver role-aligned energy correlation with

G

in the 15-token eight-condition design. (B,C) Matching-context and rotation contrasts for binding (filled circles) and syntax (open squares). (D) Dose-scaling direction loss for matching (circles) and alternative (squares) contexts. Intervals in A–C resample source groups; D shows reported means. Panels B–D use context-conditioned factor directions in the 16-token four-corner materials; dose norm ratios are normalized by the expected factor of 2.5.

Model	$\beta_{G}$ , analytic ( $n=32$ )	$\Delta G$ , dose $-0.5$ ( $n=64$ )	$\Delta C$ , dose $-0.5$ ( $n=64$ )
GPT-2	$-0.0175$ $[-0.1409,+0.1051]$	$+0.021$ $[-0.018,+0.058]$	$-0.432$ $[-0.540,-0.321]$
Qwen2.5	$+6.505$ $[+3.898,+9.111]$	$-1.887$ $[-2.919,-0.867]$	$+5.546$ $[+2.988,+8.114]$
Llama3.2	$-7.273$ $[-9.393,-5.086]$	$+3.678$ $[+2.892,+4.372]$	$+7.092$ $[+6.280,+7.928]$

Appendix

Table S13: Component intervention outcomes, with analytic and finite-dose protocols separated. All displayed numbers are multiplied by

10^{3}

; brackets are 95% source-group bootstrap intervals. The finite intervention subtracts half of the rank-16

AB

projection.

Donor recovery advantage; $n=64$
Model	Protocol	Donor $-$ reversed	Donor $-$ random
GPT-2	recovery	$+0.338$ $[+0.257,+0.427]$	$+0.209$ $[+0.143,+0.281]$
Qwen2.5	recovery	$+7.473$ $[+4.684,+10.356]$	$+5.319$ $[+3.468,+7.316]$
Llama3.2	recovery	$+12.181$ $[+9.845,+14.388]$	$+9.099$ $[+6.946,+11.399]$
Fixed mean-axis edit, dose $+0.5$ : trained-axis and random-direction $\Delta G$
Model	Material	Trained $AB$ axis	Equal-norm random

Appendix

Table S14: Cross-group donor recovery and fixed-axis material transfer. All numerical effects are in

10^{-3}

units with 95% source-group bootstrap intervals. Both blocks use 64 source groups. Fixed-axis transfer holds the injection vector and dose constant across binary orange/purple and mapped red/blue output pairs, with common category and query-role coding.

Model	Contribution clamped	Residual $\Delta G$	Residual $\Delta C$
GPT-2	Query	$+0.006$ $[-0.053,+0.064]$	$-0.257$ $[-0.427,-0.097]$
GPT-2	Candidate	$+0.027$ $[-0.008,+0.064]$	$-0.377$ $[-0.509,-0.249]$
GPT-2	Both	$+0.031$ $[-0.005,+0.067]$	$-0.292$ $[-0.430,-0.161]$
Qwen2.5	Query	$-2.477$ $[-3.785,-1.145]$	$+5.431$ $[+2.170,+8.711]$
Qwen2.5	Candidate	$-2.385$ $[-3.559,-1.182]$	$+5.384$ $[+2.529,+8.424]$
Qwen2.5	Both	$-2.288$ $[-3.460,-1.109]$	$+4.967$ $[+2.108,+7.936]$

Appendix

Table S15: Residual

AB

half-removal effect after direct contributions are clamped to the clean baseline (

\times 10^{3}

;

n=32

paired source groups). These are finite edited-minus-clean responses, not derivatives. Brackets are 95% source-group bootstrap intervals.

Figure S8: Relational transfer under distinct protocols. (A) Donor recovery against reversed (filled) and random (open) refills. (B) Analytic derivatives in 32 groups. (C) Half-projection removal in 64 groups:

\Delta G

(filled) and

\Delta C

(open). (D) Fixed training-mean axis (filled) and equal-norm random (open) edits across binary and mapped materials, using orange/purple and red/blue output pairs with the same category and query-role coding. (E) Residual half-removal effects under direct query, candidate or joint clean clamps in 32 groups. Error bars retain the original 95% source-group intervals.

Clamped operation	$\beta_{G}$	$\Delta\beta_{G}$	$\Delta\beta_{C}$	$p_{\rm Holm}$
Qwen2.5: local derivatives at the common clean state
none	$+5.193$ $[+3.119,+7.281]$	$+0.000$ $[+0.000,+0.000]$	$+0.000$ $[+0.000,+0.000]$	—
attn5	$+5.193$ $[+3.119,+7.281]$	$+0.000$ $[+0.000,+0.000]$	$+0.000$ $[+0.000,+0.000]$	—
attn13	$+5.193$ $[+3.119,+7.281]$	$+0.000$ $[+0.000,+0.000]$	$+0.000$ $[+0.000,+0.000]$	—
attn21	$+2.104$ $[+0.396,+3.815]$	$-3.089$ $[-3.907,-2.289]$	$-2.062$ $[-2.717,-1.388]$	0.00024
attn23	$+5.173$ $[+3.269,+7.112]$	$-0.019$ $[-0.412,+0.336]$	$+1.428$ $[+0.710,+2.207]$	0.92256

Appendix

Table S16: All fixed operations in the 64-group late-clamp extension. Derivatives and shifts are in

10^{-3}

units; brackets are 95% source-group bootstrap intervals. Holm values apply to the single-node family. Pre-injection controls and the native row have exactly zero shifts.

Forward-graph gains
Graph / path class	Qwen2.5	Llama3.2
Native	$+5.193$ $[+3.119,+7.281]$	$-7.425$ $[-8.930,-5.902]$
Attention clamp	$+1.625$ $[+0.773,+2.481]$	$-0.750$ $[-1.033,-0.486]$
MLP clamp	$+8.365$ $[+5.161,+11.712]$	$-13.369$ $[-15.727,-11.035]$
Joint clamp	$+4.165$ $[+2.809,+5.581]$	$-0.685$ $[-0.871,-0.509]$
Additive tangent-path totals

Appendix

Table S17: Joint graph derivatives and their additive tangent-path re-expression (

10^{-3}

units;

n=64

per model). Brackets are paired source-group 95% intervals. Native and frozen-final-denominator path totals use the same selected sets and source groups.

Figure S9: Late-component transfer. (A) Twelve single-node clamp shifts on one scale,

\Delta\beta_{G}=\beta_{\rm clamp}-\beta_{0}=-\tau

. (B) Native and jointly clamped graph gains. (C) Four additive tangent-path totals under native (filled circles) and frozen-final-denominator (open squares) readout. Points and 95% source-group intervals use all 64 groups per model; the selected attention and MLP sets define the path categories.

Static write (raw margin) and net dynamic transmission ( $10^{-3}$ )
Component	Static write	Net transmission
Qwen2.5 mlp28	$+0.286$ $[+0.204,+0.378]$	$-5.307$ $[-7.257,-3.483]$
Llama3.2 attn13	$-0.308$ $[-0.349,-0.271]$	$+2.373$ $[+1.925,+2.820]$
Qwen2.5 Attention 23 net transmission ( $10^{-3}$ )
Stratum	Net transmission
early	$+2.310$ $[+1.561,+3.088]$

Appendix

Table S18: Static/dynamic and direction controls (

n=64

). Units are stated separately for each block. Each interval resamples complete source groups.

Derivative estimates and intervals ( $10^{-3}$ )
Quantity	Qwen2.5	Llama3.2
Native $\beta_{G}$	$+5.193$ $[+3.119,+7.281]$	$-7.425$ $[-8.930,-5.902]$
Frozen-denominator $\beta_{G}$	$+6.065$ $[+3.948,+8.246]$	$-7.655$ $[-9.138,-6.141]$
Native $\beta_{C}$	$-10.371$ $[-15.498,-5.346]$	$-14.262$ $[-15.875,-12.637]$
Maximum absolute numerical differences (unscaled original units)
Clean-clamp margin	$0$	$0$

Appendix

Table S19: Final-denominator and numerical checks over all 64 source groups. The derivative block gives estimates and 95% intervals; the audit block gives maximum absolute discrepancies.

Figure S10: Direction, attribution and scale controls. (A) Absolute

AB

clamp shift minus the within-group random magnitude reference. (B) Static write versus net dynamic transmission

\tau

for two components. (C) Qwen2.5 Attention 23 early/late cancellation. (D) Native and frozen-final-denominator

\beta_{G}

. Error bars are 95% source-group bootstrap intervals. Static writes and dynamic derivatives use their stated coordinates.

Model	Layer	Task	Shared RMS	Separate RMS	Difference [95% CI]
GPT-2	2	syntax	14.6733	1.0790	$+13.5943$ $[+11.0088,+16.3434]$
GPT-2	2	binding	4.2288	0.9165	$+3.3123$ $[+2.8922,+3.7573]$
GPT-2	6	syntax	2.1464	0.2265	$+1.9200$ $[+1.5096,+2.3765]$
GPT-2	6	binding	0.3662	0.0885	$+0.2777$ $[+0.2281,+0.3295]$
GPT-2	10	syntax	0.5386	0.0256	$+0.5130$ $[+0.4416,+0.5958]$
GPT-2	10	binding	0.2307	0.0122	$+0.2185$ $[+0.1691,+0.2797]$

Appendix

Table S20: Shared-versus-separated receiver interaction. Values are in

10^{-6}

margin units; contrast intervals use source-group bootstrap. GPT-2 uses 128 source groups per task and averages route-wise RMS; Qwen2.5/Llama3.2 use 64 groups per task and pool routes before computing RMS. Qwen2.5/Llama3.2 contrast intervals use the saved geometry/function reanalysis; shared and separate means retain the route summaries.

Figure S11: Complete receiving-point comparisons. (A) Shared (filled) and separated (open) mean interaction RMS for all 18 model–task–depth comparisons. (B) Row-matched shared-minus-separated differences and 95% source-group intervals; every lower endpoint is positive. Both axes are logarithmic. GPT-2 uses 128 groups per task with route-wise RMS averaging; Qwen2.5 and Llama3.2 use 64 groups per task with pooled-route RMS. The comparisons are paired within each model.

Model / contrast	First quantity	Second quantity
Source removal: edited minus clean output (raw margin units)
Model / operation	$\Delta G$	$\Delta C$
GPT-2 / candidate_zero	$-0.00218$ $[-0.00888,+0.00426]$	$+0.00749$ $[-0.02166,+0.03550]$
GPT-2 / query_zero	$+0.00135$ $[+0.00047,+0.00223]$	$-0.03767$ $[-0.04024,-0.03515]$
Qwen2.5 / candidate_zero	$+0.08621$ $[+0.07220,+0.10134]$	$+0.32187$ $[+0.25856,+0.39050]$
Qwen2.5 / query_zero	$-0.01671$ $[-0.02053,-0.01307]$	$+0.00516$ $[-0.00092,+0.01111]$

Appendix

Table S21: Source-position interventions, fact-order contrasts and baseline write balance. All estimates use 64 paired source groups, with 95% bootstrap intervals. Single-source removal intervals retain the original projection study; joint-removal, order and write intervals use the saved follow-up analysis of the same source groups.

Figure S12: Source and order effects. (A,B) Candidate (filled circles), query (open squares) and joint (diamonds) removal in

G

and

C

. (C) Candidate removal for early (filled) and late (open) facts. (D) Paired late-minus-early removal effect. (E) Early/late unedited

G

. (F) Static late attention (filled) and MLP (open) sums. All error bars are 95% source-group intervals. The source, order and static comparisons retain their separate operations.

Model / depth or task	Measurement	Estimate [95% interval]
GPT-2 L2	Future deletion	$0.002995$ $[0.002907,0.003086]$
GPT-2 L6	Future deletion	$0.006913$ $[0.006611,0.007233]$
GPT-2 L10	Future deletion	$0.006089$ $[0.005672,0.006495]$
Qwen2.5 L5	Future deletion	$0.001971$ $[0.001853,0.002097]$
Qwen2.5 L14	Future deletion	$0.001873$ $[0.001782,0.001970]$
Qwen2.5 L23	Future deletion	$0.001728$ $[0.001622,0.001845]$

Appendix

Table S22: Future-deletion distances and factor readouts in statistical source projections. Deletion subtracts the complete q7 estimate at later source positions from the fixed full-query inverse sum, using the main calibration budgets; projected accuracies use prefix source spans, the original controlled training/development split and the full confirmation set. Brackets are 95% source-group or document-bootstrap intervals.

Figure S13: Source-coordinate carrying and transport. A , Deletion of q7’s estimated source coordinates at $s>7$ from the fixed full-query inverse sum, across three depths per model; intervals are document-bootstrap 95% intervals. B , Factor accuracies in prefix source spans, using $P_{16}/Q_{16}$ and model-specific higher ranks (32 in Qwen2.5; 43 in Llama3.2). S/B denote syntax/binding. C , Raw and independently centered cross-source cosine at every inverse node, with the query fixed at q14. Node 0 is the input; the largest node index is the selected hidden endpoint. D , Direct and evaluator-composed relation accuracy in matched GPT-2 binding groups. E , Llama3.2 L3–L8 same-window centered cosine, followed by controlled binding-factor difference cosines at q14. Natural vectors and controlled differences use distinct paired comparisons. All intervals are inherited from the corresponding frozen summaries.

Model / depth	Query / node	Raw cosine	Centered cosine
GPT-2 L6	all, node 0	$0.5372$ $[0.5264,0.5482]$	$0.0364$ $[0.0158,0.0577]$
GPT-2 L6	all, node 1	$0.9327$ $[0.9290,0.9364]$	$0.0465$ $[-0.0150,0.1055]$
GPT-2 L6	all, node 2	$0.9603$ $[0.9566,0.9638]$	$0.0606$ $[-0.0288,0.1434]$
GPT-2 L6	all, node 3	$0.9810$ $[0.9790,0.9828]$	$0.0771$ $[-0.0422,0.1945]$
GPT-2 L6	all, node 4	$0.9695$ $[0.9662,0.9727]$	$0.0683$ $[-0.0338,0.1692]$
GPT-2 L6	all, node 5	$0.9535$ $[0.9478,0.9588]$	$0.0721$ $[-0.0202,0.1642]$

Appendix

Table S23: Natural background across the complete adjacent inverse chain. Raw and centered cross-source cosines use fixed mismatched windows resampled by source document; centering uses independent documents and the same query/node. The last two rows instead pair the same windows across depth, explicitly separating that comparison from cross-source alignment.

Measurement	Two conditions or value	Contrast / interpretation
Relation, hidden receiver	84.62 / 78.61	$6.01$ $[3.86,8.11]$
Relation, recipient source spans	87.06 / 100.00	$-12.94$ $[-14.65,-11.23]$
Relation, future source spans	68.21 / 80.81	$-12.60$ $[-14.36,-10.84]$
GPT-2 suffix cosine	0.4900 / 0.8812	Before / after alignment
Qwen2.5 permutation cosine gain	0.5478 / 0.7102	$0.1624$ $[0.1341,0.1920]$
Llama3.2 permutation cosine gain	0.8624 / 0.9366	$0.0741$ $[0.0467,0.1022]$

Appendix

Table S24: Relations, source order and complementary readout rules. The first three rows give GPT-2 direct/composed relation accuracies (percent) in 512 matched groups and their paired difference (percentage points). Composition follows separate attribute decoding by the evaluator. The last two rows reuse the same 32 P0 prefix representations with a frozen factor readout and a four-way cosine retrieval rule.

Figure S14: Complete layout-conditioned combination of hidden receivers at q12–q15. Each cell is cross-entropy of the fixed new single-point baseline minus that of the adapted combination. Rows separate P0, P1 and pooled evaluation views; columns separate models. The common color bar uses a symmetric logarithmic scale centered at zero, with a linear interval of $[-0.05,0.05]$ . Labels give rounded means, using scientific notation for small nonzero values. Diamonds mark positive 95% group-bootstrap intervals against the single-point, repeated-point and shuffled-extra-position controls. The panel count is 36 per evaluation view.

Model / depth	Task / view	Adapted increment	Correction adaptation	All controls
GPT-2 L2	SA / P0	$-0.039$ $[-0.069,-0.010]$	$0.410$ $[0.374,0.448]$	No
GPT-2 L6	SA / P0	$0.017$ $[0.008,0.025]$	$0.475$ $[0.408,0.545]$	Yes
GPT-2 L10	SA / P0	$0.008$ $[0.006,0.011]$	$0.001$ $[-0.005,0.007]$	Yes
GPT-2 L2	SB / P0	$-0.012$ $[-0.015,-0.008]$	$2.434$ $[2.242,2.619]$	No
GPT-2 L6	SB / P0	$0.030$ $[0.011,0.049]$	$0.377$ $[0.294,0.458]$	Yes
GPT-2 L10	SB / P0	$0.034$ $[0.018,0.051]$	$0.477$ $[0.381,0.566]$	Yes

Appendix

Table S25: Hidden-receiver combination at q12–q15: all 36 panels in each of the P0, P1 and pooled evaluation views. Adapted increment is CE(new single point) minus CE(new-baseline/new-correction combination). Correction adaptation holds the new single point fixed and compares old versus new correction training. All-controls requires positive 95% intervals against the single point, repeated-point and shuffled-position controls. Small nonzero estimates and interval bounds retain additional digits. Complete template/position/vocabulary strata accompany the source data.

Model / task / budget	Update / comparison	Source cosine or metric	Response cosine / effect
GPT-2	Both tasks / natural	Fixed minus joint NMSE	$0.2236$ $[0.1766,0.2609]$
Qwen2.5	Both tasks / natural	Fixed minus joint NMSE	$0.0570$ $[0.0491,0.0651]$
Llama3.2	Both tasks / natural	Fixed minus joint NMSE	$-0.0009$ $[-0.0036,0.0017]$
GPT-2 B input-fixed	Fixed J	Not separately reported	$-0.4736$ $[-0.5571,-0.3955]$
GPT-2 B nat.	Fixed J	Not separately reported	$-0.4748$ $[-0.5579,-0.3969]$
GPT-2 B input-fixed	Joint H/J	Not separately reported	$0.4779$ $[0.4131,0.5375]$

Appendix

Table S26: Original joint-endpoint reinjection. The first three rows report aggregate fixed-minus-joint response NMSE. Subsequent rows separate source and descendant-response cosines by task (S, syntax; B, binding), budget and update. All injections retain only the changed source span and use dose 0.25. Natural amplitude preserves the masked direction. Qwen2.5/Llama3.2 equal budgets match its norm to

\lVert\Delta x\rVert

; GPT-2 input-fixed budgets instead set it to

0.25\lVert M_{\mathrm{source}}x\rVert

before the dose. Brackets are source-group 95% intervals.

Figure S15: Endpoint reinjection, synthetic module updates and layout. A , Original reinjection source cosine, response cosine and response NMSE in separate axes; open/filled markers denote fixed geometry/joint updates. Natural and equal-source-norm budgets are separately labeled; the NMSE axis uses a symmetric logarithmic scale with a linear region up to 1. B , Synthetic endpoint fixed-minus-arm NMSE by task and module, followed by layout-specific binding value-minus-routing contrasts. S/B denote syntax/binding. C , Direction/amplitude decomposition and final-normalization coordinates of the binding contrast. Natural forward and fixed-denominator measurements are separate conditions. Intervals are 95% source-group bootstrap intervals.

Model / task	Budget / update	Response NMSE	Source cosine	Response cosine
GPT-2 B	equal / Attention	$0.670$ $[0.614,0.734]$	$0.154$ $[0.145,0.164]$	$0.697$ $[0.625,0.764]$
GPT-2 B	equal / Fixed J	$1.321$ $[1.227,1.417]$	$0.007$ $[0.003,0.011]$	$-0.366$ $[-0.491,-0.233]$
GPT-2 B	equal / Joint H/J	$0.546$ $[0.507,0.586]$	$0.329$ $[0.320,0.336]$	$0.816$ $[0.765,0.865]$
GPT-2 B	equal / Norm/MLP	$0.837$ $[0.757,0.926]$	$0.237$ $[0.229,0.246]$	$0.513$ $[0.418,0.607]$
GPT-2 B	equal / Routing	$0.762$ $[0.664,0.876]$	$0.176$ $[0.166,0.186]$	$0.589$ $[0.498,0.679]$
GPT-2 B	equal / Routing + Norm	$0.607$ $[0.557,0.657]$	$0.329$ $[0.321,0.337]$	$0.734$ $[0.671,0.795]$

Appendix

Table S27: Synthetic endpoint updates under both source-amplitude budgets. Each model/task contains 32 source groups; four flips are averaged before source-group bootstrap. Source cosine, descendant cosine and descendant NMSE are distinct estimands. Attention, normalization/MLP and routing/value terms update complete derivative endpoints. Local-edge variants (eight groups per task), layout contrasts and final-normalization comparisons are retained in the associated source data and text.

Model / task / predictor	Local $-$ global	Global NMSE	Local NMSE	Random neighbors
GPT-2 Syntax H	$-0.1142$ $[-0.1191,-0.1091]$	$0.8601$ $[0.8320,0.8870]$	$0.7459$ $[0.7219,0.7697]$	$0.8550$ $[0.8279,0.8812]$
GPT-2 Syntax J	$-0.0929$ $[-0.0998,-0.0859]$	$0.8681$ $[0.8397,0.8958]$	$0.7751$ $[0.7425,0.8078]$	$0.8711$ $[0.8428,0.8987]$
GPT-2 Syntax H+J	$-0.0690$ $[-0.0743,-0.0637]$	$0.7869$ $[0.7648,0.8076]$	$0.7179$ $[0.6925,0.7433]$	$0.8627$ $[0.8351,0.8896]$
GPT-2 Binding H	$0.0000$ $[-0.0002,0.0002]$	$0.8758$ $[0.8717,0.8798]$	$0.8758$ $[0.8718,0.8798]$	$0.8803$ $[0.8757,0.8850]$
GPT-2 Binding J	$0.0091$ $[0.0038,0.0144]$	$0.4865$ $[0.4655,0.5082]$	$0.4956$ $[0.4715,0.5201]$	$0.8803$ $[0.8757,0.8849]$
GPT-2 Binding H+J	$-0.0272$ $[-0.0319,-0.0223]$	$0.5469$ $[0.5300,0.5645]$	$0.5197$ $[0.4996,0.5400]$	$0.8803$ $[0.8757,0.8849]$

Appendix

Table S28: Frozen predictor comparisons, exact response-error decomposition and scale controls. The first block reports all H/J/H+J neighborhoods and their global/local/random-neighborhood errors. The second separates squared norm, twice the response projection and cosine. The final block reports the total error after both arms receive one nonnegative scalar fitted on other source groups; bootstrap refits that scalar. These scale controls supplement the frozen original comparison.

Descriptor / measurement	GPT-2	Qwen2.5	Llama3.2
Architecture
Normalization	LayerNorm	RMSNorm	RMSNorm
Feed-forward module	GELU MLP	SwiGLU	SwiGLU
Position encoding	Learned absolute	RoPE	RoPE
Attention grouping	Multi-head	Grouped-query	Grouped-query
Explicit Q/K/V bias	Present	Present	Absent

Appendix

Table S29: Architecture descriptors and measured coordinates. Final-state entries are descriptive means over 512 sentences in 64 eight-condition source groups. The cosine compares the exact candidate-margin gradient with the candidate axis under a fixed final-normalization scale; LayerNorm centering is retained. Each Q/K/V cell gives bias norm / mean relative size, computed per state using the bias-free projection of the native normalized input (1,024 query/prediction states per layer). Mixed-precision checks use FP64 weights and fixed coefficients with native FP32 normalization, softmax and rotary operations.

Figure S16: Local response prediction separates direction from scale. A , Frozen H/J/H+J neighborhood comparisons across all models and tasks. Negative local-minus-global NMSE favors the local predictor. B , Local-minus-global changes in squared norm, twice the projection onto the native response, total NMSE and cosine. The error identity is evaluated before aggregation. C , Original and independently scalar-adapted local-minus-global error, with prediction directions fixed. Each arm receives one nonnegative scalar fitted outside the held-out source-group fold. Horizontal intervals are 95% source-group bootstrap intervals with the scalars refitted in each resample.

LLM Interpretability

Large language model (LLM) interpretability studies how internal representations and computations produce outputs, and how they can be explained or controlled.

See all 791 papers

Sep 10, 2026

From Parameters to Answers: How LLMs Retrieve and Use Their Internal Knowledge

How does a language model's dependence on query-routing information and target knowledge change as it answers a question? We study this question through layerwise interventions on the hidden state at the end of the question. Across Qwen, Llama, and Gemma, we compare country-continent questions with noun, adjective, and code answers while keeping several fitted measurements distinct. A pair-conditioned request direction describes which country is queried in natural single-country questions; a global request direction describes first- versus second-country requests in paired questions; separate selection candidates test control among contents already available in the hidden state. A diagnostic reanalysis of frozen Qwen natural-question states shows that the pair-conditioned direction grows stronger before interventions on it begin to alter later fitted knowledge, with this causal window opening while answer-supporting content is still forming. The paired three-model trajectories are not uniform: Gemma shows a partially overlapping mid-layer routing-content profile, whereas Llama has no sustained routing-effect window under the same gates. In the paired protocol, dependence on the global request direction decreases from fixed earlier to later layer sets while dependence on fitted content persists. A matched Qwen comparison shows that the pair-conditioned direction retains a late effect, so this operational handoff concerns the global fitted direction rather than all request information. These results separate early readability, natural strength, causal steering, and later content dependence.

Wenkang Wei, Yuan Fang, Renhe Jiang +2

University of Science and Technology of China Hefei, China · Singapore Management University Singapore, Singapore · The University of Tokyo Tokyo, Japan +1

Knowledge Retrieval in Language Models LLM Interpretability Factual Knowledge in Language Models Causal Interventions in Language Models Language Model Probing

Ask PDF

Aug 4, 2026

Test, then Route: How Language Models Execute In-Context Conditional Rules Across Models and Languages

When a language model follows an in-context conditional rule such as "if P(x) then A else B," does it assemble a runtime circuit with one module that tests the predicate and another that routes the answer? We probe this with activation patching under a four-donor design whose two swapped-rule donors make the condition and the answer word disagree, so each layer reveals which of the two it carries. Across three open models from two families and six languages sharing one fixed item bank, a mid-stack residual band carries the predicate's truth value: patching it reroutes the answer with predicate-outcome flip near 1.0 and mapping flip near 0.0, meeting a strict pre-specified isolation criterion in 17 of 18 cells, and the same localization holds across five predicate families. The router shows the opposite profile. A learned subspace flips A and B near-perfectly within the trained pair yet transfers to a new pair at approximately 0 in every model, while in Gemma-3-4B (the only model probed cross-lingually) it transfers at approximately 0.98 to the same pair in other languages. Under every probe we ran, the router direction is token-bound and non-transferable (largely answer-readout in Gemma, pair-specific in Qwen) rather than an abstract routing module. Test is modular; under these probes, route is not.

Luxshan Thavarasa, Sivasuthan Sukumar

Independent Researcher, Colombo, Sri Lanka · Department of Computer Science and Engineering, University of Moratuwa, Sri Lanka

In-Context Learning Circuits in Language Models

Ask PDF

Jun 11, 2026

Localizing Anchoring Pathways in Language Models

Irrelevant numbers in a prompt can shift language model judgments, producing anchoring effects in numerical reasoning. We study where this anchor-sensitive signal is carried inside language models using a controlled multiple-choice setup with shared answer options. We define a logit-difference metric comparing the correct answer option with the answer option corresponding to the anchor, and validate that it tracks behavioral anchoring. Using attribution-based circuit localization on 7B--8B Qwen and Llama base and instruction-tuned models, we find that edge-level methods recover this signal more faithfully than node-level methods. Low- and high-anchor circuits transfer strongly within a model, suggesting shared pathway structure across anchor direction. However, sparse transfer across base and instruction-tuned variants is less reliable, indicating that post-training changes which pathways matter most. Overall, our results provide a mechanistic account of how anchoring-related decision signals are carried inside language models.

Hillary N. Owusu, Sarah Wiegreffe, Naomi H. Feldman

University of Maryland, College Park

Numerical Reasoning in Language Models LLM Interpretability Circuits in Language Models

Ask PDF

Abstract

LLM Interpretability

Explore similar work

From Parameters to Answers: How LLMs Retrieve and Use Their Internal Knowledge

Test, then Route: How Language Models Execute In-Context Conditional Rules Across Models and Languages

Localizing Anchoring Pathways in Language Models