Activation-Space Order-Swap Geometry: A Site-Asymmetry Audit

Anqi Peter Li

cs.LGAug 26, 2026

Activation-Space Order-Swap Geometry: A Site-Asymmetry Audit

Authors: Anqi Peter Li

Organizations: Substrate Labs

Abstract

Order-dependent activation statistics are often interpreted as evidence of interaction, but that interpretation can be confounded by where interventions enter the network. We introduce a no-fit site-asymmetry audit. For a twice-differentiable readout, the open-path order-swap decomposes into a canonical additive response measured by single interventions and an antisymmetrized second difference free of first-order and pure self-curvature terms to second order. Across six open-weight language-model families, the single-intervention baseline explains 84.3-97.7 percent of the bracket norm (mean 93.7 percent), while the no-interaction self-curvature term is 1.8-5.2 times larger than the corrected residual in the two families with the plus/minus injection split. The corrected residual clears a generic-interaction null in three of six families under a confound-free prompt split and two of six after configuration robustness. A known-positive surrogate recovers planted mixed interaction, while a matched site-separation test changes the baseline share and a random architecture reproduces the first-order regime. The same estimator transfers to released non-language references: trained residual fractions fall below a fixed Gaussian-direction null in 11/12 contrasts (5/6 ViT-B/16, 6/6 ResNet-50), a portability check rather than pooled evidence. The contribution is a reusable measurement criterion: run the single-intervention baseline before reading an order-swap vector as interaction or geometric structure; if it explains the vector, form the second difference instead. All claims are scoped to activation-space interventions at distinct sites; we do not claim that representation geometry is globally Abelian.

Figures & tables

Figure 1: The two central results. (a) Norm of the order-swap bracket against injection coefficient

\alpha

, normalised at

\alpha=1

. All six families track

\alpha^{1}

(dashed), the scaling of a term with no interaction, rather than

\alpha^{2}

(dotted); fitted exponents

\gamma\in[0.886,1.043]

. (b) The unsigned shared-argument ratio for the corrected residual under the confound-free prompt-split design (solid), against the generic-interaction null (dashed red), a calibrated trait-varying first-order arm, and an uncalibrated constant-Jacobian floor (the linear slack, which has no free parameter to calibrate; Appendix D ). Stars mark the

3

6

families that clear the null — Llama

1.599

, OLMo

2.059

, Qwen

1.642

against the

1.324

bar; DeepSeek, Gemma and Mistral fall to linear-slack level. The dotted outline is the same statistic on the full sample, where all pairs share one prompt set: the gap is what the shared-prompt confound was worth. We plot the number we claim, not the more favourable full-sample one. The trait-varying arm is clipped at the axis top and annotated with its true value.

Figure 2: Three-dimensional schematic of the measurement, not model data. (a) Opposite site orders give different Jacobian-weighted paths and an endpoint chord

\mathrm{Br}_{ij}

. (b) The tangent-plane prediction

\widehat{\mathcal{L}}

is separated from the second-difference curvature

D_{ij}

. (c) Four legs can have zero net injected vector but a nonzero first-order readout residual when site Jacobians differ. Endpoint order and loop closure are diagnostics to measure, not evidence of interaction by themselves.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

family	sites	$\\|\mathcal{L}\\|/\\|\mathrm{Br}\\|$	$\\|\mathcal{S}\\|/\\|\mathrm{Br}\\|$	resid ( $\mathcal{L}$ )	resid (full)	ident. err
Llama	$(0.25,0.50)$	0.898	0.320	0.325	0.061	$1.3$ e- $7$
	$(0.50,0.75)$	0.990	0.130	0.134	0.041	$5.7$ e- $8$
	$(0.25,0.75)$	0.910	0.338	0.339	0.030	$1.9$ e- $7$
OLMo	$(0.25,0.50)$	0.999	0.092	0.105	0.052	$5.6$ e- $8$
	$(0.50,0.75)$	0.991	0.122	0.136	0.063	$6.3$ e- $8$
	$(0.25,0.75)$	0.998	0.109	0.120	0.055	$5.2$ e- $8$

Appendix

Table 1: Second-order split from the

\pm

injection design, two families, 120 pairs, three injection configurations.

\|\mathcal{S}\|/\|\mathrm{Br}\|

is the no-interaction second-order term

\tfrac{1}{2}(H_{11}-H_{22})[(v_{i},v_{i})-(v_{j},v_{j})]

as a fraction of the bracket. resid,

\mathcal{L}

only removes the first-order term alone; resid, full removes

\widehat{\mathcal{L}}=\mathcal{L}+\mathcal{S}

. The gap between them is what

\mathcal{S}

contributes.

family	layers	$\cos$	resid	$\cos_{\mathrm{sym}}$	$\cos_{\mathrm{swap}}$
DeepSeek-7B	$(0.25,0.50)$	0.9978	0.064	$-0.048$	$-0.9978$
	$(0.50,0.75)$	0.9869	0.157	$-0.008$	$-0.9869$
	$(0.25,0.75)$	0.9959	0.086	$-0.049$	$-0.9959$
Gemma-2-9B	$(0.25,0.50)$	0.9986	0.052	$-0.057$	$-0.9986$
	$(0.50,0.75)$	0.9980	0.060	$-0.166$	$-0.9980$
	$(0.25,0.75)$	0.9989	0.042	$0.016$	$-0.9989$

Appendix

Table 2: Zero-parameter first-order prediction against the measured bracket, per family and injection configuration.

\cos

is the agreement between

\widehat{\mathcal{L}}

(Section 3 , no fitted parameters) and the measured bracket; resid is the fraction of the bracket’s norm it leaves, which by that section’s expansion is the antisymmetrized mixed second derivative plus an unbounded

O(\alpha^{3})

remainder. We do not measure the cubic share on models, so resid is a finite-scale mixture, not an interaction estimate or a one-sided bound without explicit remainder control.

\cos_{\mathrm{sym}}

is the symmetric-combination control, which should carry no bracket signal;

\cos_{\mathrm{swap}}

is the site-swap control, which should equal

-\cos

exactly. Both hold in every cell.

family	observed	linear slack	generic interaction (p95)	trait-varying Jacobian
DeepSeek	1.256	1.000	1.313 (1.400)	6.082
Gemma	1.730	0.998	1.310 (1.408)	5.983
Llama	2.718	1.000	1.345 (1.420)	6.497
Mistral	2.578	1.000	1.382 (1.502)	6.632
OLMo	2.798	1.000	1.291 (1.386)	5.351
Qwen	1.710	0.998	1.303 (1.422)	5.802

Appendix

Table 3: Unsigned shared-argument ratio against three nulls, 40 draws per family. The generic-interaction and trait-varying-Jacobian arms are calibrated so their residual is the same fraction of the bracket as the observed residual; the linear slack arm is not calibrated, and cannot be — its residual is

0.007

–

0.023

of the bracket against an observed

0.029

–

0.099

, because a constant-Jacobian bracket is annihilated exactly (Appendix D ). It is a floor, not a matched competitor. Brackets are generated from the true directions and fitted on independently estimated ones. The observation is bracketed on both sides: above a generic interaction in 5 of 6 families (its own

95

th percentile, in parentheses), and far below a first-order model whose Jacobian varies with the injected traits, in 6 of 6.

family	raw cross-half	within-half	cross-half	clears $1.324$
DeepSeek-7B	2.803	1.061	1.010	no
Gemma-2-9B	2.513	1.265	1.017	no
Llama-3.1-8B	3.250	1.960	1.599	yes
Mistral-7B-v0.3	3.550	1.087	1.016	no
OLMo-7B	5.838	2.434	2.059	yes
Qwen2.5-7B	2.026	1.806	1.642	yes

Appendix

Table 4: Prompt-split replication on disjoint prompt halves, all six families, layer pair

(0.25,0.50)

, 48 prompts split into two disjoint halves of 24. Cross-half is the confound-free statistic: brackets computed on half A are compared against brackets computed on half B, so no prompt-level fluctuation is shared between the two sides. Three of six clear the generic antisymmetric null of

1.324

. The three that do not sit within

0.02

of the

1.00

of pure linear slack. The null is recomputed under the design it gates. The

1.324

bar is calibrated on the full

48

-prompt sample, while the cross-half statistic it gates is computed on

24

-prompt halves. Because each null arm’s knob is chosen so its residual fraction matches the observed one, and halving the prompts raises that observed fraction, the bar is not automatically transportable across the two designs. We therefore recomputed it: the halved-sample observed residual fractions (

0.035

0.337

, against

0.029

0.099

at full sample) give a recalibrated bar of

1.325

, and the same three families clear it. The bar is close to unmoved because the generic-interaction arm’s shared-argument ratio is insensitive to the noise level over this range, not because the recomputation was skipped. Llama

1.599

, OLMo

2.059

and Qwen

1.642

therefore clear a bar calibrated on their own design. scripts/halved_null.py ; record experiments/controls/halved_null.json . The script reproduces the committed full-sample null bit-for-bit (

1.3239312925407296

, to

2\times 10^{-16}

) before computing anything new, so it is answering with the same instrument that produced the published bar. This does not touch the paper’s central negative claim, which never uses this null.

Neural Network Interpretability

Neural network interpretability studies how models represent information and make predictions, and how to explain, test, or control their internal computations.

See all 330 papers

The Representational Limit of Scalar Interactions: An Interventional Decomposition
Jun 17, 2026Potito Aghilar, Sabino Roccotelli, Stanislao Fidanza +3Feature Interaction Modeling Representation Disentanglement
Polytechnic University of Bari · University of Bari Aldo Moro
Right Answer, Wrong Mechanism: Detecting Pernicious Divergence in Causal Interventions
Sep 30, 2026Beiming Liu, Minjie ChenCircuits in Language Models Causal Interpretability
PetroChina Southwest Oil and Gasfield Company, China
Conflicting Supervision Moves Commitment, Not Capability: A 12.29σ arrangement effect that is exactly zero under a convention-agnostic score
Sep 23, 2026Wenhui ChenLearning Rate Scheduling
University of Macau

Jun 17, 2026J/K move · Enter open · S save

The Representational Limit of Scalar Interactions: An Interventional Decomposition

Potito Aghilar, Sabino Roccotelli, Stanislao Fidanza, Vito Walter Anelli+2

Polytechnic University of Bari · University of Bari Aldo Moro

Feature Interaction Modeling Representation Disentanglement Shapley Value Attribution Causal Interpretability

Open paper AskPDF arXiv

Signed pairwise interaction scores fundamentally conflate uniqueness (U), redundancy (R), and synergy (S). We prove this on a minimal 3-way XOR structural causal model: faithful indices such as Shapley-Taylor return zero per pair, whereas projective indices such as Shapley Interaction spread the third-order effect into pair scalars that conflate the three mechanisms. We introduce Stochastic Hi-Fi, a post-hoc, retraining-free predictability decomposition that estimates per-feature U/R/S profiles by interventional masked inference. The estimator provides exact interventional semantics, finite-sample Monte Carlo bounds, strict variance reduction from coupled diamond sampling, and uniform finite-vocabulary convergence. Across tabular SCMs, Stochastic Hi-Fi recovers structure missed by scalar baselines (up to 411x larger interaction-magnitude recovery ratios). It also separates redundant and synergistic heads in the GPT-2 IOI circuit. On NIH ChestX-ray14, Stochastic Hi-Fi matches GradCAM on Pointing Game and improves substantially on Deletion AUC.

Abstract

Neural Network Interpretability

Explore similar work

The Representational Limit of Scalar Interactions: An Interventional Decomposition

Right Answer, Wrong Mechanism: Detecting Pernicious Divergence in Causal Interventions

Conflicting Supervision Moves Commitment, Not Capability: A 12.29σ arrangement effect that is exactly zero under a convention-agnostic score