From Attention Masks to Inert Zero-Vector Tokens: OAttention and O-Closure for Token Dynamics

Heyang Gong

cs.AIAug 21, 2026

From Attention Masks to Inert Zero-Vector Tokens: OAttention and O-Closure for Token Dynamics

Authors: Heyang Gong

Abstract

Attention masks are relation-level controls: they specify which query--source pairs may interact. They do not provide a representation-carried token state that is non-participating at the attention boundary. We assign each token hidden carrier $h_i$ an active-presence coefficient $p_i=\lVert h_i\rVert^2/(τ+\lVert h_i\rVert^2)$ . The same coefficient has two roles: it gates information emitted by token $i$ , and it determines the mass with which token $i$ enters computations shared with other tokens. OAttention is the support-coupled attention realization of this rule. It gates the receiver output by $p_i$ and weights source $j$ by $p_j$ in both the attention numerator and partition, while retaining the standard score, visibility relation, exponential competition, and value aggregation. This makes the zero-vector token a zero element and yields exact null-receiver, null-source insertion, self-attention insertion, and empty-support properties. The same token-level presence gives local O-components (OFFN, ONorm, and OInject), presence-weighted OStandardize, the O-Closure law $M(H\oplus0)=M(H)\oplus0$ , and an OTransformer by residual and compositional closure. The canonical operator is checked by contract tests and a GPU evaluation. In a zero-fine-tuning retrofit of a cloned pretrained TabPFN v3 regressor, calibrated hidden-carrier OAttention and Full-O variants change mean RMSE by $+0.088\%$ and $+0.177\%$ , respectively, over 18 matched dataset--seed cases. A two-block ablation shows that OAttention alone does not preserve a NULL state through ordinary host components, whereas the OTransformer path does. These are scoped tests of exactness, active-path compatibility, and compositional necessity; they do not establish universal no-loss, arbitrary-host closure, learned attraction to the origin, or a general semantics for missing values.

Figures & tables

Property	Result
Canonical equation	Output and weight $L_{\infty}$ error $\leq 8.94\times 10^{-8}$ against direct evaluation
Null insertion	Old-output $L_{\infty}\leq 4.47\times 10^{-8}$ ; old-weight $L_{\infty}\leq 5.96\times 10^{-8}$
Exact null boundary	Inserted receiver output and source weight: $0$
Empty/all-null support	Output and weights finite exact zero
Interfaces	6/6 dtype–head cases finite; MHA, GQA, fp32, bf16, fp16, cross-attention, and tested gradients

Table 1: Independent hidden-carrier OAttention operator evaluation. Errors are maxima on an NVIDIA GB10 GPU.

Dataset	Base RMSE	OA RMSE	Full-O RMSE	Base $R^{2}$	OA $R^{2}$	Full-O $R^{2}$
bodyfat	1.231549	1.230964	1.232571	.961179	.961228	.961112
concrete	4.127869	4.133669	4.129971	.936922	.936758	.936862
cpu_small	3.023569	3.022008	3.023061	.971290	.971319	.971299
kin8nm	.091274	.091292	.091278	.877050	.877005	.877038
space_ga	.093484	.093366	.093445	.740970	.741612	.741182
tecator	.422813	.424202	.424954	.999087	.999082	.999078

Table 2: Near-identity hidden-carrier retrofit of pretrained TabPFN v3 (

\tau=10^{-8}

). Values are means over three seeds. RMSE is in the original target units;

R^{2}

is computed on each test split.

Arm	Host components	Max inserted-state $L_{\infty}$	Interpretation
Standard	Softmax, additive encoding, affine norm, biased FFN	7.982	Ordinary host
OA-only	Hidden-carrier OA; other components ordinary	7.983	Attention alone is insufficient
OTransformer	Hidden-carrier OA, OInject, ONorm, OFFN	0	Declared O-closed path
Structural control	Hidden-carrier OA, no additive encoding, affine-free norm, bias-free FFN	0	Separately initialized comparator

Table 3: Scoped OTransformer study. Maxima are over the evaluated task–seed–zero-count matrix for

k>0

. The structural control is not part of the initialization-matched three-arm comparison.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Rows	Features	Classes	Task
Iris	150	4	3	classification
Wine	178	13	3	classification
Breast Cancer Wisconsin	569	30	2	classification
Digits	1,797	64	10	classification
Diabetes	442	10	–	regression

Appendix

Table 4: Real datasets used in the learned-model evaluations.

Arm	Attention	Encoding and normalization	FFN
Standard	softmax	additive encoding; affine LayerNorm	biased
OA-only	hidden-carrier OAttention	additive encoding; affine LayerNorm	biased
OTransformer	hidden-carrier OAttention	OInject; ONorm-wrapped affine LayerNorm	OFFN-wrapped biased FFN
Structural control	hidden-carrier OAttention	no additive encoding; affine-free LayerNorm	bias-free, O-gated

Appendix

Table 5: Arms in the scoped OTransformer experiment. The first three share exact initialization; the structural control is initialized separately.

Property	Result
Finiteness	15/15 seed–shape cases finite
Exact zero boundaries	Null query, empty support, and fully masked output and weights: $0$
Null-source insertion	Old-output $L_{\infty}\leq 2.4\times 10^{-7}$ ; old-weight $L_{\infty}\leq 6.0\times 10^{-8}$
Self-attention insertion	Old-output $L_{\infty}\leq 4.8\times 10^{-7}$ ; inserted-query output $0$
Near-origin behavior	Log–log output/query slope $1.9997$ over scales $10^{-7}$ to $10^{-5}$
Active path	Relative output $L_{\infty}\leq 3.7\times 10^{-7}$ at the evaluated default scale

Appendix

Table 6: Numerical verification of the Q/K/V-boundary design variation. The nonzero insertion errors are floating-point residuals on pre-existing outputs or weights.

Dataset	OA $\Delta$ RMSE	Full-O $\Delta$ RMSE	OA $\Delta R^{2}$	Full-O $\Delta R^{2}$
bodyfat	+0.532%	+95.354%	+0.000028	+0.005431
concrete	+0.162%	+24.442%	-0.000178	-0.034037
cpu_small	-0.054%	+5.238%	+0.000031	-0.002766
kin8nm	-0.112%	+39.797%	+0.000277	-0.117006
space_ga	-0.126%	+17.324%	+0.000649	-0.096142
tecator	+0.429%	+100.143%	-0.000008	-0.002602

Appendix

Table 7: Finite-

\tau

stress matrix on the cloned pretrained TabPFN v3 host. Deltas are relative to the unmodified checkpoint.

Task	Metric	Standard	Hidden-carrier OA	$\Delta$
Iris	accuracy	.9111	.9111	+.0000
Wine	accuracy	.9630	.9630	+.0000
Breast Cancer	accuracy	.9708	.9708	+.0000
Digits	accuracy	.8972	.8954	-.0019
Synthetic classification	accuracy	.6889	.6900	+.0011
Diabetes	RMSE	54.6873	54.6782	-.0092

Appendix

Table 8: Canonical hidden-carrier adapter rerun. Deltas are hidden-carrier OAttention minus standard softmax.

Inserted	Prediction RMS shift		Inserted-state $L_{\infty}$
tokens	Standard	Hidden-carrier OA	Standard	Hidden-carrier OA
1	$7.39\times 10^{-2}$	$3.69\times 10^{-5}$	1.83–7.42	0
4	$2.18\times 10^{-1}$	$1.64\times 10^{-4}$	1.84–7.42	0

Appendix

Table 9: Test-time insertion over five datasets and three seeds. Prediction shift is RMS over test predictions; state is the maximum absolute inserted coordinate.

Task	Metric	$k=0$	$k=1$	$k=2$	$k=4$
Iris	accuracy	.8667/.9222	.8667/.9111	.8778/.9111	.9333/.8889
Wine	accuracy	.9444/.9444	.9444/.9444	.9444/.9444	.9444/.9444
Breast Cancer	accuracy	.9678/.9678	.9678/.9678	.9678/.9678	.9678/.9678
Digits	accuracy	.9204/.9213	.9194/.9213	.9194/.9213	.9194/.9213
Diabetes	RMSE	62.6683/62.6689	62.7226/62.6680	62.7453/62.6682	62.8580/62.6688

Appendix

Table 10: Mean test metric over three seeds in the canonical hidden-carrier training-time zero-column study, reported as standard/hidden-carrier OAttention. For Diabetes, lower RMSE is better.

Attention Mechanisms

Attention mechanisms let neural networks weight information from other tokens or regions according to relevance, enabling context-dependent processing.

See all 517 papers

Jun 6, 2026

A Unifying View of Attention Sinks: From Mechanisms to Architectural Interventions

When attention concentrates on a single token, a sink, what is the model actually computing? Attention sinks are ubiquitous in softmax transformers, yet this shared visual signature can hide fundamentally different algorithms. We show that visually similar sink patterns can reflect two distinct mechanisms: (i) adaptive nop, where a head suppresses its update by routing to a null token, and (ii) broadcast, where a sink aggregates and redistributes global information. Each mechanism leaves distinct traces (nop-sinks exhibit negligible value norms; broadcast sinks induce low-rank outputs), which we formalize on synthetic tasks and use to derive practical diagnostics. Applied to pretrained vision transformers, these diagnostics reveal that both mechanisms exist at scale: sinks transition from CLS in early layers to patches in deeper layers and concentrate in specialized heads. Causal interventions further connect these signatures to near-null suppression and shared residual contributions. We then use architectural interventions to show how these computations can be reorganized: gating eliminates detected nop-like sinks but increases broadcast-like sinks, registers relocate rather than remove sink computation, and our position-free global pathway provides an explicit route for shared communication that reduces the broadcast-like sinks induced by gating. On dense probes, combining gating with the global pathway gives the strongest results among the tested variants, despite retaining some broadcast-like sinks. Overall, we find that the same attention pattern can reflect two very different computations, and that effective intervention depends not only on identifying the computation, but also on providing architectural alternatives through which the model can reorganize it.

Lukas Fesser*, Mozes Jacobs*, Thomas Fel* +2

Kempner Institute Harvard University Cambridge, MA

Transformer Interpretability Vision Transformer Attention Sinks Multi-Head Attention Attention Head Specialization

Ask PDF

Feb 19, 2026

The Anxiety of Influence: Bloom Filters in Transformer Attention Heads

Some transformer attention heads appear to function as membership testers, dedicating themselves to answering the question "has this token appeared before in the context?" We identify these heads across four language models (GPT-2 small, medium, and large; Pythia-160M) and show that they form a spectrum of membership-testing strategies. Two heads (L0H1 and L0H5 in GPT-2 small) function as high-precision membership filters with false positive rates of 0-4% even at 180 unique context tokens -- well above the

d_\text{head} = 64

bit capacity of a classical Bloom filter. A third head (L1H11) shows the classic Bloom filter capacity curve: its false positive rate follows the theoretical formula

p \approx (1 - e^{-kn/m})^k

with

R^2 = 1.0

and fitted capacity

m \approx 5

bits, saturating by

n \approx 20

unique tokens. A fourth head initially identified as a Bloom filter (L3H0) was reclassified as a general prefix-attention head after confound controls revealed its apparent capacity curve was a sequence-length artifact. Together, the three genuine membership-testing heads form a multi-resolution system concentrated in early layers (0-1), taxonomically distinct from induction and previous-token heads, with false positive rates that decay monotonically with embedding distance -- consistent with distance-sensitive Bloom filters. These heads generalize broadly: they respond to any repeated token type, not just repeated names, with 43% higher generalization than duplicate-token-only heads. Ablation reveals these heads contribute to both repeated and novel token processing, indicating that membership testing coexists with broader computational roles. The reclassification of L3H0 through confound controls strengthens rather than weakens the case: the surviving heads withstand the scrutiny that eliminated a false positive in our own analysis.

Peter Balogh

Attention Head Analysis Attention Head Specialization Transformer Interpretability

Ask PDF

Mar 12, 2026

Why Attend to Everything? Focus is the Key

Standard attention scales quadratically with sequence length. Efficient attention methods reduce this O(n^2) cost, but when retrofitted into pretrained models, they often degrade perplexity, downstream accuracy, or both. We introduce Focus, a method that learns which token pairs matter. Focus adds a small set of learnable centroids--as few as 148K parameters per layer--that act as gates: only token pairs belonging to the same centroid group attend to each other over long ranges. Focus is composable: it can be added to any pretrained model by training only the centroids while keeping all original weights frozen. Experiments show that composing Focus onto pretrained models yields zero degradation on downstream benchmarks across model sizes from 124M to 70B parameters and five attention architectures. Surprisingly, sparse Focus attention outperforms full attention at 124M scale (30.3 vs. 31.4 perplexity) and matches full attention when trained from scratch at 7B scale (13.82 vs. 13.89). Focus is also fast: top-k group membership gives a 2x speedup with better quality than the original pretrained model. Using our FlashAttention decomposition, Focus achieves an 8.6x speedup at 1M tokens without custom kernels.

Hengshuai Yao, Xing Chen, Ahmed Murtadha +8

Sapient Intelligence · University of Alberta · Tsinghua University

Attention Mechanisms Efficient Attention Attention Approximation Sparse Attention Long-Context Modeling

Ask PDF

Abstract

Attention Mechanisms

Explore similar work

A Unifying View of Attention Sinks: From Mechanisms to Architectural Interventions

The Anxiety of Influence: Bloom Filters in Transformer Attention Heads

Why Attend to Everything? Focus is the Key