From Attention Masks to Inert Zero-Vector Tokens: OAttention and O-Closure for Token Dynamics

Heyang Gong

cs.AIAug 21, 2026

From Attention Masks to Inert Zero-Vector Tokens: OAttention and O-Closure for Token Dynamics

Authors: Heyang Gong

Abstract

Attention masks are relation-level controls: they specify which query--source pairs may interact. They do not provide a representation-carried token state that is non-participating at the attention boundary. We assign each token hidden carrier $h_i$ an active-presence coefficient $p_i=\lVert h_i\rVert^2/(τ+\lVert h_i\rVert^2)$ . The same coefficient has two roles: it gates information emitted by token $i$ , and it determines the mass with which token $i$ enters computations shared with other tokens. OAttention is the support-coupled attention realization of this rule. It gates the receiver output by $p_i$ and weights source $j$ by $p_j$ in both the attention numerator and partition, while retaining the standard score, visibility relation, exponential competition, and value aggregation. This makes the zero-vector token a zero element and yields exact null-receiver, null-source insertion, self-attention insertion, and empty-support properties. The same token-level presence gives local O-components (OFFN, ONorm, and OInject), presence-weighted OStandardize, the O-Closure law $M(H\oplus0)=M(H)\oplus0$ , and an OTransformer by residual and compositional closure. The canonical operator is checked by contract tests and a GPU evaluation. In a zero-fine-tuning retrofit of a cloned pretrained TabPFN v3 regressor, calibrated hidden-carrier OAttention and Full-O variants change mean RMSE by $+0.088\%$ and $+0.177\%$ , respectively, over 18 matched dataset--seed cases. A two-block ablation shows that OAttention alone does not preserve a NULL state through ordinary host components, whereas the OTransformer path does. These are scoped tests of exactness, active-path compatibility, and compositional necessity; they do not establish universal no-loss, arbitrary-host closure, learned attraction to the origin, or a general semantics for missing values.

Figures & tables

Property	Result
Canonical equation	Output and weight $L_{\infty}$ error $\leq 8.94\times 10^{-8}$ against direct evaluation
Null insertion	Old-output $L_{\infty}\leq 4.47\times 10^{-8}$ ; old-weight $L_{\infty}\leq 5.96\times 10^{-8}$
Exact null boundary	Inserted receiver output and source weight: $0$
Empty/all-null support	Output and weights finite exact zero
Interfaces	6/6 dtype–head cases finite; MHA, GQA, fp32, bf16, fp16, cross-attention, and tested gradients

Table 1: Independent hidden-carrier OAttention operator evaluation. Errors are maxima on an NVIDIA GB10 GPU.

Dataset	Base RMSE	OA RMSE	Full-O RMSE	Base $R^{2}$	OA $R^{2}$	Full-O $R^{2}$
bodyfat	1.231549	1.230964	1.232571	.961179	.961228	.961112
concrete	4.127869	4.133669	4.129971	.936922	.936758	.936862
cpu_small	3.023569	3.022008	3.023061	.971290	.971319	.971299
kin8nm	.091274	.091292	.091278	.877050	.877005	.877038
space_ga	.093484	.093366	.093445	.740970	.741612	.741182
tecator	.422813	.424202	.424954	.999087	.999082	.999078

Table 2: Near-identity hidden-carrier retrofit of pretrained TabPFN v3 (

\tau=10^{-8}

). Values are means over three seeds. RMSE is in the original target units;

R^{2}

is computed on each test split.

Arm	Host components	Max inserted-state $L_{\infty}$	Interpretation
Standard	Softmax, additive encoding, affine norm, biased FFN	7.982	Ordinary host
OA-only	Hidden-carrier OA; other components ordinary	7.983	Attention alone is insufficient
OTransformer	Hidden-carrier OA, OInject, ONorm, OFFN	0	Declared O-closed path
Structural control	Hidden-carrier OA, no additive encoding, affine-free norm, bias-free FFN	0	Separately initialized comparator

Table 3: Scoped OTransformer study. Maxima are over the evaluated task–seed–zero-count matrix for

k>0

. The structural control is not part of the initialization-matched three-arm comparison.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Rows	Features	Classes	Task
Iris	150	4	3	classification
Wine	178	13	3	classification
Breast Cancer Wisconsin	569	30	2	classification
Digits	1,797	64	10	classification
Diabetes	442	10	–	regression

Appendix

Table 4: Real datasets used in the learned-model evaluations.

Arm	Attention	Encoding and normalization	FFN
Standard	softmax	additive encoding; affine LayerNorm	biased
OA-only	hidden-carrier OAttention	additive encoding; affine LayerNorm	biased
OTransformer	hidden-carrier OAttention	OInject; ONorm-wrapped affine LayerNorm	OFFN-wrapped biased FFN
Structural control	hidden-carrier OAttention	no additive encoding; affine-free LayerNorm	bias-free, O-gated

Appendix

Table 5: Arms in the scoped OTransformer experiment. The first three share exact initialization; the structural control is initialized separately.

Property	Result
Finiteness	15/15 seed–shape cases finite
Exact zero boundaries	Null query, empty support, and fully masked output and weights: $0$
Null-source insertion	Old-output $L_{\infty}\leq 2.4\times 10^{-7}$ ; old-weight $L_{\infty}\leq 6.0\times 10^{-8}$
Self-attention insertion	Old-output $L_{\infty}\leq 4.8\times 10^{-7}$ ; inserted-query output $0$
Near-origin behavior	Log–log output/query slope $1.9997$ over scales $10^{-7}$ to $10^{-5}$
Active path	Relative output $L_{\infty}\leq 3.7\times 10^{-7}$ at the evaluated default scale

Appendix

Table 6: Numerical verification of the Q/K/V-boundary design variation. The nonzero insertion errors are floating-point residuals on pre-existing outputs or weights.

Dataset	OA $\Delta$ RMSE	Full-O $\Delta$ RMSE	OA $\Delta R^{2}$	Full-O $\Delta R^{2}$
bodyfat	+0.532%	+95.354%	+0.000028	+0.005431
concrete	+0.162%	+24.442%	-0.000178	-0.034037
cpu_small	-0.054%	+5.238%	+0.000031	-0.002766
kin8nm	-0.112%	+39.797%	+0.000277	-0.117006
space_ga	-0.126%	+17.324%	+0.000649	-0.096142
tecator	+0.429%	+100.143%	-0.000008	-0.002602

Appendix

Table 7: Finite-

\tau

stress matrix on the cloned pretrained TabPFN v3 host. Deltas are relative to the unmodified checkpoint.

Task	Metric	Standard	Hidden-carrier OA	$\Delta$
Iris	accuracy	.9111	.9111	+.0000
Wine	accuracy	.9630	.9630	+.0000
Breast Cancer	accuracy	.9708	.9708	+.0000
Digits	accuracy	.8972	.8954	-.0019
Synthetic classification	accuracy	.6889	.6900	+.0011
Diabetes	RMSE	54.6873	54.6782	-.0092

Appendix

Table 8: Canonical hidden-carrier adapter rerun. Deltas are hidden-carrier OAttention minus standard softmax.

Inserted	Prediction RMS shift		Inserted-state $L_{\infty}$
tokens	Standard	Hidden-carrier OA	Standard	Hidden-carrier OA
1	$7.39\times 10^{-2}$	$3.69\times 10^{-5}$	1.83–7.42	0
4	$2.18\times 10^{-1}$	$1.64\times 10^{-4}$	1.84–7.42	0

Appendix

Table 9: Test-time insertion over five datasets and three seeds. Prediction shift is RMS over test predictions; state is the maximum absolute inserted coordinate.

Task	Metric	$k=0$	$k=1$	$k=2$	$k=4$
Iris	accuracy	.8667/.9222	.8667/.9111	.8778/.9111	.9333/.8889
Wine	accuracy	.9444/.9444	.9444/.9444	.9444/.9444	.9444/.9444
Breast Cancer	accuracy	.9678/.9678	.9678/.9678	.9678/.9678	.9678/.9678
Digits	accuracy	.9204/.9213	.9194/.9213	.9194/.9213	.9194/.9213
Diabetes	RMSE	62.6683/62.6689	62.7226/62.6680	62.7453/62.6682	62.8580/62.6688

Appendix

Table 10: Mean test metric over three seeds in the canonical hidden-carrier training-time zero-column study, reported as standard/hidden-carrier OAttention. For Diabetes, lower RMSE is better.

Attention Mechanisms

Attention mechanisms let neural networks weight information from other tokens or regions according to relevance, enabling context-dependent processing.

See all 517 papers

A Unifying View of Attention Sinks: From Mechanisms to Architectural Interventions
Jun 6, 2026Lukas Fesser*, Mozes Jacobs*, Thomas Fel* +2Transformer Interpretability Vision Transformer
Kempner Institute Harvard University Cambridge, MA
The Anxiety of Influence: Bloom Filters in Transformer Attention Heads
Feb 19, 2026Peter BaloghAttention Head Analysis Attention Head Specialization
Why Attend to Everything? Focus is the Key
Mar 12, 2026Hengshuai Yao, Xing Chen, Ahmed Murtadha +8Attention Mechanisms Efficient Attention
Sapient Intelligence · University of Alberta · Tsinghua University

Jun 6, 2026J/K move · Enter open · S save

A Unifying View of Attention Sinks: From Mechanisms to Architectural Interventions

Lukas Fesser*, Mozes Jacobs*, Thomas Fel*, Andy Keller+1

Kempner Institute Harvard University Cambridge, MA

Transformer Interpretability Vision Transformer Attention Sinks Multi-Head Attention Attention Head Specialization

Open paper AskPDF arXiv

When attention concentrates on a single token, a sink, what is the model actually computing? Attention sinks are ubiquitous in softmax transformers, yet this shared visual signature can hide fundamentally different algorithms. We show that visually similar sink patterns can reflect two distinct mechanisms: (i) adaptive nop, where a head suppresses its update by routing to a null token, and (ii) broadcast, where a sink aggregates and redistributes global information. Each mechanism leaves distinct traces (nop-sinks exhibit negligible value norms; broadcast sinks induce low-rank outputs), which we formalize on synthetic tasks and use to derive practical diagnostics. Applied to pretrained vision transformers, these diagnostics reveal that both mechanisms exist at scale: sinks transition from CLS in early layers to patches in deeper layers and concentrate in specialized heads. Causal interventions further connect these signatures to near-null suppression and shared residual contributions. We then use architectural interventions to show how these computations can be reorganized: gating eliminates detected nop-like sinks but increases broadcast-like sinks, registers relocate rather than remove sink computation, and our position-free global pathway provides an explicit route for shared communication that reduces the broadcast-like sinks induced by gating. On dense probes, combining gating with the global pathway gives the strongest results among the tested variants, despite retaining some broadcast-like sinks. Overall, we find that the same attention pattern can reflect two very different computations, and that effective intervention depends not only on identifying the computation, but also on providing architectural alternatives through which the model can reorganize it.

Abstract

Attention Mechanisms

Explore similar work

A Unifying View of Attention Sinks: From Mechanisms to Architectural Interventions

The Anxiety of Influence: Bloom Filters in Transformer Attention Heads

Why Attend to Everything? Focus is the Key