Attention Function as an Intrinsic Inductive Bias: How Models' Behavior Diverges in Novel Contexts

Dong Gyun Kang; Megha Thukral; Kwangsoo Kim

cs.LGSep 30, 2026

Attention Function as an Intrinsic Inductive Bias: How Models' Behavior Diverges in Novel Contexts

Authors: Dong Gyun Kang, Megha Thukral, Kwangsoo Kim

Organizations: College of Computing, Georgia Institute of Technology · Department of Transdisciplinary Medicine, Seoul National University Hospital

Abstract

Developmental psychology holds that certain priors are given to infants prior to experience rather than induced from data, and that the influence of such priors is suppressed under strong, well-constrained conditions but reasserts itself under weak ones. We ask whether an analogous principle holds for the Transformer: can the activation function given to attention heads serve as an intrinsic inductive bias? We propose Mixture of Function Attention (MoFA), a parameter-free modification to multi-head attention that fixes a ratio of softmax and sigmoid heads before training. Across five ratios, a 124M-parameter GPT-2 model, and five seeds, we find that this given ratio has little effect in-distribution -- differences between ratios are statistically negligible for moderate mixtures and remain small even at the extremes -- but its influence re-emerges sharply under zero-shot distribution shift across 15 out-of-distribution domains. Perplexity gaps between ratios widen by more than an order of magnitude on several domains, and the best-performing ratio tracks a single axis of domain structure, separating short, informal text (softmax-favoring) from technical, long-form text (sigmoid-favoring), that explains 78.3% of the variance in domain response. This reorganization is visible at the head level: sigmoid heads show an accelerating drop in attention entropy as their ratio increases, while softmax heads respond more modestly, yielding a consistent division of labor between the two head types. Our results suggest that activation choice functions as a given prior whose influence is masked in-distribution and re-emerges out-of-distribution.

Figures & tables

Softmax	Sigmoid	Val Loss	Val PPL	p
12	0	$3.0568\pm 0.0007$	$21.26\pm 0.01$	—
9	3	$3.0571\pm 0.0015$	$21.26\pm 0.03$	$0.693$
6	6	$3.0621\pm 0.0017$	$21.37\pm 0.04$	$0.001^{*}$
3	9	$3.0673\pm 0.0011$	$21.49\pm 0.02$	$<0.001^{*}$
0	12	$3.0814\pm 0.0027$	$21.79\pm 0.06$	$<0.001^{*}$

Table 1: Internal validation perplexity on OpenWebText by sigmoid head ratio, averaged over 5 seeds (

\pm

std). Paired

t

-test against the softmax-only baseline; ∗

p<0.01

Domain	$H_{12:0}$	$H_{9:3}$	$H_{6:6}$	$H_{3:9}$	$H_{0:12}$
SST-2	$82.82_{\pm 1.55}^{*}$	$82.99_{\pm 1.28}$	$83.72_{\pm 0.85}$	$85.48_{\pm 1.08}$	$88.93_{\pm 3.97}^{\dagger}$
Tweets	$133.65_{\pm 2.18}^{*}$	$138.60_{\pm 3.85}$	$140.10_{\pm 4.08}$	$146.70_{\pm 3.82}$	$152.92_{\pm 12.00}^{\dagger}$
Reddit	$38.27_{\pm 0.06}$	$38.17_{\pm 0.17}$	$38.14_{\pm 0.24}^{*}$	$38.38_{\pm 0.24}$	$39.39_{\pm 0.48}^{\dagger}$
PTB	$103.13_{\pm 3.70}$	$100.95_{\pm 2.71}$	$99.05_{\pm 4.61}^{*}$	$109.35_{\pm 10.13}$	$113.09_{\pm 6.64}^{\dagger}$
News	$24.47_{\pm 0.08}^{*}$	$24.57_{\pm 0.08}$	$24.75_{\pm 0.06}$	$24.79_{\pm 0.27}$	$24.81_{\pm 0.21}^{\dagger}$
Dialogue	$30.57_{\pm 0.25}$	$30.17_{\pm 0.30}^{*}$	$30.34_{\pm 0.46}$	$30.36_{\pm 0.25}$	$30.87_{\pm 0.35}^{\dagger}$

Table 2: Zero-shot perplexity (mean

\pm

std over 5 seeds) across OOD domains by sigmoid head count. Columns denote

H_{\text{sfx}}{:}H_{\text{sig}}

, the number of softmax vs. sigmoid heads out of 12 total. All models trained on OpenWebText with identical data order and initialization. ∗ marks the best ratio per domain; † marks the worst. Domains are ordered from most softmax-leaning to most sigmoid-leaning. No single configuration dominates across all domains.

Figure 1: OOD domains ranked by their first principal component (PC1) of row-normalized zero-shot perplexity across the five ratios ( $78.3\%$ variance explained). Domains separate into a softmax-leaning group (short, informal text) and a sigmoid-leaning group (technical, long-form text); conversational domains (Dialogue, Empathetic) sit closest to the boundary. Table 2 follows this ordering.

	Softmax heads		Sigmoid heads
Transition	$\Delta$	$p$	$\Delta$	$p$
$H_{12:0}\to H_{9:3}$	$-0.013$	$0.522$	—	—
$H_{9:3}\to H_{6:6}$	$+0.031$	$0.439$	$-0.051$	$0.047^{*}$
$H_{6:6}\to H_{3:9}$	$-0.146$	$0.035^{*}$	$-0.109$	$0.003^{*}$
$H_{3:9}\to H_{0:12}$	—	—	$-0.361$	$<0.001^{*}$

Table 3: Paired

t

-tests (across 5 seeds) comparing domain-averaged head entropy between consecutive ratios.

\Delta

is the change from the first to the second ratio; ∗ marks

p<0.05

Panel of Figure 2 — Figure 2: Per-layer attention entropy, averaged across 15 OOD domains, for softmax heads (left) and sigmoid heads (right) under each ratio. Dashed black lines with square markers denote the pure endpoints ( $H_{12:0}$ left, $H_{0:12}$ right); solid colored lines denote the mixed ratios.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Domain	HF Identifier	Split	Source
Code	code_search_net (python)	test	Husain et al. [2019]
Novel	bookcorpus	train[:5%]	Zhu et al. [2015]
News	cnn_dailymail (3.0.0)	test	See et al. [2017]
PubMed	ccdv/pubmed-summarization	test	Cohan et al. [2018]
arXiv	ccdv/arxiv-summarization	test	Cohan et al. [2018]
Wiki	wikitext (103-v1)	test	Merity et al. [2016]

Appendix

Table 4: Dataset sources for the 15 OOD evaluation domains.

Explore similar work

Multi-Head Self Attention is a Parameter Identification Mechanism
Sep 1, 2026W. Ross MorrowMulti-Head Attention Linear Attention
Ablation-Reversible Heads Don't Transfer: A Stress Test for Mechanistic Role Claims in Transformers
Jun 6, 2026Philip QuirkeTransformer Attention Transformer Architectures
Martian
Where Computation Lives Inside TabPFN: Causal Localisation of Attention Head Function
Jun 11, 2026Atharva Gupta, Dhruv Kumar, Murari Mandal +1Tabular Foundation Models Tabpfn
Department of Computer Science, Birla Institute of Technology and Science, Pilani, India · School of Computer Engineering, Kalinga Institute of Industrial Technology, Bhubaneswar, India · Birla AI Labs, Office of Ananya Birla, Aditya Birla Group, India

Sep 1, 2026, cs.LGJ/K move · Enter open · S save

Multi-Head Self Attention is a Parameter Identification Mechanism

W. Ross Morrow

Multi-Head Attention Linear Attention Subspace

Open paper AskPDF arXiv

We prove that a multi-head scaled dot product attention can be viewed as a parameter identification strategy. The ratio of unidentified parameters to the total number of parameters scales like the reciprocal of the number of heads (

1/2 \to 1/(2H)

), meaning models with more heads are structurally more identified. A subtle side effect of the mathematics observation that attention can never be fully identified. Similarly we also show that some bias terms can have no effect on softmax-based attention layers in both the single- and multiple-head settings, though this is mostly a curiosity that should have a marginal effect on model size and model training/prediction efficiency. We also touch on modern improvements to transformers including RoPE and GQA from this perspective, illustrating how those as well can improve the ratio of

meaningful'' parameters to all parameters. Simple numerical examples demonstrate that training can indeed involve updates that overlap model-invariant subspaces that arise from a lack of identification. As part of our experiments we use a

rebalancing'' approach that can ``fix'' updates that overlap unindentified subspaces but do not try to present evidence this should actually be adopted. Instead we simply view our numerical results as exploring and confirming the theoretical results. As a whole we discuss a purely mathematical/statistical explanation, identification, for why specific architectural choices in transformers may have improved performance.

Abstract

Explore similar work

Multi-Head Self Attention is a Parameter Identification Mechanism

Ablation-Reversible Heads Don't Transfer: A Stress Test for Mechanistic Role Claims in Transformers

Where Computation Lives Inside TabPFN: Causal Localisation of Attention Head Function