Attention Function as an Intrinsic Inductive Bias: How Models' Behavior Diverges in Novel Contexts

Dong Gyun Kang; Megha Thukral; Kwangsoo Kim

cs.LGSep 30, 2026

Attention Function as an Intrinsic Inductive Bias: How Models' Behavior Diverges in Novel Contexts

Authors: Dong Gyun Kang, Megha Thukral, Kwangsoo Kim

Organizations: College of Computing, Georgia Institute of Technology · Department of Transdisciplinary Medicine, Seoul National University Hospital

Abstract

Developmental psychology holds that certain priors are given to infants prior to experience rather than induced from data, and that the influence of such priors is suppressed under strong, well-constrained conditions but reasserts itself under weak ones. We ask whether an analogous principle holds for the Transformer: can the activation function given to attention heads serve as an intrinsic inductive bias? We propose Mixture of Function Attention (MoFA), a parameter-free modification to multi-head attention that fixes a ratio of softmax and sigmoid heads before training. Across five ratios, a 124M-parameter GPT-2 model, and five seeds, we find that this given ratio has little effect in-distribution -- differences between ratios are statistically negligible for moderate mixtures and remain small even at the extremes -- but its influence re-emerges sharply under zero-shot distribution shift across 15 out-of-distribution domains. Perplexity gaps between ratios widen by more than an order of magnitude on several domains, and the best-performing ratio tracks a single axis of domain structure, separating short, informal text (softmax-favoring) from technical, long-form text (sigmoid-favoring), that explains 78.3% of the variance in domain response. This reorganization is visible at the head level: sigmoid heads show an accelerating drop in attention entropy as their ratio increases, while softmax heads respond more modestly, yielding a consistent division of labor between the two head types. Our results suggest that activation choice functions as a given prior whose influence is masked in-distribution and re-emerges out-of-distribution.

Figures & tables

Softmax	Sigmoid	Val Loss	Val PPL	p
12	0	$3.0568\pm 0.0007$	$21.26\pm 0.01$	—
9	3	$3.0571\pm 0.0015$	$21.26\pm 0.03$	$0.693$
6	6	$3.0621\pm 0.0017$	$21.37\pm 0.04$	$0.001^{*}$
3	9	$3.0673\pm 0.0011$	$21.49\pm 0.02$	$<0.001^{*}$
0	12	$3.0814\pm 0.0027$	$21.79\pm 0.06$	$<0.001^{*}$

Table 1: Internal validation perplexity on OpenWebText by sigmoid head ratio, averaged over 5 seeds (

\pm

std). Paired

t

-test against the softmax-only baseline; ∗

p<0.01

Domain	$H_{12:0}$	$H_{9:3}$	$H_{6:6}$	$H_{3:9}$	$H_{0:12}$
SST-2	$82.82_{\pm 1.55}^{*}$	$82.99_{\pm 1.28}$	$83.72_{\pm 0.85}$	$85.48_{\pm 1.08}$	$88.93_{\pm 3.97}^{\dagger}$
Tweets	$133.65_{\pm 2.18}^{*}$	$138.60_{\pm 3.85}$	$140.10_{\pm 4.08}$	$146.70_{\pm 3.82}$	$152.92_{\pm 12.00}^{\dagger}$
Reddit	$38.27_{\pm 0.06}$	$38.17_{\pm 0.17}$	$38.14_{\pm 0.24}^{*}$	$38.38_{\pm 0.24}$	$39.39_{\pm 0.48}^{\dagger}$
PTB	$103.13_{\pm 3.70}$	$100.95_{\pm 2.71}$	$99.05_{\pm 4.61}^{*}$	$109.35_{\pm 10.13}$	$113.09_{\pm 6.64}^{\dagger}$
News	$24.47_{\pm 0.08}^{*}$	$24.57_{\pm 0.08}$	$24.75_{\pm 0.06}$	$24.79_{\pm 0.27}$	$24.81_{\pm 0.21}^{\dagger}$
Dialogue	$30.57_{\pm 0.25}$	$30.17_{\pm 0.30}^{*}$	$30.34_{\pm 0.46}$	$30.36_{\pm 0.25}$	$30.87_{\pm 0.35}^{\dagger}$

Table 2: Zero-shot perplexity (mean

\pm

std over 5 seeds) across OOD domains by sigmoid head count. Columns denote

H_{\text{sfx}}{:}H_{\text{sig}}

, the number of softmax vs. sigmoid heads out of 12 total. All models trained on OpenWebText with identical data order and initialization. ∗ marks the best ratio per domain; † marks the worst. Domains are ordered from most softmax-leaning to most sigmoid-leaning. No single configuration dominates across all domains.

Figure 1: OOD domains ranked by their first principal component (PC1) of row-normalized zero-shot perplexity across the five ratios ( $78.3\%$ variance explained). Domains separate into a softmax-leaning group (short, informal text) and a sigmoid-leaning group (technical, long-form text); conversational domains (Dialogue, Empathetic) sit closest to the boundary. Table 2 follows this ordering.

	Softmax heads		Sigmoid heads
Transition	$\Delta$	$p$	$\Delta$	$p$
$H_{12:0}\to H_{9:3}$	$-0.013$	$0.522$	—	—
$H_{9:3}\to H_{6:6}$	$+0.031$	$0.439$	$-0.051$	$0.047^{*}$
$H_{6:6}\to H_{3:9}$	$-0.146$	$0.035^{*}$	$-0.109$	$0.003^{*}$
$H_{3:9}\to H_{0:12}$	—	—	$-0.361$	$<0.001^{*}$

Table 3: Paired

t

-tests (across 5 seeds) comparing domain-averaged head entropy between consecutive ratios.

\Delta

is the change from the first to the second ratio; ∗ marks

p<0.05

Panel of Figure 2 — Figure 2: Per-layer attention entropy, averaged across 15 OOD domains, for softmax heads (left) and sigmoid heads (right) under each ratio. Dashed black lines with square markers denote the pure endpoints ( $H_{12:0}$ left, $H_{0:12}$ right); solid colored lines denote the mixed ratios.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Domain	HF Identifier	Split	Source
Code	code_search_net (python)	test	Husain et al. [2019]
Novel	bookcorpus	train[:5%]	Zhu et al. [2015]
News	cnn_dailymail (3.0.0)	test	See et al. [2017]
PubMed	ccdv/pubmed-summarization	test	Cohan et al. [2018]
arXiv	ccdv/arxiv-summarization	test	Cohan et al. [2018]
Wiki	wikitext (103-v1)	test	Merity et al. [2016]

Appendix

Table 4: Dataset sources for the 15 OOD evaluation domains.

Explore similar work

Sep 1, 2026cs.LG

Multi-Head Self Attention is a Parameter Identification Mechanism

We prove that a multi-head scaled dot product attention can be viewed as a parameter identification strategy. The ratio of unidentified parameters to the total number of parameters scales like the reciprocal of the number of heads (

1/2 \to 1/(2H)

), meaning models with more heads are structurally more identified. A subtle side effect of the mathematics observation that attention can never be fully identified. Similarly we also show that some bias terms can have no effect on softmax-based attention layers in both the single- and multiple-head settings, though this is mostly a curiosity that should have a marginal effect on model size and model training/prediction efficiency. We also touch on modern improvements to transformers including RoPE and GQA from this perspective, illustrating how those as well can improve the ratio of

meaningful'' parameters to all parameters. Simple numerical examples demonstrate that training can indeed involve updates that overlap model-invariant subspaces that arise from a lack of identification. As part of our experiments we use a

rebalancing'' approach that can ``fix'' updates that overlap unindentified subspaces but do not try to present evidence this should actually be adopted. Instead we simply view our numerical results as exploring and confirming the theoretical results. As a whole we discuss a purely mathematical/statistical explanation, identification, for why specific architectural choices in transformers may have improved performance.

W. Ross Morrow

Multi-Head Attention Linear Attention Subspace

Ask PDF

Jun 6, 2026cs.AI

Ablation-Reversible Heads Don't Transfer: A Stress Test for Mechanistic Role Claims in Transformers

In mechanistic interpretability, attention heads are commonly elevated to role claims (e.g., "this head represents addition") when they are necessary for a behavior, encode it linearly, and recover that behavior when restored after ablation. We show this evidence is insufficient: across three 7-8B instruction-tuned models and five computation families, heads passing all three checks routinely fail to transfer the computation when their activations are patched into a different prompt under matched controls. We introduce KID (Knowing / Intent / Doing), a role-assignment lens for attention heads, and pair it with a three-stage pipeline: capability-selective screening (CSS), singular value decomposition (SVD), and activation transduction under matched controls. Our results document a preliminary role taxonomy (including prompt-trajectory stabilizers, answer-side logit-bias heads, and soft computation-pattern carriers) and show that the same-answer control (a transduction target sharing the answer string but not the requested computation) is an underused check that exposes broad state transfer masquerading as semantic specificity.

Philip Quirke

Martian

Transformer Attention Transformer Architectures Mechanistic Interpretability Ablation Role Stress Testing+1

Ask PDF

Jun 11, 2026cs.LG

Where Computation Lives Inside TabPFN: Causal Localisation of Attention Head Function

We present the first causal mechanistic analysis of a tabular foundation model, investigating how TabPFN 2.5's feature wise attention heads distribute computation across layers. Using activation patching, ablation, and attention entropy across two synthetic regression datasets, we find clear temporal specialisation: one head's causal necessity dominates that of the others by 2 to 5 times at peak layer, with its dominant layer shifting across tasks of different complexity, while the remaining heads exhibit symmetric late layer profiles. Attention entropy and patching provide convergent evidence for the computationally active layers of the dominant head. We additionally investigate inference time steerability via contrastive activation steering, which fails to transfer across samples. We attribute this result to TabPFN's in context learning mechanism, which encodes task structure through context dependent attention rather than the stable parametric directions that make steering tractable in language models.

Atharva Gupta, Dhruv Kumar, Murari Mandal +1

Department of Computer Science, Birla Institute of Technology and Science, Pilani, India · School of Computer Engineering, Kalinga Institute of Industrial Technology, Bhubaneswar, India · Birla AI Labs, Office of Ananya Birla, Aditya Birla Group, India

Tabular Foundation Models Tabpfn Linear Activation Steering Inference-Time Steering

Ask PDF