cs.LGNov 2, 2026

Exact Fusion of Softmax Attention Heads Can Require Exponentially Many Atoms

Authors: Hiroyasu Usami, Keisuke Hara, Ayato Tsuboi, Naohiko Matsuda

Organizations: Department of Computer Science, Graduate School of Engineering, Chubu University, Kasugai, Aichi 487-8501, Japan · Heat Transfer Research Department, Research & Innovation Center, Mitsubishi Heavy Industries, Ltd., Takasago, Hyogo 676-8686, Japan

Abstract

Fusing independently normalized attention heads can increase the number of explicit key-value entries, or atoms, needed to preserve their output. We prove a sharp separation for Wd(q)=∑j=1dσ(qj)W_d(q)=\sum_{j=1}^dσ(q_j): its dd two-atom heads use 2d2d head-atom terms, whereas its minimum single-head representation has exactly 2d2^d atoms, even with arbitrary fixed replacement keys, scalar values, and positive masses. The same lower bound holds for every fixed uniform error below 1/21/2 on Rd\mathbb R^d, and the threshold is sharp. The proof uses exposed-key normal cones; exact equality on any nonempty open query domain inherits the bound. The resulting atom bound for a fixed head count is tight when that count divides dd. A normalized family retains its exponential exact minimum while approaching a constant on fixed-radius query balls. For pretrained-model caches, a classical affine-rank bound and a certified instance exclude exact vector preservation at a specified atom count. Learned-cache comparisons on two disjoint banks of 64 contexts show budget-dependent reversals and calibration effects.

Figures & tables

Explore similar work

CardsList
  1. Counting with Attention and LayerNorm: Norm Scaling and Representation Geometry

    Oct 12, 2026Ayyub Shaffy, Manoj Dhakal, Safal Shrestha +1Self-AttentionLayer Normalization

  2. Softmax Attention with Any Temperature in Fixed Dimension: Matching the Best Known Maximum-Inner-Product Bounds

    Nov 4, 2026Zixuan Huang, Guozhang ChenEfficient AttentionSoftmax Attention