In standard attention, three separately learned projections decide how strongly a token attends to each neighbor (WQ, WK) and how the attended features are transformed before aggregation (WV). We study Quat-Sphere-Vision (QSV), a sparse spherical vision model that replaces this projection triple with a single learned unit quaternion per token: the relative quaternion rij=qi∗⊗qj supplies both the attention logit Re(rij) and a sandwich-product feature transport x↦rij⊗x⊗rij∗, with messages passed over sparse kNN graphs on concentric Fibonacci spheres. Ablations that change only the targeted component show the two roles to be asymmetric. Removing the transport reduces test accuracy by about four percentage points on CIFAR-10 and CIFAR-100 (single runs per CIFAR-100 variant), while replacing the learned attention weights with uniform averaging leaves it essentially unchanged. Parameter-matched controls then remove the geometry itself: standard attention on the same graph exceeds QSV (mean 87.3% vs. 85.9%), and the same model on a flat 2D lattice reaches 91.1%, within 2.1 points of a ResNet-20 trained under the same pipeline (single run). In the coupled kernel, nearly all of the learned pairwise computation resides in the transport channel.
Figures & tables
Figure 1: (a) Scaled dot-product attention. (b) The QSV coupled kernel: one relative quaternion rij=qi∗⊗qj (Eq. ( 1 )) yields both the attention weight ( ℜ(rij) scaled by a learned βh ) and the message transport mij=rij⊗xj⊗rij∗ ; the output is zi=∑jCijmij (§ 3.2 ).
Figure 2: QSV in one view. (a) Images are sampled onto the outer shell by the Lambert patch embedding, propagated inward over sparse cross-shell kNN edges through five concentric Fibonacci shells, aggregated in two within-shell rounds on the last shell, and pooled. (b) The coupled kernel: one relative quaternion rij (Eq. 1 ) supplies both the routing weight and the feature transport of Eq. 2 ; the labeled costs are the ablation results of § 4.2 .
Model
Params
Test acc. (%)
ResNet-20 ( He et al. 2016 )
0.27M
91.3
ResNet-56 ( He et al. 2016 )
0.85M
93.0
ViT-Lite-7/4 ( Hassani et al. 2021 )
3.7M
93.6
CCT-7/3 × 1 ( Hassani et al. 2021 )
3.8M
96.5
Matched attention (§ 4.4 )
0.57M
87.3
Flat-lattice control (§ 4.5 )
0.57M
91.1
Table 1: Context: small models trained on CIFAR-10. Baseline numbers are as reported in the cited papers and use different training pipelines and augmentation; they are included for scale, not as a controlled comparison. Our attention entries are seed means (QSV best run across configurations: 86.9 ); the same-pipeline ResNet-20 is a single run. The two spherical models sit several points below convolutional baselines; the flat-lattice control closes most of that gap (§ 4.5 ).
Variant
CIFAR-10 Δ
CIFAR-100 Δ
Full kernel (baseline)
85.9 (—)
60.1 (—)
Transport → identity
81.7 ( −4.3 )
56.0 ( −4.1 )
Routing → uniform 1/k
85.7 ( −0.2 )
59.7 ( −0.4 )
Routing on ∣ℜ(r)∣
86.3 ( +0.3 )
—
Geometric routing
86.0 ( +0.0 )
—
Both roles disabled
64.1 ( −21.8 )
—
Table 2: Kernel ablations under the default configuration on CIFAR-10 and CIFAR-100. CIFAR-10 entries are mean test accuracy over 2 – 3 seeds; CIFAR-100 entries are single runs. The change from the full kernel is in parentheses; only the targeted component changes per row. CIFAR-100 runs differ only in class count and repulsion neighbors ( 8→16 ; an untuned legacy heuristic, applied to every CIFAR-100 run). Deltas are from unrounded accuracies; per-seed values are listed in the appendix. The joint-disable and frozen-field rows are single runs at seed 42 . † Delta from the matched attention baseline ( 87.3 CIFAR-10 mean; 63.4 CIFAR-100 single run; § 4.4 ), not from QSV.
Figure 3: Routing-temperature probe (CIFAR-10, seed 42 ; one run per intervention). Lines: means over the logged cross-shell temperatures (four of 16 heads per shell); bands: min–max. Warm-started at β0=5 , temperatures decay back to the prior’s fixed point ( β≈0.79 , dashed) at baseline accuracy; freezing β=5 throughout costs −0.4 , within seed noise (§ 4.3 ).
The Transformer model architecture has become one of the most widely used in deep learning and the attention mechanism is at its core. The standard attention formulation uses a softmax operation applied to a scaled dot product between query and key vectors. We explore the role played by norms of the queries and keys, which can cause training instabilities when they arbitrarily increase. We demonstrate how this can happen even in simple Transformer models, in the presence of easy-to-learn spurious patterns in the data. We propose a new attention formulation, QUEry-modulated Spherical aTtention (QUEST), that constrains the keys to a hyperspherical latent space, while still allowing individual tokens to flexibly control the sharpness of the attention distribution. QUEST can be easily used as a drop-in replacement for standard attention. We focus on vision applications while also exploring other domains to highlight the method's generality. We show that (1) QUEST trains without instabilities and (2) produces models with improved performance (3) that are robust to data corruptions and adversarial attacks.
Hariprasath Govindarajan, Per Sidén, Jacob Roll +1
Linköping University, Sweden · Qualcomm Auto Ltd Sweden Filial
Quaternion neural networks are parameter-efficient and model multidimensional dependencies by representing four related features as a single entity. However, existing quaternion self-attention computes component-wise scores and applies independent softmax operations to each component, which increases the computational cost and allows attention distributions to diverge across components. We propose a shared-score quaternion self-attention mechanism that computes a single real-valued score using the quaternion inner product and applies a shared attention distribution across all components. This reduces score-computation multiplications by 75% and the number of softmax operations from four to one. We prove that, when queries and keys are produced by quaternion linear projections that induce component pre-mixing, the component-wise and shared scores lie in the same interaction subspace, indicating that independent component-wise attention primarily re-parameterizes the same interactions rather than expanding the feature interaction space. In speech enhancement, our method reduces inference time by up to 44.3% on a GPU and 58.1% on a CPU while maintaining quality, with consistent trends across vision and natural language processing.
Shogo Yamauchi, Tohru Nitta, Hideaki Tamori
The Asahi Shimbun Company, Tokyo, Japan · Tokyo Woman’s Christian University, Tokyo, Japan.
The O(N2) complexity of attention over N tokens remains a computational bottleneck in transformer models. Vector-Quantized (VQ) attention reduces this to O(MN) by representing keys with M codewords, but applies uniform codebook capacity regardless of where attention mass concentrates: high-attention regions of key space may be coarsely approximated while low-attention regions waste representational capacity. We propose Adaptive Vector-Quantized (AVQ) Attention, which adaptively allocates codebook capacity based on attention importance. Starting from a small set of codewords, our method identifies the most important codes during the forward pass and refines them with pre-learned child codewords, achieving fine-grained quantization where it matters most while maintaining coarse quantization elsewhere. We develop an implementation using custom Triton kernels that enables the full adaptive refinement process, including importance scoring, child codeword insertion, and parent contribution replacement, to be carried out within the tiled computation paradigm of Flash Attention with minimal overhead. Our approach maintains O(MN) complexity while achieving improved accuracy-efficiency trade-offs compared to fixed-codebook VQ-attention.
Winfried van den dool, Patrick Forré, Amir Habibian +2
QUVA Lab, University of Amsterdam, The Netherlands · AMLab, Informatics Institute, University of Amsterdam, The Netherlands · AI4Science Lab, University of Amsterdam, The Netherlands +3