Learned Queries and Keys Are All You Need: Replacing the Value Projection with Structured Transforms

Ene Meco; Emadeldeen Hamdan; A. Enis Cetin

cs.LGSep 29, 2026

Learned Queries and Keys Are All You Need: Replacing the Value Projection with Structured Transforms

Authors: Ene Meco, Emadeldeen Hamdan, A. Enis Cetin

Organizations: University of Illinois Chicago, Chicago, IL 60607

Abstract

To reduce the number of parameters and cache memory requirements of transformers we introduce dual-headed transformers instead of three heads. We studied Walsh-Hadamard Transform (WHT), Discrete Cosine Transform (DCT), Discrete Fourier Transform, filterbank based Shearlet Transform, and Multiplication-Avoiding (MA) operators to construct dual heads. We combine spatial patches and their orthogonal transforms (or Shearlet and MA operators) in a structure similar to the attention block. We obtained better results than triple headed transformers in ImageNet. Extensive simulation examples are presented.

Figures & tables

Unit	Complexity
QKV	$O(4Nd^{2}+2N^{2}d)$
QKK	$O(3Nd^{2}+2N^{2}d)$
QK-DCTK	$O(3Nd^{2}+2N^{2}d+Nd^{2}/H)$
QK-WHTK	$O(3Nd^{2}+2N^{2}d+Nd\log d_{k})$
QK-ShearK	$O(3Nd^{2}+2N^{2}d+C_{\mathcal{S}})$
QK-MFQK	$O(3Nd^{2}+2N^{2}d)$

Table 1: Computational complexity of the attention units.

Configuration	Parameters	Test Accuracy (%)
Regular Attention: $A(Q,K)V$	546,186	$76.035\pm 0.530$
$A(Q,K)K$	480,138	$76.155\pm 0.035$
$A(Q,K)\operatorname{DCT}(K)$	480,138	$76.140\pm 0.806$
$A(Q,K)\operatorname{HT}(K)$	480,138	$76.830\pm 0.764$
$A(Q,K)\mathcal{S}(K)$	480,634	$\mathbf{77.320\pm 0.354}$
$A(Q,\mathcal{S}(K))\mathcal{S}(K)$	480,634	$77.310\pm 0.976$

Table 2: Test accuracy and parameter count of the evaluated attention configurations.

Dataset	Attention	Top-1 (%)	Top-5 (%)	Params (M)	Mult. (M)
C10	Softmax ViT	81.92	98.82	2.694	182.85
	Normalized MF–tanh	77.80	98.70	2.471	168.47
Tiny-IN	Softmax ViT	40.39	65.35	2.758	184.66
	Normalized MF–tanh	35.40	60.32	2.536	170.28

Table 3: Comparison of conventional ViT and normalized MF–tanh attention. Top-1 accuracy is the best accuracy on the fixed internal validation split from one training run (seed 0). “Mult.” counts general multiplications and excludes sign, absolute-value, addition, tanh, reduction, and division operations. C10 represent the CIFAR10 dataset and Tiny-IN represent Tiny ImageNet dataset.

Explore similar work

Rethinking Attention Output Projection: Structured Hadamard Transforms for Efficient Transformers
Mar 9, 2026Shubham Aggarwal, Lokendra KumarMulti-Head Attention Efficient Transformer Inference
A Smaller Transformer in Your Transformer
Sep 17, 2026Dhananjay Tomar, Marius Aasan, Andreas Kleppe +1Transformer Inference Vision Transformer
Institute for Cancer Genetics and Informatics Oslo University Hospital Oslo, Norway · Department of Informatics University of Oslo Oslo, Norway · SFI Visual Intelligence UiT The Arctic University of Norway Tromsø, Norway
Do Transformers Need Three Projections? Systematic Study of QKV Variants
Jun 1, 2026Ali Kayyam, Anusha Madan Gopal, M Anthony LewisEfficient Transformer Inference Transformer Attention
BrainChip Inc., Laguna Hills, CA, USA.

Mar 9, 2026J/K move · Enter open · S save

Rethinking Attention Output Projection: Structured Hadamard Transforms for Efficient Transformers

Shubham Aggarwal, Lokendra Kumar

Multi-Head Attention Efficient Transformer Inference

Open paper AskPDF arXiv

The dense output projection in multi head attention scales quadratically with model dimension, contributing significantly to parameter count, memory footprint, and inference cost. We propose replacing this projection with a fixed, parameter free Walsh Hadamard Transform (WHT) followed by a diagonal affine transformation. This approach eliminates approximately 25 percent of attention parameters per block while maintaining global cross-head interaction through an orthogonal, norm-preserving transformation. Our results demonstrate that WHT augmented models exhibit a steeper validation loss curve relative to training FLOPs compared to dense baselines, suggesting superior compute utilization during training. Crucially, we show that efficiency gains including reduced memory footprint and increased throughput grow monotonically with model size, batch size, and sequence length. We evaluate performance across both prefill and decoding stages, finding that the structured transform consistently outperforms dense projections as complexity increases. Our findings indicate that replacing dense projections with structured transforms allows for more compute-efficient architectures that achieve lower loss than dense models at an equivalent training budget.

Abstract

Explore similar work

Rethinking Attention Output Projection: Structured Hadamard Transforms for Efficient Transformers

A Smaller Transformer in Your Transformer

Do Transformers Need Three Projections? Systematic Study of QKV Variants