Real-Valued Hyperdimensional Sequence Representations with Hadamard Product Binding and Shift Equivariance

Kenny Schlegel; Dmitri A. Rachkovskij; Denis Kleyko; Amy Loutfi; Stefan Streif; Evgeny Osipov

cs.AIAug 28, 2026

Real-Valued Hyperdimensional Sequence Representations with Hadamard Product Binding and Shift Equivariance

Authors: Kenny Schlegel, Dmitri A. Rachkovskij, Denis Kleyko, Amy Loutfi, Stefan Streif, Evgeny Osipov

Organizations: Chemnitz University of Technology, Germany · Luleå University of Technology, Sweden · Örebro University, Sweden · Institute of Information Technologies and Systems, Ukraine ·

Abstract

Encoding temporal order is a fundamental requirement for sequence representations in Hyperdimensional Computing. Fractional Power Encoding provides similarity-preserving position vectors whose inner products approximate shift-invariant kernels, and it supports shift-equivariant transformations of encoded sequence representations. However, standard formulations of Fractional Power Encoding are primarily designed for binding operations such as circular convolution or complex-valued multiplication, which limits their compatibility with Hadamard product binding of real-valued vectors. This paper develops real-valued position encodings motivated by Random Fourier Features, aiming to retain the desirable properties of Fractional Power Encoding while supporting Hadamard-based operations. We propose three real-valued position-encoding variants: a real-valued baseline based on the inverse Fourier transform, and Sinusoid and Cosine-only representations derived from Random Fourier Features. Among them, the Sinusoid variant provides an explicit algebraic shift operator, allowing temporal shifts to be applied directly to the vector-encoded sequence representation without re-encoding the shifted sequence. Experiments on time-series classification datasets show that the proposed real-valued representations achieve performance comparable to standard Fractional Power Encoding while enabling computationally efficient Hadamard product binding. The Sinusoid variant offers the most favorable trade-off, combining efficient real-valued implementation with exact shift-equivariant transformations.

Figures & tables

Property	Circular Convolution / Complex Hadamard	Variant A: Real Hadamard + Original FPE	Variant B: Real Hadamard + Sinusoidal	Variant C: Real Hadamard + Cosine
Representation of features $\mathbf{f}_{t}$ and positions $\mathbf{p}_{t}$ ( Eq. 2 )	$\mathbf{f}_{t}\in\mathbb{C}^{D}$ and $\mathbf{p}_{t}=\mathbf{c}(t\mathchar 59\relax\beta)$ , or $\mathbf{f}_{t}\in\mathbb{R}^{D}$ and $\mathbf{p}_{t}=\mathbf{p}(t\mathchar 59\relax\beta)$	$\mathbf{f}_{t}\in\mathbb{R}^{D}$ and $\mathbf{p}_{t}=\mathbf{p}(t\mathchar 59\relax\beta)$	$\mathbf{f}_{t}\in\mathbb{R}^{D}$ and $\mathbf{p}_{t}=\boldsymbol{\psi}(t\mathchar 59\relax\beta)$	$\mathbf{f}_{t}\in\mathbb{R}^{D}$ and $\mathbf{p}_{t}=\boldsymbol{\chi}(t\mathchar 59\relax\beta)$
Implementation of FPE	$\begin{aligned} \mathbf{p}(t\mathchar 59\relax\beta)&=\mathcal{F}^{-1}(\mathbf{c}^{\beta t})\\ \mathbf{c}(t\mathchar 59\relax\beta)&=\mathbf{c}^{\beta t}\mathord{\mathchar 59\relax}\ \text{with}\\ \mathbf{c}&=e^{i\boldsymbol{\omega}}\end{aligned}$	$\begin{aligned} \mathbf{p}(t\mathchar 59\relax\beta)&=\mathcal{F}^{-1}(\mathbf{c}^{\beta t+1})\\ \text{with}\ \mathbf{c}&=e^{i\boldsymbol{\omega}}\end{aligned}$	$\begin{aligned} \boldsymbol{\psi}(t\mathord{\mathchar 59\relax}\beta)&=[\sin(\boldsymbol{\omega}t\beta)\mathchar 59\relax\\ &\hskip 9.24994pt\cos(\boldsymbol{\omega}t\beta)]\end{aligned}$	$\begin{aligned} \boldsymbol{\chi}(t\mathord{\mathchar 59\relax}\beta)&=\sqrt{2}\cos(\boldsymbol{\omega}t\beta+\mathbf{b})\mathchar 59\relax\\ &b_{i}\sim\mathcal{U}[-\pi\mathord{\mathchar 59\relax}\pi]\end{aligned}$
	with $\omega_{i}\overset{\text{i.i.d.}}{\sim}\mathcal{P}$ , for $i=1\mathord{\mathchar 59\relax}\ldots\mathord{\mathchar 59\relax}D$
Hadamard binding compatibility (R1)	$\odot$ (complex Hadamard) or $\circledast$ (circular convolution)	$\odot$ (real-valued Hadamard)	$\odot$ (real-valued Hadamard)	$\odot$ (real-valued Hadamard)
Similarity kernel approximation (R2)	preserved via complex exponential representation	preserved (constant offset in exponent)	preserved via RFF approximation	preserved via RFF approximation
Position vector norm (R3)	constant due to unit-magnitude complex phasors	constant after inverse Fourier transform	constant by Sinusoidal embedding	approximately constant by Cosine embedding

Table 1: Comparison of sequence representation approaches: implementation, equivariance, and sequence encoding approach.

Model	HDC	Num DS	Mean ACC	Worst ACC	Relative Change in ACC		Relative Change in Error
					Mean	Max	Mean	Min
MiniROCKET		38	80.79	29.27
HDC original HRR	HRR	38	82.22	32.61	1.43	14.42	-8.0	-40.0
Standard FPE	MAP	38	82.23	30.56	1.44	14.00	-6.0	-77.0
Sinusoid FPE	MAP	38	82.22	30.36	1.43	14.06	-6.0	-76.0
Cosine FPE	MAP	38	82.24	30.55	1.45	14.77	-6.0	-77.0

Table 2: Datasets with more than

\pm 5\%

relative change in ACC between ROCKET variants Dempster et al. (2021) ; Dempster et al. (2023) and the corresponding HDC-based approaches. “ Num DS ” denotes the number of datasets in the respective subset. All values, except for AUPRC, are reported as percentages. The last four columns show the relative changes in ACC and Error. Each HDC model refers to the HDC-based implementation using a particular positional vector variant applied to the corresponding original (non-HDC) model in the same row.

Figure 4: Example evaluation of the shift equivariance property of the Sinusoid representation in the HDC-MiniROCKET framework. A synthetic signal with a single temporal peak and the same signal shifted by 10 time steps.

Figure 5: Cosine similarity between the explicitly re-encoded representation $\mathbf{r}_{\mathrm{re}\mathord{\mathchar 59\relax}s}$ and the equivariantly shifted representation $\mathbf{r}_{\diamond\mathord{\mathchar 59\relax}s}$ for different candidate shift values $s$ . The peak value of $1.0$ at the true shift $s=10$ confirms that the equivariant transformation produces the same representation as explicit re-encoding.

Positional Encoding

Positional encoding gives Transformers signals about token order, distance, and spatial or temporal relationships.

See all 117 papers

Spiking Sequence Machines and Transformers
May 1, 2026Joy BoseTransformer Attention Spiking Neural Networks
Independent Researcher Bengaluru, India
BiHDTrans: binary hyperdimensional transformer for efficient multivariate time series classification
Sep 29, 2025Jingtao Zhang, Yi Liu, Qi Shen +1Efficient Neural Network Inference Transformer
School of Biomedical Engineering, Shenzhen Campus of Sun Yat-sen University · College of Medicine and Biological Information Engineering, Northeastern University
HyperSpace: A Generalized Framework for Spatial Encoding in Hyperdimensional Representations
Apr 16, 2026Shay Snyder, Andrew Capodieci, David Gorsich +1Hyperdimensional Computing Vector Symbolic Architectures
George Mason University · George Mason University, Fairfax, Virginia, USA · Neya Robotics +3

May 1, 2026J/K move · Enter open · S save

Spiking Sequence Machines and Transformers

Joy Bose

Independent Researcher Bengaluru, India

Transformer Attention Spiking Neural Networks Positional Encoding Sequence Modeling

Open paper AskPDF arXiv

Sequence learning reduces to similarity-based retrieval over a temporally indexed representation space, a constraint on any sequence model, not a property of a specific architecture. We show that a spiking Sparse Distributed Memory sequence machine (2007) and the transformer (2017) independently instantiate the same five functional operations (encoding, context maintenance, associative retrieval, storage, and decoding), with cosine similarity as the shared retrieval primitive in both. We formalise a Phase-Latency Isomorphism showing that sinusoidal positional phase and spike timing are linearly related, and prove that dot product attention is invariant to this mapping up to a global scale factor on the positional component (Lemma 1). Empirically, frequency-compressed positional encoding fails to converge on a positionally demanding copy task, while a learned rank-based embedding matches or exceeds sinusoidal encoding, indicating that the critical property for positional representation is distance discriminability under dot-product similarity, not sinusoidal form. Time, phase, and rank are three instantiations of the same computational primitive, an ordered index whose structure survives similarity-based retrieval.

Abstract

Positional Encoding

Explore similar work

Spiking Sequence Machines and Transformers

BiHDTrans: binary hyperdimensional transformer for efficient multivariate time series classification

HyperSpace: A Generalized Framework for Spatial Encoding in Hyperdimensional Representations