Real-Valued Hyperdimensional Sequence Representations with Hadamard Product Binding and Shift Equivariance
Authors: Kenny Schlegel, Dmitri A. Rachkovskij, Denis Kleyko, Amy Loutfi, Stefan Streif, Evgeny Osipov
Organizations: Chemnitz University of Technology, Germany · Luleå University of Technology, Sweden · Örebro University, Sweden · Institute of Information Technologies and Systems, Ukraine · Linköping University, Sweden
Encoding temporal order is a fundamental requirement for sequence representations in Hyperdimensional Computing. Fractional Power Encoding provides similarity-preserving position vectors whose inner products approximate shift-invariant kernels, and it supports shift-equivariant transformations of encoded sequence representations. However, standard formulations of Fractional Power Encoding are primarily designed for binding operations such as circular convolution or complex-valued multiplication, which limits their compatibility with Hadamard product binding of real-valued vectors. This paper develops real-valued position encodings motivated by Random Fourier Features, aiming to retain the desirable properties of Fractional Power Encoding while supporting Hadamard-based operations. We propose three real-valued position-encoding variants: a real-valued baseline based on the inverse Fourier transform, and Sinusoid and Cosine-only representations derived from Random Fourier Features. Among them, the Sinusoid variant provides an explicit algebraic shift operator, allowing temporal shifts to be applied directly to the vector-encoded sequence representation without re-encoding the shifted sequence. Experiments on time-series classification datasets show that the proposed real-valued representations achieve performance comparable to standard Fractional Power Encoding while enabling computationally efficient Hadamard product binding. The Sinusoid variant offers the most favorable trade-off, combining efficient real-valued implementation with exact shift-equivariant transformations.
Figures & tables
Property
Circular Convolution / Complex Hadamard
Variant A: Real Hadamard + Original FPE
Variant B: Real Hadamard + Sinusoidal
Variant C: Real Hadamard + Cosine
Representation of features ft and positions pt ( Eq. 2 )
ft∈CD and pt=c(t\mathchar59β) , or ft∈RD and pt=p(t\mathchar59β)
with ωi∼i.i.d.P , for i=1\mathchar59…\mathchar59D
Hadamard binding compatibility (R1)
⊙ (complex Hadamard) or ⊛ (circular convolution)
⊙ (real-valued Hadamard)
⊙ (real-valued Hadamard)
⊙ (real-valued Hadamard)
Similarity kernel approximation (R2)
preserved via complex exponential representation
preserved (constant offset in exponent)
preserved via RFF approximation
preserved via RFF approximation
Position vector norm (R3)
constant due to unit-magnitude complex phasors
constant after inverse Fourier transform
constant by Sinusoidal embedding
approximately constant by Cosine embedding
Table 1: Comparison of sequence representation approaches: implementation, equivariance, and sequence encoding approach.
Model
HDC
Num DS
Mean ACC
Worst ACC
Relative Change in ACC
Relative Change in Error
Mean
Max
Mean
Min
MiniROCKET
38
80.79
29.27
HDC original HRR
HRR
38
82.22
32.61
1.43
14.42
-8.0
-40.0
Standard FPE
MAP
38
82.23
30.56
1.44
14.00
-6.0
-77.0
Sinusoid FPE
MAP
38
82.22
30.36
1.43
14.06
-6.0
-76.0
Cosine FPE
MAP
38
82.24
30.55
1.45
14.77
-6.0
-77.0
Table 2: Datasets with more than ±5% relative change in ACC between ROCKET variants Dempster et al. (2021) ; Dempster et al. (2023) and the corresponding HDC-based approaches. “ Num DS ” denotes the number of datasets in the respective subset. All values, except for AUPRC, are reported as percentages. The last four columns show the relative changes in ACC and Error. Each HDC model refers to the HDC-based implementation using a particular positional vector variant applied to the corresponding original (non-HDC) model in the same row.
Figure 4: Example evaluation of the shift equivariance property of the Sinusoid representation in the HDC-MiniROCKET framework. A synthetic signal with a single temporal peak and the same signal shifted by 10 time steps.
Figure 5: Cosine similarity between the explicitly re-encoded representation rre\mathchar59s and the equivariantly shifted representation r⋄\mathchar59s for different candidate shift values s . The peak value of 1.0 at the true shift s=10 confirms that the equivariant transformation produces the same representation as explicit re-encoding.
Sequence learning reduces to similarity-based retrieval over a temporally indexed representation space, a constraint on any sequence model, not a property of a specific architecture. We show that a spiking Sparse Distributed Memory sequence machine (2007) and the transformer (2017) independently instantiate the same five functional operations (encoding, context maintenance, associative retrieval, storage, and decoding), with cosine similarity as the shared retrieval primitive in both. We formalise a Phase-Latency Isomorphism showing that sinusoidal positional phase and spike timing are linearly related, and prove that dot product attention is invariant to this mapping up to a global scale factor on the positional component (Lemma 1). Empirically, frequency-compressed positional encoding fails to converge on a positionally demanding copy task, while a learned rank-based embedding matches or exceeds sinusoidal encoding, indicating that the critical property for positional representation is distance discriminability under dot-product similarity, not sinusoidal form. Time, phase, and rank are three instantiations of the same computational primitive, an ordered index whose structure survives similarity-based retrieval.
The proliferation of Internet-of-Things (IoT) devices has led to an unprecedented volume of multivariate time series (MTS) data, requiring efficient and accurate processing for timely decision-making in resource-constrained edge environments. Hyperdimensional (HD) computing, with its inherent efficiency and parallelizability, has shown promise in classification tasks but struggles to capture complex temporal patterns, while Transformers excel at sequence modeling but incur high computational and memory overhead. We introduce BiHDTrans, an efficient neurosymbolic binary hyperdimensional Transformer that integrates self-attention into the HD computing paradigm, unifying the representational efficiency of HD computing with the temporal modeling power of Transformers. Empirically, BiHDTrans outperforms state-of-the-art (SOTA) HD computing models by at least 14.47% and achieves 6.67% higher accuracy on average than SOTA binary Transformers. With hardware acceleration on FPGA, our pipelined implementation leverages the independent and identically distributed properties of high-dimensional representations, delivering 39.4 times lower inference latency than SOTA binary Transformers. Theoretical analysis shows that binarizing in holographic high-dimensional space incurs significantly less information distortion than directly binarizing neural networks, explaining BiHDTrans's superior accuracy. Furthermore, dimensionality experiments confirm that BiHDTrans remains competitive even with a 64% reduction in hyperspace dimensionality, surpassing SOTA binary Transformers by 1-2% in accuracy with 4.4 times less model size, as well as further reducing the latency by 49.8% compare to the full-dimensional baseline. Together, these contributions bridge the gap between the expressiveness of Transformers and the efficiency of HD computing, enabling accurate, scalable, and low-latency MTS classification.
Jingtao Zhang, Yi Liu, Qi Shen +1
School of Biomedical Engineering, Shenzhen Campus of Sun Yat-sen University · College of Medicine and Biological Information Engineering, Northeastern University
Vector Symbolic Architectures (VSAs) provide a well-defined algebraic framework for compositional representations in hyperdimensional spaces. We introduce HyperSpace, an open-source framework that decomposes VSA systems into modular operators for encoding, binding, bundling, similarity, cleanup, and regression. Using HyperSpace, we analyze and benchmark two representative VSA backends: Holographic Reduced Representations (HRR) and Fourier Holographic Reduced Representations (FHRR). Although FHRR provides lower theoretical complexity for individual operations, HyperSpaces modularity reveals that similarity and cleanup dominate runtime in spatial domains. As a result, HRR and FHRR exhibit comparable end-to-end performance. Differences in memory footprint introduce additional deployment trade-offs where HRR requires approximately half the memory of FHRR vectors. By enabling modular, system-level evaluation, HyperSpace reveals practical trade-offs in VSA pipelines that are not apparent from theoretical or operator-level comparisons alone.
Shay Snyder, Andrew Capodieci, David Gorsich +1
George Mason University · George Mason University, Fairfax, Virginia, USA · Neya Robotics +3