Organizations: Tampere University, Tampere, Finland · Tampere Institute for Advanced Study, Tampere University, Finland · Intel Corporation, Santa Clara, CA, United States
Overparameterized machine learning (ML) methods such as neural networks may be prohibitively resource intensive for devices with limited computational capabilities. Hyperdimensional computing (HDC) is an emerging resource efficient and low-complexity ML method that allows hardware efficient implementations of (re-)training and inference procedures. In this paper, we propose a maximum-margin HDC classifier, which significantly outperforms baseline HDC methods on several benchmark datasets. Our method leverages a formal relation between HDC and support vector machines (SVMs) that we established for the first time. Our findings may inspire novel HDC methods with potentially more hardware-oriented implementations compared to SVMs, thus enabling more efficient learning solutions for various intelligent resource-constrained applications.
Figures & tables
Notation
Description
N
Number of data points
d
Dimensionality of data points
X∈Rd×N
Set of d -dimensional data points
y∈RN
Vector of corresponding labels, yj∈{−1,+1}
xj∈Rd
Individual i -th data point
yj
Label of individual i -th data point
TABLE I: Notations and descriptions used in this work
Fig. 1: Comparison of performance of maximum-margin HDC (blue), linear C -SVM (orange), OnlineHD (green), perceptron-based HDC (red), and other algorithms.
Hyperparameter
Dataset
MNIST
FASHION
UCI HAR
All methods
HD representation size, D
5000
Batch size
1000
Maximum-margin HDC
Learning rate, α
1⋅10−5
TABLE II: Parameters of considered HDC and SVM methods.
Classifier
Enc. type
Format
Accuracy
MNIST
FASHION
UCI HAR
HDC methods
Binary MM-HDC (OvO)
NL
FP32
0.979
0.885
0.948
OnlineHD
NL
FP32
0.966
0.859
0.934
Perceptron-based HDC
NL
FP32
0.966
0.858
0.933
FedHDC [ 12 , 13 ]
NL
BP/FP32
0.968
-
0.945
TABLE III: Classification performance comparison of MM-HDC with the baselines in terms of mean accuracy scores at the last epoch.
Fig. 2: Performance comparison of maximum-margin HDC (blue), linear C -SVM (orange), OnlineHD (green), and perceptron-based HDC (red) for MNIST dataset and different sizes of hypervectors D .