Asymptotic Properties of Support Vector Machines in High-Dimension, Low-Sample-Size Settings under a Spiked Model

Yugo Nakayama

stat.MLSep 30, 2026

Asymptotic Properties of Support Vector Machines in High-Dimension, Low-Sample-Size Settings under a Spiked Model

Authors: Yugo Nakayama

Organizations: Independent researcher

Abstract

In this paper, we consider asymptotic properties of the support vector machine (SVM) in high-dimension, low-sample-size (HDLSS) settings under a spiked model. The existing theory of the SVM in the HDLSS context relies on the geometric representation of HDLSS data, which requires that the eigenvalues of the covariance matrices are not dominant. We first show that the geometric representation does not hold under the spiked model. We show that the Gram matrix of HDLSS data converges in distribution to a random matrix, namely, the HDLSS data converge to a random configuration in a finite-dimensional space whose dimension is given by the number of the spikes. We show that the misclassification rates of the SVM do not tend to zero, that is, the SVM does not hold the consistency property. We also show that the bias-corrected SVM (BC-SVM) does not give preferable performance in this setting because the bias term itself should be modified. In order to overcome such difficulties, we propose a spike-corrected SVM (SC-SVM). We show that the SC-SVM holds the consistency property when the sample size goes to infinity, and that the growth of the sample size is essential in the sense that any projection-based procedure fails when the sample size is fixed. Finally, we check the performance of the classifiers by numerical simulations.

Figures & tables

Term	Theorem 4.1 (training vs training)	Theorem 6.1 (training vs new)
(I) deterministic	$\varepsilon_{i}\varepsilon_{k}\delta/4$	$\varepsilon_{i}\delta/4$ (since $\varepsilon_{0}=1$ )
(II), (III) linear	Lemma 3.3 on both sides	Lemma 3.3 on both sides, one being $z_{0}$
(IV), distinct	( 3.1 ) or Lemma 3.2	always this case
(IV), identical	( 3.2 ), giving $+c_{i0}$	never occurs, no $c_{i0}$

Table 1. Correspondence between the proofs of Theorems 4.1 and 6.1 .

$c$	$\delta$	limit ( 7.2 )	simulation	difference
0.1	0.0	0.5009	0.4993	0.0016
0.1	1.0	0.0112	0.0122	0.0010
0.1	4.0	0.0000	0.0000	0.0000
0.3	0.0	0.5009	0.4998	0.0011
0.3	1.0	0.1093	0.1091	0.0002
0.3	4.0	0.0039	0.0047	0.0008

Table 2. The limiting misclassification rate ( 7.2 ) and the simulated

E\{e(1)\}

for

d=20000

in the setting (E).

$d$	$\\|G-G^{*}\\|_{F}$	s.e.	ratio to previous
100	0.29956	0.00783	–
1,000	0.09099	0.00245	3.292
10,000	0.03138	0.00077	2.900
100,000	0.01039	0.00027	3.021
1,000,000	0.00299	0.00007	3.471

Table 3. The Frobenius norm

\|G-G^{*}\|_{F}

and the ratio of the successive values.

Figure 1. Average of

e(1)+e(2)

for the hard-margin SVM vs.

d

(

n_{1}=n_{2}=2

\delta=0.25

$n$	$c=0.8$	$c=0.5$
2	0.2684	0.3467
3	0.0771	0.1589
5	0.0154	0.0424
10	0.0000	0.0003

Table 4. The average of

e(1)+e(2)

for

d=4{,}000

and

\delta=0.25

$n_{1}$	$n_{2}$	$c_{1}$	$c_{2}$	$\kappa/\theta$	$\kappa_{*}/\theta$	$b$ (s.e.)	$A$	pred. $\kappa$	pred. $\kappa_{*}$
5	5	0.2	0.8	0.0000	0.1200	$-0.0899$ (0.0017)	1.644	0.0000	$-0.0987$
4	4	0.1	0.7	0.0000	0.1500	$-0.0921$ (0.0029)	1.492	0.0000	$-0.1119$
10	5	0.4	0.7	$-0.1000$	0.0000	$0.0071$ (0.0011)	1.766	0.0883	0.0000
8	4	0.3	0.65	$-0.1250$	0.0000	$0.0103$ (0.0015)	1.669	0.1043	0.0000
8	4	0.5	0.5	$-0.1250$	$-0.0625$	$0.0557$ (0.0012)	1.657	0.1035	0.0518

Table 5. The measured intercept

b

and the two predictions in ( 11.2 ).

	SVM	BC-SVM( $\kappa$ )	BC-SVM( $\kappa_{*}$ )	SC-SVM	SC-SVM (split)
non-spiked	1.0000 (0.0000)	0.0000 (0.0000)	0.0000 (0.0000)	0.0015 (0.0001)	0.0000 (0.0000)
$c=0.2$	0.7887 (0.0135)	0.2504 (0.0255)	0.1600 (0.0200)	0.1973 (0.0206)	0.3684 (0.0279)
$c=0.5$	0.3137 (0.0191)	0.4071 (0.0383)	0.1062 (0.0175)	0.1185 (0.0171)	0.2858 (0.0273)
$c=0.8$	0.0347 (0.0099)	0.4946 (0.0454)	0.0220 (0.0082)	0.0170 (0.0055)	0.0710 (0.0131)

Table 6. The average of

e(1)+e(2)

for

d=20{,}000

n_{1}=12

n_{2}=6

and

\delta=0.05

. Standard errors in parentheses.

Figure 2. Average of

e(1)+e(2)

for

d=20{,}000

c=0.8

\delta=0.01

, as

(n_{1},n_{2})

grows.

Feature	non-spiked model	spiked model
eigenvalue condition	$\tr(\Sigma_{i}^{2})/\theta_{i}^{2}\to 0$	$\lambda_{is}/\theta\to c_{is}\in(0,1)$
data configuration	deterministic regular simplex	random configuration in $\mathbb{R}^{1+m_{1}+m_{2}}$
Gram matrix	deterministic limit	random $G^{*}$
positive definiteness	from the geometric representation	only from the diagonal $D$
SVM solution	deterministic, closed form	random limit
noise of $x_{0}$	contributes as a diagonal term	does not contribute

Table 7. Comparison between the non-spiked model and the spiked model.

Explore similar work

Elite-Driven Support Vector Machines for Classification
Apr 28, 2026Mohammad Jafari Jozani, Bahram MoeinianfarSupport Vector Machines Empirical Risk Minimization
Department of Statistics, University of Manitoba, Winnipeg, Manitoba, CANADA, R3T2N2
Robust and sparse support vector machine via hybrid truncated loss for supervised classification
Jun 4, 2026Yuliang Yang, Chen Chen, Yuxiang Liu +1Support Vector Machines Convex Loss
School of Science, Beijing Forestry University, No.35 Qinghua East Road, 100083 Haidian, Beijing, China · Translational Cancer Research Center, Peking University First Hospital, No. 8 Xishiku Street, 100034 Xicheng, Beijing, China
High-dimensional ridgeless least squares interpolation under spiked covariance structures
Aug 7, 2026Zhijun Liu, Dandan JiangKernel Ridge Regression Covariance
College of Sciences, Northeastern University, China. · School of Mathematics and Statistics, Xi’an Jiao Tong University, China.

Apr 28, 2026, stat.MLJ/K move · Enter open · S save

Elite-Driven Support Vector Machines for Classification

Mohammad Jafari Jozani, Bahram Moeinianfar

Department of Statistics, University of Manitoba, Winnipeg, Manitoba, CANADA, R3T2N2

Support Vector Machines Empirical Risk Minimization Mle-Bench Lite

Open paper AskPDF arXiv

Support vector machines (SVMs) are a standard tool for binary classification, but their classical formulations are purely data-driven and offer no direct way to encode trusted benchmark models or structured preferences on selected subsets of the data. We propose Elite-Driven Support Vector Machines (EDSVM), a general framework that augments regularized empirical risk minimization by guiding the slack variables for a curated set of elite observations (typically the union of support vectors from one or more reference SVMs). EDSVM combines the usual slack loss with a deviation penalty that shrinks new slacks toward benchmark slack values, defining a localized, margin-aligned notion of proximity to reference models, unlike global function penalties in knowledge distillation or teacher-student methods, and without requiring privileged features as in SVM+/LUPI. Within this framework we develop two concrete models, C-EDSVM and LS-EDSVM, based respectively on hinge-type and squared-slack losses. For both variants we derive dual quadratic programs that can be implemented with modest modifications of standard SVM solvers, and we give simple sufficient conditions under which the induced margin losses are classification calibrated. Simulation studies and experiments on several UCI benchmarks show that EDSVMs closely track the behaviour induced by reference SVMs while achieving predictive performance that is competitive with, and sometimes better than, C-SVM, LINEX-SVM, and LS-SVM.

Abstract

Explore similar work

Elite-Driven Support Vector Machines for Classification

Robust and sparse support vector machine via hybrid truncated loss for supervised classification

High-dimensional ridgeless least squares interpolation under spiked covariance structures