stat.MLApr 28, 2026

Elite-Driven Support Vector Machines for Classification

Authors: Mohammad Jafari Jozani, Bahram Moeinianfar

Organizations: Department of Statistics, University of Manitoba, Winnipeg, Manitoba, CANADA, R3T2N2

Abstract

Support vector machines (SVMs) are a standard tool for binary classification, but their classical formulations are purely data-driven and offer no direct way to encode trusted benchmark models or structured preferences on selected subsets of the data. We propose Elite-Driven Support Vector Machines (EDSVM), a general framework that augments regularized empirical risk minimization by guiding the slack variables for a curated set of elite observations (typically the union of support vectors from one or more reference SVMs). EDSVM combines the usual slack loss with a deviation penalty that shrinks new slacks toward benchmark slack values, defining a localized, margin-aligned notion of proximity to reference models, unlike global function penalties in knowledge distillation or teacher-student methods, and without requiring privileged features as in SVM+/LUPI. Within this framework we develop two concrete models, C-EDSVM and LS-EDSVM, based respectively on hinge-type and squared-slack losses. For both variants we derive dual quadratic programs that can be implemented with modest modifications of standard SVM solvers, and we give simple sufficient conditions under which the induced margin losses are classification calibrated. Simulation studies and experiments on several UCI benchmarks show that EDSVMs closely track the behaviour induced by reference SVMs while achieving predictive performance that is competitive with, and sometimes better than, C-SVM, LINEX-SVM, and LS-SVM.

Explore similar work

CardsList
Jun 4, 2026cs.LG

Robust and sparse support vector machine via hybrid truncated loss for supervised classification

The support vector machine (SVM) is a widely used classifier, but choosing an appropriate loss function remains difficult. Convex losses such as the hinge loss and least-squares loss are sensitive to outliers, while bounded non-convex losses often lead to high computational cost. To address this, we propose a hybrid truncated loss function (LhtL_{\mathrm{ht}}) that is both sparse and bounded, and build the LhtL_{\mathrm{ht}}-SVM model for single-view classification. We introduce the P-stationary point and use it to establish the first-order necessary and sufficient optimality conditions. Based on these conditions, we design an alternating direction method of multipliers with a working-set strategy that reduces computational cost and achieves global convergence. We further extend LhtL_{\mathrm{ht}}-SVM to multi-view learning by adding structural information and view weights, resulting in MvLhtL_{\mathrm{ht}}-SVM, which follows both the consensus and complementarity principles. Experiments on synthetic, real-world, and image datasets show that LhtL_{\mathrm{ht}}-SVM achieves higher accuracy with fewer support vectors and better noise robustness than five single-view methods, while MvLhtL_{\mathrm{ht}}-SVM outperforms six multi-view methods in accuracy, precision, recall, and F1-score.
Yuliang Yang, Chen Chen, Yuxiang Liu +1
Loss FunctionMulti-ViewConvex LossSupervised Learning
AskPDF
Aug 2, 2026cs.LG

Data-Driven Pinball-Loss Selection for Vertically Distributed Elastic-Net SVMs

The pinball-loss support vector machine is robust, but its asymmetry parameter is usually fixed in advance. We propose a data-driven elastic-net support vector machine that learns simplex-constrained weights over candidate pinball losses while retaining one classifier. The weighted loss is equivalent to a pinball loss with a data-dependent effective parameter. An empirical oracle inequality shows that, when weight regularization and simplex truncation vanish, the classifier objective at a global minimizer does not exceed that of the best fixed candidate; otherwise, the excess is explicitly bounded. For high-dimensional data, we develop a column-partitioned variable-splitting solver. It converges with a best-iterate O(1/T)O(1/T) squared-step residual rate. Under common initialization and global parameters, any column partition produces, in exact arithmetic, the same iterates and solution as centralized training. Experiments assess predictive behavior, numerical equivalence, and multi-process scalability.
Xiaofei Wu, Kai Qi, Rongmei Liang
Loss Function
AskPDF
Jun 20, 2026cs.LG

Sequential Minimal Optimization Algorithm for One-Class Support Vector Machines With Privileged Information

One of the powerful techniques in data modeling is accounting for features that are available at the training stage, but are not available when the trained model is used to classify or predict test data -- the Learning Using Privileged Information paradigm (LUPI). Sequential Minimal Optimization (SMO) methods have been developed for supervised Support Vector Machines (SVM), unsupervised one-class SVM, and SVM with privileged information (SVM+). The missing brick in this research has long been a one-class SVM with privileged information (OC-SVM+). In this paper, we propose an SMO algorithm for OC-SVM+ that significantly outperforms non-sequential algorithms for training the OC-SVM+ model. Its finite-time convergence is established. The experiments show how privileged information affects a descriptive domain in the space of original features. Comparative benchmark tests demonstrate that our algorithm is superior over interior point algorithms.
Andrey Lange, Dmitry Smolyakov, Evgeny Burnaev
Semi-Supervised Learning
AskPDF