Representative Learning for Distributed Support Vector Machines
Organizations: Department of Mathematics, University of Alabama at Birmingham · Department of Statistics and Actuarial Science, University of Iowa
Abstract
Distributed support vector machine (SVM) learning is challenging under heterogeneous local data and limited communication. We develop a Representative Learning framework based on Lifted Empirical Risk Minimization, replacing each covariate by a local Gaussian distribution. For linear SVMs, the resulting expected hinge loss has an exact closed form, is smooth for nondegenerate lifting covariances, and converges to the ordinary hinge loss as the covariance vanishes. The lifted formulation enables clouded representatives that encode both location and covariance. We study a model-independent Mean Representative (MR) and a Score-Matching Representative (SMR) that matches the class-wise lifted gradient at its construction parameter. We show that SMR covariance scaling is stable with respect to within-group heterogeneity and that the gradient discrepancy between full data and frozen representatives is controlled by the maximum within-group diameter. Under strong convexity and smoothness, this yields an inexact-gradient convergence guarantee. Experiments under heterogeneous class allocation show that lifted representatives remain competitive with centralized solutions, while SMR more closely recovers the centralized lifted-SVM optimizer. We further extend the framework to nonlinear SVMs through a shared kernel dictionary, with experiments on Ionosphere and Australian Credit Approval showing analogous behavior. These results provide a distributional representative framework for communication-efficient distributed SVM learning under heterogeneity.
Figures & tables
| Regime | Objective | Iterations to | Final accuracy |
| Standard | Lifted hinge | 22.5 | 0.9353 |
| Ordinary hinge | 81.4 | 0.9356 | |
| Few balanced | Lifted hinge | 24.9 | 0.9648 |
| Ordinary hinge | 130.1 | 0.9648 | |
| Heavy imbalance | Lifted hinge | 46.2 | 0.9765 |
| Ordinary hinge | 81.4 | 0.9771 |
| Method | ||||
| MR (Hinge) | ||||
| SMR (Hinge) | ||||
| SMR (Lifted) | ||||
| MR (Lifted) | ||||
| FedAvg (Lifted) | ||||
| FedProx (Lifted) |
| Communication regime | ||||
| Moderate | ||||
| Single communication | ||||
| Frequent |
| Setting | Method | ||||
| Low overlap | SMR (Lifted) | ||||
| SMR (Hinge) | |||||
| MR (Lifted) | |||||
| MR (Hinge) | |||||
| High overlap | SMR (Lifted) | ||||
| SMR (Hinge) |
| Dataset | Method | |||
| Ionosphere | MR (Hinge) | |||
| SMR (Hinge) | ||||
| SMR (Lifted) | ||||
| MR (Lifted) | ||||
| FedAvg (Lifted) | ||||
| FedProx (Lifted) |
| Dataset | |||
| Ionosphere | |||
| Australian |
| Dataset | Method | |||
| Ionosphere | MR (Hinge) | |||
| MR (Lifted) | ||||
| SMR (Hinge) | ||||
| SMR (Lifted) | ||||
| Australian | MR (Hinge) | |||
| MR (Lifted) |
Support Vector Machines
Support vector machines learn margin-based classifiers and regressors using linear or kernel-defined decision boundaries.
See all 46 papers