stat.MLOct 31, 2026

Representative Learning for Distributed Support Vector Machines

Authors: Joseph A. Casey, Boxiang Wang, Keren Li

Organizations: Department of Mathematics, University of Alabama at Birmingham · Department of Statistics and Actuarial Science, University of Iowa

Abstract

Distributed support vector machine (SVM) learning is challenging under heterogeneous local data and limited communication. We develop a Representative Learning framework based on Lifted Empirical Risk Minimization, replacing each covariate by a local Gaussian distribution. For linear SVMs, the resulting expected hinge loss has an exact closed form, is smooth for nondegenerate lifting covariances, and converges to the ordinary hinge loss as the covariance vanishes. The lifted formulation enables clouded representatives that encode both location and covariance. We study a model-independent Mean Representative (MR) and a Score-Matching Representative (SMR) that matches the class-wise lifted gradient at its construction parameter. We show that SMR covariance scaling is stable with respect to within-group heterogeneity and that the gradient discrepancy between full data and frozen representatives is controlled by the maximum within-group diameter. Under strong convexity and smoothness, this yields an inexact-gradient convergence guarantee. Experiments under heterogeneous class allocation show that lifted representatives remain competitive with centralized solutions, while SMR more closely recovers the centralized lifted-SVM optimizer. We further extend the framework to nonlinear SVMs through a shared kernel dictionary, with experiments on Ionosphere and Australian Credit Approval showing analogous behavior. These results provide a distributional representative framework for communication-efficient distributed SVM learning under heterogeneity.

Figures & tables

Explore similar work

Feb 20, 2025

Bringing Generative Learning to Representation Learning: Self-Supervised Transfer Learning as Distribution Matching

Most self-supervised learning objectives defend against collapse but leave the target representation law unspecified. We formulate representation learning as Distribution Matching (DM), learning an augmentation-invariant encoder whose push-forward law matches an explicit reference law. The reference law specifies what the learned representation distribution should look like, while the matching mechanism determines how this target is matched. Within DM, generative learning and representation learning can be viewed as directionally inverse distributional processes: generative learning maps a tractable reference to data, whereas representation learning maps data to a designed reference law. For the Mallows-distance instantiation studied here, we connect the population objective to class-centre separation and classification error and derive a finite-sample bound on downstream classification error. Simulations and image benchmarks show manifold rectification, fine-grained structure and transferability.
Apr 20, 2026

Semantic-based Distributed Learning for Diverse and Discriminative Representations

In large-scale distributed scenarios, increasingly complex tasks demand more intelligent collaboration across networks, requiring the joint extraction of structural representations from data samples. However, conventional task-specific approaches often result in nonstructural embeddings, leading to collapsed variability among data samples within the same class, particularly in classification tasks. To address this issue and fully leverage the intrinsic structure of data for downstream applications, we propose a novel distributed learning framework that ensures both diverse and discriminative representations. For independent and identically distributed (i.i.d.) data, we reformulate and decouple the global optimization function by introducing constraints on representation variance. The update rules are then derived and simplified using a primal-dual approach. For non-i.i.d. data distributions, we tackle the problem by clustering and virtually replicating nodes, allowing model updates within each cluster using block coordinate descent. In both cases, the resulting optimal solutions are theoretically proven to maintain discriminative and diverse properties, with a guaranteed convergence for i.i.d. conditions. Additionally, semantic information from representations is shared among nodes, reducing the need for common neural network architectures. Finally, extensive simulations on MNIST, CIFAR-10 and CIFAR-100 confirm the effectiveness of the proposed algorithms in capturing global structural representations.
May 31, 2026

Local MixVR: Breaking the Communication-Sample Dependence in Distributed Learning

Communication overhead is a crucial bottleneck in scalable distributed learning. While existing methods aim to efficiently utilize data points, such as Local SGD, Minibatch SGD, and their accelerated variants, they still exhibit communication-round complexity that scales with the total number of samples NN. In this paper, we introduce Local MixVR, a distributed framework that integrates local updates with variance-reduction techniques to mitigate local noise. We show that Local MixVR is the first distributed method to eliminate the dependence of communication complexity on NN, achieving a complexity that scales only with the number of workers MM. In common regimes where M<O(N1/4)M<O\left(N^{1/4}\right), Local MixVR outperforms the state-of-the-art Minibatch Accelerated SGD baseline, bridging a long-standing gap in distributed optimization and establishing a new paradigm for communication-efficient training.