cs.LGAug 2, 2026

Factorized AdaBoost.MH Achieves the Same Convergence Rate as AdaBoost.MH

Authors: Xin ZouJingyuan Xu

Abstract

{AdaBoost.MH} reduces multi-class classification to a collection of binary subproblems and enjoys the classical boosting-type convergence guarantee under a weak learning condition. A more structured variant, Factorized {AdaBoost.MH}, uses base classifiers of the form h(x)=αvφ(x)\mathbf{h}(x)=α\mathbf{v} \bm{\varphi}(x), where a single binary classifier φ\bm{\varphi} is shared across all classes and the label dependence is carried by a vote vector v{±1}K\mathbf{v} \in\{\pm1\}^K. This factorization is algorithmically attractive and achieves better performance in practice, but its convergence depends on whether one can always choose a vote vector with sufficiently large induced binary weight mass. Previous work resolved this question with a lower bound max{1/n,1/2K}\max\{1/n,1/\sqrt{2K}\}, which still leaves a dimension-dependent slowdown relative to the original {AdaBoost.MH} analysis. In this paper, we sharpen this combinatorial step. For the minimax quantity Wn,K\mathfrak{W}_{n,K} governing the factorized edge, we prove Wn,K=Cmin{n+1,K}\mathfrak{W}_{n,K} = C_{\min\{n+1,K\}}, where Cq=1C_q=1 for q=1q=1, Cq=q/(3q4)C_q=q/(3q-4) for even q2q\ge2, and Cq=(q+1)/(3q1)C_q=(q+1)/(3q-1) for odd q2q\ge2. Since Cq1/3C_q\downarrow 1/3, our bounds show that Wn,K=Θ(1)\mathfrak{W}_{n,K}=Θ(1) uniformly over nn and KK. Consequently, Factorized {AdaBoost.MH} achieves the same boosting-type convergence rate as {AdaBoost.MH} up to a universal constant factor, removing the previously suggested additional dependence on nn or KK in the number of boosting rounds.

Explore similar work

Jul 29, 2026cs.LG

Tight Generalization Bound for AdaBoost

In this paper we show that the generalization error of AdaBoost is Θ(dln(nγ2/d)nγ2+ln(1/δ)n)Θ\big(\tfrac{d\ln(nγ^{2}/d)}{nγ^2}+\tfrac{\ln(1/δ)}{n}\big), where γγ is the advantage guaranteed by the weak learner, dd is the VC-dimension of the class containing the weak hypotheses, nn is the sample size, and δδ is the confidence parameter. The contribution of this paper is the upper bound; the matching lower bound follows from prior work. The upper bound proof follows by combining the known fact that AdaBoost outputs a voting classifier whose voting function has zero empirical γ/2γ/2-margin loss with what is, to the best of our knowledge, a new margin-based generalization bound for voting classifiers.
Mikael Møller Høgsgaard
Jun 9, 2026stat.ML

Near-Exponential Convergence Rates for kNN Classification based on Boltzmann Margin

Convergence-rate analysis for classifiers is often conducted under either Tsybakov margin or Massart margin. The former is a relatively weak condition that typically yields polynomial rates, while the latter is substantially stronger but can guarantee exponential rates. In this paper, we introduce a new condition, called Boltzmann margin, that bridges the gap between these two regimes. It is weaker than Massart margin, generally stronger than Tsybakov margin, and can imply many of their properties under suitable conditions. We apply Boltzmann margin to the analysis of kNN classifiers and establish the first near-exponential convergence rates for kNN classification. We also present extensions of the main results and provide numerical evidence supporting the main theoretical implications.
Luyuan Yang, Shayan Shafaei, Chao Lan
Aug 30, 2026cs.LG

Multiclass Linear Perceptrons with Multiplicative Margins

This paper introduces a family of multiclass linear Perceptron classifiers with a multiplicative margin mechanism (MMPerc), as an alternative to standard margin-free and additive margin Perceptrons. The multiplicative formulation enforces classification confidence by requiring the true class score to exceed that of competing classes by a specified fraction of itself, rather than by a fixed additive threshold. This avoids dependence on score magnitudes arising from varied norms of data and class weight vectors. We propose several architectural and algorithmic variants of MMPerc, derive associated loss functions and mistake bounds for both linearly separable and non-separable data, and analyze key design considerations, including bias, margin threshold selection, and training modes. Extensive experiments on synthetic and real datasets show that MMPerc classifiers typically outperform the standard Perceptron, as well as classic baselines such as Support Vector Machines and Ridge classifiers. Owing to their simplicity, minimalistic design, and computational efficiency, MMPerc classifiers are promising candidates for conventional machine learning tasks, linear evaluation of Deep Neural Networks, integration with Hyperdimensional Computing / Vector Symbolic Architecture representations, and deployment in resource-constrained applications.
Dmitri Rachkovskij, Evgeny Osipov, Olexander Volkov +2