cs.LGJul 16, 2026

Regularity-Aware Stochastic MGDA with Adaptive Per-Iterate Conflict-Avoidant Update

Authors: Chentong Huang, Lisha Chen

Organizations: Dept. of Electrical and Computer Engineering, University of Rochester, Rochester, NY 14627

Abstract

Multi-objective learning (MOL) aims to optimize multiple objectives simultaneously. The multi-gradient descent algorithm (MGDA) constructs a common descent or conflict-avoidant (CA) direction by solving a gradient-aggregation subproblem. In stochastic settings, the vanilla stochastic MGDA method, SMG, replaces full-batch gradients with mini-batch estimates, causing bias in the resulting CA direction. Existing analyses control this error using continuity of the CA-direction mapping, but do not distinguish how its sensitivity varies with the geometry of the MGDA subproblem. Motivated by this limitation, we show that the MGDA direction map is Lipschitz continuous under reduced nondegeneracy, whereas its general 1/21/2-Hölder exponent on bounded sets is sharp. This geometry motivates the Multi-objective Regularity-aware (MoRe) method, which retains adaptive MGDA aggregation in the regular regime and switches to a fixed-weight linear scalarization (LS) near degeneracy. The fallback branch skips the MGDA subproblem solve, which can reduce aggregation cost on fallback iterations while avoiding a potentially noise-sensitive update. Theoretically, we establish a per-iterate CA guarantee, and a O~(T−1)\widetilde{\mathcal O}(T^{-1}) convergence rate, improving upon the previously established O~(T−1/4)\widetilde{\mathcal O}(T^{-1/4}) SMG guarantee, where O~\widetilde{\mathcal O} hides logarithmic factors. Experiments demonstrate favorable multi-task performance and convergence behavior consistent with the theoretical rate.

Figures & tables

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. A Unified Framework for Gradient Aggregation in Multi-Objective Optimization

    May 28, 2026Zeou Hu, Kelvin Ho, Yaoliang YuGradient DescentStochastic Optimization Convergence

  2. Geometry-Aware Multi-Objective Optimization: Norm-Constrained Oracles with Entropic Weight Updates

    Oct 9, 2026El Mahdi Chayti, Alireza Mirrokni, Martin JaggiLMO MethodsMulti-Objective Optimization

  3. From Dual Tracking to Clipping: Provably Faster Distributionally Robust Multi-Objective Optimization

    May 7, 2026Yufeng Yang, Fangning Zhuo, Ziyi Chen +2Gradient ClippingGradient Descent