Distributional Learning

Follow this topic Ask about this topic

Recent momentum

-74%

11 papers in the last 28 days · 0.3% of indexed attention

Twelve weeks of publication activity for this topic as it is defined today.

Weekly history

Recent digests

What was published in this topic, kept on the site without email delivery.

Period ending 2026-09-14

6 new papers

A weekly snapshot of new work published in Distributional Learning.

Period ending 2026-09-07

3 new papers

A weekly snapshot of new work published in Distributional Learning.

306 papers

Latest in Distributional Learning

Open your feed →

The Limits and Potentials of Local SGD for Distributed Heterogeneous Learning with Intermittent Communication
May 19, 2024Kumar Kshitij Patel, Margalit Glasgow, Ali Zindari +5Stochastic Gradient Descent Heterogeneity
Convergence Analysis of Sequential Federated Learning on Heterogeneous Data
Nov 6, 2023Yipeng Li, Xinchen LyuOptimization Convergence Analysis Distributional Learning
Consistent Distributed Ranking of Generative Models via Kernel Distances
Oct 18, 2023Zixiao Wang, Farzan Farnia, Zhenghao Lin +2Generalizable Ranking Purpose Model Evaluation
GFlowNets and variational inference
Oct 2, 2022Esmeralda S. Whitammer, Salem Lahlou, Tristan Deleu +5Variational Inference Generative Flow Networks
Class-wise Contribution Estimation via Logit Maximization for Federated Learning
Date pendingAsim Ukaye, Nurbek Tastan, Mubarak Abdu-Aguye +1Extreme Class Imbalance Distributional Learning
Guided Adversarial Robust Transfer Learning with Source Mixing
Date pendingXin Xiong, Zijian Guo, Tianxi CaiTransfer Learning Adversarial Transferability

May 19, 2024 · cs.LGJ/K move · Enter open · S save

The Limits and Potentials of Local SGD for Distributed Heterogeneous Learning with Intermittent Communication

Kumar Kshitij Patel, Margalit Glasgow, Ali Zindari, Lingxiao Wang+4

Local SGD is a popular optimization method in distributed learning, often outperforming other algorithms in practice, including mini-batch SGD. Despite this success, theoretically proving the dominance of local SGD in settings with reasonable data heterogeneity has been difficult, creating a significant gap between theory and practice. In this paper, we provide new lower bounds for local SGD under existing first-order data heterogeneity assumptions, showing that these assumptions are insufficient to prove the effectiveness of local update steps. Furthermore, under these same assumptions, we demonstrate the min-max optimality of accelerated mini-batch SGD, which fully resolves our understanding of distributed optimization for several problem classes. Our results emphasize the need for better models of data heterogeneity to understand the effectiveness of local SGD in practice. Towards this end, we consider higher-order smoothness and heterogeneity assumptions, providing new upper bounds that imply the dominance of local SGD over mini-batch SGD when data heterogeneity is low.

Stochastic Gradient Descent Heterogeneity Distributional Learning Optimality Upper Bounds

Open paper Ask about it PDF

Recent momentum

Recent digests

6 new papers

3 new papers

Latest in Distributional Learning

The Limits and Potentials of Local SGD for Distributed Heterogeneous Learning with Intermittent Communication

Convergence Analysis of Sequential Federated Learning on Heterogeneous Data

Consistent Distributed Ranking of Generative Models via Kernel Distances

GFlowNets and variational inference

Class-wise Contribution Estimation via Logit Maximization for Federated Learning

Guided Adversarial Robust Transfer Learning with Source Mixing