Gradient Aggregation

Gradient aggregation techniques aim to efficiently combine gradients from multiple sources, improving the performance and robustness of machine learning models. Current research focuses on developing sophisticated aggregation strategies, such as those incorporating trust weighting based on model similarity or performance metrics, Bayesian uncertainty quantification for improved gradient sensitivity, and stochastic aggregation methods to mitigate issues like gradient vanishing. These advancements are crucial for addressing challenges in federated learning, multi-task learning, and adversarial robustness, ultimately leading to more efficient and reliable machine learning systems across diverse applications.

Papers

April 15, 2024

Personalized Collaborative Fine-Tuning for On-Device Large Language Models
Nicolas Wagner, Dongyang Fan, Martin Jaggi
Large Language Model Language Model Fine Tuning Data Distribution Gradient Aggregation Cooperative Fine Tuning

February 6, 2024

Bayesian Uncertainty for Gradient Aggregation in Multi-Task Learning
Idan Achituve, Idit Diamant, Arnon Netzer, Gal Chechik, Ethan Fetaya
Multi Task Learning Natural Gradient Bayesian Inference Inference Task Bayesian Uncertainty Aggregated Gradient Gradient Aggregation

August 11, 2023

Enhancing Generalization of Universal Adversarial Perturbation through Gradient Aggregation
Xuannan Liu, Yaoyao Zhong, Yuhang Zhang, Lixiong Qin, Weihong Deng
Aggregated Gradient Adversarial Instance Enhancing Generalization Gradient Aggregation

January 17, 2023

Negative Flux Aggregation to Estimate Feature Attributions
Xin Li, Deng Pan, Chengyin Li, Yao Qiang, Dongxiao Zhu
Deep Neural Network Deep Learning Architecture Feature Attribution Attribution Map Aggregated Gradient Gradient Aggregation DNN Prediction

November 29, 2022

Scalable Hierarchical Over-the-Air Federated Learning
Seyed Mohammad Azimi-Abarghouyi, Viktoria Fodor
Potential Scalability Air Aggregation Downlink Communication Gradient Aggregation

June 6, 2022

Interference Management for Over-the-Air Federated Learning in Multi-Cell Wireless Networks
Zhibin Wang, Yong Zhou, Yuanming Shi, Weihua Zhuang
Federated Learning Air Federated Learning Uplink Control Multi Cell Interference Management Gradient Aggregation

April 29, 2022

Bridging Differential Privacy and Byzantine-Robustness via Model Aggregation
Heng Zhu, Qing Ling
Native Robustness Differential Privacy Model Aggregation Byzantine Robust Gradient Aggregation

November 26, 2021

Random-reshuffled SARAH does not need a full gradient computations
Aleksandr Beznosikov, Martin Takáč
Stochastic Gradient Descent Stochastic Gradient Stochastic Recursive Gradient Gradient Aggregation