Gradient Subspace

Gradient subspace methods focus on identifying and exploiting low-dimensional structures within the high-dimensional spaces of model parameters and gradients, primarily to improve efficiency and address challenges in machine learning. Current research emphasizes applications in continual learning (reducing catastrophic forgetting), reinforcement learning (enhancing training efficiency), and federated learning (reducing communication overhead and enabling unlearning). These techniques offer significant potential for improving the scalability, efficiency, and robustness of various machine learning algorithms across diverse applications.

Papers

October 30, 2024

Why Gradient Subspace? Identifying and Mitigating LoRA's Bottlenecks in Federated Fine-Tuning of Large Language Models
Navyansh Mahla, Ganesh Ramakrishnan
Fine Tuning Low Rank Adaptation Major Challenge Bottleneck Rank Aggregation Gradient Subspace

March 4, 2024

Gradient Correlation Subspace Learning against Catastrophic Forgetting
Tammuz Dubnov, Vishal Thengane
Catastrophic Forgetting Class Incremental Learning Efficient Continual Learning Gradient Subspace

January 12, 2024

Identifying Policy Gradient Subspaces
Jan Schneider, Pierre Schumacher, Simon Guist, Le Chen, Daniel Häufle, Bernhard Schölkopf, Dieter Büchler
Policy Gradient Efficient Reinforcement Learning Second Order Optimization Deep Policy Gradient Subspace

November 25, 2023

Class Gradient Projection For Continual Learning
Cheng Chen, Ji Zhang, Jingkuan Song, Lianli Gao
Continual LEArning Catastrophic Forgetting Gradient Update Gradient Projection Gradient Subspace

October 26, 2023

Low-Dimensional Gradient Helps Out-of-Distribution Detection
Yingwen Wu, Tao Li, Xinwen Cheng, Jie Yang, Xiaolin Huang
Distribution Detection Gradient Norm Gradient Information Low Dimensional Representation Gradient Subspace

March 2, 2023

Choosing Public Datasets for Private Machine Learning via Gradient Subspace Distance
Xin Gu, Gautam Kamath, Zhiwei Steven Wu
Gradient Descent Available Datasets Private Stochastic Private Machine Learning Private Model Gradient Subspace

February 24, 2023

Subspace based Federated Unlearning
Guanghao Li, Li Shen, Yan Sun, Yue Hu, Han Hu, Dacheng Tao
Unlearning Framework Harmful Unlearning Federated Unlearning Subspace Prototype Gradient Subspace Gradient Space

June 20, 2022

GiDR-DUN; Gradient Dimensionality Reduction -- Differences and Unification
Andrew Draganov, Tyrus Berry, Jakob Rødsgaard Jørgensen, Katrine Scheel Nellemann, Ira Assent, Davide Mottin
Dimensionality Reduction Qualitative Difference Anti Unification SNE Algorithm Interpretable Embeddings Gradient Subspace

June 16, 2022

Gradient Descent for Low-Rank Functions
Romain Cosson, Ali Jadbabaie, Anuran Makur, Amirhossein Reisizadeh, Devavrat Shah
Gradient Descent Low Rank Simple Function Low Rank Structure Gradient Subspace

February 1, 2022

Recycling Model Updates in Federated Learning: Are Gradient Subspaces Low-Rank?
Sheikh Shams Azam, Seyyedali Hosseinalipour, Qiang Qiu, Christopher Brinton
Centralized Training Advanced Textile Recycling Gradient Subspace Gradient Space