Subspace Training

Subspace training focuses on improving the efficiency and robustness of deep neural network training by restricting the model's weight updates to lower-dimensional subspaces. Current research explores efficient subspace identification methods, such as trainable weight averaging, and applies these techniques to enhance various architectures, including Vision Transformers (ViTs), within frameworks like federated learning and adversarial training. This approach offers significant advantages, including reduced computational cost, improved generalization, and increased resilience to data heterogeneity and adversarial attacks, impacting both the scalability and reliability of machine learning models.