Weight Perturbation

Weight perturbation, the deliberate or incidental alteration of model weights, is a research area focusing on improving model robustness, generalization, and efficiency in various machine learning contexts. Current research investigates the effects of weight perturbations on deep neural networks (DNNs), large language models (LLMs), and linear solvers, employing techniques like multiplicative perturbations, adversarial weight perturbation, and random weight perturbation to achieve goals such as enhanced robustness to corruptions, improved pruning stability, and better out-of-distribution detection. These studies highlight the vulnerability of some models to even small perturbations and underscore the need for developing more robust and reliable algorithms, impacting areas like medical image analysis and federated learning.