Natural Gradient
Natural gradients are a powerful tool in optimization, aiming to improve the efficiency and stability of training complex models by accounting for the underlying geometry of the parameter space. Current research focuses on applying natural gradient methods to diverse areas, including distributed learning (e.g., through gradient compression and efficient client selection), inverse problems (using diffusion models), and neural network training (e.g., via regularization and novel optimizers like DiffGrad and AdEMAMix). These advancements have significant implications for improving the performance and robustness of machine learning models across various applications, from image processing and medical image analysis to scientific computing and federated learning.
Papers
SODA: Self-organizing data augmentation in deep neural networks -- Application to biomedical image segmentation tasks
Arnaud Deleruyelle, John Klein, Cristian Versari
Gradient boosting machines and careful pre-processing work best: ASHRAE Great Energy Predictor III lessons learned
Clayton Miller, Liu Hao, Chun Fu