Learned Optimizers

Learned optimizers (LOs) aim to replace traditional, hand-designed optimization algorithms with machine-learned counterparts, ultimately accelerating and improving the training of machine learning models and solving complex optimization problems. Current research focuses on improving the generalization capabilities of LOs across diverse tasks and architectures, exploring novel architectures like Transformers and employing techniques such as meta-learning, PAC-Bayesian bounds, and hybrid approaches combining learned and classical optimizers. This field holds significant promise for reducing computational costs in training large models, enhancing the efficiency of various optimization problems in diverse fields like robotics and control systems, and providing theoretical guarantees for the performance of learned optimization algorithms.