First Order Optimization

First-order optimization (FOO) focuses on efficiently finding the minimum or maximum of a function using only its gradient information, a cornerstone of many machine learning and scientific computing applications. Current research emphasizes improving FOO's efficiency and applicability to complex problems, including developing novel algorithms like those incorporating parallel processing, agentic approaches using reinforcement learning, and techniques handling non-convex constraints and streaming data. These advancements are crucial for accelerating training of large models, optimizing complex systems like robots, and enhancing the performance of various algorithms across diverse fields.