Online Gradient Descent
Online gradient descent (OGD) is an iterative optimization method used to efficiently update model parameters based on sequentially arriving data, aiming to minimize cumulative error over time. Current research focuses on improving OGD's efficiency and robustness in various settings, including handling outliers, adapting to non-convex loss functions, and managing memory constraints for large-scale models like LLMs, often employing techniques like subspace descent and variance reduction. These advancements are significant for applications requiring real-time learning from streaming data, such as online control systems, recommendation systems, and large language model training, where efficient and robust optimization is crucial.
20papers
Papers
February 18, 2025
February 3, 2025
December 26, 2024
August 23, 2024
February 6, 2024
February 2, 2024
October 21, 2023
October 10, 2023
January 17, 2023