Zeroth Order

Zeroth-order optimization (ZOO) focuses on minimizing or maximizing functions where gradient information is unavailable or computationally expensive to obtain, relying instead on function evaluations. Current research emphasizes developing efficient ZOO algorithms for various applications, including federated learning, large language model fine-tuning, and adversarial attacks, often incorporating techniques like variance reduction, momentum, and adaptive step sizes to improve convergence and efficiency. The significance of ZOO lies in its ability to address optimization challenges in high-dimensional spaces and scenarios with privacy constraints, enabling progress in areas like memory-efficient deep learning and privacy-preserving machine learning.