Nonconvex Stochastic Optimization

The field studies how to find useful approximate solutions when objectives may have many local minima and gradients are estimated from sampled data, as in neural-network training. Common approaches include stochastic gradient descent (SGD), momentum and variance reduction, adaptive preconditioning, and robust methods such as gradient clipping for heavy-tailed noise; related work handles constraints, minimax objectives, and federated or asynchronous training. Researchers compare methods using stationarity measures and oracle or communication complexity, alongside experiments on tasks such as image classification, language-model training, and distributionally robust learning. A central tension is that strong guarantees often depend on assumptions about smoothness and noise that practical training may violate, while methods that work well empirically can be difficult to analyze or tune.