Deep Learning Optimization

It studies how parameter updates, learning-rate and batch schedules, regularization, and model structure shape neural-network training. Common approaches range from SGD and AdamW to adaptive preconditioners such as Shampoo, curvature-aware and sharpness-aware methods, and matrix or spectral optimizers such as Muon; theory examines convergence, stability, and geometry, while systems work reduces memory, communication, and step time. Evaluation spans vision and tabular models, language-model pretraining, diffusion, and continual learning, measuring accuracy and loss alongside time to target and memory use. Optimizer rankings depend on architecture, batch size, training horizon, and compute budget, leaving a tension between per-step progress, wall-clock efficiency, generalization, and reliable tuning.