Hessian Free

Hessian-free methods aim to overcome the computational burden of using second-order information (Hessian matrices) in optimization problems prevalent in machine learning, particularly for large-scale models. Current research focuses on developing efficient algorithms that approximate or avoid direct Hessian computation in various contexts, including Bayesian deep learning, bi-level optimization, meta-learning, and unlearning, often employing techniques like first-order approximations, stochastic gradient descent with partial Hessian updates, and novel reformulations of optimization problems. These advancements enhance the scalability and practicality of second-order optimization methods, leading to improved performance in diverse applications such as neural architecture search, recommender systems, and robust model training against adversarial attacks.