Gradient Descent
Gradient descent is an iterative optimization algorithm used to find the minimum of a function by repeatedly taking steps proportional to the negative of the gradient. Current research focuses on improving its efficiency and robustness, particularly in high-dimensional spaces and with non-convex functions, exploring variations like stochastic gradient descent, proximal methods, and natural gradient descent, often within the context of deep learning models and other complex architectures. These advancements are crucial for training increasingly complex machine learning models and improving their performance in various applications, from image recognition to scientific simulations. A key area of investigation involves understanding and mitigating issues like vanishing/exploding gradients, overfitting, and the impact of data characteristics on convergence.
Papers
Gradient-Based Non-Linear Inverse Learning
Abhishake, Nicole Mücke, Tapio Helin
Optimization Insights into Deep Diagonal Linear Networks
Hippolyte Labarrière, Cesare Molinari, Lorenzo Rosasco, Silvia Villa, Cristian Vega
Condensed Stein Variational Gradient Descent for Uncertainty Quantification of Neural Networks
Govinda Anantha Padmanabha, Cosmin Safta, Nikolaos Bouklas, Reese E. Jones
Technical Report for ICML 2024 TiFA Workshop MLLM Attack Challenge: Suffix Injection and Projected Gradient Descent Can Easily Fool An MLLM
Yangyang Guo, Ziwei Xu, Xilie Xu, YongKang Wong, Liqiang Nie, Mohan Kankanhalli
Task-Specific Preconditioner for Cross-Domain Few-Shot Learning
Suhyun Kang, Jungwon Park, Wonseok Lee, Wonjong Rhee