Optimization Trajectory

Optimization trajectory analysis focuses on understanding and improving the path taken by parameters during the training of machine learning models, aiming to enhance efficiency and performance. Current research explores techniques like weight scaling for neural fields, Hessian suppression in meta-learning, and trajectory reweighting for adversarial training, often employing gradient-based methods and analyzing the geometry of the loss landscape. These advancements are significant because they lead to faster training, improved generalization, and more robust models across various applications, including few-shot learning and design optimization. The insights gained are crucial for developing more efficient and effective machine learning algorithms.

Papers

October 7, 2024

Fast Training of Sinusoidal Neural Fields via Scaling Initialization
Taesun Yeom, Sangyoon Lee, Jaeho Lee
Neural Network Neural Field Faster Training Spectral Bias Sinusoidal Activation Optimization Trajectory

October 4, 2023

SHOT: Suppressing the Hessian along the Optimization Trajectory for Gradient-Based Meta-Learning
JunHoo Lee, Jayeon Yoo, Nojun Kwak
Hessian Matrix Gradient Based Meta Learning Hypothesis Transfer Optimization Trajectory

June 25, 2023

Enhancing Adversarial Training via Reweighting Optimization Trajectory
Tianjin Huang, Shiwei Liu, Tianlong Chen, Meng Fang, Li Shen, Vlaod Menkovski, Lu Yin, Yulong Pei, Mykola Pechenizkiy
Adversarial Training Better Adversarial Robustness Adversarial Weight Perturbation Optimization Trajectory

June 20, 2023

No Wrong Turns: The Simple Geometry Of Neural Networks Optimization Paths
Charles Guille-Escuret, Hiroki Naganuma, Kilian Fatras, Ioannis Mitliagkas
Loss Landscape Dynamic Optimization First Order Optimization Geometric Theory Optimization Trajectory

May 29, 2023

Aligning Optimization Trajectories with Diffusion Models for Constrained Design Generation
Giorgio Giannone, Akash Srivastava, Ole Winther, Faez Ahmed
Diffusion Model Generative Model Topology Optimization Constrained Generation Data Driven Design Optimization Trajectory

March 31, 2023

Scalable Bayesian Meta-Learning through Generalized Implicit Gradients
Yilang Zhang, Bingcong Li, Shijian Gao, Georgios B. Giannakis
Meta Learning Inner Optimization Scalable Bayesian Optimization Trajectory Implicit Gradient