Paper ID: 2410.05871

A second-order-like optimizer with adaptive gradient scaling for deep learning

Jérôme Bolte (TSE-R), Ryan Boustany (TSE-R), Edouard Pauwels (TSE-R, IRIT-ADRIA), Andrei Purica

In this empirical article, we introduce INNAprop, an optimization algorithm that combines the INNA method with the RMSprop adaptive gradient scaling. It leverages second-order information and rescaling while keeping the memory requirements of standard DL methods as AdamW or SGD with this http URL having recalled our geometrical motivations, we provide quite extensive experiments. On image classification (CIFAR-10, ImageNet) and language modeling (GPT-2), INNAprop consistently matches or outperforms AdamW both in training speed and accuracy, with minimal hyperparameter tuning in large-scale settings. Our code is publicly available at \url{this https URL}.

Submitted: Oct 8, 2024