Growth Operator

Growth operators are techniques for efficiently training increasingly larger deep learning models, particularly transformers, by leveraging knowledge from smaller pretrained versions. Current research focuses on developing sophisticated growth operators that preserve both the model's function and its training dynamics, optimizing growth schedules across multiple model dimensions (depth and width), and applying these methods to both language and vision models. These advancements significantly reduce the computational cost of training large models, accelerating progress in various fields and making the development of even more powerful AI systems more feasible.

Papers

May 4, 2023

Masked Structural Growth for 2x Faster Language Model Pre-training
Yiqun Yao, Zheng Zhang, Jing Li, Yequan Wang
Language Model Structure Enhancement Growth Operator

March 2, 2023

Learning to Grow Pretrained Models for Efficient Transformer Training
Peihao Wang, Rameswar Panda, Lucas Torroba Hennigen, Philip Greengard, Leonid Karlinsky, Rogerio Feris, David Daniel Cox, Zhangyang Wang, Yoon Kim
LeArning Abstract Vision Transformer Transformer Training Scaling Transformer Growth Operator

March 11, 2022

Staged Training for Transformer Language Models
Sheng Shen, Pete Walsh, Kurt Keutzer, Jesse Dodge, Matthew Peters, Iz Beltagy
Training Data Transformer Language Model Autoregressive Language Model Growth Operator

Growth Operator

Papers

Masked Structural Growth for 2x Faster Language Model Pre-training

Learning to Grow Pretrained Models for Efficient Transformer Training

Staged Training for Transformer Language Models