Scaling Transformer

Scaling Transformer models focuses on improving their efficiency and performance by increasing their size and capabilities while mitigating the associated computational costs. Current research emphasizes optimizing attention mechanisms (e.g., through FlashAttention variants) to reduce quadratic scaling with sequence length, exploring novel architectures like Stormer for specific applications (e.g., weather forecasting), and developing techniques to efficiently train larger models (e.g., using parameter mapping from smaller pretrained models). These advancements are significant because they enable the application of Transformers to increasingly complex tasks and larger datasets in various fields, including natural language processing, computer vision, and scientific modeling, ultimately leading to improved accuracy and efficiency.

Papers

November 11, 2024

Understanding Scaling Laws with Statistical and Approximation Theory for Transformer Neural Networks on Intrinsically Low-dimensional Data
Alex Havrilla, Wenjing Liao
Scaling Law Transformer Based Large Language Model Statistical Query Intrinsic Dimension Approximation Theory Low Dimensional Data Training Dataset Size Scaling Transformer

November 4, 2024

Ask, and it shall be given: Turing completeness of prompting
Ruizhong Qiu, Zhe Xu, Wenxuan Bao, Hanghang Tong
Large Language Model Probabilistic Completeness Finite Data Scaling Transformer

October 30, 2024

TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters
Haiyang Wang, Yue Fan, Muhammad Ferjad Naeem, Yongqin Xian, Jan Eric Lenssen, Liwei Wang, Federico Tombari, Bernt Schiele
Many Parameter Strong Scaling Semantic Token Input Token Scalable Architecture Scaling Transformer

December 6, 2023

Scaling transformer neural networks for skillful and reliable medium-range weather forecasting
Tung Nguyen, Rohan Shah, Hritik Bansal, Troy Arcomano, Romit Maulik, Veerabhadra Kotamarthi, Ian Foster, Sandeep Madireddy, Aditya Grover
Neural Network Deep Learning Forecasting Performance Weather Forecasting Medium Range Atmospheric Dynamic Scaling Transformer

September 26, 2023

Boosting High Resolution Image Classification with Scaling-up Transformers
Yi Wang
Data Augmentation Training Data Cross Fold Model Prediction Average Resolution Image Classification Scaling Transformer

July 17, 2023

FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
Tri Dao
Attention Layer Efficient Attention GPU Memory Magic Cube Partition Expert Parallelism FlashAttention 2 Scaling Transformer

April 19, 2023

Scaling Transformer to 1M tokens and beyond with RMT
Aydar Bulatov, Yuri Kuratov, Yermek Kapushev, Mikhail S. Burtsev
Pre Trained Transformer Language Modeling Task Scaling Transformer

March 17, 2023

MedNeXt: Transformer-driven Scaling of ConvNets for Medical Image Segmentation
Saikat Roy, Gregor Koehler, Constantin Ulrich, Michael Baumgartner, Jens Petersen, Fabian Isensee, Paul F. Jaeger, Klaus Maier-Hein
Medical Image Segmentation High Performing 1D ConvNet ConvNeXt Model Scalable Neural Large Kernel Standard Convolutional Scaling Transformer

March 2, 2023

Learning to Grow Pretrained Models for Efficient Transformer Training
Peihao Wang, Rameswar Panda, Lucas Torroba Hennigen, Philip Greengard, Leonid Karlinsky, Rogerio Feris, David Daniel Cox, Zhangyang Wang, Yoon Kim
LeArning Abstract Vision Transformer Transformer Training Scaling Transformer Growth Operator

November 24, 2021

Sparse is Enough in Scaling Transformers
Sebastian Jaszczur, Aakanksha Chowdhery, Afroz Mohiuddin, Łukasz Kaiser, Wojciech Gajewski, Henryk Michalewski, Jonni Kanerva
Many Sparse Large Transformer Model Transformer Prediction Scaling Transformer