Large Transformer Model

Large transformer models are deep learning architectures achieving state-of-the-art results across diverse tasks, primarily by leveraging self-attention mechanisms to process sequential data. Current research focuses on improving their efficiency through techniques like pruning, knowledge distillation, quantization, and novel factorization methods, while also exploring adaptive training strategies and hardware-aware co-design to optimize performance on resource-constrained platforms. These advancements are crucial for broadening the accessibility and applicability of large transformer models in various scientific domains and practical applications, including natural language processing, computer vision, and even areas like EEG analysis and music generation.

Papers

September 30, 2024

Comprehensive Performance Modeling and System Design Insights for Foundation Models
Shashank Subramanian, Ermal Rrapaj, Peter Harrington, Smeet Chheda, Steven Farrell, Brian Austin, Samuel Williams, Nicholas Wright, Wahid Bhimji
Foundation Model Generative AI Transformer Model Large Transformer Model High Performance Computing System Performance Model

September 21, 2024

On Importance of Pruning and Distillation for Efficient Low Resource NLP
Aishwarya Mirashi, Purva Lingayat, Srushti Sonavane, Tejas Padhiyar, Raviraj Joshi, Geetanjali Kale
Natural Language Processing Low Resource Language Low Resource Edge Pruning Importance Aware Mutual Distillation Large Transformer Model Marathi Corpus

July 22, 2024

Empirical Capacity Model for Self-Attention Neural Networks
Aki Härmä, Marcin Pietrasik, Anna Wilbik
Large Transformer Model Model Capacity General Transformer Memorization Capability

February 28, 2024

Large Language Models As Evolution Strategies
Robert Tjarko Lange, Yingtao Tian, Yujin Tang
Large Language Model Context Learning Evolution Strategy Large Transformer Model Recombination Operator

February 8, 2024

RepQuant: Towards Accurate Post-Training Quantization of Large Transformer Models via Scale Reparameterization
Zhikai Li, Xuewen Liu, Jing Zhang, Qingyi Gu
Post Training Quantization Large Transformer Model Accurate Quantization Large Scale Transformer

December 20, 2023

DSFormer: Effective Compression of Text-Transformers by Dense-Sparse Weight Factorization
Rahul Chand, Yashoteja Prabhu, Pratyush Kumar
Transformer Architecture Large Transformer Model Factorization Machine Text Transformation Sparse Factorization Weight Factorization

December 14, 2023

Weight subcloning: direct initialization of transformers using larger pretrained ones
Mohammad Samragh, Mehrdad Farajtabar, Sachin Mehta, Raviteja Vemulapalli, Fartash Faghri, Devang Naik, Oncel Tuzel, Mohammad Rastegari
Vision Transformer Transformer Megatron Decepticons Transformer Model Balancing Weight Large Transformer Model Tiny Transformer One 2 Structured Initialization

November 21, 2023

Equipping Pretrained Unconditional Music Transformers with Instrument and Genre Controls
Weihan Xu, Julian McAuley, Shlomo Dubnov, Hao-Wen Dong
Symbolic Music Genre Classification Symbolic Music Generation Musical Instrument Large Transformer Model Music Transformer

September 29, 2023

Federated Learning with Differential Privacy for End-to-End Speech Recognition
Martin Pelikan, Sheikh Shams Azam, Vitaly Feldman, Jan "Honza" Silovsky, Kunal Talwar, Tatiana Likhomanenko
Automatic Speech Recognition Differential Privacy End to End Recognition Benchmark Large Transformer Model

August 20, 2023

Large Transformers are Better EEG Learners
Bingxin Wang, Xiaowen Fu, Yuan Lan, Luchan Zhang, Wei Zheng, Yang Xiang
Pre Trained Pre Trained Vision Transformer Large Transformer Model Large Scale Transformer EEG Task

July 10, 2023

FedYolo: Augmenting Federated Learning with Pretrained Transformers
Xuechen Zhang, Mingchen Li, Xiangyu Chang, Jiasi Chen, Amit K. Roy-Chowdhury, Ananda Theertha Suresh, Samet Oymak
Pre Trained Transformer Large Transformer Model General Purpose Model

June 26, 2023

Supervised Pretraining Can Learn In-Context Reinforcement Learning
Jonathan N. Lee, Annie Xie, Aldo Pacchiano, Yash Chandak, Chelsea Finn, Ofir Nachum, Emma Brunskill
Reinforcement Learning Markov Decision Process Large Transformer Model Supervised Pre Training Context Dataset

March 24, 2023

EdgeTran: Co-designing Transformers for Efficient Inference on Mobile Edge Platforms
Shikhar Tuli, Niraj K. Jha
Transformer Architecture Transformer Model Efficient Inference Efficient Transformer Large Transformer Model Transformer Performance Edge Platform

March 4, 2023

A Fast Training-Free Compression Framework for Vision Transformers
Jung Hwan Heo, Arash Fayyazi, Mahdi Nazemi, Massoud Pedram
Vision Transformer Token Pruning Large Transformer Model

February 1, 2023

The geometry of hidden representations of large transformer models
Lucrezia Valeriani, Diego Doimo, Francesca Cuturello, Alessandro Laio, Alessio Ansuini, Alberto Cazzaniga
Meaningful Representation Geometric Analysis Semantic Structure Intrinsic Dimension Large Transformer Model Large Scale Transformer Semantic Content Protein Language

November 5, 2022

Privacy-Preserving Models for Legal Natural Language Processing
Ying Yin, Ivan Habernal
Full Model Differential Privacy Transformer Model Transformer Language Model Large Transformer Model Legal Natural Language Processing Adversarial Privacy

October 7, 2022

Automatic Discovery of Composite SPMD Partitioning Strategies in PartIR
Sami Alabed, Dominik Grewe, Juliana Franco, Bart Chrzaszcz, Tom Natan, Tamara Norman, Norman A. Rink, Dimitrios Vytiniotis, Michael Schaarschmidt
Efficient Pipeline Different PaRT Large Transformer Model Large Deep Learning Model Compiler Optimization Automatic Discovery

September 30, 2022

Self-Distillation for Further Pre-training of Transformers
Seanie Lee, Minki Kang, Juho Lee, Sung Ju Hwang, Kenji Kawaguchi
Vision Transformer Transformer Megatron Decepticons Pre Trained Model Pre Trained Self Distillation Large Transformer Model

September 24, 2022

In-context Learning and Induction Heads
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, Chris Olah
Context Learning Large Transformer Model Head Transformer Context Learning Ability

September 6, 2022

EnergonAI: An Inference System for 10-100 Billion Parameter Transformer Models
Jiangsu Du, Ziming Liu, Jiarui Fang, Shenggui Li, Yongbin Li, Yutong Lu, Yang You
Transformer Based Model Large Transformer Model Pipeline Parallelism Inference System Fast Transformer