Transformer Inference

Transformer inference focuses on optimizing the execution of transformer-based models, aiming to reduce latency, memory usage, and power consumption while maintaining accuracy. Current research emphasizes efficient implementations on specialized hardware like FPGAs, employing techniques such as model compression (pruning, quantization), algorithmic optimizations (parallel decoding, dynamic pruning), and novel architectures (linear-cost transformers). These advancements are crucial for deploying large-scale transformer models in resource-constrained environments and real-time applications, impacting fields ranging from natural language processing and computer vision to scientific computing.

Papers

March 27, 2023

TransCODE: Co-design of Transformers and Accelerators for Efficient Training and Inference
Shikhar Tuli, Niraj K. Jha
Transformer Megatron Decepticons Scientific Inference Transformer Architecture Transformer Model Efficient Training Transformer Inference Accelerator Tuning Transformer Accelerator Video Transcoding

February 28, 2023

AccelTran: A Sparsity-Aware Accelerator for Dynamic Inference with Transformers
Shikhar Tuli, Niraj K. Jha
Transformer Megatron Decepticons Transformer Model Transformer Inference Dynamic Inference Accelerator Architecture Transformer Accelerator Sparse Accelerator

February 27, 2023

Full Stack Optimization of Transformer Inference: a Survey
Sehoon Kim, Coleman Hooper, Thanakul Wattanawong, Minwoo Kang, Ruohan Yan, Hasan Genc, Grace Dinh, Qijing Huang, Kurt Keutzer, Michael W. Mahoney, Yakun Sophia Shao, Amir Gholami
Timely Survey Transformer Architecture Transformer Inference Domain Specific Accelerator

February 17, 2023

ViTA: A Vision Transformer Inference Accelerator for Edge Applications
Shashank Nag, Gourav Datta, Souvik Kundu, Nitin Chandrachoodan, Peter A. Beerel
Transformer Model Swin Transformer Hardware Accelerator Transformer Inference Edge Application MLP Layer

January 23, 2023

AttMEMO : Accelerating Transformers with Memoization on Big Memory Systems
Yuan Feng, Hyeran Jeon, Filip Blagojevic, Cyril Guyot, Qing Li, Dong Li
Training Data Transformer Megatron Decepticons Transformer Architecture Transformer Inference Attention Computation Data Memorization Memory System

December 6, 2022

Vision Transformer Computation and Resilience for Dynamic Inference
Kavya Sreedhar, Jason Clemons, Rangharajan Venkatesan, Stephen W. Keckler, Mark Horowitz
Vision Transformer Deep Learning Model Transformer Architecture Tiny Refinement Elicit Resilience Transformer Inference CNN Accelerator Dynamic Inference

November 18, 2022

Castling-ViT: Compressing Self-Attention via Switching Towards Linear-Angular Attention at Vision Transformer Inference
Haoran You, Yunyang Xiong, Xiaoliang Dai, Bichen Wu, Peizhao Zhang, Haoqi Fan, Peter Vajda, Yingyan Lin
Vision Transformer Self Attention Softmax Attention ViT Lens Transformer Inference Linear Angular Attention

November 17, 2022

Zero-Shot Dynamic Quantization for Transformer Inference
Yousef El-Kurdi, Jerry Quinn, Avirup Sil
NLP Task Multiplier Free Quantization Model Quantization Transformer Inference Dynamic Quantization Accuracy Loss Holdout Set

November 9, 2022

Efficiently Scaling Transformer Inference
Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Anselm Levskaya, Jonathan Heek, Kefan Xiao, Shivani Agrawal, Jeff Dean
Transformer Model Transformer Based Model Inference Efficiency Transformer Inference Large Deep Learning Model Efficient Generative

November 2, 2022

MPCFormer: fast, performant and private Transformer inference with MPC
Dacheng Li, Rulin Shao, Hongyi Wang, Han Guo, Eric P. Xing, Hao Zhang
Inference Latency Transformer Inference Private Inference Inference Service Multi Party Computation Sl MPC

October 27, 2022

Fast DistilBERT on CPUs
Haihao Shen, Ofir Zafrir, Bo Dong, Hengyu Meng, Xinyu Ye, Zhe Wang, Yi Ding, Hanwen Chang, Guy Boudoukh, Moshe Wasserblat
Transformer Model Transformer Based Language Model Transformer Inference Fast Transformer

July 11, 2022

STI: Turbocharge NLP Inference at the Edge via Elastic Pipelining
Liwei Guo, Wonkyo Choe, Felix Xiaozhu Lin
Natural Language Inference Extreme Edge Natural Language Processing Model Exact Inference Transformer Inference Device Inference Memory Management Pipeline Parallelism

July 4, 2022

I-ViT: Integer-only Quantization for Efficient Vision Transformer Inference
Zhikai Li, Qingyi Gu
Vision Transformer Transformer Inference Integer Quantization Dyadic Decomposition

June 30, 2022

DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale
Reza Yazdani Aminabadi, Samyam Rajbhandari, Minjia Zhang, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Jeff Rasley, Shaden Smith, Olatunji Ruwase, Yuxiong He
Transformer Model Transformer Based Model Efficient Inference Transformer Inference Neural Network Inference Global Scale GPU Inference

June 1, 2022

THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryption
Tianyu Chen, Hangbo Bao, Shaohan Huang, Li Dong, Binxing Jiao, Daxin Jiang, Haoyi Zhou, Jianxin Li, Furu Wei
Transformer Model Homomorphic Encryption Transformer Inference X Klaim Inherent Privacy Privacy Preserving Inference

January 29, 2022

Decepticons: Corrupted Transformers Breach Privacy in Federated Learning for Language Models
Liam Fowl, Jonas Geiping, Steven Reich, Yuxin Wen, Wojtek Czaja, Micah Goldblum, Tom Goldstein
Language Model Transformer Megatron Decepticons Privacy Attack Transformer Inference Private Text Attack Vector Keystroke Prediction

December 28, 2021

APRIL: Finding the Achilles' Heel on Privacy for Vision Transformers
Jiahao Lu, Xi Sheryl Zhang, Tianli Zhao, Xiangyu He, Jian Cheng
Vision Transformer Privacy Policy PRIvacy Leakage Transformer Inference Gradient Leakage Gradient Leakage Attack

December 3, 2021

NN-LUT: Neural Approximation of Non-Linear Operations for Efficient Transformer Inference
Joonsang Yu, Junki Park, Seongmin Park, Minsoo Kim, Sihwa Lee, Dong Hyun Lee, Jungwook Choi
Transformer Model Transformer Inference Neural Network Approximation Simple Neural Network Hardware Approximation Nonlinear Operation