Tensor Core

Tensor cores are specialized processing units within GPUs designed to accelerate matrix multiplication, a fundamental operation in many machine learning and scientific computing tasks. Current research focuses on optimizing tensor core utilization for various applications, including large language models (LLMs), genome-wide association studies (GWAS), and deep neural networks, often employing techniques like mixed-precision arithmetic, sparse matrix operations, and novel data formats to improve efficiency and reduce memory footprint. These advancements significantly impact the scalability and speed of computationally intensive applications, enabling faster inference for LLMs, more efficient analysis of large datasets in genomics, and improved performance in deep learning model training and deployment.

Papers

October 21, 2024

Beyond 2:4: exploring V:N:M sparsity for efficient transformer inference on GPUs
Kang Zhao, Tao Yuan, Han Bao, Zhenfeng Su, Chang Gao, Zhaofeng Sun, Zichen Liang, Liping Jing, Jianfei Chen
Sparsity Increase Single GPU Sparse Transformer Transformer Inference Tensor Core

September 26, 2024

Efficient Arbitrary Precision Acceleration for Large Language Models on GPU Tensor Cores
Shaobo Ma, Chao Fang, Haikuo Shao, Zhongfeng Wang
Large Language Model LLM Inference Matrix Multiplication Tensor Core

September 3, 2024

Toward Capturing Genetic Epistasis From Multivariate Genome-Wide Association Studies Using Mixed-Precision Kernel Ridge Regression
Hatem Ltaief, Rabab Alomairy, Qinglei Cao, Jie Ren, Lotfi Slim, Thorsten Kurth, Benedikt Dorschner, Salim Bougouffa, Rached Abdelkhalak, David E. Keyes
Kernel Ridge Regression Low Precision Tensor Core Genome Wide Association Study

August 12, 2024

LUT Tensor Core: Lookup Table Enables Efficient Low-Bit LLM Inference Acceleration
Zhiwen Mo, Lei Wang, Jianyu Wei, Zhichen Zeng, Shijie Cao, Lingxiao Ma, Naifeng Jing, Ting Cao, Jilong Xue, Fan Yang, Mao Yang
Language Model Large Language Model Inference Lookup Table Tensor Core

August 2, 2024

FT K-means: A High-Performance K-means on GPU with Fault Tolerance
Shixun Wu, Yitong Ding, Yujia Zhai, Jinyang Liu, Jiajun Huang, Zizhe Jian, Huangliang Dai, Sheng Di, Bryan M. Wong, Zizhong Chen, Franck Cappello
Single GPU K Mean Fault Tolerance Tensor Core

July 30, 2024

Toward Efficient Permutation for Hierarchical N:M Sparsity on GPUs
Seungmin Yu, Xiaodie Yi, Hayun Lee, Dongkun Shin
Single GPU Sparse Network Structured Sparsity Machine Permutation Hierarchical Network Tensor Core

January 25, 2024

FP6-LLM: Efficiently Serving Large Language Models Through FP6-Centric Algorithm-System Co-Design
Haojun Xia, Zhen Zheng, Xiaoxia Wu, Shiyang Chen, Zhewei Yao, Stephen Youn, Arash Bakhtiari, Michael Wyatt, Donglin Zhuang, Zhongzhu Zhou, Olatunji Ruwase, Yuxiong He, Shuaiwen Leon Song
Co Design Weight Quantization Tensor Core Floating Point Quantization

November 22, 2023

Scalable CP Decomposition for Tensor Learning using GPU Tensor Cores
Zeliang Zhang, Zhuo Liu, Susan Liang, Zhiyuan Wang, Yifan Zhu, Chen Ding, Chenliang Xu
Tensor Based Tensor Core CP Decomposition

November 10, 2023

FlashFFTConv: Efficient Convolutions for Long Sequences with Tensor Cores
Daniel Y. Fu, Hermann Kumbong, Eric Nguyen, Christopher Ré
Long Sequence Fast Fourier Transform Tensor Core Fast Convolution Fast Fourier Convolution Long Convolution

October 9, 2023

Look-Up mAI GeMM: Increasing AI GeMMs Performance by Nearly 2.5x via msGeMM
Saeed Maleki
Artificial Intelligence System Performance Artificial Intelligence Model Low Precision Tensor Core CUDA Kernel

October 3, 2023

VENOM: A Vectorized N:M Format for Unleashing the Power of Sparse Tensor Cores
Roberto L. Castro, Andrei Ivanov, Diego Andrade, Tal Ben-Nun, Basilio B. Fraguela, Torsten Hoefler
Real Power Vectorization Method Tensor Core Sparse Vector

September 29, 2023

Benchmarking and In-depth Performance Study of Large Language Models on Habana Gaudi Processors
Chengming Zhang, Baixi Sun, Xiaodong Yu, Zhen Xie, Weijian Zheng, Kamil Iskra, Pete Beckman, Dingwen Tao
Transformer Model Transformer Based Model Benchmark Platform Performance Analysis Transformer Training Tensor Core

September 19, 2023

Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity
Haojun Xia, Zhen Zheng, Yuchao Li, Donglin Zhuang, Zhongzhu Zhou, Xiafei Qiu, Yong Li, Wei Lin, Shuaiwen Leon Song
Cost Effective Large Generative Model LLM Framework Unstructured Sparsity Tensor Core Sparse Matrix Multiplication

August 25, 2023

Training normalizing flows with computationally intensive target probability distributions
Piotr Bialas, Piotr Korcyl, Tomasz Stebel
Training Data Normalizing Flow Field Theory Reparameterization Method General Flow Target Distribution Tensor Core Reparametrization Trick

January 15, 2023

Quantum-inspired tensor network for Earth science
Soronzonbold Otgonbaatar, Dieter Kranzlmüller
Deep Learning Physic Informed Neural Network Tensor Decomposition Tensor Core

September 14, 2022

Efficient Quantized Sparse Matrix Operations on Tensor Cores
Shigang Li, Kazuki Osawa, Torsten Hoefler
High Efficiency Sparse Matrix Sparse Transformer Tensor Core Precision Matrix Sparse Operation

February 23, 2022

Generative modeling via tensor train sketching
YH. Hur, J. G. Hoskins, M. Lindsey, E. M. Stoudenmire, Y. Khoo
Generative Modeling Singular Value Decomposition Tensor Train Tensor Core

February 11, 2022

Learning from distinctive candidates to optimize reduced-precision convolution program on tensor cores
Junkyeong Choi, Hyucksung Kwon, Woongkyu Lee, Jungwook Choi, Jieun Lim
LeArning Abstract Matrix Multiplication Convolution Operation Tensor Core Multiply Accumulate Fast Convolution Candidate Summary

January 21, 2022

APack: Off-Chip, Lossless Data Compression for Efficient Deep Learning Inference
Alberto Delmas Lascorz, Mostafa Mahmoud, Andreas Moshovos
Lossless Compression Deep Learning Inference Tensor Core Chip Memory

December 3, 2021

TC-GNN: Bridging Sparse GNN Computation and Dense Tensor Cores on GPUs
Yuke Wang, Boyuan Feng, Zheng Wang, Guyue Huang, Yufei Ding
Single GPU GNN Framework Tensor Core DNN Workload GNN Acceleration

Tensor Core

Papers

Beyond 2:4: exploring V:N:M sparsity for efficient transformer inference on GPUs

Efficient Arbitrary Precision Acceleration for Large Language Models on GPU Tensor Cores

Toward Capturing Genetic Epistasis From Multivariate Genome-Wide Association Studies Using Mixed-Precision Kernel Ridge Regression

LUT Tensor Core: Lookup Table Enables Efficient Low-Bit LLM Inference Acceleration

FT K-means: A High-Performance K-means on GPU with Fault Tolerance

Toward Efficient Permutation for Hierarchical N:M Sparsity on GPUs

FP6-LLM: Efficiently Serving Large Language Models Through FP6-Centric Algorithm-System Co-Design

Scalable CP Decomposition for Tensor Learning using GPU Tensor Cores

FlashFFTConv: Efficient Convolutions for Long Sequences with Tensor Cores

Look-Up mAI GeMM: Increasing AI GeMMs Performance by Nearly 2.5x via msGeMM

VENOM: A Vectorized N:M Format for Unleashing the Power of Sparse Tensor Cores

Benchmarking and In-depth Performance Study of Large Language Models on Habana Gaudi Processors

Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity

Training normalizing flows with computationally intensive target probability distributions

Quantum-inspired tensor network for Earth science

Efficient Quantized Sparse Matrix Operations on Tensor Cores

Generative modeling via tensor train sketching

Learning from distinctive candidates to optimize reduced-precision convolution program on tensor cores

APack: Off-Chip, Lossless Data Compression for Efficient Deep Learning Inference

TC-GNN: Bridging Sparse GNN Computation and Dense Tensor Cores on GPUs