Transformer Architecture

Transformer architectures are a dominant deep learning paradigm, primarily known for their self-attention mechanism enabling efficient processing of sequential data like text and time series. Current research focuses on addressing the quadratic time complexity of self-attention through alternative architectures (e.g., state space models like Mamba) and optimized algorithms (e.g., local attention, quantized attention), as well as exploring the application of transformers to diverse domains including computer vision, robotics, and blockchain technology. These efforts aim to improve the efficiency, scalability, and interpretability of transformers, leading to broader applicability and enhanced performance across numerous fields.

Papers

May 17, 2024

Layer-Condensed KV Cache for Efficient Inference of Large Language Models
Haoyi Wu, Kewei Tu
Large Language Model Language Model Transformer Architecture Large Scale Language Model Efficient Inference Key Value Cache

May 15, 2024

Modeling Bilingual Sentence Processing: Evaluating RNN and Transformer Architectures for Cross-Language Structural Priming
Demi Zhang, Bushi Xiao, Chao Gao, Sangpil Youm, Bonnie J Dorr
Recurrent Neural Network Transformer Architecture Multilingual Task Structural Priming Sentence Processing

May 14, 2024

Improving Transformers with Dynamically Composable Multi-Head Attention
Da Xiao, Qingye Meng, Shengping Li, Xingyuan Yuan
Transformer Megatron Decepticons Human Attention Transformer Architecture Multi Head Attention Composable Framework

May 8, 2024

Transformer Architecture for NetsDB
Subodh Kamble, Kunal Sunil Kasodekar
Transformer Architecture Transformer Model Multi Head Self Attention New Database Dynamic Embeddings

May 7, 2024

S3Former: Self-supervised High-resolution Transformer for Solar PV Profiling
Minh Tran, Adrian De Luis, Haitao Liao, Ying Huang, Roy McCann, Alan Mantooth, Jack Cothren, Ngan Le
Transformer Architecture Self Supervised Transformer Photovoltaic System Solar Panel

May 6, 2024

Transformer models as an efficient replacement for statistical test suites to evaluate the quality of random numbers
Rishabh Goel, YiZi Xiao, Ramin Ramezani
Deep Learning Model Transformer Architecture Transformer Model Inherent Randomness Random Number Generator Quantum Generator

May 2, 2024

Early Transformers: A study on Efficient Training of Transformer Models through Early-Bird Lottery Tickets
Shravan Cheekati
Natural Language Processing Study Feature Transformer Architecture Transformer Model Transformer Based Model Efficient Training Lottery Ticket Training Strategy Transformer Era

April 29, 2024

Revolutionizing Traffic Sign Recognition: Unveiling the Potential of Vision Transformers
Susano Mingwin, Yulong Shisu, Yongshuai Wanwag, Sunshin Huing
Convolutional Neural Network Vision Transformer Transformer Architecture Full Potential Traffic Sign Recognition Sign Classification

April 28, 2024

April 25, 2024

From Cognition to Computation: A Comparative Review of Human Attention and Transformer Architectures
Minglu Zhao, Dehong Xu, Tao Gao
Attention Mechanism Human Attention Transformer Architecture Computation Method Human Cognition Cognitive Learning Comparative Review Cognitive Approach Smooth Attention

April 24, 2024

A Comprehensive Survey on Evaluating Large Language Model Applications in the Medical Industry
Yining Huang, Keke Tang, Meilian Chen, Boyuan Wang
Language Understanding Transformer Architecture Comprehensive Survey Healthcare System Clinical Text Medical Domain Language Model Application

April 23, 2024

Transformers Can Represent $n$-gram Language Models
Anej Svete, Ryan Cotterell
Language Model Transformer Megatron Decepticons Transformer Architecture Representational Capacity Gram Language Model

April 19, 2024

Transformer-Based Classification Outcome Prediction for Multimodal Stroke Treatment
Danqing Ma, Meng Wang, Ao Xiang, Zongqing Qi, Qin Yang
Transformer Based Transformer Architecture Transformer Model Multi Modal Fusion Stroke Patient Multi Modal Approach Transformer Neural Network Architecture

April 18, 2024

Towards Multi-modal Transformers in Federated Learning
Guangyu Sun, Matias Mendieta, Aritra Dutta, Xin Li, Chen Chen
Federated Learning Transformer Architecture Vision Language Task Multi Modal Transformer Modality Discrepancy

April 15, 2024

Transformer Architecture

Papers

Layer-Condensed KV Cache for Efficient Inference of Large Language Models

Modeling Bilingual Sentence Processing: Evaluating RNN and Transformer Architectures for Cross-Language Structural Priming

Improving Transformers with Dynamically Composable Multi-Head Attention

Transformer Architecture for NetsDB

S3Former: Self-supervised High-resolution Transformer for Solar PV Profiling

Transformer models as an efficient replacement for statistical test suites to evaluate the quality of random numbers

U-DiTs: Downsample Tokens in U-Shaped Diffusion Transformers

A Combination of BERT and Transformer for Vietnamese Spelling Correction

Exploring Extreme Quantization in Spiking Language Models

Early Transformers: A study on Efficient Training of Transformer Models through Early-Bird Lottery Tickets

Revolutionizing Traffic Sign Recognition: Unveiling the Potential of Vision Transformers

Transfer Learning and Transformer Architecture for Financial Sentiment Analysis

Exploring the Robustness of In-Context Learning with Noisy Labels

From Cognition to Computation: A Comparative Review of Human Attention and Transformer Architectures

A Comprehensive Survey on Evaluating Large Language Model Applications in the Medical Industry

Transformers Can Represent $n$-gram Language Models

Transformer-Based Classification Outcome Prediction for Multimodal Stroke Treatment

Towards Multi-modal Transformers in Federated Learning

Transformers, Contextualism, and Polysemy

State Space Model for New-Generation Network Alternative to Transformers: A Survey