Transformer XL

Transformer XL, a memory-augmented transformer architecture, addresses the limitations of standard transformers in handling long sequences by employing a recurrence mechanism and relative positional encoding. Current research focuses on applying Transformer XL and its variants to diverse tasks, including robotic learning, medical image analysis (e.g., MRI for knee replacement prediction, ultrasound denoising), time series forecasting (e.g., dengue fever, typhoon intensity), and natural language processing (e.g., large language model deployment on mobile devices). This adaptability demonstrates the model's significance for improving efficiency and accuracy across numerous fields, impacting both scientific understanding and practical applications.

Papers

July 8, 2024

STMR: Spiral Transformer for Hand Mesh Reconstruction
Huilong Xie, Wenwei Song, Wenxiong Kang, Yihong Lin
Relation Pattern Transformer XL Hand Mesh Reconstruction Slice Sampling Vertex Feature

May 24, 2024

Transformer-XL for Long Sequence Tasks in Robotic Learning from Demonstration
Gao Tianci
Convolutional Neural Network Long Short Term Memory Robot Learning Sensor Data Textual Demonstration Transformer XL Sequence Task

May 5, 2024

MR-Transformer: Vision Transformer for Total Knee Replacement Prediction Using Magnetic Resonance Imaging
Chaojie Zhang, Shengjia Chen, Ozkan Cigdem, Haresh Rengaraj Rajamohan, Kyunghyun Cho, Richard Kijowski, Cem M. Deniz
Vision Transformer Magnetic Resonance Imaging Transformer Based Deep Transformer XL Knee Magnetic Resonance Total Knee Replacement

March 29, 2024

Transformer-Lite: High-efficiency Deployment of Large Language Models on Mobile Phone GPUs
Luchang Li, Sheng Qian, Jie Lu, Lunxi Yuan, Rui Wang, Qin Xie
Large Language Model Large Language Model Inference Inference Speed Transformer XL Uncertainty Aware Deployment Mobile GPUs Mobile Inference

March 10, 2024

FWin transformer for dengue prediction under climate and ocean influence
Nhat Thanh Tran, Jack Xin, Guofa Zhou
Long Term Prediction Climate Domain Transformer XL Dengue Prediction Dengue Outbreak

November 28, 2023

Typhoon Intensity Prediction with Vision Transformer
Huanxin Chen, Pengshuai Yin, Huichou Huang, Qingyao Wu, Ruirui Liu, Xiatian Zhu
Convolutional Neural Network Vision Transformer Transformer XL Patch Level Representation Tropical Cyclone Intensity

November 14, 2023

Memory-efficient Stochastic methods for Memory-based Transformers
Vishwajit Kumar Vishnu, C. Chandra Sekhar
Two Stage Training Transformer XL Memory Based Transformer Skip Transformer

October 26, 2023

ZeroQuant-HERO: Hardware-Enhanced Robust Optimized Post-Training Quantization Framework for W8A8 Transformers
Zhewei Yao, Reza Yazdani Aminabadi, Stephen Youn, Xiaoxia Wu, Elton Zheng, Yuxiong He
Neural Network Post Training Quantization Quantization Technique Dynamic Quantization Transformer XL W4A8 Quantization Intensive Operator

October 5, 2023

A Complementary Global and Local Knowledge Network for Ultrasound denoising with Fine-grained Refinement
Zhenyu Bu, Kai-Ni Wang, Fuxing Zhao, Shengxiao Li, Guang-Quan Zhou
Ultrasound Image Transformer XL Speckle Noise Speckle Reduction

June 21, 2023

MSW-Transformer: Multi-Scale Shifted Windows Transformer Networks for 12-Lead ECG Classification
Renjie Cheng, Zhemin Zhuang, Shuxin Zhuang, Lei Xie, Jingfeng Guo
Multi Scale Electrocardiogram Signal Electrocardiogram Classification Electrocardiogram Feature Transformer XL Electrocardiogram Dataset

May 2, 2023

AxWin Transformer: A Context-Aware Vision Transformer Backbone with Axial Windows
Fangjian Lin, Yizhe Ma, Sitong Wu, Long Yu, Shengwei Tian
Vision Task Pay Attention Transformer XL Window Selection Downstream Segmentation

November 23, 2022

An ensemble of VisNet, Transformer-M, and pretraining models for molecular property prediction in OGB Large-Scale Challenge @ NeurIPS 2022
Yusong Wang, Shaoning Li, Zun Wang, Xinheng He, Bin Shao, Tie-Yan Liu, Tong Wang
Large Scale Molecular Graph Molecular Property Prediction 3D Pre Training Vision Transformer Network Transformer XL Graph Regression Open Graph Benchmark Quantum Chemical Property

November 21, 2022

PS-Transformer: Learning Sparse Photometric Stereo Network using Self-Attention Mechanism
Satoshi Ikehata
Self Attention Image Feature Photometric Stereo Transformer XL

October 27, 2022

Working Alliance Transformer for Psychotherapy Dialogue Classification
Baihan Lin, Guillermo Cecchi, Djallel Bouneffouf
Transformer Based Mental State Transformer XL Psychotherapy Session

September 8, 2022

W-Transformers : A Wavelet-based Transformer Framework for Univariate Time Series Forecasting
Lena Sasal, Tanujit Chakraborty, Abdenour Hadid
Transformer Based Transformer Model Temporal Dependency Time Series Representation Learning Univariate Time Series Forecasting Transformer Based Neural Network Transformer XL

April 16, 2022

Towards Lightweight Transformer via Group-wise Transformation for Vision-and-Language Tasks
Gen Luo, Yiyi Zhou, Xiaoshuai Sun, Yan Wang, Liujuan Cao, Yongjian Wu, Feiyue Huang, Rongrong Ji
Swin Transformer Vision Language Task Transformer Based Network Image Transformer Lightweight Transformer Transformer XL

January 5, 2022

Lawin Transformer: Improving Semantic Segmentation Transformer with Multi-Scale Representations via Large Window Attention
Haotian Yan, Chuang Zhang, Ming Wu
Vision Transformer Semantic Segmentation Hierarchical Vision Transformer Multi Scale Representation Window Attention Transformer XL

December 24, 2021

Distinguishing Transformative from Incremental Clinical Evidence: A Classifier of Clinical Research using Textual features from Abstracts and Citing Sentences
Xuanyu Shi, Jian Du
Textual Feature Incremental Task Transformer XL Clinical Research Clinical Evidence Transformative Impact Late Breaking Abstract

December 13, 2021

Roof-Transformer: Divided and Joined Understanding with Knowledge Enhancement
Wei-Lin Liao, Cheng-En Su, Wei-Yun Ma
Language Model Knowledge Graph Human Understanding BERT Model Knowledge Enhancement Transformer XL BERT Encoder

Transformer XL

Papers

STMR: Spiral Transformer for Hand Mesh Reconstruction

Transformer-XL for Long Sequence Tasks in Robotic Learning from Demonstration

MR-Transformer: Vision Transformer for Total Knee Replacement Prediction Using Magnetic Resonance Imaging

Transformer-Lite: High-efficiency Deployment of Large Language Models on Mobile Phone GPUs

FWin transformer for dengue prediction under climate and ocean influence

Typhoon Intensity Prediction with Vision Transformer

Memory-efficient Stochastic methods for Memory-based Transformers

ZeroQuant-HERO: Hardware-Enhanced Robust Optimized Post-Training Quantization Framework for W8A8 Transformers

A Complementary Global and Local Knowledge Network for Ultrasound denoising with Fine-grained Refinement

MSW-Transformer: Multi-Scale Shifted Windows Transformer Networks for 12-Lead ECG Classification

AxWin Transformer: A Context-Aware Vision Transformer Backbone with Axial Windows

An ensemble of VisNet, Transformer-M, and pretraining models for molecular property prediction in OGB Large-Scale Challenge @ NeurIPS 2022

PS-Transformer: Learning Sparse Photometric Stereo Network using Self-Attention Mechanism

Working Alliance Transformer for Psychotherapy Dialogue Classification

W-Transformers : A Wavelet-based Transformer Framework for Univariate Time Series Forecasting

Towards Lightweight Transformer via Group-wise Transformation for Vision-and-Language Tasks

Lawin Transformer: Improving Semantic Segmentation Transformer with Multi-Scale Representations via Large Window Attention

Distinguishing Transformative from Incremental Clinical Evidence: A Classifier of Clinical Research using Textual features from Abstracts and Citing Sentences

Roof-Transformer: Divided and Joined Understanding with Knowledge Enhancement