Transformer Encoders

Transformer encoders are neural network architectures designed to process sequential data by leveraging self-attention mechanisms to capture long-range dependencies. Current research focuses on improving their efficiency, particularly for long sequences, through techniques like progressive token length scaling and optimized hardware acceleration, as well as exploring their expressivity and limitations in various applications. These advancements are driving significant improvements in diverse fields, including natural language processing, computer vision, and speech recognition, by enabling more accurate and efficient models for tasks such as machine translation, image segmentation, and speech diarization.

Papers

January 25, 2023

Tighter Bounds on the Expressivity of Transformer Encoders
David Chiang, Peter Cholak, Anand Pillay
Neural Network Transformer Encoders Behavior Expressivity Style First Order Logic Tighter Bound Existential Quantifier

January 18, 2023

HSTFormer: Hierarchical Spatial-Temporal Transformers for 3D Human Pose Estimation
Xiaoye Qian, Youbao Tang, Ning Zhang, Mei Han, Jing Xiao, Ming-Chun Huang, Ruei-Sung Lin
3D Human 3D Human Pose Estimation Transformer Encoders Pose Sequence Multi Level Fusion Hierarchical Spatial Temporal Transformer

November 20, 2022

Understanding and Improving Knowledge Distillation for Quantization-Aware Training of Large Transformer Encoders
Minsoo Kim, Sihwa Lee, Sukjin Hong, Du-Seong Chang, Jungwook Choi
Knowledge Distillation Human Understanding Quantization Aware Training Transformer Encoders Transformer Encoder Model Social Attention

November 18, 2022

FiE: Building a Global Probability Space by Leveraging Early Fusion in Encoder for Open-Domain Question Answering
Akhil Kedia, Mohd Abbas Zaidi, Haejun Lee
Generative Model Open Domain Question Answering QA Datasets Transformer Encoders Probability Space

October 24, 2022

Effective Pre-Training Objectives for Transformer-based Autoencoders
Luca Di Liello, Matteo Gabburo, Alessandro Moschitti
Transformer Based Transformer Based Model Pre Training Transformer Encoders Token Generation Pre Training Objective

October 6, 2022

Damage Control During Domain Adaptation for Transducer Based Automatic Speech Recognition
Somshubra Majumdar, Shantanu Acharya, Vitaly Lavrukhin, Boris Ginsburg
Domain Adaptation Automatic Speech Recognition Model Adaptation Speech Recognition Model Transformer Encoders Google Speech Command Transformer Transducer

September 20, 2022

Hierarchical Temporal Transformer for 3D Hand Pose Estimation and Action Recognition from Egocentric RGB Videos
Yilin Wen, Hao Pan, Lei Yang, Jia Pan, Taku Komura, Wenping Wang
Action Recognition Hand Pose Estimation Transformer Encoders Egocentric RGB

September 15, 2022

Number of Attention Heads vs Number of Transformer-Encoders in Computer Vision
Tomas Hrycej, Bernhard Bermeitinger, Siegfried Handschuh
Computer Vision Transformer Architecture Numerical Data Attention Head Transformer Encoders Generalization Capability Parallel Transformer

September 13, 2022

CNN-Trans-Enc: A CNN-Enhanced Transformer-Encoder On Top Of Static BERT representations for Document Classification
Charaf Eddine Benarab, Shenglin Gui
Ticket BERT BERT Embeddings Transformer Encoders Document Classification Convolutional Transformer TOp Front TopMost

August 24, 2022

Cats: Complementary CNN and Transformer Encoders for Segmentation
Hao Li, Dewei Hu, Han Liu, Jiacheng Wang, Ipek Oguz
Convolutional Neural Network Segmentation Based Approach Medical Image Segmentation Task 3D Medical Image Segmentation Transformer Encoders Convolutional Encoder 2 Cat

July 27, 2022

A Variational AutoEncoder for Transformers with Nonparametric Variational Information Bottleneck
James Henderson, Fabio Fehr
Transformer Megatron Decepticons Variational Autoencoders Transformer Encoders Variational Information Bottleneck Transformer Embeddings

June 25, 2022

SC-Transformer++: Structured Context Transformer for Generic Event Boundary Detection
Dexiang Hong, Xiaoqi Ma, Xinyao Wang, Congcong Li, Yufei Wang, Longyin Wen
Optical Flow Transformer Encoders Context Transformer Generic Event Boundary Detection Strip Transformer

June 16, 2022

A Language Model With Million Sample Context For Raw Audio Using Transformer Architectures
Prateek Verma
Language Model Transformer Architecture Latent Representation Neural Architecture Transformer Encoders Autoregressive Generative Model Raw Audio Context Sample

April 13, 2022

Formal Language Recognition by Hard Attention Transformers: Perspectives from Circuit Complexity
Yiding Hao, Dana Angluin, Robert Frank
Synthesized View Transformer Encoders Complexity Class Circuit Complexity Language Recognition

April 1, 2022

UNetFormer: A Unified Vision Transformer Model and Pre-Training Framework for 3D Medical Image Segmentation
Ali Hatamizadeh, Ziyue Xu, Dong Yang, Wenqi Li, Holger Roth, Daguang Xu
Convolutional Neural Network Vision Transformer Pre Training 3D Medical Image Segmentation Transformer Encoders Liver Tumor Segmentation Vision Transformer Model Transformer Based Decoder

March 3, 2022

Multi-Tailed Vision Transformer for Efficient Inference
Yunke Wang, Bo Du, Wenyuan Wang, Chang Xu
Vision Transformer Efficient Inference Self Attention Layer Transformer Encoder Transformer Encoders

February 14, 2022

Source Code Summarization with Structural Relative Position Guided Transformer
Zi Gong, Cuiyun Gao, Yasheng Wang, Wenchao Gu, Yun Peng, Zenglin Xu
Code Summarization Transformer Encoders Abstract Syntax Tree Dependency Structure Position Detection Transformer

February 7, 2022

January 17, 2022

Continual Transformers: Redundancy-Free Attention for Online Inference
Lukas Hedegaard, Arian Bakhtiarnia, Alexandros Iosifidis
Human Attention Transformer Encoders Dot Product Attention Online Inference Token Sequence Sparse Token Transformer Based Continual Learning

Transformer Encoders

Papers

Tighter Bounds on the Expressivity of Transformer Encoders

HSTFormer: Hierarchical Spatial-Temporal Transformers for 3D Human Pose Estimation

Understanding and Improving Knowledge Distillation for Quantization-Aware Training of Large Transformer Encoders

FiE: Building a Global Probability Space by Leveraging Early Fusion in Encoder for Open-Domain Question Answering

Effective Pre-Training Objectives for Transformer-based Autoencoders

Damage Control During Domain Adaptation for Transducer Based Automatic Speech Recognition

Hierarchical Temporal Transformer for 3D Hand Pose Estimation and Action Recognition from Egocentric RGB Videos

Number of Attention Heads vs Number of Transformer-Encoders in Computer Vision

CNN-Trans-Enc: A CNN-Enhanced Transformer-Encoder On Top Of Static BERT representations for Document Classification

Cats: Complementary CNN and Transformer Encoders for Segmentation

A Variational AutoEncoder for Transformers with Nonparametric Variational Information Bottleneck

SC-Transformer++: Structured Context Transformer for Generic Event Boundary Detection

A Language Model With Million Sample Context For Raw Audio Using Transformer Architectures

Formal Language Recognition by Hard Attention Transformers: Perspectives from Circuit Complexity

UNetFormer: A Unified Vision Transformer Model and Pre-Training Framework for 3D Medical Image Segmentation

Multi-Tailed Vision Transformer for Efficient Inference

Source Code Summarization with Structural Relative Position Guided Transformer

HeadPosr: End-to-end Trainable Head Pose Estimation using Transformer Encoders

LwPosr: Lightweight Efficient Fine-Grained Head Pose Estimation

Continual Transformers: Redundancy-Free Attention for Online Inference