Transformer Encoder Model

Transformer encoder models are a class of deep learning architectures designed to process sequential data by identifying complex relationships between elements within a sequence. Current research focuses on improving their efficiency, particularly for long sequences, through techniques like replacing attention mechanisms with faster alternatives (e.g., Fourier transforms) and optimizing training strategies such as pretraining and knowledge distillation. These models are proving highly effective across diverse applications, including time series forecasting (e.g., in healthcare and weather prediction), natural language processing, and image analysis (e.g., table detection), demonstrating their broad utility and impact on various scientific fields.

Papers

May 27, 2024

Assessing the significance of longitudinal data in Alzheimer's Disease forecasting
Batuhan K. Karaman, Mert R. Sabuncu
Alzheimer'S Disease Importance Aware Epidemic Forecasting Longitudinal Data Longitudinal Analysis Transformer Encoder Model

May 22, 2024

WeatherFormer: A Pretrained Encoder Model for Learning Robust Weather Representations from Small Datasets
Adib Hasan, Mardavij Roozbehani, Munther Dahleh
Pre Trained Small Datasets Continuous Weather Transformer Encoder Model Multi Weather

February 12, 2024

ClusterTabNet: Supervised clustering method for table detection and table structure recognition
Marek Polewczyk, Marco Spinaci
Deep Learning Based R CNN Table Structure Recognition Table Detection OCR Annotation Transformer Encoder Model Supervised Clustering

August 19, 2023

A Transformer-based Framework For Multi-variate Time Series: A Remaining Useful Life Prediction Use Case
Oluwaseyi Ogunfowora, Homayoun Najjaran
Time Series Transformer Model Transformer Based Framework Useful Life Encoder Decoder Transformer Transformer Encoder Model

December 16, 2022

Assessing the Impact of Sequence Length Learning on Classification Tasks for Transformer Encoder Models
Jean-Thomas Baillargeon, Luc Lamontagne
Machine Learning Model Global Impact Transformer Architecture Classification Task Sequence Length Transformer Encoder Model

November 20, 2022

Understanding and Improving Knowledge Distillation for Quantization-Aware Training of Large Transformer Encoders
Minsoo Kim, Sihwa Lee, Sukjin Hong, Du-Seong Chang, Jungwook Choi
Knowledge Distillation Human Understanding Quantization Aware Training Transformer Encoders Transformer Encoder Model Social Attention

September 26, 2022

Fast-FNet: Accelerating Transformer Encoder Models via Efficient Fourier Layers
Nurullah Sevim, Ege Ozan Özyedek, Furkan Şahinuç, Aykut Koç
Transformer Based Transformer Based Model Transformer Based Language Model Transformer Encoder Model Fourier Layer

Transformer Encoder Model

Papers

Assessing the significance of longitudinal data in Alzheimer's Disease forecasting

WeatherFormer: A Pretrained Encoder Model for Learning Robust Weather Representations from Small Datasets

ClusterTabNet: Supervised clustering method for table detection and table structure recognition

A Transformer-based Framework For Multi-variate Time Series: A Remaining Useful Life Prediction Use Case

Assessing the Impact of Sequence Length Learning on Classification Tasks for Transformer Encoder Models

Understanding and Improving Knowledge Distillation for Quantization-Aware Training of Large Transformer Encoders

Fast-FNet: Accelerating Transformer Encoder Models via Efficient Fourier Layers