Complex Transformer

Complex Transformer models, while powerful in various machine learning tasks like time series forecasting and image segmentation, present challenges related to computational cost and complexity. Current research focuses on developing more efficient architectures, such as lightweight MLP-Mixer models and hybrid CNN-Transformer approaches, aiming to reduce resource demands while maintaining or improving performance. These efforts are driven by the need to deploy these powerful models in resource-constrained environments and to improve the accessibility of advanced AI techniques for broader applications. The ultimate goal is to achieve a balance between model accuracy and efficiency, making complex Transformers more practical and widely applicable.

Papers

August 8, 2024

Transformer Explainer: Interactive Learning of Text-Generative Models
Aeree Cho, Grace C. Kim, Alexander Karpekov, Alec Helbling, Zijie J. Wang, Seongmin Lee, Benjamin Hoover, Duen Horng Chau
Generative Model Transformer Megatron Decepticons Transformer Based Interactive Learning Modern Generative AI Complex Transformer

January 5, 2024

UnetTSF: A Better Performance Linear Complexity Time Series Prediction Model
Li Chu, Xiao Bingjia, Yuan Qiping
Transformer Based Model U Net Time Series Data Time Series Prediction Complex Transformer

June 16, 2023

Building Blocks for a Complex-Valued Transformer Architecture
Florian Eilers, Xiaoyi Jiang
Transformer Architecture Layer Normalization Complex Valued Neural Network Building Block Complex Signal Vector Attention Complex Transformer

June 14, 2023

TSMixer: Lightweight MLP-Mixer Model for Multivariate Time Series Forecasting
Vijay Ekambaram, Arindam Jati, Nam Nguyen, Phanwadee Sinthong, Jayant Kalagnanam
Multivariate Time Series Forecasting Long Term Forecasting MLP Mixer Time Series Foundation Model Patch Transformer Complex Transformer

April 28, 2023

DIAMANT: Dual Image-Attention Map Encoders For Medical Image Segmentation
Yousef Yeganeh, Azade Farshad, Peter Weinberger, Seyed-Ahmad Ahmadi, Ehsan Adeli, Nassir Navab
Medical Image Segmentation Convolutional Layer Transformer Based Architecture Hybrid Transformer Vision Transformer Network Complex Transformer

November 9, 2022

Efficient Large-scale Audio Tagging via Transformer-to-CNN Knowledge Distillation
Florian Schmid, Khaled Koutini, Gerhard Widmer
Knowledge Distillation Audio Spectrogram Transformer Efficient CNN Audio Tagging Complex Transformer