Transformer Based Pre Trained Model
Transformer-based pre-trained models are revolutionizing various fields by providing powerful, generalizable representations for data like text, images, and tabular data. Current research focuses on improving efficiency, particularly addressing the quadratic complexity of self-attention through techniques like state space models and hybrid attention mechanisms, as well as exploring alternative classification heads beyond traditional MLPs. These advancements are enabling applications in diverse areas such as medical image analysis, natural language processing, and Internet of Things traffic classification, improving accuracy and efficiency in resource-constrained settings.
Papers
Comparative Performance Analysis of Transformer-Based Pre-Trained Models for Detecting Keratoconus Disease
Nayeem Ahmed, Md Maruf Rahman, Md Fatin Ishrak, Md Imran Kabir Joy, Md Sanowar Hossain Sabuj, Md. Sadekur Rahman
FourierKAN outperforms MLP on Text Classification Head Fine-tuning
Abdullah Al Imran, Md Farhan Ishmam