Transformer Network

Transformer networks are a class of deep learning models designed to process sequential data by leveraging self-attention mechanisms, enabling the capture of long-range dependencies within the data. Current research focuses on optimizing transformer architectures for efficiency and generalization, including exploring sparse connections, pruning techniques, and specialized hardware acceleration, as well as adapting them for diverse applications beyond natural language processing, such as image analysis, time series prediction, and signal processing. This versatility makes transformers a powerful tool across numerous scientific fields and practical applications, driving advancements in areas ranging from medical image analysis to autonomous driving and energy management.

Papers

March 24, 2022

Keypoints Tracking via Transformer Networks
Oleksii Nasypanyi, Francois Rameau
Transformer Network Keypoint Detector Supervised Keypoint Vision Transformer Network Sparse Keypoints Keypoints Representation Keypoint Tracking

March 22, 2022

Under the Hood of Transformer Networks for Trajectory Forecasting
Luca Franco, Leonardo Placidi, Francesco Giuliari, Irtiza Hasan, Marco Cristani, Fabio Galasso
Transformer Network Sequence Modeling Bidirectional Transformer Trajectory Forecasting Future Multimodal

March 20, 2022

Delta Keyword Transformer: Bringing Transformers to the Edge through Dynamically Pruned Multi-Head Self-Attention
Zuzana Jelčicová, Marian Verhelst
Transformer Megatron Decepticons Extreme Edge Transformer Network Multi Head Self Attention Dynamic Pruning

March 12, 2022

Joint CNN and Transformer Network via weakly supervised Learning for efficient crowd counting
Fusen Wang, Kai Liu, Fei Long, Nong Sang, Xiaofeng Xia, Jun Sang
LeArning Abstract Transformer Network Crowd Counting Joint Neural Supervised Counting

February 19, 2022

Do Transformers know symbolic rules, and would we know if they did?
Tommi Gröndahl, Yujia Guo, N. Asokan
Transformer Megatron Decepticons Transformer Network Sequence to Sequence Task Associative Recall Symbolic Task

February 18, 2022

Spatio-Temporal Outdoor Lighting Aggregation on Image Sequences using Transformer Networks
Haebom Lee, Christian Homeyer, Robert Herzog, Jan Rexilius, Carsten Rother
Transformer Network Image Sequence Ego Motion Estimation Lighting Estimation Scene Lighting

February 16, 2022

ActionFormer: Localizing Moments of Actions with Transformers
Chenlin Zhang, Jianxin Wu, Yin Li
Transformer Megatron Decepticons Video Understanding Transformer Network Temporal Action Localization Past Action Temporal Moment Local Self Attention

February 8, 2022

January 4, 2022

PyramidTNT: Improved Transformer-in-Transformer Baselines with Pyramid Architecture
Kai Han, Jianyuan Guo, Yehui Tang, Yunhe Wang
Vision Transformer Transformer Network Pyramid Transformer Transformer Baseline Fine Scale Gaussian Pyramid Level

December 16, 2021

Trees in transformers: a theoretical analysis of the Transformer's ability to represent trees
Qi He, João Sedoc, Jordan Rodu
Natural Language Processing Transformer Megatron Decepticons Transformer Based Transformer Architecture Theoretical Analysis Tree Specie Critique Ability Transformer Network Light Weighed Backbone

November 26, 2021

BCH-NLP at BioCreative VII Track 3: medications detection in tweets using transformer networks and multi-task learning
Dongfang Xu, Shan Chen, Timothy Miller
Multi Task Learning Text Classification Twitter Tweet Transformer Network Sequence Labeling Automatic Extraction Quantum Natural Language Processing

November 22, 2021

November 19, 2021

Rethinking Query, Key, and Value Embedding in Vision Transformer under Tiny Model Constraints
Jaesin Ahn, Jiuk Hong, Jeongwoo Ju, Heechul Jung
Vision Transformer Computer Vision Transformer Based Model Transformer Network Net Present Value ImageNet 1k Constraint Compliance

November 13, 2021

Session-aware Item-combination Recommendation with Transformer Network
Tzu-Heng Lin, Chen Gao
Transformer Network Transformer Based Network Session Based Item Combination Click Prediction Big Data Cup

Transformer Network

Papers

Keypoints Tracking via Transformer Networks

Under the Hood of Transformer Networks for Trajectory Forecasting

Delta Keyword Transformer: Bringing Transformers to the Edge through Dynamically Pruned Multi-Head Self-Attention

Joint CNN and Transformer Network via weakly supervised Learning for efficient crowd counting

Do Transformers know symbolic rules, and would we know if they did?

Spatio-Temporal Outdoor Lighting Aggregation on Image Sequences using Transformer Networks

ActionFormer: Localizing Moments of Actions with Transformers

TransformNet: Self-supervised representation learning through predicting geometric transformations

Efficacy of Transformer Networks for Classification of Raw EEG Data

PyramidTNT: Improved Transformer-in-Transformer Baselines with Pyramid Architecture

Trees in transformers: a theoretical analysis of the Transformer's ability to represent trees

BCH-NLP at BioCreative VII Track 3: medications detection in tweets using transformer networks and multi-task learning

Ice hockey player identification via transformers and weakly supervised learning

DBIA: Data-free Backdoor Injection Attack against Transformer Networks

Rethinking Query, Key, and Value Embedding in Vision Transformer under Tiny Model Constraints

Session-aware Item-combination Recommendation with Transformer Network