Transformer Module

Transformer modules are self-contained units within larger transformer networks, designed to improve efficiency, adaptability, and performance. Current research focuses on optimizing these modules for various tasks, including knowledge distillation between modules (m2mKD), flexible model scaling through linear expansion (TLEG), and efficient compression techniques for resource-constrained environments. This modular approach enhances the flexibility and scalability of transformer architectures, leading to improvements in diverse applications such as image classification, autonomous driving behavior analysis, and medical image segmentation.

Papers

October 24, 2024

Ubiquitous Field Transportation Robots with Robust Wheel-Leg Transformable Modules
Haoran Wang, Cunxi Dai, Siyuan Wang, Ximan Zhang, Zheng Zhu, Xiaohan Liu, Jianxiang Zhou, Zhengtao Liu, Zhenzhong Jia
Field Robotics Transformer Module Wheel Leg

February 26, 2024

m2mKD: Module-to-Module Knowledge Distillation for Modular Transformers
Ka Man Lo, Yiming Liang, Wenyu Du, Yuantao Fan, Zili Wang, Wenhao Huang, Lei Ma, Jie Fu
Knowledge Distillation Neural Architecture Transformer Module Modular Neural Module Wise

December 9, 2023

Transformer as Linear Expansion of Learngene
Shiyu Xia, Miaosen Zhang, Xu Yang, Ruiming Chen, Haokun Chen, Xin Geng
Transformer Based Learngene Layer Transformer Module

October 21, 2023

Exploring Driving Behavior for Autonomous Vehicles Based on Gramian Angular Field Vision Transformer
Junwei You, Ying Chen, Zhuoyu Jiang, Zhangchi Liu, Zilin Huang, Yifeng Ding, Bin Ran
Autonomous Vehicle Driver Behavior ViT Model Driving Algorithm Transformer Module Gramian Angular Field

June 4, 2023

Modular Transformers: Compressing Transformers into Modularized Layers for Flexible Efficient Inference
Wangchunshu Zhou, Ronan Le Bras, Yejin Choi
Transformer Megatron Decepticons Transformer Model Multi Layer Model Compression Efficient Inference Sequence to Sequence Model Encoder Decoder Framework Transformer Module

February 1, 2023

An Empirical Study on the Transferability of Transformer Modules in Parameter-Efficient Fine-Tuning
Mohammad Akbar-Tajari, Sara Rajaee, Mohammad Taher Pilehvar
Pre Trained Model Empirical Study Parameter Efficient Fine Tuning Task Transferability Pre Trained BERT Transformer Module

August 22, 2022

PoseBERT: A Generic Transformer Module for Temporal 3D Human Modeling
Fabien Baradel, Romain Brégier, Thibault Groueix, Philippe Weinzaepfel, Yannis Kalantidis, Grégory Rogez
Human Pose Estimation Transformer Module

May 12, 2022

Lifting the Curse of Multilinguality by Pre-training Modular Transformers
Jonas Pfeiffer, Naman Goyal, Xi Victoria Lin, Xian Li, James Cross, Sebastian Riedel, Mikel Artetxe
Reversal Curse Language Specific Cross Lingual Performance Multilingual Pre Trained Model Transformer Module Language Specific Module

January 27, 2022

Transformer Module Networks for Systematic Generalization in Visual Question Answering
Moyuru Yamada, Vanessa D'Amario, Kentaro Takemoto, Xavier Boix, Tomotake Sasaki
Visual Question Answering Systematic Generalization Neural Module Transformer Module

November 8, 2021

Mixed Transformer U-Net For Medical Image Segmentation
Hongyi Wang, Shiao Xie, Lanfen Lin, Yutaro Iwamoto, Xian-Hua Han, Yen-Wei Chen, Ruofeng Tong
Vision Transformer Medical Image Segmentation Self Attention U Net Transformer Module

Transformer Module

Papers

Ubiquitous Field Transportation Robots with Robust Wheel-Leg Transformable Modules

m2mKD: Module-to-Module Knowledge Distillation for Modular Transformers

Transformer as Linear Expansion of Learngene

Exploring Driving Behavior for Autonomous Vehicles Based on Gramian Angular Field Vision Transformer

Modular Transformers: Compressing Transformers into Modularized Layers for Flexible Efficient Inference

An Empirical Study on the Transferability of Transformer Modules in Parameter-Efficient Fine-Tuning

PoseBERT: A Generic Transformer Module for Temporal 3D Human Modeling

Lifting the Curse of Multilinguality by Pre-training Modular Transformers

Transformer Module Networks for Systematic Generalization in Visual Question Answering

Mixed Transformer U-Net For Medical Image Segmentation