Shallow Transformer

Shallow transformers, possessing fewer layers than their deeper counterparts, are becoming a significant area of research in machine learning, driven by the need for faster and more efficient models while maintaining performance. Current work focuses on understanding their theoretical properties, including convergence and generalization capabilities, and exploring their effectiveness in various applications like text retrieval, video action detection, and in-context learning. This research is important because it addresses the computational limitations of large language models, potentially enabling deployment on resource-constrained devices and improving the efficiency of existing applications.

Papers

March 29, 2024

Shallow Cross-Encoders for Low-Latency Retrieval
Aleksandr V. Petrov, Sean MacAvaney, Craig Macdonald
Transformer Based Cross Encoder Shallow Transformer

February 1, 2024

Theoretical Understanding of In-Context Learning in Shallow Transformers with Unstructured Data
Yue Xing, Xiaofeng Lin, Chenheng Xu, Namjoon Suh, Qifan Song, Guang Cheng
Large Language Model Context Learning Transformer Architecture Theoretical Understanding Attention Layer Unstructured Data Shallow Transformer

December 4, 2023

Adapting Short-Term Transformers for Action Detection in Untrimmed Videos
Min Yang, Huan Gao, Ping Guo, Limin Wang
Vision Transformer Temporal Transformer Temporal Action Detection Action Detection Video Transformer Untrimmed Video Shallow Transformer

November 2, 2023

On the Convergence of Encoder-only Shallow Transformers
Yongtao Wu, Fanghui Liu, Grigorios G Chrysos, Volkan Cevher
Transformer Based Early Stage Convergence Neural Tangent Kernel Shallow Transformer

October 2, 2023

Linear attention is (maybe) all you need (to understand transformer optimization)
Kwangjun Ahn, Xiang Cheng, Minhak Song, Chulhee Yun, Ali Jadbabaie, Suvrit Sra
Transformer Based Transformer Model Linear Attention Linear Transformer Transformer Training Shallow Transformer

May 30, 2023

Grokking of Hierarchical Structure in Vanilla Transformers
Shikhar Murty, Pratyusha Sharma, Jacob Andreas, Christopher D. Manning
Transformer Language Model Hierarchical Structure Grokking Phenomenon Vanilla Transformer Neural Sequence Model Shallow Transformer

February 12, 2023

A Theoretical Understanding of Shallow Vision Transformers: Learning, Generalization, and Sample Complexity
Hongkang Li, Meng Wang, Sijia Liu, Pin-yu Chen
LeArning Abstract Vision Transformer Strong Generalization Theoretical Understanding Sample Complexity Attention Map Self Attention Layer Self Attention Module Shallow Transformer

January 30, 2023

Looped Transformers as Programmable Computers
Angeliki Giannou, Shashank Rajput, Jy-yong Sohn, Kangwook Lee, Jason D. Lee, Dimitris Papailiopoulos
Transformer Based Transformer Network Layer Transformer Encoder Layer Shallow Transformer Programmable Rail

June 1, 2022

B2T Connection: Serving Stability and Performance in Deep Transformers
Sho Takase, Shun Kiyono, Sosuke Kobayashi, Jun Suzuki
System Performance Back Propagation Core Stability Layer Normalization Deep Transformer Shallow Transformer