Self Attention
Self-attention is a mechanism in neural networks that allows the model to weigh the importance of different parts of the input data when processing it, enabling the capture of long-range dependencies and contextual information. Current research focuses on improving the efficiency of self-attention, particularly in vision transformers and other large models, through techniques like low-rank approximations, selective attention, and grouped query attention, aiming to reduce computational costs while maintaining accuracy. These advancements are significantly impacting various fields, including computer vision, natural language processing, and time series analysis, by enabling more efficient and powerful models for tasks such as image restoration, text-to-image generation, and medical image segmentation.
Papers
Spatiotemporal Learning of Brain Dynamics from fMRI Using Frequency-Specific Multi-Band Attention for Cognitive and Psychiatric Applications
Sangyoon Bae, Junbeom Kwon, Shinjae Yoo, Jiook ChaSeoul National University●University of Texas at Austin●Brookhaven National LaboratoryObject Isolated Attention for Consistent Story Visualization
Xiangyang Luo, Junhao Cheng, Yifan Xie, Xin Zhang, Tao Feng, Zhou Liu, Fei Ma, Fei YuGuangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)●Tsinghua University●Sun Yat-sen University
Attention Distillation: A Unified Approach to Visual Characteristics Transfer
Yang Zhou, Xu Gao, Zichong Chen, Hui HuangShenzhen UniversityEnhancing Transformer with GNN Structural Knowledge via Distillation: A Novel Approach
Zhihua Duan, Jialin WangChina Telecom Shanghai Company●Ferret Relationship Intelligence