Global Attention
Global attention mechanisms in machine learning aim to capture long-range dependencies and global context within data, improving model performance beyond local interactions. Current research focuses on developing efficient global attention models, particularly within transformer architectures, and integrating them with local attention for a balanced approach, addressing computational limitations through techniques like grouped attention, linear attention, and efficient sampling methods. These advancements are significantly impacting various fields, including computer vision (e.g., image segmentation, object detection), natural language processing (e.g., long-context understanding), and graph learning (e.g., node classification), by enhancing model accuracy and scalability. The resulting models demonstrate improved performance on complex tasks and large datasets.
Papers
Dilated-UNet: A Fast and Accurate Medical Image Segmentation Approach using a Dilated Transformer and U-Net Architecture
Davoud Saadati, Omid Nejati Manzari, Sattar Mirzakuchaki
SACANet: scene-aware class attention network for semantic segmentation of remote sensing images
Xiaowen Ma, Rui Che, Tingfeng Hong, Mengting Ma, Ziyan Zhao, Tian Feng, Wei Zhang