Cross Attention
Cross-attention is a mechanism that allows neural networks to relate information from different parts of an input, such as relating words in a sentence to pixels in an image, or aligning audio and video streams. Current research focuses on improving the efficiency and effectiveness of cross-attention in various applications, including image generation, video processing, and multimodal learning, often employing transformer architectures or state-space models like Mamba. This attention mechanism is proving crucial for enhancing performance in tasks requiring the integration of diverse data sources, leading to improvements in areas such as scene change detection, style transfer, and multimodal emotion recognition. The resulting advancements have significant implications for various fields, including computer vision, natural language processing, and healthcare.
Papers
A multimodal method based on cross-attention and convolution for postoperative infection diagnosis
Xianjie Liu, Hongwei Shi
Cross-Attention is Not Enough: Incongruity-Aware Dynamic Hierarchical Fusion for Multimodal Affect Recognition
Yaoting Wang, Yuanchao Li, Paul Pu Liang, Louis-Philippe Morency, Peter Bell, Catherine Lai