Mixture of Attention

Mixture-of-attention (MoA) techniques aim to improve the efficiency and performance of large language models (LLMs) and other deep learning architectures by selectively combining different attention mechanisms or pathways. Current research focuses on developing algorithms that dynamically weight or switch between various attention strategies, such as sparse attention for resource efficiency or personalized attention for improved task-specific performance, often within a mixture-of-experts framework. These advancements offer significant potential for reducing computational costs, enhancing model generalization across diverse tasks, and improving the quality and diversity of generated outputs in applications ranging from natural language processing to image generation.

Papers

September 29, 2024

Learning Attentional Mixture of LoRAs for Language Model Continual Learning
Jialin Liu, Jianhua Wu, Jie Liu, Yutai Duan
Large Language Model Language Model Continual LEArning Continual Learning Continual Learning Benchmark Run LoRA Run Mixture of Attention

June 21, 2024

MoA: Mixture of Sparse Attention for Automatic Large Language Model Compression
Tianyu Fu, Haofeng Huang, Xuefei Ning, Genghan Zhang, Boju Chen, Tianqi Wu, Hongyi Wang, Zixiao Huang, Shiyao Li, Shengen Yan, Guohao Dai, Huazhong Yang, Yu Wang
Large Language Model Mixture Component Sparse Attention Attention Mask Long Context Benchmark Mixture of Attention

April 17, 2024

MoA: Mixture-of-Attention for Subject-Context Disentanglement in Personalized Image Generation
Kuan-Chieh Wang, Daniil Ostashev, Yuwei Fang, Sergey Tulyakov, Kfir Aberman
Text to Image Diffusion Model Attention Layer Personalized Image Generation Mixture of Attention

December 11, 2023

U-MixFormer: UNet-like Transformer with Mix-Attention for Efficient Semantic Segmentation
Seul-Ki Yeom, Julian von Klitzing
U Net Transformer Decoder Efficient Semantic Segmentation Transformer Based UNet Mixture of Attention U MixFormer Outperforms SegFormer Multi Scale Context

June 8, 2023

Attention Weighted Mixture of Experts with Contrastive Learning for Personalized Ranking in E-commerce
Juan Gong, Zhenlin Chen, Chaoyi Ma, Zhuojian Xiao, Haonan Wang, Guoyu Tang, Lin Liu, Sulong Xu, Bo Long, Yunjiang Jiang
Contrastive Learning Expert Knowledge E Commerce Ranking Model User Representation Ranking Task Personalized Ranking Mixture of Attention

April 17, 2023

Attention Mixtures for Time-Aware Sequential Recommendation
Viet-Anh Tran, Guillaume Salha-Galvan, Bruno Sguerra, Romain Hennequin
Transformer Architecture Sequential Recommendation Temporal Attention Sequential Recommender System Temporal Context Mixture of Attention

November 14, 2022

Evade the Trap of Mediocrity: Promoting Diversity and Novelty in Text Generation via Concentrating Attention
Wenhao Li, Xiaoyuan Yi, Jinyi Hu, Maosong Sun, Xing Xie
Text Generation Human Attention Transformer Architecture Diversity Awareness Sparse Attention Novelty Search Attention Regularization Average Performance Mixture of Attention

June 5, 2022

A Simple Meta-learning Paradigm for Zero-shot Intent Classification with Mixture Attention Mechanism
Han Liu, Siyang Zhao, Xiaotong Zhang, Feng Zhang, Junjie Sun, Hong Yu, Xianchao Zhang
Meta Learning Dialogue Datasets Zero Shot Classification Task Mixture of Attention Zero Shot Intent Classification

May 24, 2022

ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Prompts
Akari Asai, Mohammadreza Salehi, Matthew E. Peters, Hannaneh Hajishirzi
Prompt Tuning Soft Prompt Multi Task Training Manual Effort Mixture of Attention

Mixture of Attention

Papers

Learning Attentional Mixture of LoRAs for Language Model Continual Learning

MoA: Mixture of Sparse Attention for Automatic Large Language Model Compression

MoA: Mixture-of-Attention for Subject-Context Disentanglement in Personalized Image Generation

U-MixFormer: UNet-like Transformer with Mix-Attention for Efficient Semantic Segmentation

Attention Weighted Mixture of Experts with Contrastive Learning for Personalized Ranking in E-commerce

Attention Mixtures for Time-Aware Sequential Recommendation

Evade the Trap of Mediocrity: Promoting Diversity and Novelty in Text Generation via Concentrating Attention

A Simple Meta-learning Paradigm for Zero-shot Intent Classification with Mixture Attention Mechanism

ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Prompts