Mixture Component
Mixture component models are a powerful class of machine learning techniques that combine multiple specialized models (experts) to improve performance and efficiency on complex tasks. Current research focuses on developing novel architectures, such as mixtures of experts (MoE), and applying them to diverse fields including natural language processing, computer vision, and signal processing, often incorporating techniques like low-rank adaptation (LoRA) for parameter efficiency. These advancements are significant because they enable the creation of larger, more capable models while mitigating computational costs and improving generalization across heterogeneous datasets, leading to improved accuracy and efficiency in various applications.
Papers
MoDification: Mixture of Depths Made Easy
Chen Zhang, Meizhi Zhong, Qimeng Wang, Xuantao Lu, Zheyu Ye, Chengqiang Lu, Yan Gao, Yao Hu, Kehai Chen, Min Zhang, Dawei Song
Provable In-context Learning for Mixture of Linear Regressions using Transformers
Yanhao Jin, Krishnakumar Balasubramanian, Lifeng Lai
Efficiently Democratizing Medical LLMs for 50 Languages via a Mixture of Language Family Experts
Guorui Zheng, Xidong Wang, Juhao Liang, Nuo Chen, Yuping Zheng, Benyou Wang
Tighter Risk Bounds for Mixtures of Experts
Wissam Akretche, Frédéric LeBlanc, Mario Marchand
Mixture of Experts Made Personalized: Federated Prompt Learning for Vision-Language Models
Jun Luo, Chen Chen, Shandong Wu
A Federated Distributionally Robust Support Vector Machine with Mixture of Wasserstein Balls Ambiguity Set for Distributed Fault Diagnosis
Michael Ibrahim, Heraldo Rozas, Nagi Gebraeel, Weijun Xie
Collaborative and Efficient Personalization with Mixtures of Adaptors
Abdulla Jasem Almansoori, Samuel Horváth, Martin Takáč
Mixture of Attentions For Speculative Decoding
Matthieu Zimmer, Milan Gritta, Gerasimos Lampouras, Haitham Bou Ammar, Jun Wang