Efficient Expert

"Efficient Expert" research focuses on optimizing complex models by reducing computational costs without sacrificing performance. Current efforts concentrate on pruning large models like Mixture-of-Experts (MoE) and diffusion models into smaller, specialized "expert" sub-networks, often employing techniques like evolutionary algorithms or hierarchical feature selection to achieve this. These advancements are significant for deploying large language models and generative AI on resource-constrained devices, improving efficiency in various applications ranging from image generation to natural language processing. The ultimate goal is to enable the practical deployment of powerful models across a wider range of hardware and applications.

Papers

December 18, 2024

A Survey on Inference Optimization Techniques for Mixture of Experts Models
Jiacheng Liu, Peng Tang, Wenfeng Wang, Yuhang Ren, Xiaofeng Hou, Pheng-Ann Heng, Minyi Guo, Chao Li
Timely Survey Full Model Mixture Component Expert Knowledge Mixture of Expert Input Mixture Merging Expert Efficient Expert

September 23, 2024

Mixture of Efficient Diffusion Experts Through Automatic Interval and Sub-Network Selection
Alireza Ganjdanesh, Yan Kang, Yuchen Liu, Richard Zhang, Zhe Lin, Heng Huang
Diffusion Model Mixture Component Probabilistic Model Time Interval Efficient Expert

July 1, 2024

Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costs
Enshu Liu, Junyi Zhu, Zinan Lin, Xuefei Ning, Matthew B. Blaschko, Shengen Yan, Guohao Dai, Huazhong Yang, Yu Wang
Large Language Model Language Model System Performance Many Sparse Inference Cost Fast Inference Based Pruning Sparse Mixture of Expert Efficient Expert

March 8, 2024

Select High-Level Features: Efficient Experts from a Hierarchical Classification Network
André Kelm, Niels Hannemann, Bruno Heberle, Lucas Schmidt, Tim Rolff, Christian Wilms, Ehsan Yaghoubi, Simone Frintrop
Neural Network Hierarchical Network Low Level Feature Level Feature Efficient Expert

August 23, 2022

A Stochastic Variance Reduced Gradient using Barzilai-Borwein Techniques as Second Order Information
Hardik Tankaria, Nobuo Yamashita
Natural Gradient Stochastic Gradient Second Order Barzilai Borwein Technique Variability Collapse Efficient Expert

August 5, 2022

An Efficient Person Clustering Algorithm for Open Checkout-free Groceries
Junde Wu, Yu Zhang, Rao Fu, Yuanpei Liu, Jing Gao
Practical Algorithm Graph Convolutional Network Nearest Neighbor Dense Graph Open Checkout Free Grocery Efficient Expert

June 17, 2022

Optimization of Temperature and Relative Humidity in an Automatic Egg Incubator Using Mamdani Interference System
Pramit Dutta, Nafisa Anjum
Optimization Purpose Temperature Measurement Fuzzy System Chicken Egg Efficient Expert

Efficient Expert

Papers

A Survey on Inference Optimization Techniques for Mixture of Experts Models

Mixture of Efficient Diffusion Experts Through Automatic Interval and Sub-Network Selection

Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costs

Select High-Level Features: Efficient Experts from a Hierarchical Classification Network

A Stochastic Variance Reduced Gradient using Barzilai-Borwein Techniques as Second Order Information

An Efficient Person Clustering Algorithm for Open Checkout-free Groceries

Optimization of Temperature and Relative Humidity in an Automatic Egg Incubator Using Mamdani Interference System