MLP Architecture

Multi-layer perceptron (MLP) architectures are experiencing a resurgence in machine learning, driven by efforts to create efficient and effective models for various tasks, including image classification, point cloud processing, and natural language processing. Current research focuses on developing novel MLP designs, such as those incorporating addition and shift operations for improved efficiency, dynamic mixing mechanisms for adaptive feature fusion, and specialized modules to enhance local and global information processing. These advancements are significant because they offer alternatives to computationally expensive methods like transformers and convolutions, potentially leading to faster inference times and improved performance on resource-constrained devices while also providing insights into the fundamental capabilities of MLPs.

Papers

April 1, 2022

Adaptive hybrid speech coding with a MLP LPC structure
Marcos Faundez-Zanuy
Adaptive Importance Nonlinear Model Speech Model Nonlinear Prediction MLP Architecture Hybrid Text

March 14, 2022

Efficient Language Modeling with Sparse all-MLP
Ping Yu, Mikel Artetxe, Myle Ott, Sam Shleifer, Hongyu Gong, Ves Stoyanov, Xian Li
Language Model Many Sparse Mixture of Expert MLP Architecture Dense Transformer

February 17, 2022

MLP-ASR: Sequence-length agnostic all-MLP architectures for speech recognition
Jin Sakuma, Tatsuya Komatsu, Robin Scheibler
Automatic Speech Recognition Speech Recognition Self Attention Multi Layer Multi Layer Perceptron MLP Architecture

February 14, 2022

Mixing and Shifting: Exploiting Global and Local Dependencies in Vision MLPs
Huangjie Zheng, Pengcheng He, Weizhu Chen, Mingyuan Zhou
Multi Layer World Event Multi Layer Perceptron Two Shift MLP Architecture Vision Benchmark Mixing Process Local Dependency

February 9, 2022

pNLP-Mixer: an Efficient all-MLP Architecture for Language
Francesco Fusco, Damian Pascual, Peter Staar, Diego Antognini
Human Language MLP Mixer MLP Architecture

January 28, 2022

DynaMixer: A Vision MLP Architecture with Dynamic Mixing
Ziyu Wang, Wenhao Jiang, Yiming Zhu, Li Yuan, Yibing Song, Wei Liu
Global Mixing MLP Architecture Multi Level Fusion MLP Like Model Simple MLP

December 8, 2021

MLP Architectures for Vision-and-Language Modeling: An Empirical Study
Yixin Nie, Linjie Li, Zhe Gan, Shuohang Wang, Chenguang Zhu, Michael Zeng, Zicheng Liu, Mohit Bansal, Lijuan Wang
Empirical Study MLP Architecture Vision and Language