MLP Layer

Multilayer perceptrons (MLPs) are a fundamental component of many neural network architectures, particularly within transformer models, where they process information between attention layers. Current research focuses on understanding MLPs' role in knowledge representation and storage within large language models, investigating redundancy within MLP layers for efficiency improvements, and exploring alternative MLP designs like MIMO-MLPs for faster processing in applications such as neural rendering. These investigations are crucial for improving the efficiency, interpretability, and performance of large-scale neural networks across various domains, from natural language processing to computer vision.

Papers

October 14, 2024

Jailbreak Instruction-Tuned LLMs via end-of-sentence MLP Re-weighting
Yifan Luo, Zhennan Zhou, Meitan Wang, Bin Dong
Large Language Model Jailbreak Attack Explanatory Inference Unsafe Prompt MLP Layer

October 10, 2024

Bilinear MLPs enable weight-based mechanistic interpretability
Michael T. Pearce, Thomas Dooms, Alice Rigg, Jose M. Oramas, Lee Sharkey
Deep Learning Mechanistic Interpretability MLP Layer Bilinear Convolutional Neural Network

October 9, 2024

SparseGrad: A Selective Method for Efficient Fine-tuning of MLP Layers
Viktoriia Chekalina, Anna Rudenko, Gleb Mezentsev, Alexander Mikhalev, Alexander Panchenko, Ivan Oseledets
Transformer Model Parameter Efficient Fine Tuning Efficient Fine Tuning Sparse Structure Fine Tuned BERT Sequential Selection MLP Layer

June 22, 2024

What Matters in Transformers? Not All Attention is Needed
Shwai He, Guoheng Sun, Zheyu Shen, Ang Li
Large Language Model Transformer Megatron Decepticons Human Attention High Similarity Attention Layer Feature Redundancy MLP Layer

February 21, 2024

Cracking Factual Knowledge: A Comprehensive Analysis of Degenerate Knowledge Neurons in Large Language Models
Yuheng Chen, Pengfei Cao, Yubo Chen, Yining Wang, Shengping Liu, Kang Liu, Jun Zhao
Large Language Model Multi Layer Comprehensive Analysis Factual Knowledge Topological Clustering MLP Layer Knowledge Neuron

October 11, 2023

An Adversarial Example for Direct Logit Attribution: Memory Management in gelu-4l
James Dao, Yeu-Tong Lau, Can Rager, Jett Janiak
Language Model Adversarial Example Adversarial Prompt Memory Management Residual Stream MLP Layer Interpretability Illusion

October 3, 2023

MIMO-NeRF: Fast Neural Rendering with Multi-input Multi-output Neural Radiance Fields
Takuhiro Kaneko
Neural Radiance Field Novel View Synthesis Multi Layer Neural Rendering Multiple Input Multiple Output Human NeRF MLP Layer

September 15, 2023

Attention-Only Transformers and Implementing MLPs with Attention Heads
Robert Huben, Valerie Morris
Transformer Architecture Attention Head MLP Layer Attention Transformer

July 21, 2023

Strip-MLP: Efficient Token Interaction for Vision MLP
Guiping Cao, Shengda Luo, Wenjian Huang, Xiangyuan Lan, Dongmei Jiang, Yaowei Wang, Jianguo Zhang
MLP Architecture MLP Layer Simple MLP

May 5, 2023

A technical note on bilinear layers for interpretability
Lee Sharkey
Neural Network Inherent Interpretability Activation Function Technical Report Bilinear Convolutional Neural Network MLP Layer

February 17, 2023

ViTA: A Vision Transformer Inference Accelerator for Edge Applications
Shashank Nag, Gourav Datta, Souvik Kundu, Nitin Chandrachoodan, Peter A. Beerel
Transformer Model Swin Transformer Hardware Accelerator Transformer Inference Edge Application MLP Layer

January 10, 2023

Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models
Peter Hase, Mohit Bansal, Been Kim, Asma Ghandeharioun
Language Model Knowledge Editing Qualitative Difference Model Editing MLP Layer

September 29, 2022

Variance Covariance Regularization Enforces Pairwise Independence in Self-Supervised Representations
Grégoire Mialon, Randall Balestriero, Yann LeCun
Self Supervised Learning Learned Representation Independent Component Analysis MLP Layer Covariance Regularization