Parameter Efficient Adapter

Parameter-efficient adapters are lightweight neural network modules designed to adapt large pre-trained models to new tasks with minimal parameter updates, thus reducing computational cost and storage requirements. Research focuses on optimizing adapter architectures, exploring techniques like low-rank matrices, Hadamard products, and hypernetworks to achieve extreme parameter efficiency while maintaining performance comparable to full fine-tuning. These methods are significantly impacting various fields, enabling efficient transfer learning in natural language processing, computer vision, and other domains where adapting large models to specific tasks is crucial. The development of more efficient and adaptable adapters continues to be a key area of investigation.

Papers

October 21, 2024

Towards Optimal Adapter Placement for Efficient Transfer Learning
Aleksandra I. Nowak, Otniel-Bogdan Mercea, Anurag Arnab, Jonas Pfeiffer, Yann Dauphin, Utku Evci
Pre Trained Model Parameter Efficient Transfer Learning Efficient Transfer Learning Adapter Tuning Hierarchical Recurrent Parameter Efficient Adapter

August 30, 2024

MoRe Fine-Tuning with 10x Fewer Parameters
Wenxuan Tan, Nicholas Roberts, Tzu-Heng Huang, Jitian Zhao, John Cooper, Samuel Guo, Chengyu Duan, Frederic Sala
Neural Architecture Search Parameter Efficient Fine Tuning Many Parameter Low Rank Adapter Parameter Efficient Adapter

July 4, 2024

Hadamard Adapter: An Extreme Parameter-Efficient Adapter Tuning Method for Pre-trained Language Models
Yuyan Chen, Qiang Fu, Ge Fan, Lun Du, Jian-Guang Lou, Shi Han, Dongmei Zhang, Zhixu Li, Yanghua Xiao
Fine Tuning Pre Trained Language Model Adapter Tuning Parameter Efficient Adapter Hadamard Adapter

July 1, 2024

HyperLoader: Integrating Hypernetwork-Based LoRA and Adapter Layers into Multi-Task Transformers for Sequence Labelling
Jesus-German Ortiz-Barajas, Helena Gomez-Adorno, Thamar Solorio
Fine Tuning Multi Task Learning Sequence Labeling Task Specific Knowledge Safe LoRA Multi Task Transformer Parameter Efficient Adapter

January 18, 2024

Adapters Mixup: Mixing Parameter-Efficient Adapters to Enhance the Adversarial Robustness of Fine-tuned Pre-trained Text Classifiers
Tuc Nguyen, Thai Le
Adversarial Attack Adversarial Example Adversarial Training Adversarial Robustness Adapter Fusion Adversarial Augmentation Parameter Efficient Adapter

November 28, 2023

PEA-Diffusion: Parameter-Efficient Adapter with Knowledge Distillation in non-English Text-to-Image Generation
Jian Ma, Chen Chen, Qingsong Xie, Haonan Lu
Knowledge Distillation Text to Image Diffusion Model Text to Image Generation Text to Image Model Parameter Efficient Adapter

October 10, 2023

Efficient Adaptation of Large Vision Transformer via Adapter Re-Composing
Wei Dong, Dawei Yan, Zhijun Lin, Peng Wang
Vision Transformer Pre Trained Model Efficient Adaptation Efficient Model Adaptation Large Pre Trained Parameter Efficient Adapter Conditional Adapter Vision Transformer Variant

October 1, 2023

PACIA: Parameter-Efficient Adapter for Few-Shot Molecular Property Prediction
Shiguang Wu, Yaqing Wang, Quanming Yao
Graph Neural Network Molecular Property Prediction Parameter Efficient Adapter Shot Molecular Property Prediction

May 26, 2023

CAILA: Concept-Aware Intra-Layer Adapters for Compositional Zero-Shot Learning
Zhaoheng Zheng, Haidong Zhu, Ram Nevatia
Pre Trained Model Semantic Space Compositional Zero Shot Learning CLIP Vision Encoder Parameter Efficient Adapter Large Scale Vision Language

May 7, 2022

Empowering parameter-efficient transfer learning by recognizing the kernel structure in self-attention
Yifan Chen, Devamanyu Hazarika, Mahdi Namazifar, Yang Liu, Di Jin, Dilek Hakkani-Tur
Pre Trained Language Model Self Attention Language Generation Parameter Efficient Transfer Learning Kernel Based Parameter Efficient Transfer Parameter Efficient Adapter Kernel Structure

May 3, 2022

Adaptable Adapters
Nafise Sadat Moosavi, Quentin Delfosse, Kristian Kersting, Iryna Gurevych
Neural Network NLP Model Parameter Efficient Adapter Efficient Adapter

December 13, 2021

VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks
Yi-Lin Sung, Jaemin Cho, Mohit Bansal
Language Model Vision Language Task Video Text Parameter Efficient Transfer Learning Video Text Task SAM2 Adapter Parameter Efficient Adapter Image to Text Task