Parameter Efficient Adaptation

Parameter-efficient adaptation (PEA) focuses on modifying only a small subset of parameters in large pre-trained models to adapt them to new tasks, thereby reducing computational costs and storage needs. Current research explores various PEA techniques, including low-rank adaptation (LoRA), adapter modules, and prompt tuning, often applied to transformer-based architectures across diverse domains like vision, language, and speech. This field is significant because it enables the practical application of massive pre-trained models to resource-constrained settings and diverse downstream tasks, improving efficiency and accessibility in various applications.

Papers

October 23, 2024

PETAH: Parameter Efficient Task Adaptation for Hybrid Transformers in a resource-limited Context
Maximilian Augustin, Syed Shakib Sarwar, Mostafa Elhoushi, Sai Qian Zhang, Yuecheng Li, Barbara De Salvo
Transformer Model Low Resource Task Adaptation Hybrid Transformer Parameter Efficient Adaptation

October 17, 2024

Parameter-efficient Adaptation of Multilingual Multimodal Models for Low-resource ASR
Abhishek Gupta, Amruta Parulekar, Sameep Chattopadhyay, Preethi Jyothi
Automatic Speech Recognition Parameter Efficient Adaptation Multilingual Multimodal Best Effort Adaptation Bilingual Multimodal

October 2, 2024

NEAT: Nonlinear Parameter-efficient Adaptation of Pre-trained Models
Yibo Zhong, Haoxiang Jiang, Lincan Li, Ryumei Nakada, Tianci Liu, Linjun Zhang, Huaxiu Yao, Haoyu Wang
Pre Trained Model Parameter Efficient Fine Tuning Efficient Fine Tuning Parameter Efficient Adaptation Pre Trained Model Weight

August 23, 2024

Open-Set Deepfake Detection: A Parameter-Efficient Adaptation Method with Forgery Style Mixture
Chenqi Kong, Anwei Luo, Peijun Bao, Haoliang Li, Renjie Wan, Zengwei Zheng, Anderson Rocha, Alex C. Kot
Deepfake Detection Face Forgery Detection Forgery Detection Parameter Efficient Adaptation

July 28, 2024

Ego-VPA: Egocentric Video Understanding with Parameter-efficient Adaptation
Tz-Ying Wu, Kyle Min, Subarna Tripathi, Nuno Vasconcelos
Video Understanding Egocentric Video Cross Modality Transfer Video Language Pre Training Egocentric Video Understanding Parameter Efficient Adaptation

May 31, 2024

Spectrum-Aware Parameter Efficient Fine-Tuning for Diffusion Models
Xinxi Zhang, Song Wen, Ligong Han, Felix Juefei-Xu, Akash Srivastava, Junzhou Huang, Hao Wang, Molei Tao, Dimitris N. Metaxas
Diffusion Model Parameter Efficient Fine Tuning Low Rank Adaptation Pre Trained Generative Model Parameter Efficient Adaptation

April 20, 2024

AdvLoRA: Adversarial Low-Rank Adaptation of Vision-Language Models
Yuheng Ji, Yue Liu, Zhicheng Zhang, Zhao Zhang, Yuting Zhao, Gang Zhou, Xingwei Zhang, Xinwang Liu, Xiaolong Zheng
Vision Language Model Adaptation Concern Parameter Efficient Adaptation Adversarial Adaptation Adaptation Method

April 6, 2024

HyperTTS: Parameter Efficient Adaptation in Text to Speech using Hypernetworks
Yingting Li, Rishabh Bhardwaj, Ambuj Mehrish, Bo Cheng, Soujanya Poria
Text Modality Speech Analysis Text to Speech Speech Synthesis Attention Hypernetworks Multi Speaker Text to Speech Speech Domain Domain Performance Parameter Efficient Adaptation

March 18, 2024

SuperLoRA: Parameter-Efficient Unified Adaptation of Multi-Layer Attention Modules
Xiangyu Chen, Jing Liu, Ye Wang, Pu Perry Wang, Matthew Brand, Guanghui Wang, Toshiaki Koike-Akino
Large Language Model Attention Module Low Rank Adaptation Tensor Factorization Parameter Efficient Adaptation

March 11, 2024

Attention Prompt Tuning: Parameter-efficient Adaptation of Pre-trained Models for Spatiotemporal Modeling
Wele Gedara Chaminda Bandara, Vishal M. Patel
Pre Trained Model Action Recognition Prompt Tuning Spatiotemporal Modeling Parameter Efficient Adaptation Non Local Attention

March 7, 2024

Discriminative Sample-Guided and Parameter-Efficient Feature Space Adaptation for Cross-Domain Few-Shot Learning
Rashindrie Perera, Saman Halgamuge
LeArning Abstract Cross Domain Unseen Domain Discriminative Reply Cross Domain Few Shot Learning Parameter Efficient Adaptation Discriminative Sample Aware Loss Function

March 1, 2024

Efficient Adapter Tuning of Pre-trained Speech Models for Automatic Speaker Verification
Mufan Sang, John H. L. Hansen
Pre Trained Model Speaker Verification Self Supervised Speech Model Parameter Efficient Transfer Learning Pre Trained Speech Model Parameter Efficient Adaptation Adapter Tuning

December 20, 2023

PPEA-Depth: Progressive Parameter-Efficient Adaptation for Self-Supervised Monocular Depth Estimation
Yue-Jiang Dong, Yuan-Chen Guo, Ying-Tian Liu, Fang-Lue Zhang, Song-Hai Zhang
Pre Trained Monocular Depth Estimation Self Supervised Monocular Depth Estimation Self Supervised Depth Parameter Efficient Adaptation Non Rigid Scene

October 27, 2023

Parameter-Efficient Methods for Metastases Detection from Clinical Notes
Maede Ashofteh Barabadi, Xiaodan Zhu, Wai Yip Chan, Amber L. Simpson, Richard K. G. Do
Self Supervised Semi Supervised Clinical Note CT Image Parameter Efficient Parameter Efficient Adaptation Task Transfer Metastasis Detection

October 6, 2023

Robust Multimodal Learning with Missing Modalities via Parameter-Efficient Adaptation
Md Kaykobad Reza, Ashley Prater-Bennette, M. Salman Asif
Multimodal Learning Missing Modality Multimodal Network Parameter Efficient Adaptation Robust Multimodal Modality Robustness Multimodal Semantic Segmentation

July 5, 2023

OpenDelta: A Plug-and-play Library for Parameter-efficient Adaptation of Pre-trained Models
Shengding Hu, Ning Ding, Weilin Zhao, Xingtai Lv, Zhen Zhang, Zhiyuan Liu, Maosong Sun
Pre Trained Model Large Pre Trained Model Plug and Play Automatic Tuning Parameter Efficient Adaptation

February 22, 2023

How Does In-Context Learning Help Prompt Tuning?
Simeng Sun, Yang Liu, Dan Iter, Chenguang Zhu, Mohit Iyyer
Context Learning Prompt Tuning Prompt Embeddings Parameter Efficient Adaptation Instruction Aware Prompt Tuning

October 13, 2022

MAPL: Parameter-Efficient Adaptation of Unimodal Pre-Trained Models for Vision-Language Few-Shot Prompting
Oscar Mañas, Pau Rodriguez, Saba Ahmadi, Aida Nematzadeh, Yash Goyal, Aishwarya Agrawal
Vision Language Large Pre Trained Model Unimodal Model Parameter Efficient Adaptation Pre Trained Unimodal

October 8, 2022

AlphaTuning: Quantization-Aware Parameter-Efficient Adaptation of Large-Scale Pre-Trained Language Models
Se Jung Kwon, Jeonghoon Kim, Jeongin Bae, Kang Min Yoo, Jin-Hwa Kim, Baeseong Park, Byeongwook Kim, Jung-Woo Ha, Nako Sung, Dongsoo Lee
Model Compression Large Scale Language Model Scale Pre Trained Language Model Hyper Tune Parameter Efficient Adaptation

July 7, 2022

Meta-Learning the Difference: Preparing Large Language Models for Efficient Adaptation
Zejiang Hou, Julian Salazar, George Polovets
Qualitative Difference Domain Adaptive Domain Specific Language Model Reparameterization Method Efficient Adaptation Parameter Efficient Adaptation

Parameter Efficient Adaptation

Papers

PETAH: Parameter Efficient Task Adaptation for Hybrid Transformers in a resource-limited Context

Parameter-efficient Adaptation of Multilingual Multimodal Models for Low-resource ASR

NEAT: Nonlinear Parameter-efficient Adaptation of Pre-trained Models

Open-Set Deepfake Detection: A Parameter-Efficient Adaptation Method with Forgery Style Mixture

Ego-VPA: Egocentric Video Understanding with Parameter-efficient Adaptation

Spectrum-Aware Parameter Efficient Fine-Tuning for Diffusion Models

AdvLoRA: Adversarial Low-Rank Adaptation of Vision-Language Models

HyperTTS: Parameter Efficient Adaptation in Text to Speech using Hypernetworks

SuperLoRA: Parameter-Efficient Unified Adaptation of Multi-Layer Attention Modules

Attention Prompt Tuning: Parameter-efficient Adaptation of Pre-trained Models for Spatiotemporal Modeling

Discriminative Sample-Guided and Parameter-Efficient Feature Space Adaptation for Cross-Domain Few-Shot Learning

Efficient Adapter Tuning of Pre-trained Speech Models for Automatic Speaker Verification

PPEA-Depth: Progressive Parameter-Efficient Adaptation for Self-Supervised Monocular Depth Estimation

Parameter-Efficient Methods for Metastases Detection from Clinical Notes

Robust Multimodal Learning with Missing Modalities via Parameter-Efficient Adaptation

OpenDelta: A Plug-and-play Library for Parameter-efficient Adaptation of Pre-trained Models

How Does In-Context Learning Help Prompt Tuning?

MAPL: Parameter-Efficient Adaptation of Unimodal Pre-Trained Models for Vision-Language Few-Shot Prompting

AlphaTuning: Quantization-Aware Parameter-Efficient Adaptation of Large-Scale Pre-Trained Language Models

Meta-Learning the Difference: Preparing Large Language Models for Efficient Adaptation