Efficient Transfer Learning

Efficient transfer learning (ETL) aims to adapt large pre-trained models to new tasks using minimal computational resources and data, focusing on reducing both the number of updated parameters and the overall training time. Current research emphasizes parameter-efficient techniques like prompt tuning, adapters, and various forms of side networks, often applied to vision-language models, transformers, and other deep learning architectures. These advancements are crucial for deploying large models in resource-constrained environments and accelerating the development of AI solutions across diverse fields, including medical imaging, industrial signal processing, and natural language processing. The ultimate goal is to achieve comparable or even superior performance to full fine-tuning with significantly reduced computational cost.

Papers

February 26, 2023

Scalable Weight Reparametrization for Efficient Transfer Learning
Byeonggeun Kim, Jun-Tae Lee, Seunghan yang, Simyung Chang
Transfer Learning Pre Trained Model Efficient Transfer Learning

December 23, 2022

Principled and Efficient Transfer Learning of Deep Models via Neural Collapse
Xiao Li, Sheng Liu, Jinxin Zhou, Xinyu Lu, Carlos Fernandez-Granda, Zhihui Zhu, Qing Qu
Fine Tuning Transfer Learning Deep Model Large Pre Trained Model Neural Collapse Efficient Transfer Learning Variability Collapse

December 6, 2022

Visual Query Tuning: Towards Effective Usage of Intermediate Representations for Parameter and Memory Efficient Transfer Learning
Cheng-Hao Tu, Zheda Mai, Wei-Lun Chao
Vision Transformer Transfer Learning Pre Trained Model Many Parameter Intermediate Representation Effective Approach Efficient Transfer Learning Visual Query

November 18, 2022

Task Residual for Tuning Vision-Language Models
Tao Yu, Zhihe Lu, Xin Jin, Zhibo Chen, Xinchao Wang
Vision Language Model Pre Trained Model Efficient Transfer Learning

November 4, 2022

October 3, 2022

Towards a Unified View on Visual Parameter-Efficient Transfer Learning
Bruce X. B. Yu, Jianlong Chang, Lingbo Liu, Qi Tian, Chang Wen Chen
Vision Task Large Vision Model Unified View Parameter Efficient Transfer Learning Efficient Transfer Learning Based Method

July 4, 2022

Revisiting Classifier: Transferring Vision-Language Models for Video Recognition
Wenhao Wu, Zhun Sun, Wanli Ouyang
Vision Language Model Pre Trained Model Vision Language Vision Task Video Recognition Video Classification Efficient Transfer Learning Classifier Re Training

June 30, 2022

Neural Networks can Learn Representations with Gradient Descent
Alex Damian, Jason D. Lee, Mahdi Soltanolkotabi
Neural Network Transfer Learning Gradient Descent Meaningful Representation Kernel Method Efficient Transfer Learning Kernel Regime

June 13, 2022

LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning
Yi-Lin Sung, Jaemin Cho, Mohit Bansal
Many Parameter Large Pre Trained Model Pre Trained Backbone Efficient Transfer Learning Parameter Efficient Transfer

June 10, 2022

A Correlation-Ratio Transfer Learning and Variational Stein's Paradox
Lu Lin, Weiyu Li
Transfer Learning Efficient Transfer Learning Stein Variational Gradient Descent Fermi Paradox

March 31, 2022

Scaling Language Model Size in Cross-Device Federated Learning
Jae Hun Ro, Theresa Breiner, Lara McConnaughey, Mingqing Chen, Ananda Theertha Suresh, Shankar Kumar, Rajiv Mathews
Language Model Neural Network Larger Language Model Efficient Transfer Learning Cross Device Federated Learning Partial Model Training

March 8, 2022

HyperPELT: Unified Parameter-Efficient Language Model Tuning for Both Language and Vision-and-Language Tasks
Zhengkun Zhang, Wenya Guo, Xiaojun Meng, Yasheng Wang, Yadao Wang, Xin Jiang, Qun Liu, Zhenglu Yang
Multi Task Learning Human Language Vision Language Task Parameter Efficient Transfer Learning Efficient Transfer Learning Learned Embeddings Hyper Representation