Efficient Transfer Learning

Efficient transfer learning (ETL) aims to adapt large pre-trained models to new tasks using minimal computational resources and data, focusing on reducing both the number of updated parameters and the overall training time. Current research emphasizes parameter-efficient techniques like prompt tuning, adapters, and various forms of side networks, often applied to vision-language models, transformers, and other deep learning architectures. These advancements are crucial for deploying large models in resource-constrained environments and accelerating the development of AI solutions across diverse fields, including medical imaging, industrial signal processing, and natural language processing. The ultimate goal is to achieve comparable or even superior performance to full fine-tuning with significantly reduced computational cost.

Papers

October 21, 2024

Towards Optimal Adapter Placement for Efficient Transfer Learning
Aleksandra I. Nowak, Otniel-Bogdan Mercea, Anurag Arnab, Jonas Pfeiffer, Yann Dauphin, Utku Evci
Pre Trained Model Parameter Efficient Transfer Learning Efficient Transfer Learning Adapter Tuning Hierarchical Recurrent Parameter Efficient Adapter

October 4, 2024

Remaining Useful Life Prediction: A Study on Multidimensional Industrial Signal Processing and Efficient Transfer Learning Based on Large Language Models
Yan Chen, Cheng Liu
Large Language Model Study Feature Useful Life Efficient Transfer Learning Industrial Machine Learning Remaining Useful Multivariate Signal Remaining Useful Life Prediction Regression Framework

August 5, 2024

FPT+: A Parameter and Memory Efficient Transfer Learning Method for High-resolution Medical Image Classification
Yijin Huang, Pujin Cheng, Roger Tam, Xiaoying Tang
Pre Trained Model Many Parameter Large Pre Trained Model Large Scale Pre Trained Model Efficient Transfer Learning

July 26, 2024

Robust and Efficient Transfer Learning via Supernet Transfer in Warm-started Neural Architecture Search
Prabhant Singh, Joaquin Vanschoren
Transfer Learning Neural Architecture Search Robust Version Differentiable Architecture Search Supernet Training Efficient Transfer Learning

July 23, 2024

AbdomenAtlas: A Large-Scale, Detailed-Annotated, & Multi-Center Dataset for Efficient Transfer Learning and Open Algorithmic Benchmarking
Wenxuan Li, Chongyu Qu, Xiaoxi Chen, Pedro R. A. S. Bassi, Yijia Shi, Yuxiang Lai, Qian Yu, Huimin Xue, Yixiong Chen, Xiaorui Lin, Yutong Tang, Yining Cao, Haoqi Han, Zheyuan Zhang, Jiawei Liu, Tiezheng Zhang, Yujiu Ma, Jincheng Wang, Guang Zhang, Alan Yuille, Zongwei Zhou
Data Set Efficient Transfer Learning Multi Center CT Volume Abdominal CT

April 15, 2024

Conditional Prototype Rectification Prompt Learning
Haoxing Chen, Yaohui Li, Zizheng Huang, Yan Hong, Zhuoer Xu, Zhangxuan Gu, Jun Lan, Huijia Zhu, Weiqiang Wang
Pre Trained Vision Language Model Task Specific Knowledge Efficient Transfer Learning

March 31, 2024

Transfer Learning with Reconstruction Loss
Wei Cui, Wei Yu
Neural Network Transfer Learning Neural Architecture Efficient Transfer Learning Reconstruction Loss Different Neural Network

March 22, 2024

Not All Attention is Needed: Parameter and Computation Efficient Transfer Learning for Multi-modal Large Language Models
Qiong Wu, Weihao Ye, Yiyi Zhou, Xiaoshuai Sun, Rongrong Ji
Human Attention Many Parameter Multi Modal Large Language Model Information Redundancy Multi Head Attention Inference Speed Efficient Transfer Learning Skip Attention

March 12, 2024

Fine-grained Prompt Tuning: A Parameter and Memory Efficient Transfer Learning Method for High-resolution Medical Image Classification
Yijin Huang, Pujin Cheng, Roger Tam, Xiaoying Tang
Pre Trained Model Many Parameter Large Scale Pre Trained Model Efficient Transfer Learning Parameter Efficient Transfer Prompt Based Fine Tuning

February 20, 2024

CST: Calibration Side-Tuning for Parameter and Memory Efficient Transfer Learning
Feng Chen
Object Detection Many Parameter Efficient Transfer Learning DST Det Side Tuning

December 20, 2023

A Closer Look at the Few-Shot Adaptation of Large Vision-Language Models
Julio Silva-Rodríguez, Sina Hajimiri, Ismail Ben Ayed, Jose Dolz
Zero Shot Large Corpus Large Vision Language Model Adaptation Concern Glance Annotation Efficient Transfer Learning Optimal Hyperparameters

December 12, 2023

Parameter Efficient Adaptation for Image Restoration with Heterogeneous Mixture-of-Experts
Hang Guo, Tao Dai, Yuanchao Bai, Bin Chen, Xudong Ren, Zexuan Zhu, Shu-Tao Xia
Image Restoration Task Parameter Efficient Transfer Learning Efficient Transfer Learning Multi Task Adaptation

November 27, 2023

Improving Adaptability and Generalizability of Efficient Transfer Learning for Vision-Language Models
Yongjin Yang, Jongwoo Ko, Se-Young Yun
Zero Shot Vision Language Model Adaptive Importance Stronger Generalizability Task Adaptation Efficient Transfer Learning

October 20, 2023

Diagnosis-oriented Medical Image Compression with Efficient Transfer Learning
Guangqi Xie, Xin Li, Xiaohan Pan, Zhibo Chen
Medical Image Efficient Transfer Learning Semantic Compression Medical Image Compression

September 12, 2023

Dynamic Visual Prompt Tuning for Parameter Efficient Transfer Learning
Chunqing Ruan, Hongjian Wang
Visual Prompt Parameter Efficient Transfer Learning Visual Prompt Tuning Efficient Transfer Learning Downstream Recognition Visual Clue

September 4, 2023

Parameter and Computation Efficient Transfer Learning for Vision-Language Pre-trained Models
Qiong Wu, Wei Yu, Yiyi Zhou, Shubin Huang, Xiaoshuai Sun, Rongrong Ji
Full Model Vision Language Many Parameter Downstream Task Efficient Transfer Learning

August 23, 2023

Efficient Transfer Learning in Diffusion Models via Adversarial Noise
Xiyu Wang, Baijiong Lin, Daochang Liu, Chang Xu
Diffusion Model Adversarial Attack Transfer Learning GAN Model Diffusion Probabilistic Model Adversarial Noise Efficient Transfer Learning

May 26, 2023

Distilling BlackBox to Interpretable models for Efficient Transfer Learning
Shantanu Ghosh, Ke Yu, Kayhan Batmanghelich
Knowledge Distillation Interpretable Model Target Domain Interpretable Feature Efficient Transfer Learning

March 26, 2023

BlackVIP: Black-Box Visual Prompting for Robust Transfer Learning
Changdae Oh, Hyeji Hwang, Hee-young Lee, YongTaek Lim, Geunyoung Jung, Jiyoung Jung, Hosik Choi, Kyungwoo Song
Large Model Visual Prompting Large Scale Pre Trained Model Shot Adaptation Efficient Transfer Learning Robust Adaptation Robust Transfer Learning

February 26, 2023

Scalable Weight Reparametrization for Efficient Transfer Learning
Byeonggeun Kim, Jun-Tae Lee, Seunghan yang, Simyung Chang
Transfer Learning Pre Trained Model Efficient Transfer Learning

Efficient Transfer Learning

Papers

Towards Optimal Adapter Placement for Efficient Transfer Learning

Remaining Useful Life Prediction: A Study on Multidimensional Industrial Signal Processing and Efficient Transfer Learning Based on Large Language Models

FPT+: A Parameter and Memory Efficient Transfer Learning Method for High-resolution Medical Image Classification

Robust and Efficient Transfer Learning via Supernet Transfer in Warm-started Neural Architecture Search

AbdomenAtlas: A Large-Scale, Detailed-Annotated, & Multi-Center Dataset for Efficient Transfer Learning and Open Algorithmic Benchmarking

Conditional Prototype Rectification Prompt Learning

Transfer Learning with Reconstruction Loss

Not All Attention is Needed: Parameter and Computation Efficient Transfer Learning for Multi-modal Large Language Models

Fine-grained Prompt Tuning: A Parameter and Memory Efficient Transfer Learning Method for High-resolution Medical Image Classification

CST: Calibration Side-Tuning for Parameter and Memory Efficient Transfer Learning

A Closer Look at the Few-Shot Adaptation of Large Vision-Language Models

Parameter Efficient Adaptation for Image Restoration with Heterogeneous Mixture-of-Experts

Improving Adaptability and Generalizability of Efficient Transfer Learning for Vision-Language Models

Diagnosis-oriented Medical Image Compression with Efficient Transfer Learning

Dynamic Visual Prompt Tuning for Parameter Efficient Transfer Learning

Parameter and Computation Efficient Transfer Learning for Vision-Language Pre-trained Models

Efficient Transfer Learning in Diffusion Models via Adversarial Noise

Distilling BlackBox to Interpretable models for Efficient Transfer Learning

BlackVIP: Black-Box Visual Prompting for Robust Transfer Learning

Scalable Weight Reparametrization for Efficient Transfer Learning