Large Scale Pre Trained Model

Large-scale pre-trained models (LPTMs) are foundational AI models trained on massive datasets, aiming to efficiently adapt to diverse downstream tasks with minimal additional training. Current research emphasizes parameter-efficient fine-tuning techniques, such as Low-Rank Adaptation (LoRA) and prompt-based methods, to reduce computational costs and improve generalization, particularly for high-resolution data and federated learning scenarios. These advancements are significantly impacting various fields, enabling progress in areas like medical image analysis, video understanding, and natural language processing by improving model efficiency and reducing the need for extensive labeled data.

Papers

April 8, 2023

FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placement
Xiaonan Nie, Xupeng Miao, Zilong Wang, Zichao Yang, Jilong Xue, Lingxiao Ma, Gang Cao, Bin Cui
Mixture of Expert Model Training DNN Framework Sparse Model Large Scale Pre Trained Model Device Placement Conditional Computation

March 28, 2023

Large-scale Pre-trained Models are Surprisingly Strong in Incremental Novel Class Discovery
Mingxuan Liu, Subhankar Roy, Zhun Zhong, Nicu Sebe, Elisa Ricci
LeArning Abstract Full Model Large Scale Lifelong Learning Large Scale Pre Trained Model Novel Class Discovery Novel Concept

March 26, 2023

BlackVIP: Black-Box Visual Prompting for Robust Transfer Learning
Changdae Oh, Hyeji Hwang, Hee-young Lee, YongTaek Lim, Geunyoung Jung, Jiyoung Jung, Hosik Choi, Kyungwoo Song
Large Model Visual Prompting Large Scale Pre Trained Model Shot Adaptation Efficient Transfer Learning Robust Adaptation Robust Transfer Learning

March 14, 2023

Good Neighbors Are All You Need for Chinese Grapheme-to-Phoneme Conversion
Jungjun Kim, Changjin Han, Gyuhyeon Nam, Gyeongsu Chae
Language Model Large Scale Pre Trained Model Thy Neighbor Character Embeddings

March 6, 2023

Angel-PTM: A Scalable and Economical Large-scale Pre-training System in Tencent
Xiaonan Nie, Yi Liu, Fangcheng Fu, Jinbao Xue, Dian Jiao, Xupeng Miao, Yangyu Tao, Bin Cui
Pre Trained Model Transformer Model System Description Deep Learning System Large Scale Pre Trained Model Large Scale Model

March 5, 2023

An Empirical Study of Pre-Trained Model Reuse in the Hugging Face Deep Learning Model Registry
Wenxin Jiang, Nicholas Synovic, Matt Hyatt, Taylor R. Schorlemmer, Rohan Sethi, Yung-Hsiang Lu, George K. Thiruvathukal, James C. Davis
Deep Neural Network Pre Trained Model Empirical Study Large Scale Pre Trained Model Model Reuse

February 16, 2023

Towards Efficient Visual Adaption via Structural Re-parameterization
Gen Luo, Minglang Huang, Yiyi Zhou, Xiaoshuai Sun, Guannan Jiang, Zhiyu Wang, Rongrong Ji
Vision Model Large Vision Model Large Scale Pre Trained Model Parameter Efficient Transfer Visual Adaptation Structural Re Parameterization

February 3, 2023

Gradient Estimation for Unseen Domain Risk Minimization with Pre-Trained Models
Byounggyu Lew, Donghyun Son, Buru Chang
Pre Trained Model Gradient Estimation Large Scale Pre Trained Model Generalization Capacity

January 27, 2023

Voting from Nearest Tasks: Meta-Vote Pruning of Pre-trained Models for Downstream Tasks
Haiyan Zhao, Tianyi Zhou, Guodong Long, Jing Jiang, Chengqi Zhang
Pre Trained Model Downstream Task Model Pruning Large Scale Pre Trained Model Pruned Model

November 7, 2022

Reconciliation of Pre-trained Models and Prototypical Neural Networks in Few-shot Named Entity Recognition
Youcheng Huang, Wenqiang Lei, Jie Fu, Jiancheng Lv
Entity Recognition Pre Trained Model Large Scale Pre Trained Model Normalization Method Prototypical Neural Network

November 3, 2022

Fine-Tuning Pre-Trained Language Models Effectively by Optimizing Subnetworks Adaptively
Haojie Zhang, Ge Li, Jia Li, Zhongjin Zhang, Yuqi Zhu, Zhi Jin
Language Model Pre Trained Language Model Large Scale Pre Trained Model Scale Pre Trained Language Model Parameter Selection

October 17, 2022

Pseudo-OOD training for robust language models
Dhanasekar Sundararaman, Nikhil Mehta, Lawrence Carin
Language Model Distribution Detection Large Scale Pre Trained Model Distribution Sample Pseudo Labeled Pseudo Label Regularization

April 17, 2022

A Survivor in the Era of Large-Scale Pretraining: An Empirical Study of One-Stage Referring Expression Comprehension
Gen Luo, Yiyi Zhou, Jiamu Sun, Xiaoshuai Sun, Rongrong Ji
Empirical Study New Era Multi Modal Fusion Large Scale Pre Trained Model Large Scale Pretraining Deep Survival Referring Expression Comprehension Expression Comprehension

February 20, 2022

$\mathcal{Y}$-Tuning: An Efficient Tuning Paradigm for Large-Scale Pre-Trained Models via Label Representation Learning
Yitao Liu, Chenxin An, Xipeng Qiu
Large Scale Pre Trained Model Parameter Efficient Tuning Label Representation P Tuning

Large Scale Pre Trained Model

Papers

FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placement

Large-scale Pre-trained Models are Surprisingly Strong in Incremental Novel Class Discovery

BlackVIP: Black-Box Visual Prompting for Robust Transfer Learning

Good Neighbors Are All You Need for Chinese Grapheme-to-Phoneme Conversion

Angel-PTM: A Scalable and Economical Large-scale Pre-training System in Tencent

An Empirical Study of Pre-Trained Model Reuse in the Hugging Face Deep Learning Model Registry

Towards Efficient Visual Adaption via Structural Re-parameterization

Gradient Estimation for Unseen Domain Risk Minimization with Pre-Trained Models

Voting from Nearest Tasks: Meta-Vote Pruning of Pre-trained Models for Downstream Tasks

Reconciliation of Pre-trained Models and Prototypical Neural Networks in Few-shot Named Entity Recognition

Fine-Tuning Pre-Trained Language Models Effectively by Optimizing Subnetworks Adaptively

Pseudo-OOD training for robust language models

A Survivor in the Era of Large-Scale Pretraining: An Empirical Study of One-Stage Referring Expression Comprehension

$\mathcal{Y}$-Tuning: An Efficient Tuning Paradigm for Large-Scale Pre-Trained Models via Label Representation Learning