Pretraining

Follow this topic Ask about this topic

Momentum

60 papers in the last four weeks, up 122% on the four weeks before. 0.6% of all new papers.

Jul 13Week of Sep 28

Latest papers 428

All topics

Large Pretraining Datasets Don't Guarantee Robustness after Fine-Tuning in Image Classification
Oct 28, 2024Jaedong Hwang, Brian Cheung, Zhang-Wei Hong +3Pretraining Imagenet
Massachusetts Institute of Technology
Realistic Continual Learning Approach using Pre-trained Models
Apr 11, 2024Nadia Nasri, Carlos Gutiérrez-Álvarez, Sergio Lafuente-Arroyo +2Replay-Based Continual Learning Continual Learning
University of Alcala, Department of Signal Theory and Communications, GRAM Research group, Alcala de Henares, 28805, Spain · Univesity of Alicante, Department of Computer Science and Artificial Intelligence, San Vicente del Raspeig, 03690, Spain
SynGhost: Invisible and Universal Task-agnostic Backdoor Attack via Syntactic Transfer
Feb 29, 2024Pengzhou Cheng, Wei Du, Zongru Wu +4Clean Label Backdoor Attack Attacker Large Language Model
School of Electronic Information and Electrical Engineering Shanghai Jiao Tong University · Department of Computer Science and Engineering Southern University of Science and Technology
DA-Cramming: Enhancing Cost-Effective Language Model Pretraining with Dependency Agreement Integration
Nov 8, 2023Martin Kuo, Jianyi Zhang, Dongting Li +1Large Language Model Pretraining Pretraining
Center for Computational Evolutionary Intelligence, Duke University
Geometry-Aware Adaptation for Pretrained Models
Jul 23, 2023Nicholas Roberts, Xintong Li, Dyah Adila +4Pretraining Imagenet
University of Wisconsin-Madison
Are We Really Doing Few-Shot Learning? A Critical Examination of Pre-Training Assumptions
Date pendingAlejandro Galan-Cuenca, Marcelo Saval-Calvo, Antonio Javier GallegoFew-Shot Learning Target Domain
University Institute for Computer Research, University of Alicante, Alicante, Spain
NS-Copilot: An LLM-Driven Agent System for Autonomous Neuroscience Analysis
Date pendingWuche Liu, Yiran Qiao, Linlin Hou +4Neuroscience Copilot
Department of Computer and Data Sciences, Case Western Reserve University, Cleveland, OH 44106, USA · Department of Mathematics and Statistics, University of West Florida, Pensacola, FL 32514, USA · Department of Computer Science, University of Central Florida, Orlando, FL 32816, USA
Diffract: Spectral View of LLM Domain Adaptation
Date pendingNikita Borodin, Maria Krylova, Artem Zabolotnyi +6Large Language Model Adaptation Pretraining
Risk AI Research Lab · Applied AI Institute

Oct 28, 2024, cs.CVJ/K move · Enter open · S save

Large Pretraining Datasets Don't Guarantee Robustness after Fine-Tuning in Image Classification

Jaedong Hwang, Brian Cheung, Zhang-Wei Hong, Akhilan Boopathy+2

Massachusetts Institute of Technology

Pretraining Imagenet

Open paper AskPDF arXiv

Large-scale pretrained models are widely leveraged as foundations for learning new specialized tasks via fine-tuning, with the goal of maintaining the general performance of the model while allowing it to gain new skills. A valuable goal for all such models is robustness: the ability to perform well on out-of-distribution (OOD) tasks. We assess whether fine-tuning preserves the overall robustness of the pretrained model in image classification, and observed that models pretrained on large datasets exhibited strong catastrophic forgetting and loss of OOD generalization. To systematically assess robustness preservation in fine-tuned models, we propose the Robustness Inheritance Benchmark (ImageNet-RIB). The benchmark, which can be applied to any pretrained model, consists of a set of related but distinct OOD (downstream) tasks and involves fine-tuning on one of the OOD tasks in the set then testing on the rest. We find that though continual learning methods help, fine-tuning reduces robustness across pretrained models. Surprisingly, models pretrained on the largest and most diverse datasets (e.g., LAION-2B) exhibit both larger robustness losses and lower absolute robustness after fine-tuning on small datasets, relative to models pretrained on smaller datasets. We observe this collapse in contrastively pretrained (CLIP) models and their fine-tuned variants, where it grows with pretraining scale; the supervised models we test do not exhibit it. These findings suggest that starting with the strongest foundation model is not necessarily the best approach for performance on specialist tasks. https://jd730.github.io/projects/ImageNet-RIB

Momentum

Related topics

Latest papers 428