Exploring Bad Minima in Transfer Learning
Organizations: Churchill College
Abstract
Transfer learning is a highly successful learning paradigm in deep learning which leverages knowledge from one task and applies it to another. Despite its success, the necessary conditions for knowledge transfer are unclear and require further research. In this dissertation, our aim is to understand transferability a little better by answering the following question: Does good generalization on a source task necessarily imply transferability to a related target task? We hypothesize the answer is negative, and attempt to find counterexamples via adversarial attacks on the transfer learning procedure. Over three types of adversarial attacks, our empirical results show a strong relationship between generalization on the source task and transferability. These results do not rule out generalization on the source task as a sufficient condition for successful transfer.
Figures & tables
| before poisoning | after poisoning | |||
| 1 epoch | 10 epochs | 100 epochs | ||
| ImageNet | - | - | 42.66% | |
| CIFAR upstream | 77.26% | 59.24% | 61.91% | 61.27% |
| CIFAR downstream | 96.4 % | 88.4% | 87.8% | 87% |
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| upstream | downstream | |||
|---|---|---|---|---|
| from scratch | LP transfer | rand. LP transfer | ||
| test accuracy | ||||
Transfer Learning
Transfer learning studies how models reuse knowledge across tasks, domains, or modalities when target data or computation are limited.
See all 455 papers