cs.LGNov 8, 2026

Full Fine-Tuning is not the Problem: Why Adam Fails and SGD Succeeds in Few-Shot CLIP Adaptation

Authors: Nairouz Mrabah, Ismail Ben Ayed, Eric Granger

Organizations: LIVIA, ILLS, Department of Systems Engineering, ETS Montreal, Canada

Abstract

Few-shot CLIP adaptation is often treated as a parameterization problem. Full fine-tuning is assumed to overfit, so parameter-efficient fine-tuning (PEFT) methods are preferred. In this paper, we show that this conclusion conflates parameterization with optimization. Across 11 datasets, 3 CLIP backbones, and 5 shot levels, dense full fine-tuning with plain SGD outperforms representative PEFT methods under the reported evaluation protocol, whereas Adam/AdamW collapses. We trace this collapse to unreliable support-set preconditioning. The few-shot second moment systematically underestimates the corresponding full-data statistic, already at initialization. Adam's inverse-square-root normalization converts this error into an oversized adaptive update. Matching Adam's first-step update budget to SGD resolves most of the collapse, whereas numerator-side alignment does not explain the failure. Shampoo and SOAP also fail, so richer preconditioning alone does not resolve the problem. Theory formalizes how few-shot estimation error is amplified into unsafe updates. SGD succeeds because it avoids unreliable preconditioners and stays close to pretrained CLIP. Interpolation probes show low-loss connections along tested directions, and curvature probes show favorable directional curvature. These results identify optimizer-induced preconditioning, not dense full fine-tuning itself, as the main source of failure in few-shot CLIP adaptation.

Figures & tables

Appendix figures & tables133 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. LE-GDA: Layer Ensemble Gaussian Discriminant Analysis for Training-Free Few-Shot CLIP Adaptation

    Oct 26, 2026Pavodi Maniamfu, Vogel Kiketa, Zezina Kuzulu Kuetu +4VLM AdaptationFew-Shot Image Classification

  2. Hold-One-Shot-Out (HOSO) for Validation-Free Few-Shot CLIP Adapters

    Mar 4, 2026Chris Vorster, Mayug Maniparambil, Noel E. O'Connor +2Few-Shot VLM AdaptationVLM Adaptation

  3. On the Few-Shot Adaptation of Vision-Language Models: A Few Gradient Signs Are All You Need

    Nov 5, 2026Nairouz Mrabah, Ismail Ben Ayed, Mahdi S. Hosseini +1Few-Shot VLM AdaptationVLM Adaptation