Full Fine-Tuning is not the Problem: Why Adam Fails and SGD Succeeds in Few-Shot CLIP Adaptation
Organizations: LIVIA, ILLS, Department of Systems Engineering, ETS Montreal, Canada
Abstract
Few-shot CLIP adaptation is often treated as a parameterization problem. Full fine-tuning is assumed to overfit, so parameter-efficient fine-tuning (PEFT) methods are preferred. In this paper, we show that this conclusion conflates parameterization with optimization. Across 11 datasets, 3 CLIP backbones, and 5 shot levels, dense full fine-tuning with plain SGD outperforms representative PEFT methods under the reported evaluation protocol, whereas Adam/AdamW collapses. We trace this collapse to unreliable support-set preconditioning. The few-shot second moment systematically underestimates the corresponding full-data statistic, already at initialization. Adam's inverse-square-root normalization converts this error into an oversized adaptive update. Matching Adam's first-step update budget to SGD resolves most of the collapse, whereas numerator-side alignment does not explain the failure. Shampoo and SOAP also fail, so richer preconditioning alone does not resolve the problem. Theory formalizes how few-shot estimation error is amplified into unsafe updates. SGD succeeds because it avoids unreliable preconditioners and stays close to pretrained CLIP. Interpolation probes show low-loss connections along tested directions, and curvature probes show favorable directional curvature. These results identify optimizer-induced preconditioning, not dense full fine-tuning itself, as the main source of failure in few-shot CLIP adaptation.
Figures & tables
Appendix figures & tables133 assets
Supplementary material from the paper’s appendix.
Appendix
| Term | Dimension | Definition |
|---|---|---|
| finite set | Trainable CLIP encoder set, . | |
| element of | Encoder index. The values and denote the image and text encoders. | |
| integer | Optimizer step, epoch, or logged checkpoint. In the full-support diagnostics, one step is one pass over the support set. | |
| Flattened vector of all trainable CLIP parameters at step . | ||
| Pretrained CLIP checkpoint before few-shot adaptation. | ||
| Final adapted checkpoint when denotes the terminal training step in a local experiment. |
| Term | Dimension | Definition |
|---|---|---|
| , | loss function; split label | is the per-example loss. indexes empirical objectives. |
| , , | datasets | Support, full-training, and test datasets, respectively. |
| , | vectors | Optimizer first-moment state and encoder-restricted dataset mean gradient, respectively. |
| , | nonnegative vectors | Optimizer second-moment state and encoder-restricted dataset second moment, respectively. |
| parameter vector | Trainable parameters at step ; the same symbol is used in drift analyses. | |
| , , | vectors | Unit SGD direction, norm-matched probe displacement, and block-restricted displacement, respectively. |
| Term | Dimension | Definition |
|---|---|---|
| nonnegative scalar | Initialization ratio . | |
| nonnegative scalar | Initialization adaptive norm ratio . | |
| nonnegative scalar | Numerator ratio . | |
| nonnegative scalar | Second-moment ratio . | |
| , | nonnegative scalars | Collapse factors for the numerator ratio and second-moment ratio between initialization and the final logged checkpoint. |
| optimizer label | Trajectory index in . |
| Term | Dimension | Definition |
|---|---|---|
| split label | Common objective index in . | |
| Linear interpolation between pretrained and final SGD checkpoints. | ||
| scalar | Line loss for . | |
| , | scalar | Minimizer of the line loss and the corresponding minimum value. |
| scalar function | Endpoint chord used to compute the interpolation barrier. | |
| scalar | Interpolation barrier above the endpoint chord. |
| Term | Dimension | Definition |
|---|---|---|
| map to | Unnormalized CLIP image encoder at step . | |
| map to | Unnormalized CLIP text encoder at step . | |
| Normalized image feature for image . | ||
| text prompt | Prompt string passed to the text encoder. | |
| Normalized text feature for prompt . | ||
| set of prompts | Prompt set for class , . |
| Term | Dimension | Definition |
|---|---|---|
| subset of | Support mini-batch at optimizer iteration . | |
| support set | Few-shot support set in the optimizer algorithms. | |
| Cross-entropy loss on . | ||
| SMD or Adam first-moment state in Appendix C. | ||
| Adam diagonal second-moment state in Appendix C. | ||
| , | vectors in | Bias-corrected Adam first and second moments. |
| Term | Dimension | Definition |
|---|---|---|
| , , | dataset abbreviations | EuroSAT, Oxford-Pets, and Describable Textures Dataset. These are the three main diagnostic datasets. |
| , | encoder abbreviations | Vision and text encoders in diagnostic figures. |
| positive integer | Shot level used to sample the support set. | |
| training split | Training split from which few-shot support sets and full-train references are formed. | |
| test split | Held-out evaluation split for each dataset. |
| Term | Dimension | Definition |
|---|---|---|
| positive integer | Training batch size in benchmark and stability experiments. | |
| positive scalar | Learning rate in optimizer sweeps and method configurations. | |
| nonnegative scalar | Weight decay coefficient in full-fine-tuning optimizer hyperparameters. | |
| , | scalars in | Adam, SMD, Shampoo, or SOAP moment coefficients. |
| positive scalar | Numerical stabilizer in AdamW and related adaptive optimizers. | |
| positive scalar | ProGrad softmax temperature in the hyperparameter table. |
| Term | Dimension | Definition |
|---|---|---|
| Weight matrix of an idealized fully connected layer in the memory model. | ||
| positive integer | Output dimension of the layer. | |
| positive integer | Input dimension of the layer in the memory model. | |
| positive integer | Low-rank dimension for LoRA, PiSSA, and GaLoRE memory counts. | |
| scalar in | Sparse update density. The sparsity ratio is . | |
| scalar count | Number of dense weight scalars in one fully connected layer. |
| Term | Dimension | Definition |
|---|---|---|
| percentage | Top-1 test accuracy in the SGD/SMD/Adam benchmark. | |
| positive integer | Shot level on the horizontal axis of few-shot results. | |
| method label | SGD with momentum. | |
| mean and standard deviation | Reported accuracy summary over random seeds. |
| Term | Dimension | Definition |
|---|---|---|
| method family label | Parameter-efficient fine-tuning family used for comparisons against dense SGD. | |
| , , , , | method labels | Representative PEFT methods in the benchmark. |
| positive integer | Shot level in the PEFT comparison tables. | |
| percentage | Top-1 test accuracy. |
| Term | Dimension | Definition |
|---|---|---|
| positive scalar | Learning rate varied in the PEFT stability sweep. | |
| epoch | Training epoch used as the horizontal axis in stability curves. | |
| , , | method labels | PEFT methods compared with SGD in the first stability subsection. |
| mean and standard deviation | Accuracy summary over three fixed seeds. |
| Term | Dimension | Definition |
|---|---|---|
| scalar in | Adam second-moment coefficient used in the stability sweep. | |
| positive integer | Preconditioning frequency for Shampoo or SOAP. | |
| positive scalar | Learning rate varied in the second-order stability sweep. | |
| epoch | Training epoch used as the horizontal axis. | |
| , | method labels | Structured preconditioned optimizers compared with SGD and Adam. |
| Term | Dimension | Definition |
|---|---|---|
| , | optimizer labels | Optimizers used inside each PEFT family and dense full fine-tuning. |
| , , | method labels | Adapter-style PEFT methods. |
| , , | method labels | Prompt-learning PEFT methods. |
| , , | method labels | Low-rank or projected-gradient PEFT methods. |
| , , , | method labels | Sparse or subset-based PEFT methods. |
| positive integer | Shot level in each ablation table. |
| Term | Dimension | Definition |
|---|---|---|
| mini-batch subset of | Diagnostic mini-batch used to estimate empirical first and second moments. | |
| coordinate index | Coordinate index in encoder , . | |
| scalar | Base-10 coordinate log-ratio of support and full-data diagonal second moments. | |
| nonnegative scalar | Mean absolute coordinate log-ratio in encoder . | |
| positive integer | Number of salient coordinates selected by top- . | |
| scalar in | Density used to set . |
| Term | Dimension | Definition |
|---|---|---|
| scalar in | Initialization cosine between and . | |
| nonnegative scalar | Initialization denominator norm ratio. | |
| nonnegative scalar | Coordinatewise denominator mismatch at initialization. | |
| nonnegative scalar | Initialization adaptive-direction norm ratio. | |
| nonnegative scalar | Mean absolute coordinate log-ratio restricted to Transformer block of encoder . | |
| , | Block-restricted oracle-denominator and full-data reference directions. |
| Term | Dimension | Definition |
|---|---|---|
| nonnegative scalar | Numerator ratio between support and full-train gradients. | |
| nonnegative scalar | Denominator-statistic ratio between support and full-train second moments. | |
| final logged checkpoint | Endpoint used to compute collapse factors. | |
| nonnegative scalar | Collapse factor . | |
| nonnegative scalar | Collapse factor . |
| Term | Dimension | Definition |
|---|---|---|
| Squared full-support gradient . | ||
| Support-batch diagonal second moment. | ||
| Encoder restriction of Adam’s uncorrected second-moment state . | ||
| Bias-corrected Adam running second moment. | ||
| Adaptive direction using the instantaneous second moment. | ||
| nonnegative scalar | Instantaneous second-moment norm ratio. |
| Term | Dimension | Definition |
|---|---|---|
| positive integer | Lag length for the short-lag denominator, set to in the experiment. | |
| Direction using current few-shot numerator and initialization denominator . | ||
| Direction using current few-shot numerator and lagged denominator . | ||
| direction identifier | Element of . | |
| scalar in | Alignment between and . | |
| nonnegative scalar | Scale ratio . |
| Term | Dimension | Definition |
|---|---|---|
| optimizer label | Trajectory index in . | |
| Update vector on trajectory . | ||
| nonnegative scalar | Global relative update budget. | |
| positive scalar | One-time learning-rate multiplier that matches Adam’s first-step budget to SGD. | |
| positive scalar | Learning rate of step-0 matched Adam. | |
| , | Trajectory-specific numerator-side geometry directions with full-data denominator. |
| Term | Dimension | Definition |
|---|---|---|
| coordinate group | Subset of trainable parameters, such as all parameters, one encoder, or one block. | |
| Parameters of group on trajectory . | ||
| nonnegative scalar | Relative parameter drift from the pretrained state. | |
| Normalized image feature on trajectory . | ||
| Normalized class-text feature on trajectory . | ||
| nonnegative scalar | Mean angular image-feature drift on split . |
| Term | Dimension | Definition |
|---|---|---|
| positive integer | Temporal lag in optimizer steps or epochs. | |
| Support gradient at step on trajectory , restricted to encoder . | ||
| Full-train gradient at step on trajectory , restricted to encoder . | ||
| scalar in | Cosine between support gradients at steps and . | |
| scalar in | Cosine between support gradient at and full-train gradient at . | |
| positive integer | Number of trainable scalar parameters in block of encoder . |
| Term | Dimension | Definition |
|---|---|---|
| optimizer family | Structured preconditioner family, usually Shampoo or SOAP. | |
| linear map | Effective structured preconditioning map. | |
| , , | Structured preconditioned directions. | |
| nonnegative scalar | Scale ratio of few-shot and full-data structured preconditioners with the same numerator. | |
| scalar in | Repaired reference alignment for the structured preconditioner. | |
| Flattened statistic used to build the structured preconditioner. |
| Term | Dimension | Definition |
|---|---|---|
| labeled example | Example in the full training split. | |
| subset of | Indices selected into the few-shot support set. | |
| Per-example encoder gradient of the cross-entropy loss. | ||
| nonnegative scalar | Squared per-example gradient coordinate . | |
| nonnegative scalar | Finite-population estimator of the few-shot second-moment coordinate. | |
| nonnegative scalar | Finite-population variance of . |
| Term | Dimension | Definition |
|---|---|---|
| scalar map | Stabilized inverse-square-root map . | |
| scalar in | Local relative-error bound used only in Proposition 2. | |
| scalar | Taylor remainder in the inverse-square-root bias expansion. | |
| scalar | Remainder normalized by . | |
| scalar | Intermediate point in Taylor’s theorem between and . | |
| scalar in | Stabilized relative denominator error. Positive values mean support underestimation. |
| Term | Dimension | Definition |
|---|---|---|
| unit vector in | Unit direction of the Adam-like update. | |
| positive scalar | Parameter-space length of the Adam-like step, . | |
| unit vector in | Generic update direction used in the step-length theory. | |
| scalar in | Alignment between and the full-train gradient. | |
| line segment | Segment . | |
| scalar | Directional curvature of at . |
| Term | Dimension | Definition |
|---|---|---|
| Diagonal preconditioner formed from . | ||
| Diagonal preconditioner formed from . | ||
| Adaptive-direction error caused by the denominator difference. | ||
| Residual error caused by using the few-shot numerator after denominator repair. | ||
| Square root of the full-data diagonal preconditioner, . | ||
| nonnegative scalar | Numerator-staleness ratio in the repaired-denominator metric. |
| Term | Dimension | Definition |
|---|---|---|
| finite set | Set of trainable encoder–Transformer-block pairs. | |
| element of | Block index, often . | |
| positive integer | Number of trainable scalar parameters in block . | |
| Diagonal projector for block . | ||
| nonnegative scalar | Block relative update budget for update vector . | |
| Raw Adam-like update. |
| Term | Dimension | Definition |
|---|---|---|
| Init-to-final interpolation between and . | ||
| scalar | Line loss for . | |
| , | scalar | Line-loss minimizer and minimum value. |
| scalar function | Endpoint chord between and . | |
| scalar | Interpolation barrier above the endpoint chord. | |
| positive scalar | Tolerance used to define the near-minimum set. |
| Term | Dimension | Definition |
|---|---|---|
| SGD update . | ||
| Norm-matched probe displacement for . | ||
| scalar in | Sampled nonnegative multiplier for a local line probe. | |
| scalar | Line loss . | |
| scalar | Minimizer of over the sampled nonnegative grid. | |
| nonnegative scalar | Largest sampled nonnegative multiplier whose loss does not exceed the unperturbed loss. It does not certify the intervening interval. |
| Term | Dimension | Definition |
|---|---|---|
| unit vector in | Normalized SGD update direction. | |
| direction family label | One of random, Adam-like, full-gradient, or top-Hessian directions. | |
| Raw comparison direction before orthogonalization. | ||
| random vector | Raw random vector before orthogonalization in the 2D surface control. | |
| unit vector | Largest-magnitude eigendirection of the support-loss Hessian. | |
| unit vector | Direction orthogonalized against . |
| Term | Dimension | Definition |
|---|---|---|
| Hessian of loss at checkpoint . | ||
| scalar | Directional curvature . | |
| unit vector | Actual normalized SGD update direction. | |
| unit vector | Normalized full-train gradient direction. | |
| unit vector | Largest-magnitude eigendirection of . | |
| unit vector | Random-direction control. |
| Term | Dimension | Definition |
|---|---|---|
| Block projector used in the blockwise SGD step. Here indexes an encoder and its Transformer depth. | ||
| Block-restricted SGD step . | ||
| scalar | Blockwise line loss for objective . | |
| scalar | Blockwise loss change from scale . | |
| scalar in | Full-loss preferred scale for block . | |
| scalar | Support–full mismatch in loss change. |
| Term | Dimension | Definition |
|---|---|---|
| , | seed labels | Independent few-shot support-set seeds. |
| , | Final SGD states from two seeds. | |
| Seed-to-seed interpolation. | ||
| scalar-valued loss | Average of the two support losses from seeds and . | |
| scalar-valued loss | Seed-pair loss for . | |
| scalar | Line loss along the seed-to-seed interpolation. |
| Term | Dimension | Definition |
|---|---|---|
| , | Support and full-train encoder-restricted gradients. | |
| scalar in | Cosine between support and full-train gradients. | |
| nonnegative scalar | Gradient norm ratio. | |
| , | Encoder-restricted support and full-train Hessians. | |
| , | unit vectors | Top Hessian eigendirections for support and full-train losses. |
| , | scalars | Top Hessian eigenvalues. |
| Term | Dimension | Definition |
|---|---|---|
| Trainable parameters of group at checkpoint . | ||
| nonnegative scalar | Relative parameter displacement of group . | |
| Image feature used in SGD feature-drift diagnostics. | ||
| Class-text feature used in SGD feature-drift diagnostics. | ||
| nonnegative scalar | Mean angular image-feature drift on split . | |
| nonnegative scalar | Mean angular text-feature drift over classes. |
| Term | Dimension | Definition |
|---|---|---|
| variant label | Full-batch, half-batch, quarter-batch, additive-noise, or momentum in I1–I3. In I4, half and quarter denote accumulation chunk sizes for full-support-sized bootstrap draws. | |
| Parameter vector of variant . | ||
| Micro-batch gradient for variant . | ||
| Full-support gradient for variant . | ||
| collection of micro-batches | Micro-batches used at checkpoint in the stochasticity diagnostic. | |
| positive integer | Number of micro-batches, , at checkpoint . |
| Operation | Setting |
| RandomResizedCrop | size , scale , bicubic interpolation |
| RandomHorizontalFlip | probability |
| Method | Method-specific settings | Optimization settings | Trainable component |
| CoOp | , , , Adam | Class-specific context tokens | |
| KgCoOp | , | , , , SGD | Class-specific context tokens with knowledge regularization |
| ProGrad | , , | , , , Adam | Class-specific context tokens with gradient projection |
| CLIP-Adapter | , | , , , Adam | Adapter module |
| TIP-Adapter | , | , , , Adam | Cache-based adapter parameters |
| LP++ | – | , , , SGD | Class-wise linear-probe coefficients |
| Method | Method-specific settings | Optimization settings |
| LoRA | rank , scaling | , , , Adam |
| PiSSA | rank , scaling | , , , Adam |
| GaLoRE | rank , refresh period | , , , Adam |
| BitFit | bias parameters only | , , , Adam |
| LayerNorm | affine scale and shift parameters only | , , , Adam |
| SO | density , support refresh period , , | , , |
| Optimizer | Optimizer-specific settings | Shared training settings |
| SGD | no optimizer state | , , |
| SMD | first-moment coefficient | , , |
| Adam | , | , , |
| Shampoo | default Shampoo settings, preconditioning frequency | , , |
| SOAP | default SOAP settings, preconditioning frequency | , , |
| Method | Training time |
| Adapter | 9 min |
| TIP-Adapter | 4 min |
| LP++ | 2 min |
| CoOp | 1 h 18 min |
| KgCoOp | 1 h 18 min |
| ProGrad | 2 h 08 min |
| SGD | Adam | SAFT | SO | LoRA | PiSSA | GaLoRE | |
| Weight | |||||||
| Gradient | |||||||
| Optimizer states |
| Method | Weight (#Vars, MiB) | Gradient (#Vars, MiB) | Opt. States (#Vars, MiB) | #Trainable | Total Mem. (MiB) |
| LoRA ( ) | M, | M, | M, | M | |
| LoRA ( ) | M, | M, | M, | M | |
| LoRA ( ) | M, | M, | M, | M | |
| LoRA ( ) | M, | M, | M, | M | |
| PiSSA ( ) | M, | M, | M, | M | |
| PiSSA ( ) | M, | M, | M, | M |
| Shots | Method | ImageNet | SUN | Aircraft | EuroSAT | Cars | Food | Pets | Flowers | Caltech | DTD | UCF | Average |
| 0 | CLIP | 62.0 | 62.1 | 18.1 | 35.8 | 60.1 | 78.9 | 85.1 | 64.0 | 91.4 | 43.0 | 61.6 | 60.2 |
| 1 | AdamW (0.1) | 7.1 0.8 | 13.4 0.6 | 1.0 0.0 | 8.6 0.8 | 0.5 0.1 | 1.8 1.0 | 6.6 3.5 | 1.1 0.4 | 22.6 11.4 | 13.7 3.6 | 1.8 0.8 | 7.1 2.1 |
| AdamW (0.01) | 6.6 1.1 | 12.9 1.4 | 1.0 0.1 | 9.8 1.0 | 0.5 0.1 | 2.2 1.4 | 6.1 3.7 | 1.2 0.4 | 20.4 10.8 | 14.1 3.4 | 1.5 0.4 | 6.9 2.2 | |
| AdamW (0.001) | 6.8 1.0 | 14.0 0.3 | 1.0 0.0 | 11.0 3.0 | 0.5 0.1 | 2.1 1.1 | 5.4 2.3 | 1.1 0.4 | 20.2 15.4 | 12.3 2.6 | 2.2 0.8 | 7.0 2.5 | |
| Adam | 12.4 2.6 | 16.6 1.7 | 1.1 0.2 | 9.1 2.0 | 0.5 0.1 | 2.1 0.8 | 5.7 3.7 | 3.6 2.5 | 30.9 21.9 | 13.7 2.6 | 1.4 0.5 | 8.8 3.5 | |
| SGD | 64.6 0.3 | 67.4 0.1 | 22.3 1.2 | 65.1 2.4 | 63.5 0.7 | 80.5 0.4 | 89.0 0.1 | 77.4 1.0 | 93.1 0.5 | 50.0 0.1 | 71.7 0.3 | 67.7 0.6 |
| Shots | Method | ImageNet | SUN | Aircraft | EuroSAT | Cars | Food | Pets | Flowers | Caltech | DTD | UCF | Average |
| 0 | CLIP | 66.7 | 62.6 | 23.9 | 42.2 | 65.5 | 85.2 | 88.3 | 67.4 | 93.3 | 44.4 | 65.2 | 64.1 |
| 1 | AdamW (0.1) | 8.7 0.6 | 5.0 2.1 | 1.3 0.4 | 15.3 1.5 | 0.5 0.1 | 1.8 0.6 | 4.9 1.6 | 7.2 3.7 | 19.7 11.0 | 13.1 3.2 | 1.5 0.7 | 7.2 2.3 |
| AdamW (0.01) | 8.7 0.5 | 3.8 2.8 | 1.0 0.0 | 15.3 1.4 | 0.5 0.1 | 1.3 0.2 | 3.9 1.3 | 7.8 2.9 | 22.7 14.0 | 13.1 2.7 | 1.6 0.7 | 7.3 2.4 | |
| AdamW (0.001) | 8.6 0.4 | 6.3 0.7 | 1.1 0.5 | 16.3 2.2 | 0.4 0.1 | 1.4 0.6 | 3.9 1.0 | 7.2 3.7 | 24.8 12.9 | 13.3 4.3 | 2.0 1.1 | 7.8 2.5 | |
| Adam | 18.3 0.9 | 19.5 2.2 | 1.1 0.1 | 11.7 3.1 | 0.7 0.1 | 1.4 0.1 | 3.3 0.6 | 1.4 0.7 | 40.3 6.1 | 13.5 1.8 | 2.3 1.8 | 10.3 1.6 | |
| SGD | 70.1 0.2 | 70.3 0.2 | 30.5 0.9 | 73.8 2.7 | 70.8 0.6 | 85.8 0.2 | 92.7 0.2 | 83.6 0.6 | 94.0 0.2 | 53.7 0.8 | 76.1 0.5 | 72.8 0.6 |
| Shots | Method | ImageNet | SUN | Aircraft | EuroSAT | Cars | Food | Pets | Flowers | Caltech | DTD | UCF | Average |
| 0 | CLIP | 73.5 | 67.7 | 30.0 | 55.1 | 76.8 | 90.3 | 93.1 | 76.2 | 95.2 | 52.4 | 73.7 | 71.3 |
| 1 | AdamW (0.1) | 14.3 0.3 | 22.0 2.0 | 1.2 0.2 | 19.9 6.0 | 0.5 0.0 | 7.8 6.3 | 22.7 9.6 | 23.5 12.4 | 52.6 3.5 | 21.0 2.1 | 33.1 5.3 | 19.9 4.3 |
| AdamW (0.01) | 14.2 1.1 | 22.9 1.9 | 1.4 0.2 | 16.7 6.6 | 0.5 0.0 | 6.0 4.7 | 23.6 9.5 | 26.0 12.8 | 56.2 3.2 | 22.5 3.1 | 33.6 1.5 | 20.3 4.0 | |
| AdamW (0.001) | 14.5 0.9 | 22.2 2.0 | 1.2 0.3 | 14.4 2.0 | 0.5 0.0 | 5.7 5.3 | 25.2 10.5 | 24.4 11.0 | 58.8 1.5 | 21.4 1.7 | 35.4 2.9 | 20.3 3.5 | |
| Adam | 14.8 0.7 | 23.5 1.2 | 1.2 0.0 | 18.3 8.2 | 0.5 0.0 | 5.2 4.0 | 17.6 11.2 | 18.8 4.7 | 44.0 17.8 | 21.5 3.2 | 33.8 4.3 | 18.1 5.0 | |
| SGD | 76.6 0.2 | 74.3 0.1 | 41.1 0.9 | 75.3 4.1 | 80.7 0.4 | 91.0 0.1 | 94.6 0.2 | 90.2 0.8 | 95.6 0.5 | 60.5 0.4 | 81.9 0.5 | 78.3 0.7 |
| Shots | Method | ImageNet | SUN | Aircraft | EuroSAT | Cars | Food | Pets | Flowers | Caltech | DTD | UCF | Average |
| 0 | CLIP | 62.0 | 62.1 | 18.1 | 35.8 | 60.1 | 78.9 | 85.1 | 64.0 | 91.4 | 43.0 | 61.6 | 60.2 |
| 1 | Adapter (Adam) | 62.8 0.1 | 63.0 0.0 | 18.5 0.2 | 39.0 1.1 | 60.7 0.1 | 79.3 0.1 | 86.1 0.1 | 64.7 0.3 | 91.6 0.0 | 43.7 0.2 | 62.6 0.3 | 61.1 0.2 |
| Adapter (SGD) | 62.0 0.0 | 62.1 0.0 | 18.1 0.0 | 35.7 0.6 | 60.1 0.1 | 78.9 0.2 | 85.3 0.2 | 63.9 0.2 | 91.5 0.1 | 42.8 0.2 | 61.6 0.1 | 60.2 0.2 | |
| TIP-Adapter (Adam) | 62.1 0.0 | 62.1 0.0 | 18.2 0.1 | 36.5 0.4 | 60.2 0.0 | 78.9 0.0 | 85.1 0.0 | 64.3 0.0 | 91.4 0.1 | 43.0 0.0 | 61.7 0.0 | 60.3 0.1 | |
| TIP-Adapter (SGD) | 62.1 0.0 | 62.1 0.0 | 18.2 0.1 | 36.3 0.4 | 60.2 0.0 | 78.9 0.0 | 85.1 0.0 | 64.2 0.0 | 91.4 0.1 | 43.0 0.0 | 61.7 0.0 | 60.3 0.1 | |
| LP++ (Adam) | 13.3 0.8 | 17.6 0.3 | 7.0 1.2 | 27.8 4.9 | 11.0 0.7 | 18.7 0.6 | 19.9 4.9 | 27.7 0.7 | 48.0 4.8 | 21.4 1.8 | 24.7 1.2 | 21.5 2.0 |
| Shots | Method | ImageNet | SUN | Aircraft | EuroSAT | Cars | Food | Pets | Flowers | Caltech | DTD | UCF | Average |
| 0 | CLIP | 66.7 | 62.6 | 23.9 | 42.2 | 65.5 | 85.2 | 88.3 | 67.4 | 93.3 | 44.4 | 65.2 | 64.1 |
| 1 | Adapter (Adam) | 67.6 0.1 | 64.3 0.1 | 24.1 0.2 | 45.6 0.7 | 66.2 0.2 | 85.4 0.1 | 89.2 0.1 | 68.6 0.4 | 93.4 0.1 | 45.0 0.1 | 66.3 0.2 | 65.1 0.2 |
| Adapter (SGD) | 66.7 0.1 | 62.6 0.1 | 23.6 0.2 | 41.2 0.3 | 65.6 0.1 | 85.2 0.1 | 88.2 0.2 | 67.4 0.2 | 93.2 0.1 | 44.2 0.2 | 65.1 0.1 | 63.9 0.1 | |
| TIP-Adapter (Adam) | 66.7 0.0 | 62.6 0.0 | 23.9 0.0 | 43.1 0.1 | 65.6 0.1 | 85.2 0.0 | 88.3 0.0 | 67.6 0.1 | 93.3 0.0 | 44.4 0.0 | 65.3 0.0 | 64.2 0.0 | |
| TIP-Adapter (SGD) | 66.7 0.0 | 62.6 0.0 | 24.0 0.0 | 42.9 0.1 | 65.5 0.1 | 85.2 0.0 | 88.3 0.0 | 67.5 0.1 | 93.3 0.0 | 44.4 0.0 | 65.3 0.0 | 64.2 0.0 | |
| LP++ (Adam) | 15.4 0.5 | 18.3 1.2 | 9.0 0.6 | 23.0 2.6 | 15.2 1.1 | 24.1 2.6 | 23.2 1.9 | 37.1 1.9 | 45.9 3.3 | 22.2 2.1 | 25.1 2.3 | 23.5 1.8 |
| Shots | Method | ImageNet | SUN | Aircraft | EuroSAT | Cars | Food | Pets | Flowers | Caltech | DTD | UCF | Average |
| 0 | CLIP | 73.5 | 67.7 | 30.0 | 55.1 | 76.8 | 90.3 | 93.1 | 76.2 | 95.2 | 52.4 | 73.7 | 71.3 |
| 1 | Adapter (Adam) | 75.1 0.1 | 70.4 0.1 | 32.0 0.2 | 59.0 1.0 | 78.0 0.1 | 90.6 0.1 | 93.4 0.1 | 77.0 0.3 | 95.2 0.1 | 53.6 0.1 | 75.5 0.3 | 72.7 0.2 |
| Adapter (SGD) | 73.4 0.0 | 67.6 0.1 | 30.2 0.1 | 55.3 0.2 | 76.9 0.1 | 90.2 0.0 | 93.2 0.1 | 76.0 0.1 | 95.2 0.1 | 52.3 0.1 | 73.8 0.1 | 71.3 0.1 | |
| TIP-Adapter (Adam) | 73.5 0.0 | 67.7 0.0 | 30.1 0.1 | 55.5 0.2 | 76.9 0.0 | 90.3 0.0 | 93.1 0.0 | 76.4 0.0 | 95.2 0.0 | 52.4 0.0 | 73.7 0.0 | 71.3 0.0 | |
| TIP-Adapter (SGD) | 73.5 0.0 | 67.7 0.0 | 30.1 0.1 | 55.4 0.1 | 76.9 0.0 | 90.3 0.0 | 93.1 0.0 | 76.3 0.0 | 95.1 0.0 | 52.4 0.0 | 73.7 0.0 | 71.3 0.0 | |
| LP++ (Adam) | 19.1 0.4 | 23.3 0.7 | 11.5 1.3 | 31.2 6.5 | 22.4 0.8 | 29.7 1.7 | 31.9 0.9 | 42.8 3.4 | 42.0 3.1 | 25.9 2.2 | 30.9 1.4 | 28.2 2.0 |
| Shots | Method | ImageNet | SUN | Aircraft | EuroSAT | Cars | Food | Pets | Flowers | Caltech | DTD | UCF | Average |
| 0 | CLIP | 62.0 | 62.1 | 18.1 | 35.8 | 60.1 | 78.9 | 85.1 | 64.0 | 91.4 | 43.0 | 61.6 | 60.2 |
| 1 | CoOp (Adam) | 62.4 0.5 | 64.1 0.2 | 21.4 1.1 | 49.2 4.4 | 60.4 0.8 | 79.0 1.2 | 85.9 0.3 | 70.8 0.9 | 91.3 0.1 | 45.2 1.4 | 68.0 1.7 | 63.4 1.1 |
| CoOp (SGD) | 63.4 0.1 | 64.4 0.2 | 14.8 7.7 | 50.7 0.9 | 59.4 0.6 | 80.2 0.5 | 88.2 0.7 | 64.6 1.7 | 90.5 0.4 | 45.0 2.4 | 65.0 0.6 | 62.4 1.4 | |
| KgCoOp (Adam) | 64.3 0.2 | 66.5 0.0 | 21.4 0.2 | 55.9 3.1 | 61.5 0.2 | 80.8 0.2 | 88.5 0.2 | 67.8 0.2 | 93.0 0.1 | 49.5 1.9 | 69.3 0.2 | 65.3 0.6 | |
| KgCoOp (SGD) | 63.5 0.2 | 65.4 0.4 | 20.6 0.6 | 51.9 0.9 | 60.4 0.2 | 80.5 0.0 | 88.5 0.5 | 65.5 0.6 | 92.2 0.3 | 47.4 1.8 | 66.6 0.2 | 63.9 0.5 | |
| ProGrad (Adam) | 54.2 4.9 | 61.9 2.0 | 16.8 1.8 | 50.7 2.6 | 59.1 1.7 | 77.0 1.6 | 82.8 4.5 | 69.0 1.0 | 90.3 0.8 | 45.1 0.8 | 66.8 1.5 | 61.2 2.1 |
| Shots | Method | ImageNet | SUN | Aircraft | EuroSAT | Cars | Food | Pets | Flowers | Caltech | DTD | UCF | Average |
| 0 | CLIP | 66.7 | 62.6 | 23.9 | 42.2 | 65.5 | 85.2 | 88.3 | 67.4 | 93.3 | 44.4 | 65.2 | 64.1 |
| 1 | CoOp (Adam) | 67.3 0.3 | 67.0 0.6 | 28.1 0.4 | 54.3 6.9 | 66.7 0.6 | 84.7 0.5 | 89.3 0.6 | 75.6 1.6 | 93.1 0.4 | 47.6 1.6 | 72.3 0.7 | 67.8 1.3 |
| CoOp (SGD) | 67.7 0.4 | 66.8 0.5 | 26.7 0.1 | 52.6 2.1 | 64.6 0.7 | 85.8 0.2 | 89.6 0.4 | 70.1 0.5 | 91.8 1.4 | 46.6 1.2 | 70.0 1.3 | 66.6 0.8 | |
| KgCoOp (Adam) | 69.3 0.2 | 68.6 0.3 | 26.8 0.2 | 60.1 2.7 | 66.5 0.2 | 86.1 0.2 | 92.3 0.3 | 72.9 0.7 | 94.1 0.4 | 52.7 0.6 | 72.6 0.8 | 69.3 0.6 | |
| KgCoOp (SGD) | 68.6 0.1 | 67.5 0.4 | 25.7 0.3 | 57.4 2.7 | 65.0 0.5 | 86.1 0.1 | 90.9 0.4 | 69.9 0.4 | 93.4 0.5 | 47.4 1.0 | 71.0 0.6 | 67.5 0.6 | |
| ProGrad (Adam) | 65.4 1.8 | 65.7 1.0 | 24.0 2.6 | 56.1 4.3 | 63.6 4.0 | 83.2 1.2 | 89.2 1.9 | 75.3 1.3 | 92.0 1.3 | 48.0 1.4 | 71.5 0.7 | 66.7 2.0 |
| Shots | Method | ImageNet | SUN | Aircraft | EuroSAT | Cars | Food | Pets | Flowers | Caltech | DTD | UCF | Average |
| 0 | CLIP | 73.5 | 67.7 | 30.0 | 55.1 | 76.8 | 90.3 | 93.1 | 76.2 | 95.2 | 52.4 | 73.7 | 71.3 |
| 1 | CoOp (Adam) | 71.2 0.6 | 68.8 0.6 | 37.2 0.4 | 58.8 3.3 | 78.1 0.9 | 88.9 0.7 | 93.4 0.3 | 89.3 1.0 | 94.5 0.3 | 53.8 1.6 | 77.6 0.4 | 73.8 0.9 |
| CoOp (SGD) | 74.7 0.2 | 69.9 0.5 | 35.8 1.5 | 60.1 3.4 | 77.6 0.2 | 90.1 0.0 | 93.9 0.2 | 80.8 1.7 | 93.3 0.8 | 53.5 1.7 | 76.4 0.6 | 73.3 1.0 | |
| KgCoOp (Adam) | 75.9 0.1 | 73.0 0.0 | 36.9 0.5 | 68.4 4.3 | 78.7 0.3 | 91.0 0.1 | 94.8 0.4 | 85.2 0.4 | 96.0 0.5 | 59.6 0.3 | 80.0 0.7 | 76.3 0.7 | |
| KgCoOp (SGD) | 75.6 0.0 | 71.6 0.2 | 34.1 0.8 | 69.1 2.0 | 77.7 0.7 | 90.9 0.2 | 93.8 0.1 | 80.9 0.5 | 95.3 0.6 | 56.6 0.4 | 76.8 0.9 | 74.8 0.6 | |
| ProGrad (Adam) | 70.9 0.6 | 69.2 0.9 | 35.3 1.0 | 59.3 4.4 | 72.7 6.5 | 88.6 0.1 | 93.4 0.2 | 86.2 1.5 | 94.2 1.2 | 54.8 1.9 | 77.4 0.7 | 72.9 1.7 |
| Shots | Method | ImageNet | SUN | Aircraft | EuroSAT | Cars | Food | Pets | Flowers | Caltech | DTD | UCF | Average |
| 0 | CLIP | 62.0 | 62.1 | 18.1 | 35.8 | 60.1 | 78.9 | 85.1 | 64.0 | 91.4 | 43.0 | 61.6 | 60.2 |
| 1 | LoRA (Adam) | 62.8 0.6 | 67.4 0.2 | 21.5 0.2 | 58.2 2.2 | 62.9 0.2 | 80.8 0.2 | 89.0 0.3 | 68.2 0.3 | 93.1 0.1 | 49.7 0.7 | 70.1 0.2 | 65.8 0.5 |
| LoRA (SGD) | 62.1 0.0 | 62.1 0.0 | 18.1 0.0 | 36.0 0.0 | 60.2 0.0 | 78.9 0.0 | 85.1 0.0 | 64.0 0.0 | 91.4 0.0 | 42.9 0.0 | 61.7 0.0 | 60.2 0.0 | |
| PiSSA (Adam) | 60.7 0.3 | 65.1 0.2 | 22.2 2.0 | 64.3 6.5 | 62.1 0.3 | 77.2 0.7 | 86.4 0.4 | 76.9 1.5 | 92.9 0.8 | 50.1 1.2 | 70.6 0.5 | 66.2 1.3 | |
| PiSSA (SGD) | 64.4 0.2 | 65.8 0.1 | 20.6 0.3 | 57.5 2.5 | 62.0 0.2 | 80.4 0.2 | 87.6 0.3 | 67.8 0.0 | 92.3 0.1 | 48.4 0.4 | 66.8 0.2 | 64.9 0.4 | |
| SGD | 64.6 0.3 | 67.4 0.1 | 22.3 1.2 | 65.1 2.4 | 63.5 0.7 | 80.5 0.4 | 89.0 0.1 | 77.4 1.0 | 93.1 0.5 | 50.0 0.1 | 71.7 0.3 | 67.7 0.6 |
| Shots | Method | ImageNet | SUN | Aircraft | EuroSAT | Cars | Food | Pets | Flowers | Caltech | DTD | UCF | Average |
| 0 | CLIP | 66.7 | 62.6 | 23.9 | 42.2 | 65.5 | 85.2 | 88.3 | 67.4 | 93.3 | 44.4 | 65.2 | 64.1 |
| 1 | LoRA (Adam) | 68.1 0.6 | 69.6 0.4 | 28.9 0.3 | 65.7 2.9 | 69.7 0.1 | 86.2 0.1 | 92.4 0.1 | 74.0 0.5 | 94.1 0.1 | 52.3 0.5 | 74.7 0.4 | 70.5 0.5 |
| LoRA (SGD) | 66.8 0.0 | 62.6 0.0 | 24.0 0.1 | 42.4 0.0 | 65.6 0.0 | 85.2 0.0 | 88.3 0.0 | 67.5 0.0 | 93.3 0.0 | 44.4 0.0 | 65.2 0.0 | 64.1 0.0 | |
| PiSSA (Adam) | 66.0 0.0 | 67.7 0.7 | 29.9 1.1 | 72.3 3.0 | 69.7 0.9 | 83.2 0.7 | 90.5 0.1 | 83.5 0.5 | 93.4 0.6 | 51.4 0.8 | 75.4 1.6 | 71.2 0.9 | |
| PiSSA (SGD) | 69.3 0.1 | 68.5 0.1 | 27.9 0.1 | 70.5 2.7 | 67.5 0.1 | 86.0 0.1 | 90.7 0.4 | 72.7 0.1 | 93.7 0.0 | 50.7 0.7 | 71.3 0.2 | 69.9 0.4 | |
| SGD | 70.1 0.2 | 70.3 0.2 | 30.5 0.9 | 73.8 2.7 | 70.8 0.6 | 85.8 0.2 | 92.7 0.2 | 83.6 0.6 | 94.0 0.2 | 53.7 0.8 | 76.1 0.5 | 72.8 0.6 |
| Shots | Method | ImageNet | SUN | Aircraft | EuroSAT | Cars | Food | Pets | Flowers | Caltech | DTD | UCF | Average |
| 0 | CLIP | 73.5 | 67.7 | 30.0 | 55.1 | 76.8 | 90.3 | 93.1 | 76.2 | 95.2 | 52.4 | 73.7 | 71.3 |
| 1 | LoRA (Adam) | 72.3 0.5 | 73.6 0.4 | 41.1 1.2 | 69.0 5.6 | 80.7 0.2 | 90.8 0.2 | 94.4 0.1 | 86.6 1.4 | 95.8 0.5 | 59.0 0.3 | 80.8 0.2 | 76.7 1.0 |
| LoRA (SGD) | 73.5 0.0 | 67.7 0.0 | 30.1 0.0 | 55.3 0.0 | 76.9 0.0 | 90.3 0.0 | 93.1 0.0 | 76.2 0.0 | 95.2 0.0 | 52.4 0.0 | 73.7 0.0 | 71.3 0.0 | |
| PiSSA (Adam) | 72.2 0.6 | 72.8 0.5 | 39.2 0.9 | 75.1 1.1 | 78.4 0.5 | 89.3 0.5 | 93.4 0.7 | 92.2 0.8 | 95.7 0.4 | 62.1 0.6 | 82.9 0.5 | 77.6 0.6 | |
| PiSSA (SGD) | 76.4 0.1 | 72.5 0.1 | 37.3 0.4 | 70.2 2.3 | 79.3 0.2 | 91.1 0.1 | 94.3 0.2 | 81.4 0.4 | 95.6 0.3 | 57.2 0.5 | 77.8 0.4 | 75.7 0.4 | |
| SGD | 76.6 0.2 | 74.3 0.1 | 41.1 0.9 | 75.3 4.1 | 80.7 0.4 | 91.0 0.1 | 94.6 0.2 | 90.2 0.8 | 95.6 0.5 | 60.5 0.4 | 81.9 0.5 | 78.3 0.7 |
| Shots | Method | ImageNet | SUN | Aircraft | EuroSAT | Cars | Food | Pets | Flowers | Caltech | DTD | UCF | Average |
| 0 | CLIP | 62.0 | 62.1 | 18.1 | 35.8 | 60.1 | 78.9 | 85.1 | 64.0 | 91.4 | 43.0 | 61.6 | 60.2 |
| 1 | LayerNorm (Adam) | 63.9 0.1 | 64.3 0.1 | 19.6 0.0 | 40.3 0.5 | 61.3 0.2 | 79.8 0.1 | 86.7 0.1 | 65.8 0.0 | 91.9 0.1 | 44.4 0.3 | 63.4 0.1 | 62.0 0.1 |
| LayerNorm (SGD) | 62.1 0.0 | 62.1 0.0 | 18.1 0.0 | 35.9 0.0 | 60.2 0.0 | 78.9 0.0 | 85.1 0.0 | 64.0 0.0 | 91.4 0.0 | 42.9 0.1 | 61.7 0.0 | 60.2 0.0 | |
| BitFit (Adam) | 64.1 0.3 | 66.8 0.1 | 22.0 0.4 | 63.0 2.9 | 62.2 0.3 | 80.3 0.4 | 88.9 0.2 | 70.2 0.6 | 93.1 0.2 | 49.9 0.1 | 70.2 0.1 | 66.4 0.5 | |
| BitFit (SGD) | 63.5 0.1 | 64.2 0.1 | 20.0 0.0 | 42.0 1.5 | 60.8 0.1 | 79.5 0.1 | 86.7 0.1 | 65.6 0.2 | 91.7 0.1 | 44.5 0.3 | 63.8 0.2 | 62.0 0.3 | |
| SAFT (Adam) | 59.7 0.7 | 64.4 0.2 | 21.0 1.0 | 66.2 1.8 | 57.5 1.0 | 78.0 0.6 | 85.3 1.1 | 73.3 0.7 | 93.3 0.7 | 47.6 0.9 | 69.7 0.6 | 65.1 0.8 |
| Shots | Method | ImageNet | SUN | Aircraft | EuroSAT | Cars | Food | Pets | Flowers | Caltech | DTD | UCF | Average |
| 0 | CLIP | 66.7 | 62.6 | 23.9 | 42.2 | 65.5 | 85.2 | 88.3 | 67.4 | 93.3 | 44.4 | 65.2 | 64.1 |
| 1 | LayerNorm (Adam) | 68.7 0.1 | 66.7 0.1 | 25.5 0.2 | 49.6 2.0 | 66.7 0.2 | 85.7 0.1 | 89.4 0.1 | 69.9 0.1 | 93.5 0.1 | 45.9 0.2 | 67.8 0.2 | 66.3 0.3 |
| LayerNorm (SGD) | 66.7 0.0 | 62.6 0.0 | 23.9 0.0 | 42.3 0.1 | 65.6 0.0 | 85.2 0.0 | 88.3 0.0 | 67.5 0.0 | 93.3 0.0 | 44.4 0.1 | 65.3 0.0 | 64.1 0.0 | |
| BitFit (Adam) | 69.4 0.2 | 69.3 0.2 | 29.5 0.3 | 72.0 1.9 | 69.1 0.4 | 85.5 0.2 | 92.3 0.3 | 74.8 0.3 | 93.9 0.2 | 52.1 0.6 | 74.6 0.4 | 71.1 0.5 | |
| BitFit (SGD) | 68.0 0.1 | 66.9 0.2 | 25.5 0.1 | 52.0 1.8 | 66.3 0.1 | 85.5 0.1 | 89.3 0.1 | 69.5 0.2 | 93.4 0.0 | 46.4 0.2 | 67.9 0.3 | 66.4 0.3 | |
| SAFT (Adam) | 66.1 0.1 | 67.2 0.3 | 27.4 0.8 | 68.9 3.5 | 64.8 0.7 | 84.1 0.4 | 89.9 0.9 | 80.7 0.9 | 93.9 0.2 | 52.2 1.0 | 73.7 0.4 | 69.9 0.8 |
| Shots | Method | ImageNet | SUN | Aircraft | EuroSAT | Cars | Food | Pets | Flowers | Caltech | DTD | UCF | Average |
| 0 | CLIP | 73.5 | 67.7 | 30.0 | 55.1 | 76.8 | 90.3 | 93.1 | 76.2 | 95.2 | 52.4 | 73.7 | 71.3 |
| 1 | LayerNorm (Adam) | 75.9 0.1 | 71.2 0.1 | 33.7 0.1 | 61.9 0.8 | 78.3 0.2 | 90.9 0.1 | 93.9 0.1 | 78.7 0.1 | 95.4 0.2 | 54.1 0.2 | 75.7 0.3 | 73.6 0.2 |
| LayerNorm (SGD) | 73.5 0.0 | 67.8 0.0 | 30.1 0.1 | 55.3 0.0 | 76.8 0.0 | 90.3 0.0 | 93.1 0.0 | 76.2 0.0 | 95.2 0.0 | 52.4 0.0 | 73.7 0.0 | 71.3 0.0 | |
| BitFit (Adam) | 75.9 0.4 | 73.6 0.1 | 39.7 0.9 | 71.4 4.1 | 80.0 0.1 | 90.9 0.3 | 94.5 0.1 | 83.2 1.0 | 95.8 0.2 | 59.5 0.4 | 80.5 0.3 | 76.8 0.7 | |
| BitFit (SGD) | 74.6 0.1 | 69.1 0.1 | 31.6 0.1 | 59.6 0.5 | 77.3 0.1 | 90.4 0.0 | 93.3 0.0 | 76.7 0.1 | 95.2 0.0 | 52.6 0.2 | 74.5 0.1 | 72.3 0.1 | |
| SAFT (Adam) | 74.2 0.6 | 73.7 0.1 | 38.4 1.6 | 74.2 3.5 | 78.5 0.0 | 89.4 0.5 | 91.0 1.9 | 89.1 0.8 | 96.2 0.8 | 59.6 1.7 | 81.3 0.8 | 76.9 1.1 |
| SGD | Adam | Adam-M0 | Recovery (%) | |
| 1 | 66.37 | 6.30 | 69.28 | 104.84 |
| 2 | 72.28 | 7.35 | 72.95 | 101.03 |
| 4 | 75.24 | 10.64 | 76.25 | 101.55 |
| 8 | 78.53 | 18.25 | 79.18 | 101.07 |
| 16 | 82.23 | 29.98 | 82.49 | 100.49 |
| Mean | 74.93 | 14.51 | 76.03 | 101.82 |
| Dataset | SGD | Adam | Adam-M0 | |
|---|---|---|---|---|
| EuroSAT | 1 | |||
| EuroSAT | 2 | |||
| EuroSAT | 4 | |||
| EuroSAT | 8 | |||
| EuroSAT | 16 | |||
| Oxford-Pets | 1 |
| Reference cosine | |||||
| 1 | 1.256 | 0.244 | 1.651 | 0.379 | 4.207 |
| 2 | 1.116 | 0.209 | 1.390 | 0.493 | 4.266 |
| 4 | 1.100 | 0.165 | 1.199 | 0.599 | 4.374 |
| 8 | 0.995 | 0.140 | 1.118 | 0.687 | 3.447 |
| 16 | 1.037 | 0.120 | 1.032 | 0.755 | 2.423 |
| Adam-M0 | Adam-Mt | Difference | |
| 1 | 73.22 | 71.58 | |
| 2 | 76.46 | 74.41 | |
| 4 | 79.29 | 77.86 | |
| 8 | 81.42 | 80.59 | |
| 16 | 84.21 | 84.00 | |
| Mean | 78.92 | 77.69 |
| Dataset | Adam-M0 | Adam-Mt | |
|---|---|---|---|
| Caltech101 | 1 | ||
| Caltech101 | 2 | ||
| Caltech101 | 4 | ||
| Caltech101 | 8 | ||
| Caltech101 | 16 | ||
| DTD | 1 |
| SGD | Adam | Adam-M0 | Adam-Mt | M0 recovery | Mt recovery | |
| (%) | (%) | |||||
| 1 | 60.06 | 1.75 | 64.64 | 60.64 | 107.85 | 101.00 |
| 4 | 68.74 | 1.62 | 70.17 | 67.68 | 102.12 | 98.41 |
| 16 | 77.91 | 8.86 | 77.97 | 78.25 | 100.07 | 100.49 |
| Mean | 68.91 | 4.07 | 70.92 | 68.86 | 103.11 | 99.92 |
| Optimizer and schedule | Accuracy | Difference | interval |
| SGD, | 74.61 | — | |
| Adam, | 15.04 | ||
| Adam, | 55.34 | ||
| Adam, | 68.39 | ||
| Adam, | 74.25 | ||
| Adam, | 76.10 |
| Optimizer | LR | Runs | Epoch 30 | Epoch 300 | Runs at epoch 300 | Peak | Last | Drop |
| Adam | 18 | 1.39 | 1.45 | 17 | 44.71 | 1.39 | 43.32 | |
| Adam | 18 | 0.89 | 1.18 | 17 | 44.78 | 1.15 | 43.62 | |
| Adam | 18 | 4.70 | 4.78 | 17 | 48.14 | 4.73 | 43.41 | |
| Adam | 18 | 64.71 | 47.30 | 17 | 77.17 | 46.96 | 30.21 | |
| Adam | 18 | 79.70 | 73.29 | 18 | 80.88 | 69.95 | 10.93 | |
| SGD | 18 | 4.15 | 8.51 | 18 | 50.00 | 8.51 | 41.49 |
| Dataset | SGD | Adam | Adam-M0 | |
| EuroSAT | 1 | |||
| EuroSAT | 4 | |||
| EuroSAT | 16 | |||
| Oxford-Pets | 1 | |||
| Oxford-Pets | 4 | |||
| Oxford-Pets | 16 |
| 1 | 1.410 | 0.919 | 0.319 | 3.492 |
| 16 | 1.033 | 1.004 | 0.137 | 1.531 |