Beyond Endpoints: Bézier Paths for Vision Foundation Model Adaptation
Organizations: University of Birmingham · University of Alabama at Birmingham · University of New South Wales · Harvard University
Abstract
Frozen vision foundation models can be adapted by learning a few prompt tokens shared across all images. When prompts are updated as data shift or tasks change, the distribution of image features moves from one endpoint to another. Prior theory asks which endpoints prompts can reach; we ask whether they can travel between them along the shortest path. Because shared prompts compete with each image's own content for attention, they often cannot. We quantify this restriction as the extra transport cost that shared attention imposes over the shortest path. In an exactly solvable attention model, the reachable features form a cone; unrolling it into a plane gives the exact minimum action for every pair of endpoints and explicit optimal prompts on short arcs. The analysis reveals a sharp threshold in content normalization, which measures how much attention each image's own tokens absorb: for same-sign endpoints, at most two distinct levels allow prompts to follow the shortest path, whereas three or more force a strictly positive extra cost unless the endpoints differ only by rescaling. This gap persists with more prompts, small content perturbations, and aligned multihead outputs. On ViT-B/16 with VTAB-1k, a quadratic Bézier prompt path that minimizes feature action reduces held-out transport action relative to linear and cross-entropy-trained paths between the same endpoints and improves intermediate and online accuracy under unlabeled distribution shift, while cross-entropy-trained paths retain higher minimum accuracy when labeled calibration data are available. Reachable endpoints do not guarantee a reachable shortest path: how prompts move matters, not only where they end. Project page: https://xixiaouab.github.io/Beyond-Endpoints/.
Figures & tables
| Relative action | Pooled diagnostics | |||||
| Path | Excess | End err. † | Build ms | Valid | ||
| Token-linear | 5.3622 | 6.0701 | 4.7986 | 1 | 24/24 | |
| Aggregate-linear | 1.4300 | 1.3963 | 0.4316 | 2 | 24/24 | |
| Cone-coordinate-linear | 1.0153 | 1.0148 | 0.0254 | 25 | 24/24 | |
| Token-linear + arclength | 4.4514 | 4.9699 | 3.7815 | 8 | 24/24 | |
| Aggregate-linear + arclength | 1.0183 | 1.0137 | 0.0265 | 10 | 24/24 | |
| Transport | Prediction quality | Implementation | ||||
|---|---|---|---|---|---|---|
| Path | Action | Min. acc. | Mean acc. | Loss barrier | End err. † | Build s |
| VPT-Shallow endpoints ( Jia et al., 2022 ) (endpoint , acc. ) | ||||||
| Token-linear | 4.80 | 65.5 | 69.0 | 0.18 | 0.2 | |
| Token-linear + arclength | 4.10 | 65.5 | 69.4 | 0.18 | 20 | |
| MC Bézier (CE) | 4.50 | 68.8 | 70.0 | 0.02 | 2400 | |
| MC polygonal (CE) | 4.40 | 68.5 | 69.8 | 0.03 | 2200 | |
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
| Normalized action and its components | ||||
|---|---|---|---|---|
| Geometric curve | Clock | Total | Route | Timing |
| Token-linear | Original | 5.7161 | 3.7107 | 1.0055 |
| Arclength | 4.7107 | 3.7107 | 0.0000 | |
| Aggregate-linear | Original | 1.4131 | 0.0160 | 0.3972 |
| Arclength | 1.0160 | 0.0160 | 0.0000 | |
| A. Distinct normalizers ( ; fixed endpoint radii ) | |||
|---|---|---|---|
| Normalized diagnostics | |||
| Setting | Prediction for | Gap † | Formula error † |
| Controlled 24-atom architecture test | Normalization / head mixing | Measured norm. gap | Endpoint residual |
|---|---|---|---|
| Shared-query scalar head (Prop. 4.1 ) | None / single head | ||
| Pre-attention LayerNorm ( , App. F.2 ) | Pre-LN / single head | ||
| Post-attention output LN ( , App. F.2 ) | Output LN / single head | ||
| Isometric vector head ( , App. F.1 ) | None / vector output | ||
| Two aligned heads, concatenated (App. F.3 ) | None / unmixed concat. | ||
| Two independent heads, summed (App. F.3 ) | None / post-attention sum |
| Top-1 accuracy (%) | Resource cost | ||||||
| Endpoint trainer | Natural | Specialized | Structured | All (19) | Trainable | Peak mem. | Latency |
| (7) | (4) | (8) | M | GiB | ms/img | ||
| Reference adaptation | |||||||
| Linear probe | 69.0 | 80.6 | 33.2 | 56.4 | 0.06 | 1.5 | 0.65 |
| Full fine-tuning | 76.2 | 85.0 | 53.4 | 68.5 | 86.0 | 10.0 | 0.65 |
| Parameter-efficient updates | |||||||
| A. Block 1 single-head midpoint diagnostics | B. Layerwise query and normalizer spread | ||||
|---|---|---|---|---|---|
| Diagnostic | Shallow | Deep | Block | Query RMS disp. | P90 P10 |
| Paired endpoint cost | 1 | ||||
| Mean separation term | 6 | ||||
| Relative term | 12 | ||||
| Positive numerical residuals | – | – | – | ||
| A. Final CLS readout stage sensitivity (mean within-task 5-pair SD across 19 tasks) | |||||
| Readout stage | Shallow | Shallow CE / FA ( ) | Deep | Deep CE / FA ( ) | |
| Before final LayerNorm | / ( ) | / ( ) | |||
| After final LayerNorm (pre-unit) | / ( ) | / ( ) | |||
| After LayerNorm + unit L2 norm | / ( ) | / ( ) | |||
| Natural (7) | ||||||||
| Endpoint trainer | CIFAR 100 | Caltech 101 | DTD | Flowers 102 | Pets | SVHN | SUN 397 | Mean |
| Linear probe | 66.0 | 78.6 | 56.8 | 87.2 | 77.1 | 72.1 | 45.2 | 69.0 |
| Full fine-tuning | 71.6 | 86.4 | 63.7 | 95.0 | 84.7 | 80.0 | 52.1 | 76.2 |
| BitFit | 71.1 | 82.9 | 61.1 | 92.4 | 80.7 | 77.8 | 49.2 | 73.6 |
| LoRA | 76.0 | 89.6 | 68.2 | 96.3 | 88.1 | 82.1 | 55.4 | 79.4 |
| AdaptFormer | 75.8 | 90.6 | 68.6 | 98.6 | 87.9 | 82.8 | 55.1 | 79.9 |
| Specialized (4) | |||||
| Endpoint trainer | Patch Camelyon | EuroSAT | RESISC45 | Retinopathy | Mean |
| Linear probe | 79.6 | 89.7 | 82.1 | 71.1 | 80.6 |
| Full fine-tuning | 84.2 | 94.7 | 86.0 | 74.9 | 85.0 |
| BitFit | 82.8 | 92.6 | 81.9 | 73.2 | 82.6 |
| LoRA | 85.7 | 94.0 | 85.1 | 76.1 | 85.2 |
| AdaptFormer | 84.1 | 94.4 | 85.0 | 78.1 | 85.4 |
| Structured (8) | |||||||||
| Endpoint trainer | CLEVR count | CLEVR distance | DMLab | KITTI distance | dSprites location | dSprites orient. | sNORB azimuth | sNORB elev. | Mean |
| Linear probe | 52.9 | 37.3 | 22.8 | 48.2 | 52.2 | 26.7 | 8.0 | 17.8 | 33.2 |
| Full fine-tuning | 74.5 | 56.2 | 45.3 | 68.8 | 71.0 | 45.9 | 27.1 | 38.5 | 53.4 |
| BitFit | 66.9 | 49.9 | 36.3 | 59.6 | 62.6 | 40.3 | 20.2 | 29.8 | 45.7 |
| LoRA | 80.5 | 61.9 | 47.8 | 73.3 | 77.0 | 52.0 | 30.7 | 44.2 | 58.4 |
| AdaptFormer | 80.1 | 63.8 | 48.7 | 72.5 | 77.0 | 53.2 | 31.9 | 43.5 | 58.8 |
| VPT mode | Endpoint | 8-segment FA action | Sampled excess | Spherical OT gap | Fixed-image spherical gap |
|---|---|---|---|---|---|
| Shallow | ( ) | ( ) | |||
| Deep | ( ) | ( ) |
| A. Fixed-endpoint transition under shift ( shifted acc.: Shallow, Deep) | |||||
| VPT mode | Entropy min. acc. (%) | FA min. acc. (%) | Labeled CE min. acc. (%) | Entropy / FA action | |
| Shallow | / | ||||
| Deep | / | ||||
Vision Foundation Model Adaptation
Vision foundation model adaptation tailors pretrained image and video models to new tasks and domains while balancing accuracy, efficiency, and generalization.
See all 352 papers