Batch-Conditioned Semantic Anchors for Robust Transductive Adaptation of Vision--Language Models
Organizations: Edge Hill University, UK
Abstract
Transductive adaptation improves vision-language models at test time by refining predictions on an unlabeled batch, but current methods face a key trade-off. Aggressive approaches exploit batch structure but can drift when batches are sparse, have prior shift, or cover little of the label space. Conservative methods better preserve CLIP text semantics, but fixed semantic references can under-adapt even when the batch provides strong corrective evidence. We propose JPTA (Joint Prior and Transport Adaptation), a prior-aware framework that casts transductive VLM adaptation as batch-conditioned semantic reference estimation. Instead of treating CLIP text prototypes as fixed priors or freely replaceable, JPTA views them as semantic references that shift toward image-side batch structure only under calibrated evidence. It estimates batch priors and soft image-side prototypes to form transported semantic references whose impact is scaled by batch reliability. Across standard benchmarks, low effective-class regimes, all-class evaluation, and online streams, JPTA consistently outperforms TransCLIP and StatA, with the largest gains on sparse and prior-shifted batches. Anchor-drift, active-support, and absent-class-mass diagnostics show that JPTA fixes effective-class mismatch without unconstrained transductive drift, indicating that robust transductive VLM adaptation should recalibrate semantic references using reliable batch evidence rather than choose between unrestricted transduction and fixed prototypes. Code is available at https://github.com/MR-Sherif/JPTA.git.
Figures & tables
| Method | Avg | ImageNet | SUN397 | Aircraft | EuroSAT | Cars | Food101 | Pets | Flowers | Caltech | DTD | UCF101 | |
| CLIP | 65.2 | 66.6 | 62.5 | 24.7 | 48.3 | 65.6 | 85.9 | 89.1 | 70.7 | 93.2 | 43.5 | 67.5 | |
| MTA | 66.6 | 69.3 | 64.8 | 27.4 | 46.9 | 68.0 | 87.2 | 89.4 | 71.7 | 94.0 | 44.4 | 69.0 | |
| Very Low (1–4) | Dirichlet | 68.5 | 79.2 | 75.7 | 28.2 | 47.2 | 68.2 | 88.1 | 87.5 | 71.2 | 88.8 | 50.3 | 69.0 |
| ZLaP | 27.5 | 14.5 | 13.0 | 8.4 | 36.6 | 23.7 | 31.9 | 57.0 | 22.4 | 52.4 | 13.0 | 29.2 | |
| TransCLIP | 38.9 | 21.6 | 21.1 | 11.6 | 45.1 | 34.7 | 59.2 | 72.4 | 36.4 | 62.3 | 26.1 | 37.7 | |
| StatA | 70.4 | 72.9 | 66.0 | 29.3 | 56.8 | 76.2 | 90.3 | 95.5 | 77.6 | 93.0 | 46.1 | 70.2 |
| Method | Avg | ImageNet | SUN397 | Aircraft | EuroSAT | Cars | Food101 | Pets | Flowers | Caltech | DTD | UCF101 | |
| Medium (5–25) | Dirichlet | 64.4 | 60.9 | 75.4 | 26.7 | 38.8 | 74.1 | 76.2 | 91.0 | 71.6 | 92.4 | 36.2 | 65.4 |
| ZLaP | 41.5 | 16.6 | 20.1 | 16.4 | 49.0 | 32.2 | 55.5 | 76.4 | 40.6 | 67.7 | 34.2 | 48.1 | |
| TransCLIP | 56.5 | 39.9 | 42.7 | 22.0 | 63.1 | 49.9 | 80.6 | 87.9 | 58.7 | 79.1 | 42.9 | 55.0 | |
| StatA | 69.7 | 70.8 | 64.5 | 28.4 | 60.4 | 74.0 | 87.5 | 93.1 | 77.5 | 92.8 | 47.1 | 70.2 | |
| JPTA (Ours) | 74.5 | 88.7 | 79.6 | 26.0 | 68.4 | 77.1 | 94.4 | 92.6 | 79.8 | 94.0 | 50.1 | 68.8 | |
| High (25–50) | Dirichlet | 45.3 | 17.3 | 37.3 | 21.0 | 37.9 | 65.4 | 46.3 | 81.3 | 46.3 | 80.5 | 21.1 | 43.6 |
| Method | Avg | ImageNet | SUN397 | Aircraft | EuroSAT | Cars | Food101 | Pets | Flowers | Caltech | DTD | UCF101 | |
| CLIP | 65.2 | 66.6 | 62.5 | 24.7 | 48.3 | 65.6 | 85.9 | 89.1 | 70.7 | 93.2 | 43.5 | 67.5 | |
| MTA | 66.6 | 69.3 | 64.8 | 27.4 | 46.9 | 68.0 | 87.2 | 89.4 | 71.7 | 94.0 | 44.4 | 69.0 | |
| All | Dirichlet | 29.5 | 66.6 | 0.7 | 15.5 | 39.9 | 21.2 | 8.5 | 76.3 | 14.3 | 44.0 | 19.4 | 18.4 |
| ZLaP | 66.4 | 68.8 | 67.2 | 26.9 | 49.1 | 67.1 | 86.9 | 87.8 | 68.7 | 90.8 | 45.5 | 71.4 | |
| TransCLIP | 70.3 | 70.4 | 68.9 | 26.9 | 66.1 | 69.5 | 87.1 | 92.5 | 76.5 | 92.7 | 48.6 | 74.1 | |
| StatA | 69.9 | 69.9 | 68.7 | 24.7 | 67.3 | 68.0 | 87.1 | 92.4 | 75.2 | 94.2 | 48.4 | 73.5 |
| Scenario | Method | Avg | ImageNet | SUN397 | Aircraft | EuroSAT | Cars | Food101 | Pets | Flowers | Caltech | DTD | UCF101 |
| Shared | CLIP | 65.2 | 66.6 | 62.5 | 24.7 | 48.3 | 65.6 | 85.9 | 89.1 | 70.7 | 93.2 | 43.5 | 67.5 |
| MTA | 66.6 | 69.3 | 64.8 | 27.4 | 46.9 | 68.0 | 87.2 | 89.4 | 71.7 | 94.0 | 44.4 | 69.0 | |
| Low | TENT | 65.8 | 66.6 | 64.5 | 24.6 | 51.8 | 65.7 | 85.9 | 89.3 | 70.6 | 93.4 | 44.0 | 67.8 |
| TDA | 67.7 | 68.3 | 66.0 | 25.4 | 60.6 | 66.9 | 86.1 | 89.6 | 72.5 | 93.4 | 45.5 | 71.0 | |
| DMN | 67.2 | 68.0 | 64.8 | 24.9 | 59.8 | 67.0 | 84.2 | 89.9 | 73.3 | 92.2 | 44.8 | 70.3 | |
| StatA | 67.0 | 66.2 | 63.6 | 24.3 | 52.3 | 67.4 | 88.0 | 92.5 | 72.7 | 94.2 | 46.8 | 68.8 |
| Regime | Full JPTA | StatA-style | Transported-only | w/o prior | w/o control | Fixed pseudo. |
| Very low – | 64.56 | 51.47 ( ) | 59.82 ( ) | 61.86 ( ) | 56.46 ( ) | 61.04 ( ) |
| Low – | 61.15 | 51.51 ( ) | 57.59 ( ) | 49.27 ( ) | 50.41 ( ) | 57.14 ( ) |
| High – | 58.28 | 49.21 ( ) | 55.34 ( ) | 54.48 ( ) | 56.11 ( ) | 56.63 ( ) |
| All classes | 59.62 | 48.72 ( ) | 56.28 ( ) | 53.01 ( ) | 55.02 ( ) | 56.35 ( ) |
| Overall mean | 60.90 | 50.23 ( ) | 57.26 ( ) | 54.66 ( ) | 54.50 ( ) | 57.79 ( ) |
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
| Regime | Batch size | Effective classes |
| Very Low | 64 | 1–4 |
| Low | 64 | 2–10 |
| Medium | 64 | 5–25 |
| Medium | 1000 | 5–25 |
| High | 1000 | 25–50 |
| Very High | 1000 | 50–100 |
| Method | Adaptation family | Batch eval. | Online eval. | No support labels | Frozen CLIP | Direct match |
| Shared reference baselines | ||||||
| CLIP Radford et al. (2021) | Zero-shot reference | ✓ | ✓ | ✓ | ✓ | ✗ |
| MTA Zanella and Ben Ayed (2024b) | Static reference baseline | ✓ | ✓ | ✓ | ✓ | ✗ |
| Frozen-feature transductive baselines used in the main batch evaluation | ||||||
| Dirichlet Martin et al. (2024) | Prior-based transductive refinement | ✓ | ✗ | ✓ | ✓ | ✓ |
| ZLaP Kalantidis et al. (2024) | Graph label propagation | ✓ | ✗ | ✓ | ✓ | ✓ |
| Symbol / parameter | Role | Selection protocol |
| batch size | fixed by regime, e.g., or | |
| active classes per batch | fixed by regime in Table 7 | |
| nearest neighbors for graph | shared with TransCLIP/StatA protocol | |
| refinement iterations | fixed on validation split | |
| CLIP logit scale | frozen CLIP value | |
| anchor weight | validation-selected |
| Regime | ||||||
| Small-batch low coverage | ||||||
| Large-batch high coverage | ||||||
| All-class | ||||||
| Online streams |
| Hyperparameter | Search space |
| (a) RN50 | |||||||||||||
| Method | Avg | ImageNet | SUN397 | Aircraft | EuroSAT | Cars | Food101 | Pets | Flowers | Caltech | DTD | UCF101 | |
| CLIP | 58.7 | 58.2 | 58.9 | 17.0 | 36.2 | 55.8 | 77.4 | 85.7 | 66.1 | 85.7 | 42.8 | 61.8 | |
| Very Low (1–4) | StatA | 65.8 | 68.2 | 63.7 | 21.1 | 43.3 | 71.3 | 87.1 | 93.1 | 74.1 | 90.1 | 45.3 | 66.2 |
| JPTA (Ours) | 73.3 | 79.1 | 76.3 | 25.5 | 45.0 | 80.6 | 96.8 | 95.5 | 75.2 | 95.2 | 54.6 | 82.3 | |
| Low (2–10) | StatA | 63.5 | 65.4 | 63.1 | 16.5 | 51.7 | 65.4 | 81.0 | 84.4 | 70.0 | 88.3 | 47.2 | 66.0 |
| JPTA (Ours) | 65.8 | 68.9 | 66.3 | 17.5 | 52.9 | 67.6 | 85.3 | 88.9 | 72.2 | 89.3 | 48.0 | 67.3 | |
| Regime | Batch time (ms) | ms/image | Throughput | Peak mem. |
| 1–4, | 419.9 | 6.561 | 152 img/s | 115.1 MB |
| 2–10, | 445.2 | 6.956 | 144 img/s | 115.1 MB |
| 5–25, | 482.2 | 7.534 | 133 img/s | 115.1 MB |
| 5–25, | 535.2 | 0.535 | 1868 img/s | 358.0 MB |
| 25–50, | 545.8 | 0.546 | 1832 img/s | 412.7 MB |
| 50–100, | 543.1 | 0.543 | 1841 img/s | 414.5 MB |
| Dataset | Regime | Full JPTA | Fixed anchor (StatA-style) | w/o prior | w/o control |
| FGVC-Aircraft | Very low (1–4), B=64 | 40.10 | 29.36 ( -10.74 ) | 38.45 ( -1.65 ) | 27.40 ( -12.70 ) |
| FGVC-Aircraft | Low (2–10), B=64 | 36.24 | 28.57 ( -7.67 ) | 32.72 ( -3.52 ) | 26.32 ( -9.92 ) |
| FGVC-Aircraft | High (25–50), B=1000 | 26.67 | 21.54 ( -5.13 ) | 27.10 ( +0.43 ) | 23.97 ( -2.70 ) |
| FGVC-Aircraft | All classes | 28.23 | 18.23 ( -10.00 ) | 27.06 ( -1.17 ) | 25.68 ( -2.55 ) |
| EuroSAT | Very low (1–4), B=64 | 70.63 | 57.47 ( -13.16 ) | 66.87 ( -3.76 ) | 64.40 ( -6.23 ) |
| EuroSAT | Low (2–10), B=64 | 66.61 | 57.74 ( -8.87 ) | 37.87 ( -28.74 ) | 54.35 ( -12.26 ) |
| Dataset | Regime | Full JPTA | Original-only (StatA-style) | Transported-only | Fixed pseudocount |
| FGVC-Aircraft | Very low (1–4), B=64 | 40.10 | 29.36 ( -10.74 ) | 36.42 ( -3.68 ) | 37.88 ( -2.22 ) |
| FGVC-Aircraft | Low (2–10), B=64 | 36.24 | 28.57 ( -7.67 ) | 33.41 ( -2.83 ) | 34.52 ( -1.72 ) |
| FGVC-Aircraft | High (25–50), B=1000 | 26.67 | 21.54 ( -5.13 ) | 25.42 ( -1.25 ) | 25.86 ( -0.81 ) |
| FGVC-Aircraft | All classes | 28.23 | 18.23 ( -10.00 ) | 25.86 ( -2.37 ) | 26.91 ( -1.32 ) |
| EuroSAT | Very low (1–4), B=64 | 70.63 | 57.47 ( -13.16 ) | 66.12 ( -4.51 ) | 66.74 ( -3.89 ) |
| EuroSAT | Low (2–10), B=64 | 66.61 | 57.74 ( -8.87 ) | 62.93 ( -3.68 ) | 59.82 ( -6.79 ) |
| Regime | ZS | Full JPTA | + oracle prior | + oracle support | + oracle trust |
| Very low (1–4), | 50.21 | 64.19 | 72.66 ( +8.47 ) | 81.90 ( +17.71 ) | 64.15 ( -0.04 ) |
| High (25–50), | 49.48 | 58.24 | 65.01 ( +6.77 ) | 68.44 ( +10.20 ) | 58.22 ( -0.02 ) |
| All classes | 49.68 | 60.13 | 65.89 ( +5.76 ) | 60.13 ( 0.00 ) | 60.11 ( -0.02 ) |
| Overall mean | 49.79 | 60.85 | 67.85 ( +7.00 ) | 73.49 ( +12.64 ) | 60.83 ( -0.03 ) |
| Regime | Full JPTA | Fixed text | Image proto. | Unweighted Proc. | Weighted Proc. | Linear map |
| Very low (1–4), | 64.19 | 45.19 ( -19.00 ) | 44.23 ( -19.96 ) | 49.16 ( -15.03 ) | 54.47 ( -9.72 ) | 48.95 ( -15.24 ) |
| High (25–50), | 58.24 | 48.20 ( -10.04 ) | 54.82 ( -3.42 ) | 53.66 ( -4.58 ) | 56.02 ( -2.22 ) | 55.14 ( -3.10 ) |
| All classes | 60.13 | 50.15 ( -9.98 ) | 58.43 ( -1.70 ) | 58.64 ( -1.49 ) | 58.86 ( -1.27 ) | 58.42 ( -1.71 ) |
| Overall mean | 60.85 | 47.85 ( -13.01 ) | 52.49 ( -8.36 ) | 53.82 ( -7.03 ) | 56.45 ( -4.40 ) | 54.17 ( -6.68 ) |
VLM Adaptation
Vision-language model adaptation tailors pretrained image-text and multimodal models to new tasks and domains while retaining transferable capabilities.
See all 564 papers