cs.CVNov 7, 2026

Permute-then-Adapt: Weak-to-Strong Contrastive Image-Text Adaptation

Authors: Jinhao Li, Sarah Erfani, Lei Feng, Guangrui Li, James Bailey, Feng Liu

Organizations: School of Computing and Information Systems, The University of Melbourne, Australia · School of Computer Science and Engineering, Southeast University, China · AWS AI Labs, USA · Department of Data Science and Artificial Intelligence, Monash University, Australia

Abstract

Image-text alignment models such as CLIP are typically trained with contrastive learning, where unpaired examples are treated as strict negatives. While this assumption introduces noise by penalising semantically related pairs, existing solutions often rely on heuristic similarity thresholds that lack statistical grounding and sensitivity to dataset-specific noise distributions. In this paper, we propose Permute-then-Adapt (PTA), a framework that leverages efficient weak models to identify missed positives while controlling Type-I error relative to a conditional permutation reference, up to calibration approximation error. Unlike standard distillation or arbitrary thresholding, PTA estimates a random-pairing reference distribution from the weak teacher's similarity scores and uses its upper tail, determined by αα, to select auxiliary positives. The calibrated positive set is then trained against by a single multi-positive contrastive objective: each anchor maximises the total probability mass it assigns to its positive set. Offline calibration with the weak model adds negligible training overhead. Experiments show gains over heuristic soft-label approaches in object recognition, cross-modal retrieval, and data efficiency. Project repository: https://github.com/JinhaoLee/Permute-then-Adapt.

Figures & tables

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. ITO: Multi-View Alignment and Training-Time Fusion for Image-Text Pretraining

    Mar 3, 2026Hanpeng Liu, Yaqian Li, Zidan Wang +6Cross-Modal LearningCross-Modal Alignment

  2. DEPICT: Scoring Text-to-Image Alignment by Answer Agreement

    Oct 2, 2026Vasco Ramos, Sandra Godinho Silva, Joao Magalhaes +2Text-Image AlignmentT2I Generation Evaluation

  3. AspectCLIP: Optimizing CLIP Representation Space via Aspect-Guided Consistency Regularization

    Jul 15, 2026Yiyang Yao, Shanglin Liu, Jianming Lv +4Vision-Language AlignmentMultimodal Contrastive Learning