Test-Time Adaptation of VLMs

Test-time adaptation adjusts a vision-language model to the data it encounters at deployment, aiming to preserve performance when images, modalities, or tasks differ from pretraining. Common approaches include prompt or adapter tuning, entropy and consistency objectives, pseudo-labeling, and reinforcement learning, alongside training-free methods that use caches, prototypes, retrieval, or feature calibration. Work spans CLIP classification and robustness benchmarks as well as detection, segmentation, medical imaging, and multimodal question answering. A central challenge is obtaining reliable adaptation signals without amplifying errors, causing drift or miscalibration, or spending more computation than the gains justify.