Underrepresented in Foundation Model Pretraining Data? A One-Shot Probe

Chris Vorster; Mayug Maniparambil; Noel E. O'Connor; Noel Murphy; Derek Molloy

cs.CVMar 4, 2026

Underrepresented in Foundation Model Pretraining Data? A One-Shot Probe

Authors: Chris Vorster, Mayug Maniparambil, Noel E. O'Connor, Noel Murphy, Derek Molloy

Organizations: ML-Labs, Dublin City University, Dublin, Ireland

Abstract

Large-scale Vision-Language Foundation Models (VLFMs), such as CLIP, now underpin a wide range of computer vision research and applications. VLFMs are often adapted to various domain-specific tasks. However, VLFM performance on novel, specialised, or underrepresented domains remains inconsistent. Evaluating VLFMs typically requires labelled test sets, which are often unavailable for niche domains of interest, particularly those from the Global South. We address this gap by proposing a highly data-efficient method to predict a VLFM's zero-shot accuracy on a target domain using only a single labelled image per class. Our approach uses a Large Language Model to generate plausible counterfactual descriptions of a given image. By measuring the VLFM's ability to distinguish the correct description from these hard negatives, we engineer features that capture the VLFM's discriminative power in its shared embedding space. A linear regressor trained on these similarity scores estimates the VLFM's zero-shot test accuracy across various visual domains with a Pearson-r correlation of 0.96. We demonstrate our method's performance across five diverse datasets, including standard benchmark datasets and underrepresented datasets from Africa. Our work provides a low-cost, reliable tool for probing VLFMs, enabling researchers and practitioners to make informed decisions about data annotation efforts before committing significant resources. The model training code, generated captions and counterfactuals are released here: https://github.com/chris-vorster/PreLabellingProbe.

Figures & tables

Fig. 1: Methodological Overview. This figure illustrates our three-stage pipeline for predicting a Vision-Language Model’s zero-shot accuracy on a target domain, using the “Ekwang” class from the African Food dataset as an example. Counterfactual Probing: A single representative image (Step 1) is used to generate a plausible caption $T_{pc}$ (Step 2) and a set of counterfactual captions $\{T_{cf_{i}}\}$ via an LLM (Steps 3-4). Similarity Scoring: The VLFM under evaluation is used to compute embeddings for the image ( $\mathcal{I}$ ) and captions (Step 5). $\{T_{L_{i}}\}$ represents the standard CLIP zero-shot text prompt for class $i$ , e.g. “a photo of Ekwang”. Two sets of similarity scores are calculated: one for the standard CLIP zero-shot prompts and another for the LLM-generated captions (Step 6). Performance Prediction: The similarity scores are used as input to a Ridge Regression Model. (Steps 7). The model is trained to estimate the VLFM’s zero-shot accuracy on the full test set using only one labelled image per class.

Dataset	Task	Classes	Focus
CIFAR-10 [ 27 ]	Object classification	10	Ten common object categories (airplane, automobile, bird, etc.) in low-resolution natural scenes.
CIFAR-100 [ 27 ]	Object classification	100	One hundred fine-grained classes grouped into 20 super-classes (e.g., maples, lobsters).
Oxford 102 Flowers [ 28 ]	Flower classification	102	Photographs of 102 flower species with diverse backgrounds, lighting, and viewpoints.
Food-101 [ 29 ]	Food classification	101	Real-world dishes (e.g., pizza, sushi, ramen) with varied preparation and noisy backgrounds.
Caltech-101 [ 30 ]	Object classification	101	Images of objects from 101 categories (e.g., faces, motorcycles, crabs) with varying scale, position, and background clutter.
Oxford-IIIT Pet [ 31 ]	Pet breed classification	37	Cat and dog breeds with diverse poses, backgrounds, and bounding box annotations.

TABLE I: Overview of Datasets Used

Fig. 2: Ground-Truth vs. Predicted Zero-Shot Accuracy. The scatter plot depicts our method’s predictions across 16 datasets. The x-axis shows the actual zero-shot accuracy of OpenCLIP-ViT-B/16, calculated on each dataset’s full test set, and the y-axis shows our predicted accuracy using only a single labelled image per class. The dashed black line indicates perfect agreement. Blue points are datasets used to train our Ridge Regression Model; red points are unseen test datasets. The strong test correlation demonstrates our method’s accuracy and generalisation, even in underrepresented domains such as African Food and Beans .

Variant	Pearson-r	RMSE
LLM-generated only	0.848922	0.144607
Vanilla CLIP prompt-based only	0.947374	0.150432
PreLabellingProbe	0.961896	0.103692

TABLE II: Test Performance Comparison of Variants

Dataset	CLIP ZS Accuracy (Full testset)	PreLabellingProbe (1-shot)	Error
African Food	38.24	41.22	2.98
Beans	39.84	26.12	-13.72
Caltech101	89.25	84.86	-4.39
CIFAR10	91.68	76.41	-15.27
Food101	83.76	74.38	-9.38

TABLE III: Full Test Set CLIP zero-shot Accuracy vs. PreLabellingProbe (Percentage Scale). Pearson-r = 0.96. Root Mean Squared Error = 10.37.

VLM Evaluation

Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.

See all 989 papers

LFM: Leveraging Foundation Models for Source-Free Universal Domain Adaptation
Jul 20, 2026Jing Li, Pan Liu, Meng Zhao +7Source-Free Domain Adaptation Unsupervised Domain Adaptation
School of Computer Science and Engineering, Tianjin University of Technology, Tianjin 300384, China · Engineering Research Center of Learning-Based Intelligent System, Ministry of Education of the People’s Republic of China, Tianjin University of Technology, Tianjin 300384, China · School of Artificial Intelligence, Tianjin University, Tianjin 300350, China +1
Zero-Shot Domain Generalization via Domain Prompt Learning
Oct 22, 2026Shiming Chen, Shuhuang Chen, Salman Khan +1Domain Generalization Multimodal Domain Generalization
Mohamed Bin Zayed University of Artificial Intelligence, UAE · Huazhong University of Science and Technology, China · Australian National University, Australia +1
Unlocking Few-Shot Capabilities in LVLMs via Prompt Conditioning and Head Selection
Mar 25, 2026Adhemar de Senneville, Xavier Bou, Jérémy Anger +2Large Vision-Language Models Zero-Shot Image Classification
Université Paris-Saclay, CNRS, ENS Paris-Saclay, Centre Borelli, France · École Polytechnique, AMIAD, France · Institut Universitaire de France

Jul 20, 2026J/K move · Enter open · S save

LFM: Leveraging Foundation Models for Source-Free Universal Domain Adaptation

Jing Li, Pan Liu, Meng Zhao, Wanli Xue+6

School of Computer Science and Engineering, Tianjin University of Technology, Tianjin 300384, China · Engineering Research Center of Learning-Based Intelligent System, Ministry of Education of the People’s Republic of China, Tianjin University of Technology, Tianjin 300384, China · School of Artificial Intelligence, Tianjin University, Tianjin 300350, China+1

Source-Free Domain Adaptation Unsupervised Domain Adaptation Domain Adaptation

Open paper AskPDF arXiv

Source-free universal domain adaptation (SF-UniDA) adapts a pre-trained source model to an unlabeled target domain under both covariate and label shifts, without access to source data. However, existing SF-UniDA methods rely on inefficient techniques such as threshold tuning and clustering. Foundation models (FMs), known for their generalization and zero-shot capabilities, remain underexplored in SF-UniDA. In this paper, we propose a framework that leverages foundation models (LFM) for SF-UniDA. We use a vision-language model (VLM) to compute similarities between target samples and text labels, including those for unknown classes generated by prompting a large language model. The label shift type is determined by analyzing the coefficient of variation of a similarity-based sample-level score. Unknown samples are identified using a binary Gaussian mixture model fitted to another similarity-based metric. Under a consensus strategy, the pseudo-labels generated by the VLM are refined by the target model initialized with the pre-trained source model, integrating knowledge from both the source domain and foundation models. Finally, these refined pseudo-labels are used to train the target model. Extensive experiments across all possible label shifts and multiple benchmarks demonstrate the effectiveness and superiority of our proposed LFM framework. Our code is available at https://github.com/iamjingli/LFM.

Abstract

VLM Evaluation

Explore similar work

LFM: Leveraging Foundation Models for Source-Free Universal Domain Adaptation

Zero-Shot Domain Generalization via Domain Prompt Learning

Unlocking Few-Shot Capabilities in LVLMs via Prompt Conditioning and Head Selection