Recognizing implicit visual and textual patterns is essential in many real-world applications of modern AI. However, tackling long-tail pattern recognition tasks remains challenging for current pre-trained foundation models such as LLMs and VLMs. While finetuning pre-trained models can improve accuracy in recognizing implicit patterns, it is usually infeasible due to a lack of training data and high computational overhead. In this paper, we propose ADAMAB, an efficient embedding calibration framework for few-shot pattern recognition. To maximally reduce the computational costs, ADAMAB trains embedder-agnostic light-weight calibrators on top of fixed embedding models without accessing their parameters. To mitigate the need for large-scale training data, we introduce an adaptive data augmentation strategy based on the Multi-Armed Bandit (MAB) mechanism. With a modified upper confidence bound algorithm, ADAMAB diminishes the gradient shifting and offers theoretically guaranteed convergence in few-shot training. Our multi-modal experiments justify the superior performance of ADAMAB, with up to 40% accuracy improvement when training with less than 5 initial data samples of each class.
Figures & tables
Figure 1: Illustration of light-weight neural similarity networks.
Figure 2: The framework of ADAMAB . ADAMAB can be applied to both visual and textual pattern recognition tasks. To calibrate a pre-trained embedder (e.g., CLIP), we train a light-weight neural similarity network with a detailed structure illustrated in Fig. 1 . We select a class with a Multi-armed Bandit, and augment the samples of this class with another pre-trained generator (e.g., diffusion models and GPT for image and text generation respectively). We alternatively augment the training data and train the model until convergence.
Dataset
MultiWD
FQS
TREC
OxfordPets
Flowers102
CUB200
Query Modality
Text
Text
Text
Image
Image
Image
Label Modality
Text
Text
Text
Text
Text
Text
# Classes
6
13
30
37
102
200
# Init Data
30 (5/class)
65 (5/class)
147 (5/class)
111 (3/class)
204 (2/class)
400 (2/class)
# Test Data
227
325
688
3558
5945
5394
Table 1: A summary of the datasets.
Model/Method
MultiWD
Forbidden Question Set
TREC
# Params
Zero-shot
Few-shot
Zero-shot
Few-shot
Zero-shot
Few-shot
GPT-4o-mini
37.89%
51.54%
80.31%
76.00%
60.03%
43.31%
n/a
Gemini2.0-Flash-Lite
39.65%
46.39%
74.77%
81.85%
54.94%
59.74%
n/a
Mistral-Small
30.40%
45.82%
77.23%
79.39%
43.02%
42.73%
24B
Llama-3.2
28.19%
37.00%
75.08%
79.69%
53.78%
60.47%
90B
Cohere-v3.5
3.08%
25.55%
54.15%
78.77%
32.70%
58.29%
n/a
Table 2: The classification accuracy of each method under both zero-shot and few-shot settings in text-query datasets. For embedding calibration methods, we also include the improvement on top of the original embedding models. We report the number of parameters as a metric of the efficiency for each method, where “n/a” denotes that the model is closed-source without parameter information.
Model/Method
OxfordPets
Flowers102
CUB200
# Params
GPT-4o-mini
79.83%
71.05%
33.00%
n/a
Gemini2.0-Flash-Lite
93.32%
85.38%
60.07%
n/a
Mistral-Small
61.22%
48.61%
38.93%
24B
Llama-3.2
78.64%
67.94%
40.06%
90B
Jina-reranker-m0
73.64%
56.60%
35.22%
2.4B
CLIP-VIT-Large
82.88%
60.99%
33.18%
0.4B
Table 3: The classification accuracy of each method in image-query datasets. For embedding calibration methods, we also include the improvement on top of the original embedding models. We report the number of parameters as a metric of the efficiency for each method, where “n/a” denotes that the model is closed-source without parameter information.
Figure 3: Calibration Accuracy with respect to the average number of training samples per class.
Figure 4: Calibration Accuracy with respect to the exploration hyperparameter α .
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Dataset
MultiWD
FQS
TREC
OxfordPets
Flowers102
CUB200
Batch Size
32
32
64
64
128
256
# Training Rounds
30
50
100
100
300
500
Δn
5
5
5
3
2
2
# Aug. Rounds
18
39
60
74
102
400
α
100
100
30
100
30
100
Appendix
Table 4: The hyperparameters for the results in Tab. 2 and Tab. 3 .