ReHARK: Refined Hybrid Adaptive RBF Kernels for Robust One-Shot Vision-Language Adaptation

Md Jahidul Islam

cs.CVMar 12, 2026

ReHARK: Refined Hybrid Adaptive RBF Kernels for Robust One-Shot Vision-Language Adaptation

Authors: Md Jahidul Islam

Organizations: Department of Electrical and Electronic Engineering, Bangladesh University of Engineering and Technology, Dhaka, Bangladesh

Abstract

The adaptation of large-scale Vision-Language Models (VLMs) like CLIP to downstream tasks with extremely limited data -- specifically in the one-shot regime -- is often hindered by a significant "Stability-Plasticity" dilemma. While efficient caching mechanisms have been introduced by training-free methods such as Tip-Adapter, these approaches often function as local Nadaraya-Watson estimators. Such estimators are characterized by inherent boundary bias and a lack of global structural regularization. In this paper, ReHARK (Refined Hybrid Adaptive RBF Kernels) is proposed as a synergistic training-free framework that reinterprets few-shot adaptation through global proximal regularization in a Reproducing Kernel Hilbert Space (RKHS). A multistage refinement pipeline is introduced, consisting of: (1) Hybrid Prior Construction, where zero-shot textual knowledge from CLIP and GPT-3 is fused with visual class prototypes to form a robust semantic-visual anchor; (2) Support Set Augmentation (Bridging), where intermediate samples are generated to smooth the transition between visual and textual modalities; (3) Adaptive Distribution Rectification, where test feature statistics are aligned with the augmented support set to mitigate domain shifts; and (4) Multi-Scale RBF Kernels, where an ensemble of kernels is employed to capture complex feature geometries across diverse scales. Superior stability and accuracy are demonstrated through extensive experiments on 11 diverse benchmarks. A new state-of-the-art for one-shot adaptation is established by ReHARK, which achieves an average accuracy of 65.83%, significantly outperforming existing baselines. Code is available at https://github.com/Jahid12012021/ReHARK.

Figures & tables

Figure 1: 1-Shot performance comparison across 11 benchmarks. The proposed ReHARK method (red line with star markers) consistently outperforms existing training-free adaptation baselines.

Figure 2: The overall architecture of the proposed ReHARK framework. Visual features and ensembled text weights (enriched by GPT3) undergo non-linear rectification before entering the core adaptation module. The system combines a refined semantic prior (global logic) with a multi-scale RBF kernel path (local adaptation) to solve for optimal adaptation coefficients in closed form.

Method	ImageNet	Caltech101	DTD	EuroSAT	FGVCAircraft	Food101	OxfordFlowers	OxfordPets	StanfordCars	SUN397	UCF101	Average
Zero-Shot CLIP	60.35	85.68	42.91	36.27	17.01	77.37	66.02	85.72	55.75	58.82	61.78	58.88
GDA	60.68	87.29	46.26	58.30	17.78	77.42	72.08	85.49	56.78	59.93	62.65	62.24
Tip-Adapter	60.58	88.09	45.90	56.76	19.06	77.54	75.06	86.02	57.11	60.85	64.40	62.85
ProKeR	60.60	88.17	47.99	59.75	20.65	77.40	78.85	86.44	56.79	59.66	65.13	63.77
ReHARK	61.88	90.13	49.23	69.19	21.45	77.55	80.82	86.34	59.18	63.53	64.83	65.83

Table 1: One-shot classification accuracy (%) comparison across 11 datasets.

Figure 3: Qualitative analysis of ReHARK 1-shot predictions across 11 benchmarks. Green labels indicate correct classifications, while red labels denote misclassifications. The model demonstrates high fidelity in diverse domains, including fine-grained objects and complex scenes.

Figure 4: t-SNE visualization of the latent space clusters generated by ReHARK. The Multi-Scale RBF kernels effectively capture the local geometry of class distributions across 11 datasets, facilitating distinct separation even with a single support sample per class.

Configuration	ImageNet	Caltech101	DTD	EuroSAT	FGVCAircraft	Food101	OxfordFlowers	OxfordPets	StanfordCars	SUN397	UCF101	Avg.
NO_Refine	61.54	89.75	49.67	68.15	21.15	77.76	80.32	85.91	58.19	63.25	64.75	65.49
NO_MULTISCALE	61.72	89.84	49.21	68.66	21.04	77.52	81.05	85.65	59.40	63.39	65.45	65.72
NO_RECTIFY	61.57	89.83	48.52	66.53	21.06	77.80	81.03	86.38	58.84	63.53	64.69	65.43
NO_AUGMENT	61.60	89.93	49.27	68.75	21.25	77.76	81.25	86.07	59.06	63.55	65.12	65.78
NO_POWER	61.78	89.99	49.47	68.11	21.04	77.15	78.06	85.83	58.50	63.24	65.31	65.32
ReHARK	62.09	90.11	49.29	68.56	21.22	77.60	80.96	86.01	58.88	63.59	64.97	65.75

Table 2: Ablation study of ReHARK components on 1-shot classification average accuracy (%) (500 trials).

Figure 5: Ablation study evaluating the impact of individual architectural components. Removing the Power Transform causes the most significant performance degradation (

65.32\%

), while the Full ReHARK configuration maintains robust accuracy (

65.75\%

) across all components.

Modality	ImageNet	Caltech101	DTD	EuroSAT	FGVCAircraft	Food101	OxfordFlowers	OxfordPets	StanfordCars	SUN397	UCF101	Avg.
ONLY_TEXT_GPT	61.06	89.70	48.72	68.63	20.25	74.79	80.16	83.44	56.98	61.55	62.27	64.32
ONLY_VISUAL	25.63	76.13	35.84	63.41	16.05	38.30	67.42	42.08	30.94	38.15	48.20	43.83
FULL SYNERGY	62.09	90.11	49.29	68.56	21.22	77.60	80.96	86.01	58.88	63.59	64.97	65.75

Table 3: Effect of different modality priors on 1-shot adaptation average accuracy (%).

Figure 6: Prior modality ablation study. Relying solely on visual priors results in a sharp performance drop to

43.83\%

. The combination of both textual and visual modalities in the full ReHARK framework yields the best average accuracy of

65.75\%

Trials	ImageNet	Caltech101	DTD	EuroSAT	FGVCAircraft	Food101	OxfordFlowers	OxfordPets	StanfordCars	SUN397	UCF101	Avg.
50	59.69	90.17	49.25	67.83	20.43	76.54	79.90	85.09	57.80	62.63	64.26	64.87
100	60.52	89.99	49.17	67.80	21.04	77.08	80.08	85.48	58.48	63.16	65.34	65.29
500	62.09	90.11	49.29	68.56	21.22	77.60	80.96	86.01	58.88	63.59	64.97	65.75
1000	61.88	90.13	49.23	69.19	21.45	77.55	80.82	86.34	59.18	63.53	64.83	65.83

Table 4: Impact of search trials on 1-shot classification accuracy (%).

Figure 7: Sensitivity analysis of the Optuna search budget. Accuracy increases with the number of trials, reaching a peak of

65.83\%

at 1,000 trials. Significant gains are observed when increasing the budget from 50 to 500 trials.

Kernel	ImageNet	Caltech101	DTD	EuroSAT	FGVCAircraft	Food101	OxfordFlowers	OxfordPets	StanfordCars	SUN397	UCF101	Avg.
LINEAR	26.26	88.38	47.77	63.23	18.54	45.21	77.51	80.05	44.99	55.47	62.50	55.45
LAPLACIAN	43.30	89.95	49.03	57.43	21.21	54.41	80.71	85.79	59.24	63.47	64.70	60.84
RBF	61.88	90.13	49.23	69.19	21.45	77.55	80.82	86.34	59.18	63.53	64.83	65.83

Table 5: Impact of different kernel types on 1-shot classification accuracy (%).

Figure 8: Performance comparison of different kernel functions across 11 benchmarks. The RBF kernel (red) consistently achieves the highest accuracy (

65.83\%

), significantly outperforming the Linear (

55.45\%

) and Laplacian (

60.84\%

) baselines, especially on ImageNet and Food101.

Explore similar work

Improving CLIP Adaptation by Breaking Tail Alignment for Source-Free Cross-Domain Few-Shot Learning
May 28, 2026Shuai Yi, Yixiong Zou, Yuhua Li +1Source-Free Domain Adaptation Few-Shot Domain Adaptation
Institute of Artificial Intelligence, Huazhong University of Science and Technology, Wuhan, China · School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, China.
HeBA: Heterogeneous Bottleneck Adapters for Robust Vision-Language Models
Mar 17, 2026Md Jahidul IslamVLM Adaptation Adapter Tuning
Department of Electrical and Electronic Engineering, Bangladesh University of Engineering and Technology, Dhaka, Bangladesh
PERL: Parameter Efficient Reasoning in CLIP Latent Space
May 18, 2026Simone Carnemolla, Salvatore Calcagno, Daniela Giordano +2VLM Adaptation Few-Shot Learning
University of Catania

May 28, 2026J/K move · Enter open · S save

Improving CLIP Adaptation by Breaking Tail Alignment for Source-Free Cross-Domain Few-Shot Learning

Shuai Yi, Yixiong Zou, Yuhua Li, Ruixuan Li

Institute of Artificial Intelligence, Huazhong University of Science and Technology, Wuhan, China · School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, China.

Source-Free Domain Adaptation Few-Shot Domain Adaptation VLM Adaptation Few-Shot Learning Vision-Language Alignment

Open paper AskPDF arXiv

Vision-Language Models (VLMs) such as CLIP demonstrate strong zero-shot generalization, but their performance significantly degrades in cross-domain scenarios with scarce target-domain training data (Cross-Domain Few-Shot Learning, CDFSL). In this paper, we focus on the target-domain few-shot finetuning in the CLIP-based CDFSL task. Prevailing finetuning paradigms uniformly align all image patch tokens with their corresponding textual embeddings. However, we find a counterintuitive phenomenon: actively pushing away certain low-similarity image tokens, termed "tail tokens", from their textual embeddings consistently improves target-domain performance. We delve into this phenomenon and provide a novel interpretation: under great domain shifts and scarce training data, the model can hardly extract semantic information from visual inputs; therefore, the common belief of alignment is valid only for tokens already containing sufficient semantic information; for tail tokens, forcing the alignment would lead to excessive overfitting to the scarce training, while breaking the alignment is more useful. Motivated by this, we propose Adaptive Tail-Head Alignment (ATHA), a novel fine-tuning strategy for CLIP that transforms the conventional uniform alignment paradigm to an adaptive alignment paradigm, with both alignment strengthening and weakening. Extensive experiments on four challenging CDFSL benchmarks validate our state-of-the-art performance. Our code is available at https://github.com/shuaiyi308/ATHA.

Abstract

Explore similar work

Improving CLIP Adaptation by Breaking Tail Alignment for Source-Free Cross-Domain Few-Shot Learning

HeBA: Heterogeneous Bottleneck Adapters for Robust Vision-Language Models

PERL: Parameter Efficient Reasoning in CLIP Latent Space