HeBA: Heterogeneous Bottleneck Adapters for Robust Vision-Language Models

Md Jahidul Islam

cs.CVMar 17, 2026

HeBA: Heterogeneous Bottleneck Adapters for Robust Vision-Language Models

Authors: Md Jahidul Islam

Organizations: Department of Electrical and Electronic Engineering, Bangladesh University of Engineering and Technology, Dhaka, Bangladesh

Abstract

Adapting large-scale Vision-Language Models (VLMs) like CLIP to downstream tasks often suffers from a "one-size-fits-all" architectural approach, where visual and textual tokens are processed uniformly by wide, generic adapters. We argue that this homogeneity ignores the distinct structural nature of the modalities -- spatial locality in images versus semantic density in text. To address this, we propose HeBA (Heterogeneous Bottleneck Adapter), a unified architectural framework that introduces modality-specific structural inductive biases. HeBA departs from conventional designs through three key architectural innovations: (1) Heterogeneity: It processes visual tokens via 2D depthwise-separable convolutions to preserve spatial correlations, while distinctively processing text tokens via dense linear projections to capture semantic relationships; (2) Bottleneck Regularization: Unlike standard expanding adapters, HeBA employs a compression bottleneck (D -> D/4) that explicitly forces the model to learn compact, robust features and acts as a structural regularizer; and (3) Active Gradient Initialization: We challenge the restrictive zero-initialization paradigm, utilizing a Kaiming initialization strategy that ensures sufficient initial gradient flow to accelerate convergence without compromising the frozen backbone's pre-trained knowledge. Extensive experiments demonstrate that HeBA's architecturally specialized design achieves superior stability and accuracy, establishing a new state-of-the-art on 11 few-shot benchmarks. Code is available at https://github.com/Jahid12012021/VLM-HeBA.

Figures & tables

Figure 1: Chronological Base-to-Novel Generalization. Novel Accuracy (blue) and Harmonic Mean (red) across 11 datasets. HeBA (Ours) sets a new state-of-the-art with 78.62% Novel Accuracy and 81.35% HM .

Figure 2: Overview of the HeBA framework. We keep the pre-trained CLIP [ 27 ] backbone frozen (indicated by ) and inject lightweight, modality-specific adapters. Left: Enriched text prompts combine standard handcrafted templates with fine-grained LLM descriptions (CuPL) to enhance semantic representation. Top: The Text Adapter employs a Bottleneck linear architecture to preserve semantic integrity while compressing dimensions. Bottom: The Visual Adapter explicitly captures spatial inductive biases using $3\times 3$ depthwise convolutions (DW-Conv). Key Innovation: Unlike prior methods, the up-projection layers utilize Active Kaiming Initialization to provide immediate gradient flow, driving rapid feature adaptation from the first iteration and mitigating zero-gradient stagnation.

Figure 3: Model-level Inductive Bias Integration in HeBA. Left: Parallel adapters are inserted into the frozen Transformer block (MSA and MLP) to learn residual corrections. Right: The Text Adapter utilizes a linear bottleneck (

D\rightarrow D/4

) to preserve semantics, while the Visual Adapter employs

3\times 3

Depthwise Convolutions to enforce spatial locality. Crucially, up-projections use Kaiming Initialization to actively stimulate learning, modulated by a dynamic scaling factor

s

Method	Year	Average (11)			ImageNet			Caltech101			Oxford Pets
Method	Year	Base	Novel	HM	Base	Novel	HM	Base	Novel	HM	Base	Novel	HM
CLIP	2021	69.34	74.22	71.70	72.43	68.14	70.22	96.84	94.00	95.40	91.17	97.26	94.12
CoOp	2022	82.69	63.22	71.66	76.47	67.88	71.92	98.00	89.81	93.73	93.67	95.29	94.47
CoOpOp	2022	80.47	71.69	75.83	75.98	70.43	73.10	97.96	93.81	95.84	95.20	97.69	96.43
ProDA	2022	81.56	72.30	76.65	75.40	70.23	72.72	98.27	93.23	95.68	95.43	97.83	96.62
KgCoOp	2023	80.73	73.60	77.00	75.83	69.96	72.78	97.72	94.39	96.03	94.65	97.76	96.18

Table 1: Comparison with state-of-the-art methods on Base-to-Novel Generalization (Part 1/3). HeBA achieves the highest average Harmonic Mean (HM).

Figure 4: Fine-grained Performance Comparison on Structure-Sensitive Datasets. We report the Novel Accuracy (top) and Harmonic Mean (bottom) on four challenging benchmarks: DTD (textures) [ 5 ] , EuroSAT (satellite imagery) [ 12 ] , Oxford Flowers (fine-grained) [ 24 ] , and UCF101 (actions) [ 29 ] . These domains require capturing local spatial correlations, which standard MLP-based adapters [ 9 ] often neglect. HeBA (Red bars) consistently outperforms the previous state-of-the-art LwEIB [ 36 ] (Blue bars) and other baselines (Gray bars).

Method	Year	ImageNet	Caltech101	OxfordPets	StanfordCars	Flowers102	Food101	FGVCAircraft	SUN397	DTD	EuroSAT	UCF101	Average
CLIP [ 27 ]	2021	66.72	92.98	89.13	65.29	71.30	86.11	24.90	62.59	44.56	47.84	66.83	65.15
CoOp [ 42 ]	2022	71.51	93.70	89.14	64.51	68.71	85.30	18.47	64.15	41.92	46.39	66.55	63.88
CoCoOp [ 41 ]	2022	71.02	94.43	90.14	65.32	71.88	86.06	22.94	67.36	45.73	45.37	68.21	65.74
MaPLe [ 18 ]	2023	70.72	93.53	90.49	65.57	72.23	86.20	24.74	67.01	46.49	48.06	68.69	66.30
P-SRC [ 19 ]	2023	71.27	93.60	90.25	65.70	70.25	86.15	23.90	67.10	46.87	45.50	68.75	65.81
HPT [ 33 ]	2024	71.72	94.20	92.63	66.33	74.84	86.21	25.68	68.75	50.87	47.36	70.50	67.74

Table 2: Comparison with state-of-the-art methods in the Cross-Dataset Evaluation setting. Models are trained on ImageNet (16 shots) and evaluated on 10 other datasets.

Method	Year	ImageNet	-V2	-S	-A	-R	Average
CLIP [ 27 ]	2021	66.73	60.83	46.15	47.77	73.96	57.18
CoOp [ 42 ]	2022	71.51	64.20	47.99	49.71	75.21	59.28
CoCoOp [ 41 ]	2022	71.02	64.07	48.75	50.63	76.18	59.91
MaPLe [ 18 ]	2023	70.72	64.07	49.15	50.90	76.98	60.27
PromptSRC [ 19 ]	2023	71.27	64.35	49.55	50.90	77.80	60.65
HPT [ 33 ]	2024	71.72	65.25	49.36	50.85	77.38	60.71

Table 3: Comparison of domain generalization on ImageNet variants. HeBA shows superior robustness on ImageNet-A (Adversarial).

Configuration	Base	Novel	HM
HeBA (Full)	84.29	78.62	81.35
w/ Zero-Initialization	84.11	78.63	81.28
w/o Spatial Bias (1D)	84.04	78.63	81.25
w/o Depthwise Conv	83.96	78.63	81.20

Table 4: Ablation study of HeBA components. Full HeBA (utilizing Kaiming Init and Spatial Depthwise Convolutions) achieves the best trade-off between Base and Novel accuracy.

$\boldsymbol{\alpha_{\text{base}}}$	$\boldsymbol{\alpha_{\text{novel}}}$	Cross-Dataset	Domain Gen.
$\boldsymbol{\alpha_{\text{base}}}$	$\boldsymbol{\alpha_{\text{novel}}}$	Avg Acc (%)	Avg Acc (%)
0.05	0.075	68.26	60.62
0.05	0.050	68.60	60.67
0.05	0.025	68.71	60.26
0.05	0.0125	68.66	59.86

Table 5: Impact of Inference-time Scaling (

\alpha_{\text{novel}}

). Models are trained with

\alpha_{\text{base}}=0.05

. Reducing scale helps for Cross-Dataset transfer, while keeping it fixed is optimal for Domain Generalization.

Explore similar work

ReHARK: Refined Hybrid Adaptive RBF Kernels for Robust One-Shot Vision-Language Adaptation
Mar 12, 2026Md Jahidul IslamVLM Adaptation Test-Time Adaptation of VLMs
Department of Electrical and Electronic Engineering, Bangladesh University of Engineering and Technology, Dhaka, Bangladesh
Cross-Modal Attention Analysis and Optimization in Vision-Language Models: A Study on Visual Reliability
Apr 19, 2026Lijie ZhouCross-Modal Learning VLM Robustness
School of Computer Science University of Nottingham Ningbo China Ningbo, China
AdaptInfer: Adaptive Token Pruning for Vision-Language Model Inference via Dynamical Text Guidance
Aug 8, 2025Weichen Zhang, Zhui Zhu, Ningbo Li +5Efficient VLM Inference Visual Token Pruning
Global Innovation Exchange, Tsinghua University, Beijing, China · Department of Automation, Tsinghua University, Beijing, China · School of Computer Science, Peking University, Beijing, China

Mar 12, 2026J/K move · Enter open · S save

ReHARK: Refined Hybrid Adaptive RBF Kernels for Robust One-Shot Vision-Language Adaptation

Md Jahidul Islam

Department of Electrical and Electronic Engineering, Bangladesh University of Engineering and Technology, Dhaka, Bangladesh

VLM Adaptation Test-Time Adaptation of VLMs Kernel Methods Few-Shot Domain Adaptation

Open paper AskPDF arXiv

The adaptation of large-scale Vision-Language Models (VLMs) like CLIP to downstream tasks with extremely limited data -- specifically in the one-shot regime -- is often hindered by a significant "Stability-Plasticity" dilemma. While efficient caching mechanisms have been introduced by training-free methods such as Tip-Adapter, these approaches often function as local Nadaraya-Watson estimators. Such estimators are characterized by inherent boundary bias and a lack of global structural regularization. In this paper, ReHARK (Refined Hybrid Adaptive RBF Kernels) is proposed as a synergistic training-free framework that reinterprets few-shot adaptation through global proximal regularization in a Reproducing Kernel Hilbert Space (RKHS). A multistage refinement pipeline is introduced, consisting of: (1) Hybrid Prior Construction, where zero-shot textual knowledge from CLIP and GPT-3 is fused with visual class prototypes to form a robust semantic-visual anchor; (2) Support Set Augmentation (Bridging), where intermediate samples are generated to smooth the transition between visual and textual modalities; (3) Adaptive Distribution Rectification, where test feature statistics are aligned with the augmented support set to mitigate domain shifts; and (4) Multi-Scale RBF Kernels, where an ensemble of kernels is employed to capture complex feature geometries across diverse scales. Superior stability and accuracy are demonstrated through extensive experiments on 11 diverse benchmarks. A new state-of-the-art for one-shot adaptation is established by ReHARK, which achieves an average accuracy of 65.83%, significantly outperforming existing baselines. Code is available at https://github.com/Jahid12012021/ReHARK.

Abstract

Explore similar work

ReHARK: Refined Hybrid Adaptive RBF Kernels for Robust One-Shot Vision-Language Adaptation

Cross-Modal Attention Analysis and Optimization in Vision-Language Models: A Study on Visual Reliability

AdaptInfer: Adaptive Token Pruning for Vision-Language Model Inference via Dynamical Text Guidance