BiCLIP: Domain Canonicalization via Structured Geometric Transformation

Pranav Mantini; Shishir K. Shah

cs.CVMar 9, 2026

BiCLIP: Domain Canonicalization via Structured Geometric Transformation

Authors: Pranav Mantini, Shishir K. Shah

Organizations: University of Houston, Department of Computer Science · The University of Oklahoma, School of Computer Science

Abstract

Recent advances in vision-language models (VLMs) have demonstrated remarkable zero-shot capabilities, yet adapting these models to specialized domains remains a significant challenge. Building on recent theoretical insights suggesting that independently trained VLMs are related by a canonical transformation, we extend this understanding to the concept of domains. We hypothesize that image features across disparate domains are related by a canonicalized geometric transformation that can be recovered using a small set of anchors. Few-shot classification provides a natural setting for this alignment, as the limited labeled samples serve as the anchors required to estimate this transformation. Motivated by this hypothesis, we introduce BiCLIP, a framework that applies a targeted transformation to multimodal features to enhance cross-modal alignment. Our approach is characterized by its extreme simplicity and low parameter footprint. Extensive evaluations across 11 standard benchmarks, including EuroSAT, DTD, and FGVCAircraft, demonstrate that BiCLIP consistently achieves state-of-the-art results. Furthermore, we provide empirical verification of existing geometric findings by analyzing the orthogonality and angular distribution of the learned transformations, confirming that structured alignment is the key to robust domain adaptation. Code is available at https://github.com/QuantitativeImagingLaboratory/BilinearCLIP

Figures & tables

Panel of Figure 1 — Figure 1: Quantitative analysis of angular distribution on DTD [ 4 ] dataset. The zero-shot CLIP (a) shows significant overlap between positive and negative pairs. A structured geometric transformation on the image features (b) reduces the overlap area significantly.

Figure 2: The BiCLIP Adaptation Framework. Unlike standard CLIP which relies on a fixed dot product, BiCLIP introduces a trainable, structured transformation matrix $\mathbf{W}$ between the image and text modalities.

Dataset	Zero-Shot	BiCLIP	Delta	Zero-Shot	BiSigLIP	Delta
	CLIP	(Ours)	$\Delta$	SigLIP	(Ours)	$\Delta$
ImageNet	68.84	71.69	+2.85	74.89	76.73	+1.83
DTD	42.82	71.01	+28.19	62.23	73.94	+11.70
EuroSAT	48.22	85.13	+36.91	35.35	77.50	+42.15
Flowers102	70.99	94.97	+23.99	81.15	96.11	+14.96
FGVCAircraft	24.60	45.21	+20.61	45.99	49.41	+3.42

Table 1: Main Results: 16-Shot Performance Comparison. We report Top-1 Accuracy (%) for Zero-Shot baselines and our proposed Bilinear adaptation (BiCLIP and BiSigLIP).

\Delta

represents the gain over the respective zero-shot baseline.

Panel of Figure 3 — Figure 3: Few-shot performance comparison on various datasets. Our methods BiCLIP (black) and BiSigLIP (red) significantly outperform existing prompt tuning baselines across 1, 2, 4, 8, and 16 shots.

Dataset	CLIP	BiCLIP	$\Delta$ (Reduction)
FGVCAircraft	0.327	0.175	0.152
OxfordPets	0.100	0.044	0.056
Flowers102	0.181	0.026	0.155
StanfordCars	0.071	0.048	0.023
Food101	0.056	0.041	0.015
DTD	0.539	0.167	0.372

Table 2: Comparison of overlap in average angular distributions between image and text embeddings for zero-shot CLIP and BiCLIP. Lower values indicate better separation.

	Normalized
Dataset	Orthogonal Error
FGVCAircraft	0.008
OxfordPets	0.007
Flowers102	0.027
StanfordCars	0.074
Food101	0.006

Table 3: Orthogonality of

\mathbf{W}

matrix: We report the normalized Frobenius norm deviation from orthogonality.

Initialization	Structure	EuroSAT	DTD	Aircraft
Random	Dense	81.04	69.63	44.88
Random	Upper Tri	84.28	69.57	44.55
Identity	Dense	82.54	70.74	45.18
Identity (Ours)	Upper Tri	85.13	71.01	45.21

Table 4: Ablation study on EuroSAT, DTD, and FGVCAircraft (16-shot). We evaluate the impact of Identity Initialization and the Upper Triangular (Upper Tri) structural constraint.

VLM Adaptation

Vision-language model adaptation tailors pretrained image-text and multimodal models to new tasks and domains while retaining transferable capabilities.

See all 545 papers

Improving CLIP Adaptation by Breaking Tail Alignment for Source-Free Cross-Domain Few-Shot Learning
May 28, 2026Shuai Yi, Yixiong Zou, Yuhua Li +1Source-Free Cross-Domain Few-Shot Learning Source-Free Domain Adaptation
Institute of Artificial Intelligence, Huazhong University of Science and Technology, Wuhan, China · School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, China.
Reviving In-domain Fine-tuning Methods for Source-Free Cross-domain Few-shot Learning
May 12, 2026Yaze Zhao, Yicong Liu, Yixiong Zou +2Source-Free Cross-Domain Few-Shot Learning Source-Free Domain Adaptation
Domain Recentering and Confidence-Weighted Prior Calibration for Vision-Language Models
Sep 24, 2026Youngeun Seol, Jimin Shin, Heeseo Yoon +1Source-Free Domain Adaptation Unsupervised Domain Adaptation
Department of Computer Science and Engineering, Ewha Womans University

May 28, 2026J/K move · Enter open · S save

Improving CLIP Adaptation by Breaking Tail Alignment for Source-Free Cross-Domain Few-Shot Learning

Shuai Yi, Yixiong Zou, Yuhua Li, Ruixuan Li

Institute of Artificial Intelligence, Huazhong University of Science and Technology, Wuhan, China · School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, China.

Source-Free Cross-Domain Few-Shot Learning Source-Free Domain Adaptation Few-Shot Domain Adaptation VLM Adaptation Few-Shot Learning Vision-Language Alignment

Open paper AskPDF arXiv

Vision-Language Models (VLMs) such as CLIP demonstrate strong zero-shot generalization, but their performance significantly degrades in cross-domain scenarios with scarce target-domain training data (Cross-Domain Few-Shot Learning, CDFSL). In this paper, we focus on the target-domain few-shot finetuning in the CLIP-based CDFSL task. Prevailing finetuning paradigms uniformly align all image patch tokens with their corresponding textual embeddings. However, we find a counterintuitive phenomenon: actively pushing away certain low-similarity image tokens, termed "tail tokens", from their textual embeddings consistently improves target-domain performance. We delve into this phenomenon and provide a novel interpretation: under great domain shifts and scarce training data, the model can hardly extract semantic information from visual inputs; therefore, the common belief of alignment is valid only for tokens already containing sufficient semantic information; for tail tokens, forcing the alignment would lead to excessive overfitting to the scarce training, while breaking the alignment is more useful. Motivated by this, we propose Adaptive Tail-Head Alignment (ATHA), a novel fine-tuning strategy for CLIP that transforms the conventional uniform alignment paradigm to an adaptive alignment paradigm, with both alignment strengthening and weakening. Extensive experiments on four challenging CDFSL benchmarks validate our state-of-the-art performance. Our code is available at https://github.com/shuaiyi308/ATHA.

Abstract

VLM Adaptation

Explore similar work

Improving CLIP Adaptation by Breaking Tail Alignment for Source-Free Cross-Domain Few-Shot Learning

Reviving In-domain Fine-tuning Methods for Source-Free Cross-domain Few-shot Learning

Domain Recentering and Confidence-Weighted Prior Calibration for Vision-Language Models