Gram-Anchored Prompt Learning for Vision-Language Models via Second-Order Statistics

Minglei Chen; Weilong Wang; Jiang Duan; Ye Deng

cs.CVApr 5, 2026

Gram-Anchored Prompt Learning for Vision-Language Models via Second-Order Statistics

Authors: Minglei Chen, Weilong Wang, Jiang Duan, Ye Deng

Organizations: All authors are with Southwestern University of Finance and Economics, Chengdu, Sichuan, China.

Abstract

Parameter-efficient prompt learning has become the de facto standard for adapting Vision-Language Models (VLMs) to downstream tasks. Existing approaches predominantly focus on aligning text prompts with first-order visual features (i.e., spatial feature maps). While effective for fine-grained semantic discrimination, we argue that relying solely on first-order information is insufficient for robust adaptation, as these spatially entangled features are highly susceptible to domain shifts and local noise. In this work, we propose Gram-Anchored Prompt Learning (GAPL) for Vision-Language Models via Second-Order Statistics, a framework that synergizes local semantic alignment with global structural consistency. Methodologically, we introduce an additional second-order statistical stream via Gram matrices that augments the standard first-order spatial interaction. By anchoring prompts to these second-order priors, our approach enables language representations to dynamically adapt to statistical distribution shifts across diverse domains. Extensive experiments indicate the effectiveness of the second-order features, and show compelling performances of GAPL on various benchmarks.

Figures & tables

Fig. 1: Illustration of first-order and anchored feature spaces. Under domain shift (e.g., photo vs. sketch), first-order conditioning may produce separated representations for the same semantic concept, whereas Gram-based anchoring introduces a second-order cue that brings them closer in the feature space.

Fig. 2: Overview of the proposed Gram-Anchored Prompt Learning (GAPL) framework. Learnable prompt tokens are inserted into the text encoder, and only the input-layer prompts are shown for clarity. GAPL contains three branches: (1) a Global Invariant Stream that aligns the prompted text feature with the global visual feature from the CLS token; (2) a Gram-Anchored Stream (purple), the core component of our method, which extracts a Gram-based second-order cue from patch tokens and uses a Gram-based Style Modulator to generate a Style Text Anchor; and (3) a Contextual-Anchored Stream (green), which uses learnable local signals to produce Contextual Text Anchors for fine-grained alignment. The three streams are optimized jointly for robust prompt adaptation across domains.

Fig. 3: Detailed architecture of the Gram-based Style Modulator and the Contextual Modulator. Left: the Gram-based Style Modulator receives a Gram-based second-order cue derived from patch tokens, keeps their diagonal Gram features as a compact image-level descriptor, and converts them into a gating vector for modulating the prompted text feature. Right: the Contextual Modulator uses the prompted text feature as a query to interact with learnable local signals, producing contextual text anchors for fine-grained alignment.

Method	Source	Target (OOD Variants)				Average
Method	ImageNet	-V2	-Sketch	-A	-R
CoOp [ 37 ]	71.51	64.20	47.99	49.71	75.21	59.28
CoCoOp [ 36 ]	71.02	64.07	48.75	50.63	76.18	59.91
KgCoOp [ 30 ]	70.66	64.10	48.97	50.69	76.70	60.12
MaPLe [ 13 ]	70.72	64.07	49.15	50.90	76.98	60.27
TCP [ 31 ]	71.20	64.60	49.50	51.20	76.73	60.51

TABLE I: Domain Generalization. GAPL is trained on ImageNet (16 shots) and directly evaluated on the target datasets. The best results are highlighted in bold .

Method	Average Performance			ImageNet			Caltech101			OxfordPets
Method	Base	Novel	HM	Base	Novel	HM	Base	Novel	HM	Base	Novel	HM
CoOp [ 37 ]	82.69	63.22	71.66	76.47	67.88	71.92	98.00	89.81	93.73	93.67	95.29	94.47
CoCoOp [ 36 ]	80.47	71.69	75.83	75.98	70.43	73.10	97.96	93.81	95.84	95.20	97.69	96.43
KgCoOp [ 30 ]	80.73	73.60	77.00	75.83	69.96	72.78	97.72	94.39	96.03	94.65	97.76	96.18
MaPLe [ 13 ]	82.28	75.14	78.55	76.66	70.54	73.47	97.74	94.36	96.02	95.43	97.76	96.58
TCP [ 31 ]	84.13	75.36	79.50	77.27	69.87	73.38	98.23	94.67	96.42	94.67	97.20	95.92

TABLE II: Base-to-novel generalization results on 11 datasets. We particularly emphasize the Average results, as they offer a more comprehensive indicator of the balance between base-class fitting and novel-class generalization across diverse benchmarks. In particular, GAPL establishes a new state of the art in average HM, mainly driven by its improved performance on novel classes without sacrificing strong base-class accuracy. The best results are highlighted in bold .

Statistic Design	Source	Cross-domain
Statistic Design	ImageNet	Average
Diagonal only	72.90	61.12
Diagonal + Variance	72.90	61.10
Full Gram Matrix	OOM

TABLE III: Ablation on the statistical design of the Gram-Anchored Stream.

$\alpha$	-V2	-Sketch	-A	-R	Average
0.1	62.40	49.00	49.00	77.10	59.38
0.3	64.23	49.77	49.70	77.50	60.30
0.5	65.85	50.13	50.30	77.53	60.95
0.7 (Final)	66.43	50.13	50.43	77.50	61.12
0.9	66.43	49.90	50.23	77.27	60.96

TABLE IV: Sensitivity analysis of the fusion weight

\alpha

in the Global Invariant Stream.

$\mathcal{G}$	$\mathcal{S}$	$\mathcal{C}$	Source	Target (OOD Variants)				Average
			ImageNet	-V2	-Sketch	-A	-R
✓			70.80	63.00	48.90	47.10	76.06	58.77
✓	✓		70.60	63.70	49.20	50.25	76.10	59.81
✓		✓	72.90	65.53	49.40	48.10	77.45	60.12
✓	✓	✓	72.90	66.43	50.13	50.43	77.50	61.12

TABLE V: Ablation study of different stream combinations.

\mathcal{G}

: Global Invariant,

\mathcal{S}

: Gram-Anchored,

\mathcal{C}

: Contextual-Anchored.

Setting	Source	Target (OOD Variants)
	ImageNet	-V2	-Sketch	-A	-R
V0 (Ours)	72.90	66.43	50.13	50.43	77.50
V12 (Deep)	73.00	65.50	48.60	47.30	74.60

TABLE VI: Comparison of Visual Prompting Depth. V0 (Ours) results are synchronized with final config.

Fig. 4: Visual specificity comparison (query point on the dog’s ear).

Fig. 5: t-SNE visualization of latent manifolds across four domains for five selected classes. (a) CLS Token (first-order global) exhibits severe domain divergence with scattered distributions. (b) Contextual Text Anchor (first-order local) remains susceptible to stylistic noise despite local patch aggregation. (c) Style Text Anchor (ours, second-order) achieves superior alignment by collapsing domain-specific variance into compact, class-discriminative clusters.

Fig. 6: Quantitative analysis of domain alignment. We visualize the feature distribution of Photo (Teal) and Sketch (Purple) domains for “Goldfish” and “Banana”.

Prompt Learning for VLMs

See all 23 papers

MMLoP: Multi-Modal Low-Rank Prompting for Efficient Vision-Language Adaptation
Feb 24, 2026Sajjad Ghiasvand, Haniyeh Ehsani Oskouie, Mahnoosh Alizadeh +1Multimodal Foundation Models Prompt Learning
University of California, Santa Barbara · University of California, Los Angeles
U-shaped Multi-granularity Learning for Vision-Language Models
Jul 16, 2026Biao Chen, Yunqian Yu, Xiangxu Zhao +3VLM Adaptation Visual Prompt Learning
University of Electronic Science and Technology of China · Hithink Research, Hangzhou, China
Local-Global Prompt Learning via Sparse Optimal Transport
Mar 9, 2026Deniz Kizaroğlu, Ülku Tuncer Küçüktas, Emre Çakmakyurdu +1VLM Adaptation Visual Prompt Learning
Graduate School of Informatics, Middle East Technical University, Ankara, Turkey · Department of Electrical and Electronics Engineering, Gazi University, Ankara, Turkey

Feb 24, 2026J/K move · Enter open · S save

MMLoP: Multi-Modal Low-Rank Prompting for Efficient Vision-Language Adaptation

Sajjad Ghiasvand, Haniyeh Ehsani Oskouie, Mahnoosh Alizadeh, Ramtin Pedarsani

University of California, Santa Barbara · University of California, Los Angeles

Multimodal Foundation Models Prompt Learning VLM Adaptation Cross-Modal Learning Vision-Language Alignment Visual Prompt Learning+1

Open paper AskPDF arXiv

Prompt learning has become a dominant paradigm for adapting vision-language models (VLMs) such as CLIP to downstream tasks without modifying pretrained weights. While extending prompts to both vision and text encoders across multiple transformer layers significantly boosts performance, it dramatically increases the number of trainable parameters, with state-of-the-art methods requiring millions of parameters and abandoning the parameter efficiency that makes prompt tuning attractive. In this work, we propose MMLoP (Multi-Modal Low-Rank Prompting), a framework that achieves deep multi-modal prompting with only 11.5K trainable parameters, comparable to early text-only methods like CoOp. MMLoP parameterizes vision and text prompts at each transformer layer through a low-rank factorization that constrains prompts to a compact subspace, providing parameter efficiency while motivating the need for our complementary regularization components. To further close the accuracy gap with state-of-the-art methods, we introduce three complementary components: a self-regulating consistency loss that anchors prompted representations to frozen zero-shot CLIP features at both the feature and logit levels, a uniform drift correction that removes the global embedding shift induced by prompt tuning to preserve class-discriminative structure, and a shared up-projection that couples vision and text prompts through a common low-rank factor to enforce cross-modal alignment. Extensive experiments across three benchmarks and 11 diverse datasets demonstrate that MMLoP achieves a highly favorable accuracy-efficiency tradeoff, outperforming the majority of existing methods including those with orders of magnitude more parameters, while achieving a harmonic mean of 79.70% on base-to-novel generalization. Code is available at https://github.com/sajjad-ucsb/MMLoP.

Abstract

Prompt Learning for VLMs

Explore similar work

MMLoP: Multi-Modal Low-Rank Prompting for Efficient Vision-Language Adaptation

U-shaped Multi-granularity Learning for Vision-Language Models

Local-Global Prompt Learning via Sparse Optimal Transport