Decoupled Mode Connectivity for Base-to-Novel Generalization in Vision-Language Models

Imad Eddine Marouf; Khalid Oublal; Enzo Tartaglione; Stéphane Lathuilière

cs.CVNov 6, 2026

Decoupled Mode Connectivity for Base-to-Novel Generalization in Vision-Language Models

Authors: Imad Eddine Marouf, Khalid Oublal, Enzo Tartaglione, Stéphane Lathuilière

Organizations: Télécom Paris, Institut Polytechnique de Paris, Palaiseau, France · École Polytechnique, Institut Polytechnique de Paris, Palaiseau, France · CNR-ISASI, Lecce, Italy · Inria, Univ. Grenoble Alpes, CNRS, LJK, France

Abstract

Prompt learning adapts vision-language models such as CLIP by optimizing a short learnable prompt. The cross-entropy objective pulls this prompt toward base-class specialization, whereas generalization to unseen classes benefits from staying close to CLIP's zero-shot feature space. Because both objectives act on the same prompt, their gradients conflict at convergence, and in our experiments no single-prompt loss design clearly improves on the cosine baseline. We propose Decoupled Mode Connectivity (DMC), which gives each objective its own prompt. Starting from a base-trained prompt, DMC trains a specialization prompt and a generalization prompt, and merges their text features with a fixed coefficient at test time, at the inference cost of a single prompt. A linear mode connectivity (LMC) corridor in text-feature space keeps the classification loss low for every classifier interpolated between the two endpoints, and a Visual Anchor KL term preserves CLIP's zero-shot class-similarity distribution of each image during specialization. We further argue that such regularizers should satisfy class-permutation invariance (CPI), and show via Fano's inequality, in an idealized model, that violating CPI lower-bounds the drop in novel accuracy by a term proportional to the mutual information between the prompt and the base-class labels. Across 11 datasets, DMC improves the average harmonic mean of base and novel accuracy over MergeTune for both CoOp and KgCoOp, and its interpolation curves pass beyond the single-prompt operating point, shifting the base-novel frontier rather than trading along it. Code is available at https://github.com/IemProg/DMC.

Figures & tables

Figure 1: Decoupled Mode Connectivity on Flowers102 (joint CE loss landscape). The 2D plane in CLIP text-feature space is anchored at $f_{w_{1}}$ (zero-shot CLIP); the x-axis spans toward $f_{\hat{w}_{2}}$ (fine-tuned CoOp checkpoint) and the y-axis is the Gram–Schmidt orthogonal component of $f_{\mathrm{gen}}$ (text features of DMC’s converged generalization prompt) relative to $f_{w_{1}}$ . Each grid point is re-normalized per class before the joint base+novel CE loss is evaluated; construction details in Appendix G . Before DMC (left): the single-prompt baseline traces the linear path ; the inset shows the base–novel trade-off, with base and novel accuracies crossing over as $\alpha$ varies. During DMC (middle): the two prompts start collapsed at $f_{\hat{w}_{2}}$ (epoch 0) and separate during training. After DMC (right): the converged DMC corridor lies inside the low-loss valley, with the operating point at $\alpha=0.20$ marked by the star ; the inset confirms base and novel accuracies are both high and stable along the corridor.

Figure 2: Empirical interpolation curves on Caltech101, SUN397, UCF101, and EuroSAT (CoOp base, 3-seed mean). Each curve traces the family of classifiers from

f_{\mathrm{gen}}

(

\alpha{=}0

, near zero-shot) to

f_{\mathrm{spec}}

(

\alpha{=}1

, fully specialized). Solid: DMC (full method); dashed: DMC w/o VA . Gray dots mark the MergeTune baseline ; blue stars mark the operating point; the green arrow shows the shift from baseline to operating point. The boxed

\Delta\mathrm{HM}

value reports the gap (operating point vs. baseline), read off the seed-averaged curves; it can differ from Table 1 by up to

0.23

HM.

Method	Avg	HM per Dataset
Method	Avg	IN	Cal	Pets	Cars	Flo	Food	Air	SUN	DTD	Euro	UCF
CLIP (ICML 21)	71.70	70.22	95.40	94.12	68.65	74.83	90.66	31.09	72.23	56.37	60.03	73.85
CoOp (IJCV 22)	71.66	71.92	93.73	94.47	68.13	74.06	85.19	28.75	72.51	54.24	68.69	67.46
KgCoOp (CVPR 23 )	77.01	72.78	96.03	96.18	73.36	83.65	91.10	34.83	78.36	64.35	73.48	79.66
MaPLe (CVPR 23)	78.55	73.47	96.02	96.58	73.47	82.56	91.38	36.50	79.75	68.16	82.35	80.77
PromptSRC (ICCV 23)	79.97	74.01	96.02	96.30	76.58	85.95	91.10	40.15	80.52	71.75	82.32	82.74

Table 1: Base-to-novel HM on 11 datasets. MergeTune and DMC: mean over 3 seeds (

\alpha=0.20

). Bold: better of MergeTune and DMC within each base method. Avg: mean of the 11 per-dataset HMs for our runs; for published, TIES and DARE rows, HM of the average Base and Novel, as in Table 3 . † : uses LLM or teacher-model knowledge.

Method	CE	$\boldsymbol{\mathcal{R}}$	LMC	HM	$\Delta$	HM	$\Delta$	HM	$\Delta$	HM	$\Delta$	HM	$\Delta$
				Flowers102		Caltech101		StanfordCars		EuroSAT		Average
CoOp	✓			80.51 $\pm$ 1.54	—	94.42 $\pm$ 0.60	—	71.98 $\pm$ 0.70	—	71.18 $\pm$ 5.75	—	79.52	—
MergeTune w/o $\mathcal{R}$	✓		✓	78.19 $\pm$ 2.40	$-2.32$	94.43 $\pm$ 1.89	$+0.01$	71.04 $\pm$ 1.21	$-0.94$	73.39 $\pm$ 4.93	$+2.21$	79.26	$-0.26$
MergeTune	✓	✓	✓	82.86 $\pm$ 0.17	$+4.67$	96.03 $\pm$ 0.21	$+1.60$	73.60 $\pm$ 0.02	$+2.56$	73.45 $\pm$ 3.48	$+0.06$	81.48	$+2.22$
DMC w/o $\mathcal{R}$	✓		✓	78.66 $\pm$ 1.43	$-4.20$	94.60 $\pm$ 0.41	$-1.43$	70.23 $\pm$ 1.89	$-3.37$	68.79 $\pm$ 3.72	$-4.66$	78.07	$-3.42$
DMC w/o VA	✓	✓	✓	83.91 $\pm$ 0.46	$+5.25$	96.25 $\pm$ 0.04	$+1.65$	73.97 $\pm$ 0.44	$+3.74$	73.24 $\pm$ 0.96	$+4.45$	81.84	$+3.77$

Table 2: Component ablation (HM, 3 seeds,

\beta=4

for all rows; batch size 128, whereas Table 1 uses 32, so values differ from it).

\Delta

: change from the row above; bold: best per column. Rows without

\mathcal{R}

omit VA.

Figure 3: Ablation: learned

f_{\mathrm{gen}}

vs. fixed

f_{w_{1}}

. Both variants retrain

\mathbf{c}_{\mathrm{spec}}

with the same loss (CE + VA + LMC), differing only in the generalization endpoint. At small

\alpha

, including

\alpha=0.20

, the learned

f_{\mathrm{gen}}

(solid) gives a higher HM than the fixed

f_{w_{1}}

(dashed).

Figure 4: Gradient conflict at convergence.

\rho=\cos(\nabla_{\mathbf{c}}\mathcal{L}_{\mathrm{CE}},\,\nabla_{\mathbf{c}}\mathcal{R})

at the converged single-prompt checkpoint, averaged over the full training set.

\rho<0

on every dataset-baseline pair; Flowers102 shows the strongest conflict (

\rho\approx{-}0.95

for CoOp).

Figure 5: DMC decoupling dynamics during training . Both prompts are initialized from

\hat{w}_{2}

. Blue :

\cos(f_{\mathrm{gen}},f_{w_{1}})

rises sharply within 10 epochs and saturates near

0.97

–

0.99

, confirming rapid convergence toward zero-shot geometry. Orange :

\cos(f_{\mathrm{spec}},f_{w_{1}})

decreases overall under CE supervision, with oscillations from the LMC coupling. Final values range from

{\sim}0.45

(Flowers102, strongest separation) to

{\sim}0.65

(OxfordPets, weakest). Purple dashed :

\cos(f_{\mathrm{gen}},\,f_{\mathrm{spec}})

reflects the feature separation, decreasing from

{\sim}1.0

0.50

–

0.67

. The shaded region visualizes the decoupling gap. Single seed, with VA; Table 10 reports the 3-seed endpoint cosines (without VA; with-VA averages in its caption).

Appendix figures & tables29 assets

Supplementary material from the paper’s appendix.

Appendix

Table 3: Base, Novel and HM on 11 datasets. For the MergeTune and DMC rows we report the best of 3 seeds , so these numbers are higher than, and not directly comparable with, the 3-seed means of Table 1 , which are the headline results; prior-work rows are published numbers. Parentheses: average-HM change relative to the base method. † : Using large language model or teacher model’s knowledge.

	CoOp		KgCoOp		MMA
Dataset	+ MergeTune	+DMC	+ MergeTune	+DMC	+ MergeTune	+DMC
ImageNet	72.89 $\pm$ 0.18	72.90 $\pm$ 0.18	72.82 $\pm$ 0.10	73.09 $\pm$ 0.26	73.15 $\pm$ 0.10	72.90 $\pm$ 0.01
Caltech101	96.27 $\pm$ 0.26	96.41 $\pm$ 0.15	95.94 $\pm$ 0.03	96.11 $\pm$ 0.08	95.75 $\pm$ 0.16	95.74 $\pm$ 0.16
OxfordPets	96.13 $\pm$ 0.66	96.04 $\pm$ 0.29	96.28 $\pm$ 0.10	96.23 $\pm$ 0.35	93.22 $\pm$ 0.11	93.38 $\pm$ 0.01
StanfordCars	73.38 $\pm$ 0.37	75.21 $\pm$ 0.10	74.90 $\pm$ 0.29	75.38 $\pm$ 0.24	70.16 $\pm$ 0.24	70.51 $\pm$ 0.04
Flowers102	83.92 $\pm$ 0.57	84.47 $\pm$ 0.58	83.39 $\pm$ 0.27	83.67 $\pm$ 0.33	75.22 $\pm$ 0.40	76.37 $\pm$ 0.23

Appendix

Table 4: Seed variability for every row we trained ourselves: HM mean

\pm

sample standard deviation over 3 seeds (DMC at

\alpha=0.20

; MergeTune at its final prompt). Per-cell companion to Table 1 , which reports the same runs as means. Cells with

\sigma_{\mathrm{HM}}>1

are bold and flagged as seed-fragile.

Figure 6: Flowers102 and OxfordPets. Flowers102 shows a shift toward higher base accuracy at nearly unchanged novel accuracy (

+0.67

HM). OxfordPets has a near-ceiling baseline; DMC is within

0.1

HM of it (

+0.09

on the plotted curves,

-0.09

in Table 4 ).

Figure 7: StanfordCars and FGVCAircraft. Both fine-grained datasets show clear operating-point shifts:

+1.96

HM on StanfordCars and

+1.89

on FGVCAircraft.

Figure 8: Food101 and DTD. Food101 has a flat frontier; DMC shifts the operating point slightly (

+0.08

HM). DTD is the only dataset in these figures where DMC is slightly below the baseline (

-0.77

HM).

Figure 9: EuroSAT and UCF101. EuroSAT shows the largest improvement among these datasets (

+2.43

HM). UCF101 achieves

+1.80

HM. (Both datasets are also shown in the main paper, Figure 2 .)

Figure 10: ImageNet — CoOp+DMC, KgCoOp+DMC, MMA+DMC. Interpolation curves on ImageNet (seed mean of the 21-point

\alpha

sweep). The blue star marks the operating point at the fixed

\alpha=0.20

(the same value used for all held-out datasets, selected on Flowers102+Caltech101). At the operating point, DMC yields +0.01 HM on CoOp+DMC (

72.89\to 72.90

), +0.27 HM on KgCoOp+DMC (

72.82\to 73.09

), and

-0.25

HM on MMA+DMC (

73.15\to 72.90

; the MMA panel shows an earlier single run, whose boxed value differs) over MergeTune. The operating-point shift on ImageNet is small relative to the specialized datasets (Figure 2 ); this is consistent with ImageNet’s high-diversity 1000-class set, where the single-prompt envelope already covers most of the achievable operating points and decoupling has less headroom than on specialized tasks like EuroSAT or StanfordCars. Unlike the other interpolation-curve figures in this appendix, only the DMC (with VA) curve is shown: a DMC-without-VA ablation was not run on ImageNet.

CoOp baseline
Dataset	$\boldsymbol{\rho}$	DMC gain
Flowers102	$-0.955$	$+0.23$
Caltech101	$-0.716$	$+0.36$
Food101	$-0.776$	$+0.08$
OxfordPets	$-0.211$	$+0.22$
KgCoOp baseline

Appendix

Table 5: Gradient conflict

\rho=\cos(\nabla_{\mathbf{c}}\mathcal{L}_{\mathrm{CE}},\,\nabla_{\mathbf{c}}\mathcal{R})

at convergence, grouped by baseline.

\rho<0

on all eight dataset–baseline pairs. The DMC gain is the HM gain over MergeTune in the runs where

\rho

was measured; it can differ from the seed means of Table 1 .

	Caltech101				Flowers102
Modification	Base	Novel	HM	$\Delta$	Base	Novel	HM	$\Delta$
Cosine score, $\lambda=8$ ( MergeTune )	97.93	94.21	96.03	—	94.21	75.46	83.80	—
Fisher-weighted cosine	97.42	95.09	96.24	$+0.21$	94.11	75.89	84.02	$+0.22$
Prompt-space EWC	97.87	94.32	96.06	$+0.03$	94.59	75.25	83.82	$+0.02$
KL path	97.87	94.10	95.95	$-0.08$	94.02	75.60	83.81	$+0.01$
ZSDD	97.48	95.41	96.43	$+0.40$	87.46	76.17	81.43	$-2.37$

Appendix

Table 6: Single-prompt loss modifications on Caltech101 and Flowers102. For each modification we report the best HM across the hyperparameter sweep;

\Delta

HM is the gap to the cosine-score baseline at

\lambda=8

. No modification improves on the baseline by clearly more than retuning

\lambda

does, and modifications that gain on one dataset typically regress on the other.

Method	Flowers102	Caltech101	StanfordCars	EuroSAT
CoOp ( $n_{\text{ctx}}=4$ , from Table 1 )	74.06	93.73	68.13	68.69
CoOp ( $n_{\text{ctx}}=8$ )	77.84 $\pm$ 0.36	92.19 $\pm$ 3.20	68.35 $\pm$ 1.23	69.95 $\pm$ 4.91
CoOp ( $n_{\text{ctx}}=16$ )	79.40 $\pm$ 1.50	95.34 $\pm$ 0.64	70.21 $\pm$ 1.05	72.29 $\pm$ 9.52
CoOp + DMC (Ours)	84.47 $\pm$ 0.58	96.41 $\pm$ 0.15	75.21 $\pm$ 0.10	75.75 $\pm$ 3.65

Appendix

Table 7: Extra single-prompt capacity does not match CoOp+DMC. Base-to-novel harmonic mean (HM) on four datasets. We train CoOp with enlarged context length

n_{\text{ctx}}\in\{8,16\}

using the same protocol as the standard

n_{\text{ctx}}=4

CoOp;

n_{\text{ctx}}=16

gives a single prompt with

16\times 512=8192

tunable parameters, strictly larger than the

2\times 4\times 512=4096

parameters of CoOp+DMC. Values are mean

\pm

sample standard deviation over 3 seeds; the

n_{\text{ctx}}=4

row is the single number reported in the main paper. CoOp+DMC still leads on every dataset; on EuroSAT the margin is within the seed spread.

	Flowers102		Caltech101
Configuration	HM	$\Delta$	HM	$\Delta$
No VA	83.91 $\pm$ 0.46	—	96.25 $\pm$ 0.04	—
Single-view VA	84.23 $\pm$ 0.16	$+0.32$	96.26 $\pm$ 0.25	$+0.01$
Two-view VA (default)	84.17 $\pm$ 0.84	$-0.06$	96.41 $\pm$ 0.15	$+0.15$

Appendix

Table 8: Decomposing the Visual Anchor (VA) benefit into KL anchoring vs. augmentation consistency. No VA disables the term entirely. Single-view VA : the student branch sees the raw image and the teacher branch one train-transform view; this is the closest variant to the single-view case of Proposition 2 , which it matches exactly only without augmentation (preset none in Table 9 ). Two-view VA is the default used in all reported experiments: both branches see independent train-transform views.

\Delta

is the HM change relative to the row above. All cells at

\alpha=0.20

\beta=4.0

, 3 seeds. Trainer: CoOp + DMC. No VA is from the component-ablation runs (batch size 128); the VA rows use batch size 32, so the first

\Delta

also includes the batch-size change.

	Flowers102			Caltech101
Preset	Base	Novel	HM	Base	Novel	HM
none	96.65 $\pm$ 0.64	73.88 $\pm$ 0.74	83.74 $\pm$ 0.70	97.89 $\pm$ 0.32	94.36 $\pm$ 0.33	96.09 $\pm$ 0.30
weak	96.68 $\pm$ 0.17	75.44 $\pm$ 0.27	84.75 $\pm$ 0.14	97.96 $\pm$ 0.38	94.58 $\pm$ 0.71	96.24 $\pm$ 0.24
default	97.02 $\pm$ 0.79	74.32 $\pm$ 1.10	84.17 $\pm$ 0.84	98.13 $\pm$ 0.30	94.76 $\pm$ 0.55	96.41 $\pm$ 0.15
strong	97.12 $\pm$ 0.24	74.73 $\pm$ 0.84	84.46 $\pm$ 0.53	98.23 $\pm$ 0.20	94.58 $\pm$ 0.28	96.37 $\pm$ 0.13

Appendix

Table 9: Sensitivity of CoOp + DMC to the visual-anchor augmentation preset. Presets are: none (raw image, no augmentation), weak (mild crop and jitter, no grayscale), default (the paper setting: RandomResizedCrop

0.7

–

1.0

, HorizontalFlip

0.5

, ColorJitter

0.4/0.4/0.2/0.1

, RandomGrayscale

0.1

), and strong (aggressive crop

0.2

–

1.0

, strong jitter, RandomGrayscale

0.2

). All cells are evaluated at

\alpha=0.20

and reported as mean

\pm

sample standard deviation over 3 seeds. Trainer: CoOp + DMC.

Figure 11: HM landscape on Flowers102. Same 2D plane and overlays as the main-paper Figure 1 , with the heatmap recolored by harmonic mean of base and novel accuracy (high HM in blue). The DMC corridor lies on the high-HM peak; the operating point at $\alpha=0.20$ sits near the corridor’s HM maximum $\approx 84$ . The single-prompt linear path along the bottom of the plane traverses an HM ridge that peaks below $\approx 80$ at every mixing weight, consistent with the single-prompt ceiling of Lemma 1 along this path.

Figure 12: HM landscape on Caltech101. Caltech101 is a near-ceiling task (zero-shot CLIP attains HM $\approx 95$ ), so the landscape varies over a much narrower range, $[92.8,96.9]$ , than Flowers102 ( $[64.9,84.6]$ ). Despite the compressed dynamic range, the same geometry is visible: the DMC corridor lies on a high-HM ridge offset from the linear single-prompt path, and the operating point at $\alpha=0.20$ sits near the ridge maximum. Flat-but-shifted performance is the expected near-ceiling behavior of decoupling: when single-prompt HM is already close to the achievable upper bound, DMC’s gain shrinks (CoOp+DMC gains $+0.14$ HM over MergeTune on Caltech101, Table 1 ).

Dataset	CoOp + DMC	KgCoOp + DMC	MMA + DMC
Caltech101	0.608 $\pm$ 0.065	0.919 $\pm$ 0.034	1.000 $\pm$ 0.000
OxfordPets	0.785 $\pm$ 0.054	0.971 $\pm$ 0.022	1.000 $\pm$ 0.000
StanfordCars	0.673 $\pm$ 0.038	0.701 $\pm$ 0.009	0.999 $\pm$ 0.000
Flowers102	0.542 $\pm$ 0.030	0.606 $\pm$ 0.027	0.997 $\pm$ 0.002
Food101	0.741 $\pm$ 0.007	0.987 $\pm$ 0.014	0.999 $\pm$ 0.000
FGVCAircraft	0.514 $\pm$ 0.058	0.521 $\pm$ 0.005	1.000 $\pm$ 0.000

Appendix

Table 10: Endpoint-collapse diagnostic for DMC. We report

\cos(f_{\text{gen}},f_{\text{spec}})

, the cosine similarity between the two DMC endpoints (class text features, averaged over classes) at the evaluated best-validation checkpoint, as 3-seed mean

\pm

sample std. Runs:

\lambda_{\text{gen}}=8

\beta=4

, without VA; with VA (the runs of Table 1 ) the averages are

0.628

0.704

and

0.996

, with SUN397 again the only MMA dataset below

0.99

(

0.964

). Values close to

1.0

mean the two endpoints have collapsed to the same solution, so the linear connector degenerates and the

\alpha

-sweep is flat. CoOp and KgCoOp endpoints remain distinct on every dataset (maximum

0.987

). MMA endpoints exceed

0.99

on 9 of 10 datasets (all but SUN397, at

0.940

); this collapse delimits DMC’s scope: on MMA it cannot close the gap to published MMA (Section 4 ).

Dataset	$\cos(f_{\text{gen}},f_{\text{spec}})$	MMA + MergeTune HM	MMA + DMC HM	Gain
Caltech101	1.000	95.75	95.74	$-0.01$
OxfordPets	1.000	93.22	93.38	$+0.16$
StanfordCars	0.999	70.16	70.51	$+0.35$
Flowers102	0.997	75.22	76.37	$+1.15$
Food101	0.999	90.18	90.33	$+0.15$
FGVCAircraft	1.000	32.72	33.10	$+0.38$

Appendix

Table 11: Per-dataset gap decomposition for MMA + DMC: endpoint cosine

\cos(f_{\text{gen}},f_{\text{spec}})

(Table 10 ) and HM of MMA + MergeTune and MMA + DMC (mean over 3 seeds, Table 1 ). ImageNet has no entry in Table 10 and is omitted; the average is over the 10 datasets shown (the 11-dataset averages are

72.58

and

73.16

). The gain does not track the residual separation (Pearson

r=0.14

between

\cos

and gain): the largest gains occur on DTD, UCF101 and Flowers102, where

\cos\geq 0.997

Config A: enlarge generalization-adapter scale
	Flowers102		DTD
Method	HM	$\cos$	HM	$\cos$
MMA-alone (published)	85.48	—	73.38	—
MMA + MergeTune	75.22	—	58.07	—
MMA + DMC (canonical)	76.37 $\pm$ 0.23	0.997 $\pm$ 0.002	59.62 $\pm$ 0.13	1.000 $\pm$ 0.000
scale $=10^{-2}$	79.18 $\pm$ 1.45	0.951 $\pm$ 0.004	59.92 $\pm$ 1.50	0.980 $\pm$ 0.002

Appendix

Table 12: MMA adapter-capacity ablation for DMC. We test two interventions against the canonical MMA+DMC endpoint collapse (

\cos(f_{\text{gen}},f_{\text{spec}})\approx 1.00

): Config A enlarges the generalization-adapter scale from

10^{-3}

10^{-2}

10^{-1}

; Config B re-initializes the generalization adapter with a fresh Kaiming draw instead of copying the base adapter. HM is base-to-novel harmonic mean at

\alpha=0.20

;

\cos

\cos(f_{\text{gen}},f_{\text{spec}})

at end of training; both reported as mean

\pm

sample std over 3 seeds.

\cos\approx 1

marks endpoint collapse. MMA-alone is the published number; MMA + MergeTune is the mean over 3 seeds from Table 1 . A scale of

10^{-2}

partly separates the endpoints (

\cos\approx 0.95

) and raises Flowers102 HM by

2.81

; a scale of

10^{-1}

separates them further (

\cos\approx 0.78

) but the enlarged endpoint overfits the base classes and HM drops. Fresh initialization does not break the collapse. No intervention closes the gap to MMA-alone.

Method	Flowers102	Caltech101	StanfordCars	EuroSAT
MMA+ MergeTune w/o $\mathcal{R}$	76.63 $\pm$ 0.64	95.75 $\pm$ 0.16	70.71 $\pm$ 0.16	61.32 $\pm$ 0.28
MMA+ MergeTune	75.22 $\pm$ 0.40	95.75 $\pm$ 0.16	70.16 $\pm$ 0.24	59.57 $\pm$ 0.48
MMA+DMC w/o $\mathcal{R}$	78.73 $\pm$ 0.46	95.77 $\pm$ 0.15	71.62 $\pm$ 0.25	63.78 $\pm$ 0.34
MMA+DMC w/o VA	76.38 $\pm$ 0.13	95.73 $\pm$ 0.17	70.57 $\pm$ 0.13	60.53 $\pm$ 0.49
MMA+DMC (full)	76.37 $\pm$ 0.23	95.74 $\pm$ 0.16	70.51 $\pm$ 0.04	60.57 $\pm$ 0.45
MMA (published)	85.48	96.15	75.70	83.87

Appendix

Table 13: Component ablation with MMA as the base method (HM, mean

\pm

sample std over 3 seeds; DMC rows at

\alpha=0.20

). Same components as Table 2 ; the two MergeTune rows use

\beta=1

, the DMC rows

\beta=4

. Bottom row: published MMA (Table 1 ).

Dataset	Rank	LoRA	LoRA+DMC	$\Delta$	$\cos$
Flowers102	8	73.03 $\pm$ 0.03	84.37 $\pm$ 0.33	$+11.34$	0.454 $\pm$ 0.010
Flowers102	16	73.18 $\pm$ 0.11	83.51 $\pm$ 0.41	$+10.33$	0.454 $\pm$ 0.015
Caltech101	8	95.69 $\pm$ 0.00	96.01 $\pm$ 0.48	$+0.32$	0.861 $\pm$ 0.146
Caltech101	16	95.68 $\pm$ 0.05	96.06 $\pm$ 0.30	$+0.38$	0.908 $\pm$ 0.092

Appendix

Table 14: LoRA on the text encoder (

Q

K

V

, all 12 layers) with and without DMC (

\lambda_{\text{gen}}=8

\beta=4

, no VA, 50 epochs). HM mean

\pm

sample std over 3 seeds; DMC at

\alpha=0.20

;

\cos=\cos(f_{\mathrm{gen}},f_{\mathrm{spec}})

. The LoRA baseline is under-tuned (Flowers102 base accuracy

69.5

Regularizer	CPI	$\boldsymbol{\delta_{\max}}$	$\boldsymbol{\delta_{\mathrm{mean}}}$	$\Delta$ HM (Cal / Flo)
Cosine score [ Wang et al., 2026 ]	✓	$1.2\times 10^{-7}$	$4\times 10^{-8}$	baseline
Visual Anchor (ours)	✓	$6\times 10^{-8}$	$2\times 10^{-8}$	$-0.04$ / $+0.36$
Drift-Penalty (ours) ‡	✓	$1.2\times 10^{-7}$	$4\times 10^{-8}$	—
ZSDD	✓	$2.4\times 10^{-7}$	$7\times 10^{-8}$	$+0.40$ / $-2.37$
Fisher-weighted cosine	$\times$	$1.4\times 10^{-2}$	$3.3\times 10^{-3}$	$+0.21$ / $+0.22$
EWC (feature space)	$\times$	$1.07$	$0.297$	—

Appendix

Table 15: Class-permutation invariance (CPI) over

10{,}000

random permutations on Caltech101 (

C=50

\Delta

HM: change of the best single-prompt configuration over the cosine baseline on Caltech101 / Flowers102 (Table 6 ); —, symmetry test only, not trained. ‡ Drift-Penalty is an

\alpha

-selection criterion (Appendix K ), not a training regularizer.

Criterion	Flowers102	Caltech101	StanfordCars	EuroSAT	Avg
Val-Acc	$-3.50$	$-0.95$	—	—	—
Proxy-HM	$-5.29$	$-1.05$	—	—	—
Gram-Align	$-2.29$	$-0.24$	—	—	—
EffRank-Max	$-9.26$	$-1.54$	—	—	—
NP $\geq$ 0.90	$-2.29$	$-0.28$	—	—	—
Drift-Pen ( $\mu{=}4$ )	$-0.25$	$-0.30$	$-0.18$	$-0.14$	$-0.22$

Appendix

Table 16:

\alpha

selection comparison (mean

\Delta

to oracle HM, 3 seeds). Fixed

\alpha=0.20

achieves the best average gap. —: not evaluated (large gap already on the development datasets).

Dataset	CoOp+DMC				KgCoOp+DMC				MMA+DMC
	$\alpha^{\star}$	HM ⋆	HM .20	$\Delta$	$\alpha^{\star}$	HM ⋆	HM .20	$\Delta$	$\alpha^{\star}$	HM ⋆	HM .20	$\Delta$
Caltech101	0.15	96.52	96.41	+0.11	0.00	96.26	96.11	+0.15	0.00	95.76	95.74	+0.02
DTD	0.20	63.54	63.54	+0.00	0.25	64.51	64.37	+0.14	0.65	59.68	59.62	+0.06
EuroSAT	0.10	75.98	75.84	+0.14	0.00	74.12	73.39	+0.73	0.95	60.77	60.57	+0.20
FGVCAircraft	0.20	35.24	35.24	+0.00	0.05	33.33	33.22	+0.11	1.00	33.21	33.10	+0.10
Food101	0.45	91.22	91.13	+0.09	0.45	91.20	91.19	+0.01	0.15	90.33	90.33	+0.00

Appendix

Table 17: Per-dataset oracle-best

\alpha^{\star}

for DMC across three baselines. HM is averaged over 3 seeds.

\Delta

\mathrm{HM}(\alpha^{\star})-\mathrm{HM}(0.20)

. Bottom rows: median

\alpha^{\star}

across the 11 datasets and range

[\min,\max]

. For CoOp and KgCoOp the median oracle

\alpha^{\star}

lies one grid step from the fixed

\alpha=0.20

; for MMA the corridor is degenerate (Appendix H ), so

\alpha^{\star}

is poorly determined.

Statistic over 55 dev pairs	Value
Pairs that pick $\alpha^{\star}\in[0.15,0.25]$	47/55 (85%)
Pairs that pick $\alpha^{\star}\leq 0.10$ or $\geq 0.35$	2/55 (4%)
Pairs with held-out mean gap $\leq 0.30$ HM	47/55 (85%)
Pairs with held-out mean gap $\leq 0.50$ HM	54/55 (98%)
Mean held-out gap across 55 pairs	0.17 HM
Median held-out gap across 55 pairs	0.15 HM

Appendix

Table 18: Dev-set reshuffle robustness for CoOp+DMC. For each of the

\binom{11}{2}=55

ways to pick a 2-dataset dev subset, we choose the fixed

\alpha

that minimizes the mean gap-to-oracle on the dev pair, then measure the mean gap-to-oracle on the remaining 9 held-out datasets. The paper’s dev pair is Flowers102+Caltech101. Gaps are to the oracle of the seed-averaged

\alpha

-curve (Table 17 ).

	Flowers102		Caltech101
Criterion	Mean $\Delta$	Worst $\Delta$	Mean $\Delta$	Worst $\Delta$
Val-Acc	$-7.19$	$-12.69$	$-0.29$	$-0.48$
Proxy-HM	$-7.19$	$-12.69$	$-0.29$	$-0.48$
Gram-Align	$-1.47$	$-1.91$	$-0.24$	$-0.45$
ValAcc $\times$ Gram	$-0.78$	$-1.22$	—	—
Drift-Pen ( $\mu{=}1$ )	$-0.78$	$-1.22$	$-0.16$	$-0.48$

Appendix

Table 19: Drift-Penalty

\mu

sweep on the development set (Flowers102, Caltech101; 3 seeds,

\lambda_{\text{gen}}=8.0

\beta=4.0

). Mean and worst-case HM gap to oracle.

$\alpha$	Pets	Food	Cars	Aircraft	SUN	DTD	EuroSAT	UCF	Avg
0.10	$-0.20$	$-0.30$	$-0.30$	$-0.66$	$-0.26$	$-1.33$	$-0.14$	$-0.33$	$-0.44$
0.15	$-0.19$	$-0.22$	$-0.20$	$-0.42$	$-0.18$	$-0.46$	$-0.18$	$-0.06$	$-0.24$
0.20	$\boldsymbol{-0.17}$	$\boldsymbol{-0.13}$	$\boldsymbol{-0.14}$	$\boldsymbol{-0.22}$	$\boldsymbol{-0.07}$	$\boldsymbol{-0.23}$	$\boldsymbol{-0.28}$	$\boldsymbol{-0.21}$	$\boldsymbol{-0.18}$
0.25	$-0.12$	$-0.09$	$-0.01$	$-0.44$	$-0.01$	$-0.43$	$-0.53$	$-0.41$	$-0.26$
0.30	$-0.15$	$-0.08$	$-0.15$	$-0.93$	$-0.02$	$-0.54$	$-0.77$	$-0.86$	$-0.44$

Appendix

Table 20: Generalization of fixed

\alpha=0.20

(selected on dev set) to 8 held-out datasets. Mean

\Delta

to oracle HM.

Figure 13: Pareto fronts for

\alpha

selection on four datasets. Stars mark Pareto knee points. As

\alpha

increases, features move right (better base fit) and down (more drift from

f_{w_{1}}

Figure 14: HM, Base accuracy, and Novel accuracy vs

\alpha

on four datasets. Each row is one dataset; columns show HM, Base, and Novel accuracy. The dashed red line marks

\alpha=0.20

. Base accuracy increases monotonically with

\alpha

(the specialization prompt dominates), while Novel accuracy decreases (generalization degrades). The HM peak reflects the trade-off between these opposing trends and lies in

\alpha\in[0.10,0.35]

across all datasets.

	Flowers102				Caltech101
$\beta$	S1	S2	S3	Mean $\pm$ Std	S1	S2	S3	Mean $\pm$ Std
1.0	84.66	82.84	85.82	84.44 $\pm$ 1.50	96.15	96.37	95.76	96.09 $\pm$ 0.31
2.0	84.38	83.06	85.63	84.36 $\pm$ 1.29	96.20	96.39	95.76	96.12 $\pm$ 0.32
4.0	84.26	83.73	84.45	84.15 $\pm$ 0.37	96.27	96.42	96.55	96.41 $\pm$ 0.14
Baseline	84.06	84.12	84.53	84.24 $\pm$ 0.25	96.00	96.24	95.92	96.05 $\pm$ 0.17

Appendix

Table 21:

\beta

(LMC path weight) sweep (

\lambda_{\text{gen}}=8.0

, 3 seeds, DMC without VA). Per-seed HM at each run’s oracle

\alpha^{\star}

; mean

\pm

sample standard deviation. Baseline : single-prompt MergeTune objective (

\lambda=10

\beta=1

), same seeds. Higher

\beta

stabilizes performance; the selected

\beta=4.0

achieves the lowest cross-seed variance among the DMC rows on both datasets.

	Flowers102				Caltech101
$\lambda_{\text{gen}}$	$\alpha^{*}$	Base	Novel	HM	$\alpha^{*}$	Base	Novel	HM
0.0	0.15	97.72	65.32	78.30	1.00	98.13	92.90	95.44
1.0	0.10	97.25	70.99	82.07	0.40	97.42	93.45	95.39
2.0	0.25	97.15	73.12	83.44	0.40	97.68	93.56	95.58
4.0	0.30	96.77	74.96	84.48	0.25	97.87	93.67	95.72
8.0	0.30	95.73	75.89	84.66	0.15	97.93	94.43	96.15

Appendix

Table 22: Effect of

\lambda_{\text{gen}}

(cosine regularizer weight) in DMC training (seed 1,

\beta=1.0

; HM at the best

\alpha^{*}

of each run). At

\lambda_{\text{gen}}=0

the anchor is removed and decoupling collapses. Selected configuration is highlighted; the bottom row reports the single-prompt baseline.

VLM Adaptation

Vision-language model adaptation tailors pretrained image-text and multimodal models to new tasks and domains while retaining transferable capabilities.

See all 551 papers

Sep 24, 2026

ProCAP: Probabilistic Cross-Attentive Prompt Learning for Vision-Language Models

Pre-trained vision-language models such as CLIP can recognize new categories via prompting, but they often struggle when labeled data are scarce or the test distribution shifts. Prompt learning adapts only a small set of parameters while keeping the backbone frozen, yet many existing multimodal prompt learners couple the visual and textual branches weakly and can be brittle in low-shot regimes. We propose ProCAP, a probabilistic cross-attentive prompt learning framework that improves cross-modal interaction and training stability without updating any CLIP weights: it learns both visual and textual prompt tokens and links them through stacked bidirectional multi-head cross-attention so the two branches refine each other across prompt depth. To reduce overfitting under limited supervision, we parameterize prompt tokens with Gaussian means and variances and regularize them with lightweight KL and L2 penalties, and we further add a compact symmetric InfoNCE head that aligns cross-attended image features with class-level text representations in a shared low-dimensional space. Across few-shot base-to-novel generalization on 11 datasets, cross-dataset transfer, and domain generalization on ImageNet shift benchmarks, ProCAP achieves strong aggregate base-to-novel performance and competitive transfer performance while keeping the CLIP backbone unchanged.

Hiwa Azeez Abbas, Fatemeh Daneshfar, Moloud Abdar

University of Kurdistan, Sanandaj, Iran · Department of Computer Engineering, University of Kurdistan, Sanandaj, Iran · CHIRP, Child Health Research Centre, The University of Queensland, Brisbane, Australia

VLM Adaptation Visual Prompt Learning Cross-Modal Learning Few-Shot Learning Domain Generalization

Ask PDF

Feb 24, 2026

MMLoP: Multi-Modal Low-Rank Prompting for Efficient Vision-Language Adaptation

Prompt learning has become a dominant paradigm for adapting vision-language models (VLMs) such as CLIP to downstream tasks without modifying pretrained weights. While extending prompts to both vision and text encoders across multiple transformer layers significantly boosts performance, it dramatically increases the number of trainable parameters, with state-of-the-art methods requiring millions of parameters and abandoning the parameter efficiency that makes prompt tuning attractive. In this work, we propose MMLoP (Multi-Modal Low-Rank Prompting), a framework that achieves deep multi-modal prompting with only 11.5K trainable parameters, comparable to early text-only methods like CoOp. MMLoP parameterizes vision and text prompts at each transformer layer through a low-rank factorization that constrains prompts to a compact subspace, providing parameter efficiency while motivating the need for our complementary regularization components. To further close the accuracy gap with state-of-the-art methods, we introduce three complementary components: a self-regulating consistency loss that anchors prompted representations to frozen zero-shot CLIP features at both the feature and logit levels, a uniform drift correction that removes the global embedding shift induced by prompt tuning to preserve class-discriminative structure, and a shared up-projection that couples vision and text prompts through a common low-rank factor to enforce cross-modal alignment. Extensive experiments across three benchmarks and 11 diverse datasets demonstrate that MMLoP achieves a highly favorable accuracy-efficiency tradeoff, outperforming the majority of existing methods including those with orders of magnitude more parameters, while achieving a harmonic mean of 79.70% on base-to-novel generalization. Code is available at https://github.com/sajjad-ucsb/MMLoP.

Sajjad Ghiasvand, Haniyeh Ehsani Oskouie, Mahnoosh Alizadeh +1

University of California, Santa Barbara · University of California, Los Angeles

Multimodal Foundation Models Prompt Learning VLM Adaptation Cross-Modal Learning Vision-Language Alignment Visual Prompt Learning+1

Ask PDF

Sep 20, 2026

PETR: Prompt Ensembling with Training-free Routing for Vision-Language Models

Prompt learning efficiently adapts vision-language models (VLMs) to downstream tasks, but gains on seen classes often come at the expense of generalization to unseen classes. To address this limitation, we propose prompt ensembling with training-free routing (PETR), whose key innovation is a carefully designed dual-prompt architecture: two complementary prompts are learned from different data and objectives to emphasize seen class discrimination and unseen-class generalization, respectively. During training, both prompts are fine-tuned using a shared frozen CLIP backbone, and statistical information is collected from the training set logits. At inference time, we determine the similarity of each test sample to seen data, and route the sample to the most appropriate prompt branch. To the best of our knowledge, this is the first prompt tuning framework that performs training-free adaptive routing based on statistical similarity. This design provides an interpretable routing signal and avoids common MoE-style routing pathologies, such as router training instability and load imbalance. Extensive experiments on 11 benchmark datasets demonstrate that our framework consistently outperforms previous methods on both seen and unseen classes, achieving new state-of-the-art results.

Weihan Cai, Hao Tan, Xinping Gao +2

State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, and also with the School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · Purple Mountain Laboratories, Nanjing, China · Beijing University of Posts and Telecommunications, Beijing, China

Zero-Shot Learning VLM Adaptation Visual Prompt Learning Prompt Learning for VLMs

Ask PDF

Abstract

VLM Adaptation

Explore similar work

ProCAP: Probabilistic Cross-Attentive Prompt Learning for Vision-Language Models

MMLoP: Multi-Modal Low-Rank Prompting for Efficient Vision-Language Adaptation

PETR: Prompt Ensembling with Training-free Routing for Vision-Language Models