PPOM: Marginalizing Patch-Grid Phase for CLIP-Based Generalizable Vision-Language Prompt Tuning

Liang Wang; Haoyang Li; Chao Wang; Guodong Long; Jing Jiang; Yan Peng

cs.CVAug 14, 2026

PPOM: Marginalizing Patch-Grid Phase for CLIP-Based Generalizable Vision-Language Prompt Tuning

Authors: Liang Wang, Haoyang Li, Chao Wang, Guodong Long, Jing Jiang, Yan Peng

Organizations: Shanghai University, Shanghai, China · University of Technology Sydney, Sydney, Australia

Abstract

Prompt tuning adapts CLIP-based vision-language models with few trainable parameters, yet its predictions remain sensitive to the spatial sampling imposed by a frozen vision transformer. In particular, non-overlapping patch tokenization makes predictions depend on the alignment (phase) between image and the patch lattice. To reduce prediction sensitivity to patch-grid alignment, we introduce Patch-Phase Orbit Marginalization (PPOM), a training-free inference operator that treats phase shift as a nuisance variable. Given a patch stride, PPOM evaluates the identity view and reflection-padded translations, pairs opposite shifts into horizontal, vertical, and diagonal antithetic families, and assigns equal mass to these families and the identity prediction to avoid view-count bias during phase integration. In summary, PPOM provides a deterministic interface between prompt adaptation and patch-grid sensitivity. Across multiple prompt-learning hosts, PPOM improves host performance without re-training.

Figures & tables

Figure 1: Impact of horizontal patch phase shift on the test set of prompt learner: (a) Host predictions show substantial phase disagreement rate, while (b) PPOM significantly reduces disagreement and improves harmonic mean (HM) accuracy.

Figure 2: Framework of PPOM. Architecture-derived Phase Orbit (Sec. 2.2 ) constructs stride-defined patch phases via reflection padding and half-patch translations $\delta$ , evaluated by the unchanged prompt-tuned host. Family-balanced Orbit Marginalization (Sec. 2.3 ) then assigns equal mass to identity, horizontal, vertical, and diagonal families for final prediction $\hat{y}$ .

	Average (11)			ImageNet			Caltech101			Oxford Pets			Stanford Cars			Flowers102
Method	Base	New	HM	Base	New	HM	Base	New	HM	Base	New	HM	Base	New	HM	Base	New	HM
CoOp	80.91	70.01	75.07	75.96	68.93	72.27	97.81	94.65	96.20	95.11	97.04	96.06	68.97	69.67	69.32	96.77	71.42	82.18
+PPOM	81.50	70.59	75.65	76.93	69.49	73.02	98.32	94.76	96.51	95.48	97.26	96.36	70.61	70.46	70.54	96.58	72.13	82.58
KgCoOp	80.32	72.16	76.02	76.18	70.26	73.10	98.06	95.63	96.83	95.06	97.93	96.47	71.84	74.77	73.28	95.92	71.42	81.87
+PPOM	80.87	72.54	76.48	77.08	70.78	73.80	98.39	96.07	97.21	95.37	98.15	96.74	73.44	75.61	74.51	96.49	72.13	82.55
PromptSRC	81.25	73.66	77.27	77.00	70.60	73.66	97.93	94.54	96.21	95.32	97.43	96.36	71.51	75.51	73.46	96.87	73.83	83.79

Table 1: Base-to-new generalization performance (%) of 4 host models w/ or w/o our PPOM on 11 datasets.

	Source	Target
Method	ImageNet	Average	Caltech	Pets	Cars	Flowers	Food	Aircraft	SUN	DTD	EuroSAT	UCF
CoOp	71.64	64.91	93.71	90.08	64.47	69.47	85.94	20.28	66.24	42.26	49.30	67.30
+PPOM	72.14	65.32	93.67	90.24	65.43	69.22	86.51	21.00	66.67	42.38	49.77	68.36
KgCoOp	70.86	65.36	93.96	90.35	65.99	70.89	86.41	23.13	67.17	47.99	40.47	67.20
+PPOM	71.53	65.80	94.16	90.68	67.02	71.50	86.87	23.76	67.79	47.46	40.46	68.36
PromptSRC	71.88	65.42	93.02	90.98	65.53	70.08	86.14	23.91	67.54	46.22	41.69	69.10

Table 2: Cross-dataset transfer of PPOM across 4 hosts on ImageNet source and the other 10 evaluation-only target datasets.

	Modules		Average (11)
	Phase Orbit	Family Balance	Base	New	HM
(1)	$\times$	$\checkmark$	85.51	75.33	80.10
(2)	$\checkmark$	$\times$	85.89	75.55	80.39
(3)	$\checkmark$	$\checkmark$	86.18	75.91	80.72

Table 3: Ablation study of components in PPOM+MMRL.

Families			Average (11)
Horizontal	Vertical	Diagonal	Base	New	HM
$\times$	$\checkmark$	$\checkmark$	85.85	75.74	80.47
$\checkmark$	$\times$	$\checkmark$	85.63	75.59	80.30
$\checkmark$	$\checkmark$	$\times$	85.87	75.36	80.27
$\checkmark$	$\checkmark$	$\checkmark$	86.18	75.91	80.72

Table 4: Ablation study of orbit family in PPOM+MMRL.

Figure 3: Case study of horizontal patch phase shift of ViT.

Model	Learnable Params	Memory (MB)	Base Task Time	Effective FPS	HM Acc.
CoOp	8K	8114.6	106m52s	12.48	72.27
+PPOM	+0	1609.2	+11m4s	37.68	73.02
MMRL	4.99M	6913.8	105m26s	12.65	74.48
+PPOM	+0	3216.6	+17m58s	23.15	75.20

Table 5: Computation cost of PPOM on ImageNet dataset.

Explore similar work

ZOMP: Zeroth-Order Multi-Modal Prompt Tuning for Vision-Language Models
Aug 8, 2026Sajjad Ghiasvand, Yifan Yang, Mahnoosh Alizadeh +1VLM Adaptation Zeroth-Order Optimization
Electrical and Computer Engineering Department, UC Santa Barbara1 · Computer Science Department, UC Santa Barbara2
FVG-PT: Adaptive Foreground View-Guided Prompt Tuning for Vision-Language Models
Mar 9, 2026Haoyang Li, Liang Wang, Siyu Zhou +5VLM Adaptation Prompt Tuning
University of Technology Sydney, NSW 2007, Australia · Shanghai University, Shanghai 200444, China
Towards Calibrating Prompt Tuning of Vision-Language Models
Feb 22, 2026Ashshak Sharifdeen, Fahad Shamshad, Muhammad Akhtar Munir +6VLM Calibration
Mohamed bin Zayed University of AI · University of Colombo · Michigan State University

Aug 8, 2026J/K move · Enter open · S save

ZOMP: Zeroth-Order Multi-Modal Prompt Tuning for Vision-Language Models

Sajjad Ghiasvand, Yifan Yang, Mahnoosh Alizadeh, Ramtin Pedarsani

Electrical and Computer Engineering Department, UC Santa Barbara1 · Computer Science Department, UC Santa Barbara2

VLM Adaptation Zeroth-Order Optimization Visual Prompt Learning

Open paper AskPDF arXiv

Fine-tuning vision-language models such as CLIP typically requires backpropagation (BP) through the full model, which is infeasible when only forward-pass access is available, as is common for memory-constrained edge devices and proprietary model deployments. Prior BP-free, zeroth-order prompt-tuning methods avoid this requirement but often tune prompts in a single modality or optimize over a search space large enough that convergence requires thousands of forward passes, which is impractical under realistic query budgets. We propose ZOMP (Zeroth-Order Multimodal Prompt tuning), a query-efficient, fully forward-only method that tunes deep prompts in both the vision and text branches of a frozen CLIP model using simultaneous perturbation stochastic approximation. ZOMP combines three ingredients: a cross-modal low-rank reparameterization that ties the two branches through a shared factor and keeps the effective search dimensionality small, a gradient-correction momentum term that stabilizes the noisy zeroth-order estimate, and a budget-indexed rank schedule that unlocks capacity as the query budget is spent. Across 13 vision-language benchmarks under a matched 5,000-query budget, ZOMP consistently outperforms prior BP-free prompt-tuning methods in both few-shot accuracy and query efficiency, and it generalizes better across base-to-new, cross-dataset transfer, and out-of-distribution settings. Our results show that jointly exploiting multimodality and low-rank structure is an effective route to practical, query-efficient BP-free prompt tuning.

Abstract

Explore similar work

ZOMP: Zeroth-Order Multi-Modal Prompt Tuning for Vision-Language Models

FVG-PT: Adaptive Foreground View-Guided Prompt Tuning for Vision-Language Models

Towards Calibrating Prompt Tuning of Vision-Language Models