MMLoP: Multi-Modal Low-Rank Prompting for Efficient Vision-Language Adaptation

Sajjad Ghiasvand; Haniyeh Ehsani Oskouie; Mahnoosh Alizadeh; Ramtin Pedarsani

cs.CVFeb 24, 2026

MMLoP: Multi-Modal Low-Rank Prompting for Efficient Vision-Language Adaptation

Authors: Sajjad Ghiasvand, Haniyeh Ehsani Oskouie, Mahnoosh Alizadeh, Ramtin Pedarsani

Organizations: University of California, Santa Barbara · University of California, Los Angeles

Abstract

Prompt learning has become a dominant paradigm for adapting vision-language models (VLMs) such as CLIP to downstream tasks without modifying pretrained weights. While extending prompts to both vision and text encoders across multiple transformer layers significantly boosts performance, it dramatically increases the number of trainable parameters, with state-of-the-art methods requiring millions of parameters and abandoning the parameter efficiency that makes prompt tuning attractive. In this work, we propose MMLoP (Multi-Modal Low-Rank Prompting), a framework that achieves deep multi-modal prompting with only 11.5K trainable parameters, comparable to early text-only methods like CoOp. MMLoP parameterizes vision and text prompts at each transformer layer through a low-rank factorization that constrains prompts to a compact subspace, providing parameter efficiency while motivating the need for our complementary regularization components. To further close the accuracy gap with state-of-the-art methods, we introduce three complementary components: a self-regulating consistency loss that anchors prompted representations to frozen zero-shot CLIP features at both the feature and logit levels, a uniform drift correction that removes the global embedding shift induced by prompt tuning to preserve class-discriminative structure, and a shared up-projection that couples vision and text prompts through a common low-rank factor to enforce cross-modal alignment. Extensive experiments across three benchmarks and 11 diverse datasets demonstrate that MMLoP achieves a highly favorable accuracy-efficiency tradeoff, outperforming the majority of existing methods including those with orders of magnitude more parameters, while achieving a harmonic mean of 79.70% on base-to-novel generalization. Code is available at https://github.com/sajjad-ucsb/MMLoP.

Explore similar work

Sep 8, 2026cs.CV

Low-Rank Prompt Learning for Vision-Language Models with Fixed-Token Bases

Prompt learning adapts CLIP to downstream recognition by replacing hand-written templates with learned continuous context vectors, which in Context Optimization (CoOp) form a dense prompt matrix

\mathbf{P}\in\mathbb{R}^{m\times d}

trained from only a few examples per class. We study whether this matrix is over-parameterized by factorizing it as

\mathbf{P}=\mathbf{B}\mathbf{A}

, which cuts the trainable prompt parameters from

md

r(m+d)

, and to

rd

once the token-side factor

\mathbf{B}

is fixed. Across seven few-shot benchmarks and two CLIP backbones, low-rank prompts match or improve dense CoOp at far fewer parameters, with the clearest gains on low-shot base-to-new generalization. We then find that the token-side factor need not be learned at all: fixing

\mathbf{B}

to a Gaussian, orthogonal, SVD-derived, or even random basis and training only the embedding-side factor

\mathbf{A}

stays on par with the fully trainable factorization, and a source-trained

\mathbf{B}

offers no advantage over a random one. A prompt-factor asymmetry and a local update-space dimension gap show why fixing

\mathbf{B}

is far less restrictive than fixing

\mathbf{A}

, and a smoothness-only guarantee certifies that optimizing

\mathbf{A}

over a fixed

\mathbf{B}

converges. In the CLIP prompt setting, the embedding-side coefficients carry the adaptation while the token basis can simply be fixed.

Tanvir Muntakim Tonoy, Sajjad Ghiasvand, Mahnoosh Alizadeh +1

Few-Shot Contrastive Language-Image Pre-Training Adaptation Prompt Optimization Asymmetry Coefficients

Ask PDF

Aug 8, 2026cs.CV

ZOMP: Zeroth-Order Multi-Modal Prompt Tuning for Vision-Language Models

Fine-tuning vision-language models such as CLIP typically requires backpropagation (BP) through the full model, which is infeasible when only forward-pass access is available, as is common for memory-constrained edge devices and proprietary model deployments. Prior BP-free, zeroth-order prompt-tuning methods avoid this requirement but often tune prompts in a single modality or optimize over a search space large enough that convergence requires thousands of forward passes, which is impractical under realistic query budgets. We propose ZOMP (Zeroth-Order Multimodal Prompt tuning), a query-efficient, fully forward-only method that tunes deep prompts in both the vision and text branches of a frozen CLIP model using simultaneous perturbation stochastic approximation. ZOMP combines three ingredients: a cross-modal low-rank reparameterization that ties the two branches through a shared factor and keeps the effective search dimensionality small, a gradient-correction momentum term that stabilizes the noisy zeroth-order estimate, and a budget-indexed rank schedule that unlocks capacity as the query budget is spent. Across 13 vision-language benchmarks under a matched 5,000-query budget, ZOMP consistently outperforms prior BP-free prompt-tuning methods in both few-shot accuracy and query efficiency, and it generalizes better across base-to-new, cross-dataset transfer, and out-of-distribution settings. Our results show that jointly exploiting multimodality and low-rank structure is an effective route to practical, query-efficient BP-free prompt tuning.

Sajjad Ghiasvand, Yifan Yang, Mahnoosh Alizadeh +1

Prompt Tuning Few-Shot Contrastive Language-Image Pre-Training Adaptation Multimodal Instruction Tuning Cross-Modal Learning Prompt Engineering Token Budgets

Ask PDF

Sep 20, 2026cs.CV

PETR: Prompt Ensembling with Training-free Routing for Vision-Language Models

Prompt learning efficiently adapts vision-language models (VLMs) to downstream tasks, but gains on seen classes often come at the expense of generalization to unseen classes. To address this limitation, we propose prompt ensembling with training-free routing (PETR), whose key innovation is a carefully designed dual-prompt architecture: two complementary prompts are learned from different data and objectives to emphasize seen class discrimination and unseen-class generalization, respectively. During training, both prompts are fine-tuned using a shared frozen CLIP backbone, and statistical information is collected from the training set logits. At inference time, we determine the similarity of each test sample to seen data, and route the sample to the most appropriate prompt branch. To the best of our knowledge, this is the first prompt tuning framework that performs training-free adaptive routing based on statistical similarity. This design provides an interpretable routing signal and avoids common MoE-style routing pathologies, such as router training instability and load imbalance. Extensive experiments on 11 benchmark datasets demonstrate that our framework consistently outperforms previous methods on both seen and unseen classes, achieving new state-of-the-art results.

Weihan Cai, Hao Tan, Xinping Gao +2

Topic pending

Ask PDF