You Only Reprogram Once: Rethinking Prolonged Training for Visual Reprogramming

Zizhao Li; Mohammed Yaqoob Ansari; Xinyu Su; Jiayang Ao; Joseph West; Kourosh Khoshelham

cs.CVSep 29, 2026

You Only Reprogram Once: Rethinking Prolonged Training for Visual Reprogramming

Authors: Zizhao Li, Mohammed Yaqoob Ansari, Xinyu Su, Jiayang Ao, Joseph West, Kourosh Khoshelham

Organizations: The University of Melbourne · Fudan University

Abstract

Visual reprogramming is a parameter-efficient method for adapting pretrained models, yet its training can remain computationally expensive: even with a frozen backbone, visual prompts are often optimized through the full model for hundreds of epochs. Before changing what the pretrained model sees, we ask whether we are fully using what it already tells us. We find that modeling the full source response can already yield strong downstream predictions without prompt optimization. Motivated by this observation, we introduce You Only Reprogram Once (YORO), which constructs a downstream predictor from the frozen response space in a single forward-only traversal. Its Bayesian Discriminant Mapping (BDM) derives a covariance-aware affine mapping from streaming class statistics, requiring no backpropagation, optimizer updates, or repeated visits to the training set. When further input adaptation helps, YORO-FP optionally refines the visual prompt for 20 epochs. BDM also extends naturally to CLIP by treating attribute-prompt similarities as source responses. Across three full-data settings, YORO improves average accuracy over the strongest prior gradient-free mapping by 18.4--24.4%. On 16-shot CLIP, it raises the four-backbone average from 71.4% to 77.2%. YORO-FP provides further gains on selected tasks, while validation often retains the one-pass predictor. These results suggest a different default for visual reprogramming: read out the frozen response first, and optimize the input only when needed.

Figures & tables

Figure 1: Stronger label mapping, less prompt training. Compared with Cai et al. (2024a) , which relies on Top-1 or Top- $K$ source-label information, our method (YORO) models the full source response. Using a single traversal of the training set, YORO already outperforms BLM trained for 200 epochs on CIFAR-100, while further visual-prompt optimization may reduce training loss without improving validation accuracy.

Figure 2: One pass first, optional prompt refinement when needed. YORO fits BDM to the full frozen response in one pass. BDM estimates target class means and a shared covariance in the source-response space, then solves these statistics into an affine label mapping. The response can be classifier logits or CLIP image–text similarities. The dashed orange path denotes optional YORO-FP refinement, where only the visual prompt is optimized and the mapping is updated between epochs.

	ResNet–18 (ImageNet–1K)								ResNeXt–101–32x8d (Instagram)
Dataset	RLM	FLM	ILM	BLM	BLM+	Deep	YORO	YORO-FP	FLM	ILM	BLM	BLM+	Deep	YORO	YORO-FP
Flowers102	11.0	20.0	27.9	44.4	50.1	76.7	88.0 $\pm$ 0.0	87.9 $\pm$ 0.3	22.5	27.9	31.5	30.1	85.2	94.0 $\pm$ 0.0	94.3 $\pm$ 0.1
DTD	16.3	32.4	35.3	42.0	43.9	49.1	59.0 $\pm$ 0.0	59.6 $\pm$ 0.5	40.3	41.4	47.8	49.4	64.0	70.3 $\pm$ 0.0	70.3 $\pm$ 0.0
UCF101	6.6	18.9	23.9	30.9	32.0	46.0	61.3 $\pm$ 0.0	61.3 $\pm$ 0.0	41.9	43.1	48.3	50.1	68.3	74.9 $\pm$ 0.0	75.4 $\pm$ 0.3
Food101	3.8	12.8	14.8	23.2	25.1	34.1	49.8 $\pm$ 0.0	49.8 $\pm$ 0.0	20.5	23.0	29.6	31.4	58.7	70.3 $\pm$ 0.0	70.3 $\pm$ 0.0
GTSRB	46.1	45.5	52.0	54.8	54.3	63.1	79.0 $\pm$ 0.1	80.2 $\pm$ 0.4	56.2	59.9	62.9	63.0	74.4	79.3 $\pm$ 0.1	79.6 $\pm$ 0.5

Table 1: Performance comparison of visual reprogramming methods with different output label mappings under the padding visual prompt setting (top-1 accuracy, %). YORO and YORO-FP report mean

\pm

sample standard deviation over three seeds. Baseline results are directly taken from Cai et al. (2024a) . YORO results are highlighted, the best prior gradient-free label mapping is shown in bold , and the gradient-based method (Deep) is shown in gray. Our methods achieve the best overall results.

	ResNet–18 (ImageNet–1K)
Dataset	ILM	BLM	BLM+	Deep	YORO	YORO-FP
Flowers102	23.2	39.2	44.1	82.4	88.8 $\pm$ 0.0	88.8 $\pm$ 0.0
DTD	29.0	40.1	43.0	48.9	62.5 $\pm$ 0.0	62.5 $\pm$ 0.0
UCF101	24.4	32.9	35.4	53.1	65.1 $\pm$ 0.0	65.1 $\pm$ 0.0
Food101	13.2	21.5	22.9	30.4	52.8 $\pm$ 0.0	52.8 $\pm$ 0.0
GTSRB	76.8	82.1	82.0	89.5	75.9 $\pm$ 0.1	84.5 $\pm$ 0.6

Table 2: Watermarking-based visual reprogramming on ResNet–18 (top-1 accuracy, %). Baseline results are taken from Cai et al. (2024a) . Our methods achieve the best overall results.

	ViT-B/32 (CLIP)						ResNet–50 (CLIP)
Dataset	VP	AR	AttrVR	DVP	YORO	YORO-FP	VP	AR	AttrVR	DVP	YORO	YORO-FP
FGVC Aircraft	24.3	21.8	24.5	26.1	36.6 $\pm$ 0.8	36.9 $\pm$ 1.2	16.2	18.6	20.7	22.1	36.2 $\pm$ 1.5	36.3 $\pm$ 1.4
Caltech101	92.3	92.7	92.0	92.9	94.0 $\pm$ 0.0	93.7 $\pm$ 0.6	80.1	86.5	89.1	89.8	91.3 $\pm$ 0.1	91.0 $\pm$ 0.4
StanfordCars	58.6	56.9	56.6	56.5	71.8 $\pm$ 0.6	71.0 $\pm$ 1.2	44.0	53.9	53.9	54.5	68.6 $\pm$ 0.2	68.5 $\pm$ 0.3
DTD	54.9	49.9	56.8	57.2	65.8 $\pm$ 1.6	65.5 $\pm$ 1.8	43.4	46.4	54.4	55.9	64.4 $\pm$ 0.2	63.8 $\pm$ 0.8
EuroSAT	85.9	85.6	88.6	88.5	82.3 $\pm$ 1.5	82.1 $\pm$ 2.4	59.7	66.6	72.0	72.2	82.9 $\pm$ 0.7	79.7 $\pm$ 5.7

Table 3: Performance comparison on 16-shot downstream classification using ViT-B/32 and ResNet–50 CLIP backbones (accuracy %). Prior results are taken from Cai et al. (2025b) . Highlighted YORO columns report mean

\pm

sample standard deviation over three seeds using DVP attribute-prompt similarities as source responses. Best results are in bold . Our methods achieve the best overall results.

Method	RN50	RN101	ViT-B/32	ViT-B/16	Overall
VP	53.5	57.5	68.3	74.4	63.4
AR	60.4	62.7	66.3	76.5	66.5
AttrVR	64.6	67.2	69.8	78.5	70.0
DVP	66.0	68.8	71.0	79.7	71.4
YORO	74.4 $\pm$ 0.2	76.8 $\pm$ 0.2	76.2 $\pm$ 0.3	81.2 $\pm$ 0.1	77.2 $\pm$ 0.1
YORO-FP	73.9 $\pm$ 0.6	76.7 $\pm$ 0.3	76.0 $\pm$ 0.2	81.4 $\pm$ 0.3	77.0 $\pm$ 0.3

Table 4: Aggregate 16-shot accuracy (%) over 11 datasets. YORO results report mean

\pm

sample standard deviation over three seeds. Overall averages the four frozen CLIP backbones equally. Our methods achieve the best overall results.

Method	Test accuracy (%)	Training time (s)
YORO	$\mathbf{94.0\pm 0.1}$	$\mathbf{17.1\pm 6.2}$
DGA	$91.8\pm 4.8$	$6562.1\pm 34.5$
AutoVP	$81.5\pm 0.4$	$2754.4\pm 21.3$
DVP	$80.9\pm 1.1$	$3714.9\pm 7.3$
AttrVR	$76.2\pm 0.6$	$2683.2\pm 7.4$
BLM	$73.1\pm 1.2$	$4996.0\pm 122.3$

Table 5: Accuracy and training time on Oxford Flowers with CLIP ViT-B/32 under the 16-shot setting. Results are mean

\pm

sample standard deviation over three seeds. YORO achieves the best accuracy with the lowest training cost. All methods in this table are rerun under the same hardware protocol.

Setting	YORO	Frozen	Refresh	Momentum
RN18–P	71.0 $\pm$ 0.0	71.1 $\pm$ 0.0	71.1 $\pm$ 0.0	71.6 $\pm$ 0.1
RN18–W	71.5 $\pm$ 0.0	73.3 $\pm$ 0.1	73.6 $\pm$ 0.1	73.9 $\pm$ 0.0
RNXT101–P	79.8 $\pm$ 0.0	80.2 $\pm$ 0.0	80.6 $\pm$ 0.1	80.9 $\pm$ 0.1
Overall	74.1 $\pm$ 0.0	74.9 $\pm$ 0.0	75.1 $\pm$ 0.1	75.4 $\pm$ 0.0

Table 6: Full-data mapping-update ablation (top-1 accuracy, %). Entries are 14-dataset macro-averages over three seeds. YORO is the one-pass predictor, while Frozen, Refresh, and Momentum denote YORO-FP variants with

\beta=1

0

, and

0.9

, respectively. P/W denote padding/watermarking.

Setting	YORO	Frozen	Refresh	Momentum
RN18–P	71.0 $\pm$ 0.0	71.1 $\pm$ 0.0	71.1 $\pm$ 0.0	71.6 $\pm$ 0.1
RN18–W	71.5 $\pm$ 0.0	73.3 $\pm$ 0.1	73.6 $\pm$ 0.1	73.9 $\pm$ 0.0
RNXT101–P	79.8 $\pm$ 0.0	80.2 $\pm$ 0.0	80.6 $\pm$ 0.1	80.9 $\pm$ 0.1
Overall	74.1 $\pm$ 0.0	74.9 $\pm$ 0.0	75.1 $\pm$ 0.1	75.4 $\pm$ 0.0

\beta=1

0

, and

0.9

, respectively. P/W denote padding/watermarking.

Backbone	YORO	Frozen	Refresh	Momentum
ResNet–50	74.4 $\pm$ 0.2	74.0 $\pm$ 0.2	73.8 $\pm$ 0.4	73.9 $\pm$ 0.6
ResNet–101	76.8 $\pm$ 0.2	76.5 $\pm$ 0.2	76.4 $\pm$ 0.0	76.7 $\pm$ 0.3
ViT-B/32	76.2 $\pm$ 0.3	76.0 $\pm$ 0.4	76.1 $\pm$ 0.2	76.0 $\pm$ 0.2
ViT-B/16	81.2 $\pm$ 0.1	80.9 $\pm$ 0.2	81.3 $\pm$ 0.3	81.4 $\pm$ 0.3
Overall	77.2 $\pm$ 0.1	76.8 $\pm$ 0.1	76.9 $\pm$ 0.1	77.0 $\pm$ 0.3

Table 7: Few-shot CLIP mapping-update ablation (top-1 accuracy, %). Entries are 11-dataset macro-averages over three seeds. YORO is the one-pass predictor, while Frozen, Refresh, and Momentum denote YORO-FP variants with

\beta=1

0

, and

0.9

, respectively.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

	ResNet–18 (ImageNet–1K)								ResNeXt–101–32x8d (Instagram)
Dataset	RLM	FLM	ILM	BLM	BLM+	Deep	YORO	YORO-FP	FLM	ILM	BLM	BLM+	Deep	YORO	YORO-FP
Flowers102	11.0	20.0	27.9	44.4	50.1	76.7	80.1	80.7	22.5	27.9	31.5	30.1	85.2	89.3	89.4
DTD	16.3	32.4	35.3	42.0	43.9	49.1	55.5	53.2	40.3	41.4	47.8	49.4	64.0	67.4	66.9
UCF101	6.6	18.9	23.9	30.9	32.0	46.0	54.1	52.9	41.9	43.1	48.3	50.1	68.3	68.4	69.0
Food101	3.8	12.8	14.8	23.2	25.1	34.1	41.7	35.5	20.5	23.0	29.6	31.4	58.7	60.9	58.2
GTSRB	46.1	45.5	52.0	54.8	54.3	63.1	47.2	57.7	56.2	59.9	62.9	63.0	74.4	58.2	67.9

Appendix

Table 9: Comparison under the legacy padding setting (top-1 accuracy, %). Following Cai et al. (2024a) , downstream images are directly padded to the pretrained input size without resizing. Despite this legacy input configuration, YORO substantially improves over BLM and BLM+ across both pretrained backbones.

Protocol	Method	Accuracy (%)	$\Delta$ (%)	No refinement	Passes / time
ResNet18 (P)	YORO	71.0 $\pm$ 0.0	–	–	$1$ / $1.0\times$
ResNet18 (P)	YORO-FP	71.6 $\pm$ 0.1	$+0.6$	54.8%	$21$ / $20.8\times$
FS-CLIP	YORO	77.2 $\pm$ 0.1	–	–	$1$ / $1.0\times$
FS-CLIP	YORO-FP	77.0 $\pm$ 0.3	$-0.2$	58.3%	$21$ / $17.5\times$

Appendix

Table 10: Efficiency–accuracy comparison of YORO and YORO-FP. Accuracy is the three-seed macro-average over 14 full-data tasks for ResNet–18 padding, or over 11 few-shot datasets and four CLIP backbones. Relative time is normalized by the corresponding YORO run. All timings use one NVIDIA A100 and exclude validation and test evaluation. “No refinement” reports the fraction of YORO-FP runs in which validation selects the original YORO checkpoint.

Backbone	BDM	Conditional 95%	Adaptive	OAS	RBLW	Empirical LDA
ResNet–50	$\mathbf{74.4}\pm\mathbf{0.2}$	$74.1\pm 0.3$	$74.0\pm 0.3$	$73.3\pm 0.2$	$73.3\pm 0.2$	$69.2\pm 0.2$
ResNet–101	$\mathbf{76.8}\pm\mathbf{0.2}$	$76.6\pm 0.3$	$76.6\pm 0.3$	$76.1\pm 0.2$	$76.1\pm 0.2$	$72.5\pm 0.4$
ViT-B/32	$\mathbf{76.2}\pm\mathbf{0.3}$	$76.1\pm 0.4$	$76.1\pm 0.4$	$75.5\pm 0.0$	$75.5\pm 0.0$	$71.6\pm 0.1$
ViT-B/16	$\mathbf{81.2}\pm\mathbf{0.1}$	$81.1\pm 0.1$	$81.0\pm 0.2$	$80.7\pm 0.1$	$80.7\pm 0.1$	$77.5\pm 0.5$
Overall	$\mathbf{77.2}\pm\mathbf{0.1}$	$77.0\pm 0.0$	$76.9\pm 0.1$	$76.4\pm 0.1$	$76.4\pm 0.1$	$72.7\pm 0.2$

Appendix

Table 11: Covariance and response-selection ablations. Results report mean

\pm

sample standard deviation over three seeds (%), averaged across 11 datasets under the 16-shot FS-CLIP protocol. Overall further averages the four CLIP backbones.

Method	Pets	Flowers	Aircraft	DTD	EuroSAT	Cars	Food	SUN	Caltech	UCF	Avg.
CARPRT	85.7	65.6	16.9	41.3	36.8	56.4	76.9	61.3	88.5	63.7	59.3
MPE	76.0	57.0	16.1	41.7	30.3	55.7	75.5	59.3	86.4	60.1	55.8
WPE	78.4	58.8	16.1	40.9	30.7	56.0	76.2	59.7	86.7	61.5	56.5
Tip-Adapter	88.1	89.9	29.8	60.9	70.5	66.8	77.8	66.9	90.2	70.6	71.2
YORO	87.5	92.6	36.2	64.4	82.9	68.6	72.8	67.6	91.3	75.1	73.9

Appendix

Table 12: Computationally efficient CLIP adaptation on ten overlapping datasets with a ResNet–50 backbone (accuracy %). We recompute each average over OxfordPets, Flowers102, FGVC Aircraft, DTD, EuroSAT, StanfordCars, Food101, SUN397, Caltech101, and UCF101. Tip-Adapter is reported by Wang et al. (2024b) , while CARPRT, MPE, and WPE are reported by Dong et al. (2026) . YORO is our three-seed result. All methods avoid iterative backbone updates, so the table provides a contextual efficiency-oriented comparison despite differences in data access.

Backbone	Input	YORO	Frozen BDM	$\Delta$ (%)	YORO-FP
ResNet–18	Padding	72.3 $\pm$ 0.1	76.0 $\pm$ 0.2	$+3.7$	77.0 $\pm$ 0.3
ResNeXt–101–32x8d	Padding	56.5 $\pm$ 0.0	63.3 $\pm$ 0.3	$+6.8$	68.6 $\pm$ 0.6
ResNet–18	Watermark	60.6 $\pm$ 0.1	83.4 $\pm$ 0.2	$+22.8$	84.7 $\pm$ 0.2

Appendix

Table 13: SVHN isolates the benefit of input adaptation. With BDM frozen, optimizing only the visual prompt improves every tested backbone–input setting. Accuracy is mean

\pm

sample standard deviation over three seeds (%).

Protocol	Strategy	Accuracy (%)	$\Delta$ (%)	95% CI	W/T/L	Refinement depth	No refinement
Full data	Frozen	$71.1\pm 0.0$	$+0.2$	$[-0.2,+0.8]$	2/7/5	$4.8\pm 7.7$	69.0%
	Refresh	$71.1\pm 0.0$	$+0.1$	$[-0.2,+0.7]$	2/10/2	$2.2\pm 5.5$	81.0%
	Momentum	$\mathbf{71.6\pm 0.1}$	$\mathbf{+0.6}$	$\mathbf{[0.0,+1.4]}$	5/6/3	$6.5\pm 7.8$	54.8%
FS-CLIP	Frozen	$76.8\pm 0.1$	$-0.3$	$[-0.5,-0.2]$	2/13/29	$3.2\pm 5.6$	66.7%
	Refresh	$76.9\pm 0.1$	$-0.3$	$[-0.5,0.0]$	6/14/24	$3.4\pm 5.7$	60.6%
	Momentum	$\mathbf{77.0\pm 0.3}$	$\mathbf{-0.2}$	$\mathbf{[-0.4,0.0]}$	8/8/28	$4.2\pm 6.3$	58.3%

Appendix

Table 14: Variants of YORO-FP mapping strategies.

\Delta

is measured relative to YORO, with 95% bootstrap confidence intervals over evaluation settings. W/T/L counts datasets for full data and dataset–backbone pairs in the few-shot CLIP setting. Refinement depth is the number of additional prompt-training epochs selected on the validation set. A depth of 0 corresponds to the original YORO checkpoint. A median depth of 0 means that validation selects YORO without further prompt training in at least half of the runs. The last column reports the fraction of runs for which validation retains the original YORO predictor. Frozen, Refresh, and Momentum use

\beta=1

0

, and

0.9

, respectively.

(a) ResNet–18 / P
Dataset	YORO	Frozen	Refresh	Momentum
Caltech101	92.0 $\pm$ 0.0	92.0 $\pm$ 0.0	92.0 $\pm$ 0.0	92.1 $\pm$ 0.1
CIFAR-10	86.2 $\pm$ 0.1	86.0 $\pm$ 0.4	86.2 $\pm$ 0.1	86.2 $\pm$ 0.1
CIFAR-100	64.8 $\pm$ 0.1	64.8 $\pm$ 0.1	64.8 $\pm$ 0.1	64.8 $\pm$ 0.1
DTD	59.0 $\pm$ 0.0	59.0 $\pm$ 0.0	59.1 $\pm$ 0.2	59.6 $\pm$ 0.5
EuroSAT	92.8 $\pm$ 0.0	92.5 $\pm$ 0.5	92.5 $\pm$ 0.5	92.5 $\pm$ 0.2

Appendix

Table 15: Per-dataset full-data mapping-update ablation (accuracy %). Each entry is the three-seed mean

\pm

sample standard deviation. P/W denote padding/watermarking. Bold marks the highest mean within each dataset and setting. These results underlie the averages in Table 7 .

$\mathcal{X}^{\rm S},\mathcal{X}^{\rm T}$	Source and target input spaces.
$\mathcal{Y}^{\rm S},\mathcal{Y}^{\rm T}$	Source and target label spaces.
$k_{\rm S}$	Number of source-response coordinates.
$k_{\rm T}$	Number of target classes.
$[q]$	Index set $\{1,\ldots,q\}$ .
$i$	Example index.
$c$	Target-class index.

Appendix

Table 16: Symbols used in YORO and YORO-FP.

$f_{\rm pre}$	Frozen classifier or CLIP encoders.
$f_{\rm in}(\cdot\mid\theta)$	Input reprogramming function with parameters $\theta$ .
$\theta_{0}$	Prompt initialization.
$\mathcal{L}_{\rm FP}$	YORO-FP objective.
$\ell_{\rm CE}$	Cross-entropy loss.
$\nabla_{\theta}$	Gradient with respect to prompt parameters.
$X$	Minibatch images.

Appendix

Table 17: Symbols used in YORO and YORO-FP (continued).

Explore similar work

Reprogramming Vision-Language Models via Structured Prompt Reparameterization
Sep 29, 2026Zizhao Li, Chengyi Cai, Mohammed Yaqoob Ansari +3Contrastive Language-Image Pre-Training Model Intra-Class
The University of Melbourne, Melbourne, Australia
ZOMP: Zeroth-Order Multi-Modal Prompt Tuning for Vision-Language Models
Aug 8, 2026Sajjad Ghiasvand, Yifan Yang, Mahnoosh Alizadeh +1Soft Prompt Tuning Frozen Contrastive Language-Image Pre-Training Visual Encoder
Electrical and Computer Engineering Department, UC Santa Barbara1 · Computer Science Department, UC Santa Barbara2
Low-Rank Prompt Learning for Vision-Language Models with Fixed-Token Bases
Sep 8, 2026Tanvir Muntakim Tonoy, Sajjad Ghiasvand, Mahnoosh Alizadeh +1Multiple Prompt Learning Frameworks Contrastive Language-Image Pre-Training Model
Department of Electrical and Computer Engineering UC Santa Barbara Santa Barbara, CA 93106, USA

Sep 29, 2026, cs.CVJ/K move · Enter open · S save

Reprogramming Vision-Language Models via Structured Prompt Reparameterization

Zizhao Li, Chengyi Cai, Mohammed Yaqoob Ansari, Feng Liu+2

The University of Melbourne, Melbourne, Australia

Contrastive Language-Image Pre-Training Model Intra-Class Reparameterization

Open paper AskPDF arXiv

Visual reprogramming adapts pretrained models to downstream tasks by modifying their input and output interfaces while keeping the backbone fixed. In vision-language models, existing methods mainly rely on intra-class prompt aggregation and do not explicitly model relationships among classes. However, fine-grained categories often exhibit highly overlapping attribute descriptions and strong inter-class correlation in the text embedding space, where discriminative cues lie in subtle low-variance components. We propose Reparameterized Inter-Class Visual Reprogramming (RVP), a structured framework that aggregates multiple text prompts within each class and applies residual correction across classes. We also show that CLIP-based visual reprogramming with input-independent linear output aggregation can be expressed as a linear mapping from frozen image embeddings to downstream logits, and use this view to design a structured reparameterization that models shared semantic components and class-specific differences. RVP uses only a single visual prompt and can be reparameterized at inference into a frozen backbone followed by a linear classifier, incurring nearly zero computational overhead. Across 11 few-shot classification benchmarks and four CLIP backbones, RVP consistently improves over prior visual reprogramming methods with comparable or better inference efficiency.

Abstract

Explore similar work

Reprogramming Vision-Language Models via Structured Prompt Reparameterization

ZOMP: Zeroth-Order Multi-Modal Prompt Tuning for Vision-Language Models

Low-Rank Prompt Learning for Vision-Language Models with Fixed-Token Bases