Prime Once, then Reprogram Locally: An Efficient Alternative to Black-Box Service Model Adaptation

Yunbei Zhang; Chengyi Cai; Feng Liu; Jihun Hamm

cs.CVApr 1, 2026

Prime Once, then Reprogram Locally: An Efficient Alternative to Black-Box Service Model Adaptation

Authors: Yunbei Zhang, Chengyi Cai, Feng Liu, Jihun Hamm

Organizations: Tulane University · University of Melbourne

Abstract

Adapting closed-box service models (i.e., APIs) for target tasks typically relies on reprogramming via Zeroth-Order Optimization (ZOO). However, this standard strategy is known for extensive, costly API calls and often suffers from slow, unstable optimization. Furthermore, we observe that this paradigm faces new challenges with modern APIs (e.g., GPT-4o). These models can be less sensitive to the input perturbations ZOO relies on, thereby hindering performance gains. To address these limitations, we propose an Alternative efficient Reprogramming approach for Service models (AReS). Instead of direct, continuous closed-box optimization, AReS initiates a single-pass interaction with the service API to prime an amenable local pre-trained encoder. This priming stage trains only a lightweight layer on top of the local encoder, making it highly receptive to the subsequent glass-box (white-box) reprogramming stage performed directly on the local model. Consequently, all subsequent adaptation and inference rely solely on this local proxy, eliminating all further API costs. Experiments demonstrate AReS's effectiveness where prior ZOO-based methods struggle: on GPT-4o, AReS achieves a +27.8% gain over the zero-shot baseline, a task where ZOO-based methods provide little to no improvement. Broadly, across ten diverse datasets, AReS outperforms state-of-the-art methods (+2.5% for VLMs, +15.6% for standard VMs) while reducing API calls by over 99.99%. AReS thus provides a robust and practical solution for adapting modern closed-box models.

Figures & tables

Panel of Figure 1 — Figure 1: (a) On the real-world GPT-4o API, ZOO-based methods show limited effectiveness, providing little to no improvement over zero-shot performance while incurring high total (training and inference) costs. (b, c) On CLIP ViT-B/16, these methods require $\sim$ 10 8 API calls and over 32 hours of training on Flowers102, yet still underperform ours, AReS, which uses only $\sim$ 10 3 API calls and 0.4 hours.

Figure 2: (a) Previous closed-box methods use Zeroth-Order Optimization (ZOO), which estimates gradients by querying the API multiple times for the same training image with random perturbations. This results in continuous API dependency: numerous calls during training and one API call per image during inference. (b) Our AReS approach performs a single-pass priming, requiring only one API call per training image to prepare a local model. This enables efficient, gradient-aware VR locally, eliminating all subsequent API costs during inference.

Panel of Table 1 — Table 1: Comparison of tuning methods for Model-as-a-Service (MaaS). The Access column indicates the strictness, distinguishing translucent-box (grey, requiring intermediate Features or Embedding access) from closed-box (black, Input/Predictions only) settings. We compare methods by: MR (Model Reprogramming-based); required Input Access and Output Access; support for VMs (Standard Vision Models) and VLMs (Vision-Language Models); Prompt type (L: Language, V: Visual); and Encoder Flexibility. Our method, AReS, uniquely offers Cost-free Inference for both VMs and VLMs while operating under the strictest closed-box constraints.

Method	Flowers	DTD	UCF	Food	GTSRB	EuroSAT	Pets	Cars	SUN	SVHN	Avg.	#API (M)	Time (h)
VR (glass-box)	86.8	62.0	74.0	81.6	65.1	90.9	90.7	66.2	66.7	60.2	74.4	16.82	9.8
Zero-shot	71.3	43.9	66.9	85.9	21.0	47.9	89.1	65.2	62.6	17.9	57.2	0.12	0
BAR	71.0	46.8	64.2	84.4	21.5	77.3	88.4	63.0	62.4	34.6	61.4	612.84	185.6
BlackVIP	70.6	45.3	68.7	85.9	21.3	73.3	89.1	65.4	64.5	44.4	62.9	754.20	197.5
LLM-Opt	67.6	45.0	59.9	78.5	21.2	48.0	87.7	56.2	60.3	20.2	54.5	5011.32	5.1
AReS	86.6	48.2	67.1	68.8	39.4	85.7	88.9	43.2	62.8	63.2	65.4	0.02	3.7

Table 2: Accuracy and Efficiency comparison on ten diverse visual recognition datasets using CLIP ViT-B/16 (Service) and ViT-B/16 (Local) in a 16-shot setting. AReS achieves superior average accuracy with a drastic reduction in resources (99.99% fewer API calls and 98% less training time). For this and all subsequent tables, Gray denotes glass-box Service VR results. #API (M): total API calls in millions (training and inference); Wall-clock Time (h): total training hours run on an RTX 6000 Ada GPU.

Method	Flowers	DTD	UCF	Food	GTSRB	EuroSAT	Pets	Cars	SUN	SVHN	Avg.	#API (M)	Time (h)
VR (glass-box)	69.6	52.1	49.7	38.7	60.6	96.7	77.1	6.2	34.0	83.3	56.8	43.4	39.6
BAR	14.8	24.3	29.5	12.2	14.1	43.1	24.9	1.0	10.6	29.6	20.4	1,724.2	668.2
BlackVIP (RN50)	15.2	42.3	34.0	14.5	13.9	67.3	63.6	3.7	22.6	30.8	30.8	2,586.2	686.9
AReS (RN50)	43.5	43.4	39.4	25.9	48.7	84.3	73.9	5.8	19.9	73.7	45.9	0.2	8.7
BlackVIP (ViT-B/32)	27.4	44.1	36.6	18.5	38.4	59.6	65.4	3.9	23.7	30.3	34.8	2,586.2	757.6
AReS (ViT-B/32)	53.6	46.9	41.4	27.3	58.8	95.6	65.0	4.7	26.9	83.5	50.4	0.2	22.4

Table 3: Accuracy and Efficiency comparison using ViT-B/16 (Service) in Full-shot setting.

Method	LLaVA				GPT-4o				Clarifai
Method	Acc. $\uparrow$	Train	Infer.	Total (#) $\downarrow$	Acc. $\uparrow$	Train	Infer.	Total ( $)$ \downarrow$	Acc. $\uparrow$	Train	Infer.	Total ( $)$ \downarrow$
glass-box	91.3	–	–	–	–	–	–	–	–	–	–	–
Zero-shot	40.1	–	8100	8100	59.4	–	14.6	14.6	–	–	–	–
BAR	34.1	$\sim 10^{6}$	8100	$\sim 10^{6}$	59.1	57.6	14.6	72.2	68.1	38.4	9.7	48.1
BlackVIP	39.4	$\sim 10^{7}$	8100	$\sim 10^{7}$	60.1	86.4	14.6	101.0	72.1	57.6	9.7	67.3
AReS	73.1	160	0	160	87.2	0.3	0	0.3	83.2	0.2	0	0.2

Table 4: Performance and Cost comparison on the open-source MLLM LLaVA, the proprietary API GPT-4o, and the commercial VM API Clarifai, evaluated on the EuroSAT 16-shot benchmark. A zero-shot baseline is not applicable for Clarifai as it is a closed-vocabulary model. Costs are shown in total API calls (#) or US dollars ($). AReS demonstrates significant gains where ZOO-based methods struggle.

Panel of Figure 3 — Figure 3: Ablation studies for AReS on the EuroSAT dataset: impact of (a) training data fraction (for VMs), (b) few-shot sample size (for VLMs), (c) priming loss function, and (d) amount of extra unlabeled data for priming on accuracy (%). Default setting involves 16-shot learning, CLIP ViT-B/16 Service, and ViT-B/16 Local encoder unless otherwise indicated by the subplot analysis.

Method	Service Backbone
	RN50	RN101	ViT-B/32	ViT-B/16	Avg.
ZS	37.5	32.6	45.2	47.9	40.8
BAR	26.9	33.5	70.3	77.2	52.0
BlackVIP (RN50)	51.3	50.8	62.9	68.5	58.4
AReS (RN50)	62.2	68.5	63.2	78.6	68.1
BlackVIP (ViT-B/16)	48.4	51.3	67.9	73.3	60.2

Table 5: Ablation study across Service/Local model backbones.

Method	Priming	VR	LP	Accuracy
Zero-shot	–	–	–	47.9
Primed	✓	–	–	45.6
Local VR	–	✓	–	70.6
Local LP	–	–	✓	73.8
Local VR+LP	–	✓	✓	80.1
AReS (Ours)	✓	✓	–	85.7

Table 6: Component analysis of Priming, Visual Reprogramming (VR), and Linear Probing (LP).

Figure 4: Illustration of Zeroth-Order Optimization (ZOO) techniques commonly used in closed-box model reprogramming. (a) BAR with Randomized Gradient-Free (RGF) method estimates gradients by querying the model with random directional perturbations. (b) BlackVIP with Simultaneous Perturbation Stochastic Approximation with Gradient Correction (SPSA-GC) approximates gradients using only two model queries with a randomly generated perturbation vector. Both approaches suffer from high query complexity and noisy gradient estimates, leading to unstable and computationally intensive optimization, especially for high-dimensional prompts.

Dataset	Full-shot Training	16-shot Training	Testing Set Size	Number of Classes
Flowers102	4,093	1,632	2,463	102
DTD	2,820	752	1,692	47
UCF101	7,639	1,616	3,783	101
Food101	50,500	1,616	30,300	101
GTSRB	39,209	688	12,630	43
EuroSAT	13,500	160	8,100	10

Table 7: Detailed dataset information.

Figure 5: Comparison of closed-box visual reprogramming approaches. (a) BAR and (b) BlackVIP rely on Zeroth-Order Optimization (ZOO) by repeatedly querying the service model with perturbed inputs (e.g., using random directions) to estimate gradients for updating the visual prompt. These methods suffer from high API call costs and potentially unstable optimization. (c) Our AReS method performs a one-time priming from the service model to a local model. Subsequent visual prompt optimization occurs efficiently on this local model using glass-box gradients, eliminating further API calls and enabling stable, cost-effective adaptation.

Method	Flowers	DTD	UCF	Food	GTSRB	EuroSAT	Pets	Cars	SUN	SVHN	Avg.	#API (M)	Time (h)
VR (glass-box)	42.3	44.4	34.4	25.4	53.1	85.0	73.9	5.0	20.5	75.1	45.9	43.4	21.9
BAR	16.9	25.3	30.2	13.9	22.5	45.7	28.7	1.5	10.7	42.1	23.8	1,724.2	313.5
BlackVIP (RN50)	21.7	39.6	31.4	14.9	26.8	68.3	61.4	3.5	12.3	42.6	32.3	2,586.2	320.5
AReS (RN50)	41.3	42.3	37.2	25.3	48.4	84.3	73.7	5.1	19.7	72.9	45.0	0.2	8.7
BlackVIP (ViT-B/32)	26.7	37.7	30.2	13.3	33.8	67.3	57.9	3.0	14.5	34.5	31.9	2,586.2	418.0
AReS (ViT-B/32)	51.3	45.8	41.5	30.2	57.9	94.9	63.3	5.1	25.6	83.6	49.9	0.2	22.3

Table 8: Accuracy and Efficiency comparison using RN101 (Service) in Full-shot setting.

Method	Flowers	DTD	UCF	Food	GTSRB	EuroSAT	Pets	Cars	SUN	SVHN	Avg.
VR (glass-box)	55.0	44.7	42.0	18.0	17.4	70.6	70.7	5.7	29.6	28.4	38.2
BAR	9.7	16.3	23.5	8.1	4.8	34.2	21.5	1.0	7.6	19.6	14.6
BlackVIP	16.1	36.1	33.8	12.5	9.1	54.9	54.5	3.9	22.1	16.2	25.9
AReS (Ours)	46.0	35.5	34.3	12.7	19.6	71.1	54.3	4.2	23.0	27.3	32.8

Table 9: Accuracy comparison using ViT-B/16 (Service) and ViT-B/32 (Local) in 16-shot setting.

Method	Flowers	DTD	UCF	Food	GTSRB	EuroSAT	Pets	Cars	SUN	SVHN	Avg.
Zero-shot	71.3	43.9	66.9	85.9	21.0	47.9	89.1	65.2	62.6	17.9	57.2
Local VR	55.0	44.7	42.0	18.0	17.4	70.6	70.7	5.7	29.6	28.4	38.2
AReS (Ours)	86.6	48.2	67.1	68.8	39.4	85.7	88.9	43.2	62.8	63.2	65.4

Table 10: Accuracy comparison between Local VR (ViT-B/16) and our method on VLMs using CLIP ViT-B/16 (Service) and ViT-B/16 (Local) in a 16-shot setting.

Service	Local	Local VR	Ours
ViT-B/16	ViT-B/32	45.3	50.4
ViT-B/16	RN50	43.9	45.9
RN101	ViT-B/32	45.3	49.9
RN101	RN50	43.9	45.0

Table 11: Accuracy comparison between Local VR and our method on VMs in Full-shot setting.

Method	# Trainable Params	Accuracy (%)
VP w/ SPSA-GC	69K	70.9
BAR	37K	77.3
BlackVIP	9K	73.3
AReS (Ours)	21K	85.7

Table 12: Comparison of trainable parameters and accuracy on the EuroSAT dataset. More parameters do not correlate with better performance for ZOO-based methods.

Method	Accuracy (%)
Local VR	70.6
BlackVIP	73.3
BlackVIP w/ Local Tuning	74.1
AReS (Ours)	85.7

Table 13: Comparison with an enhanced BlackVIP baseline on the EuroSAT dataset. The results confirm that AReS’s performance gain is primarily from its superior two-stage framework.

Method	4-shot	8-shot	16-shot	32-shot
BlackVIP	69.3	71.7	73.3	72.9
Local VR+LP	59.7	68.1	80.1	86.5
AReS (Ours)	66.1	73.4	85.7	91.6

Table 14: Accuracy (%) comparison on the EuroSAT dataset across different few-shot settings. AReS consistently outperforms the strong local baseline (Local VR+LP) as data availability increases.

Initial State	Service: RN101 (42.3%)		Service: ViT-B/16 (69.6%)
Initial State	% of Dataset	How AReS Performed	% of Dataset	How AReS Performed
Both Service & Local Correct	30.0%	29.5% kept correct	35.0%	34.0% kept correct
Only Local Correct	8.7%	7.0% kept correct	3.7%	0.5% kept correct
Only Service Correct	12.3%	10.0% newly learned	34.6%	19.0% newly learned
Both Incorrect	49.0%	4.8% newly learned	26.7%	0.1% newly learned
AReS Final Accuracy	51.3%		53.6%

Table 15: Illustrative Breakdown of AReS’s Performance Composition on Flowers102. The local model for both scenarios is ViT-B/32 (glass-box Acc: 38.7%). “Kept correct” refers to samples the local model classified correctly that AReS also classifies correctly. “Newly learned” refers to samples the local model misclassified that AReS now classifies correctly.

Method	ImageNet Acc (%)	Gain over Zero-shot (%)
VR (glass-box)	67.4	+0.7
Zero-shot	66.7	-
BAR	64.6	-2.1
VP w/ SPSA-GC	62.3	-4.4
BlackVIP	67.1	+0.4
AReS (Ours)	80.1	+13.4

Table 16: Accuracy (%) comparison on ImageNet in a 16-shot setting with CLIP ViT-B/16 as the service model. AReS significantly outperforms all baselines.

Figure 6: An example of GPT-4o’s robustness to the input perturbations used in reprogramming. When presented with a clean image (left) and a prompted image (right), GPT-4o’s textual reasoning explicitly acknowledges the visual alteration but correctly identifies the flower’s key features in both cases. This demonstrates that the model is often invariant to the input noise that ZOO-based methods rely on, motivating a strategic shift in adaptation methods.

Method	GPU Mem. (MB)	Latency (ms)	Acc. (%)
VR (glass-box)	11,937	-	90.9
Zero-shot	0	120	47.9
BAR	1,649	122	77.3
BlackVIP	2,428	170	73.3
AReS (Ours)	2,781	60	85.7

Table 17: GPU memory and inference latency comparison on EuroSAT with NVIDIA RTX 3090. AReS achieves the highest accuracy and fastest inference by running entirely locally after priming.

Method	Zero-shot	BlackVIP	AReS (Ours)
Acc. (%)	62.5	63.2	63.6

Table 18: Domain gap experiment: DermCLIP (pretrained on Derm1M)

\rightarrow

ViT-B/16 (pretrained on ImageNet) on HAM10000 under the 16-shot setting.

Top- $k$	1	2	5	all (10)
Soft	80.1	82.3	85.0	85.7
Hard	70.7	74.6	78.9	80.6

Table 19: Robustness to partial API outputs (top-

k

) on EuroSAT. “Soft” uses top-

k

class names with confidence scores; “Hard” uses only the ordered top-

k

list.

VLM Adaptation

Vision-language model adaptation tailors pretrained image-text and multimodal models to new tasks and domains while retaining transferable capabilities.

See all 545 papers

Adapting in the Dark: Efficient and Stable Test-Time Adaptation for Black-Box Models
Apr 17, 2026Yunbei Zhang, Shuaicheng Niu, Chengyi Cai +2Test-Time Adaptation Image Classification
1Tulane University · 2Nanyang Technological Univer · 3The University of Melbourne.
ReLoRA: Knowledge-Reusing Adaptation for Fast Rollout of Evolving LLM Services
May 23, 2026Yang Xu, Zihuai Xu, Hongli Xu +3LLM Serving LLM Fine-Tuning
Suzhou Institute for Advanced Research, University of Science and Technology of China, Suzhou, Jiangsu, China, 215123
Meta-Tool: Efficient Few-Shot Tool Adaptation for Small Language Models
Apr 22, 2026Sachin KumarSmall Language Models Few-Shot Prompting
LexisNexis, USA

Apr 17, 2026J/K move · Enter open · S save

Adapting in the Dark: Efficient and Stable Test-Time Adaptation for Black-Box Models

Yunbei Zhang, Shuaicheng Niu, Chengyi Cai, Feng Liu+1

1Tulane University · 2Nanyang Technological Univer · 3The University of Melbourne.

Test-Time Adaptation Image Classification

Open paper AskPDF arXiv

Test-Time Adaptation (TTA) for black-box models accessible only via APIs remains a largely unexplored challenge. Existing approaches such as post-hoc output refinement offer limited adaptive capacity, while Zeroth-Order Optimization (ZOO) enables input-space adaptation but faces high query costs and optimization challenges in the unsupervised TTA setting. We introduce BETA (Black-box Efficient Test-time Adaptation), a framework that addresses these limitations by employing a lightweight, local white-box steering model to create a tractable gradient pathway. Through a prediction harmonization technique combined with consistency regularization and prompt learning-oriented filtering, BETA enables stable adaptation with no additional API calls and negligible latency beyond standard inference. On ImageNet-C, BETA achieves a +7.1% accuracy gain on ViT-B/16 and +3.4% on CLIP, surpassing strong white-box and gray-box methods including TENT and TPT. On a commercial API, BETA achieves comparable performance to ZOO at 250x lower cost while maintaining real-time inference speed, establishing it as a practical and efficient solution for real-world black-box TTA.

Abstract

VLM Adaptation

Explore similar work

Adapting in the Dark: Efficient and Stable Test-Time Adaptation for Black-Box Models

ReLoRA: Knowledge-Reusing Adaptation for Fast Rollout of Evolving LLM Services

Meta-Tool: Efficient Few-Shot Tool Adaptation for Small Language Models