Adapting closed-box service models (i.e., APIs) for target tasks typically relies on reprogramming via Zeroth-Order Optimization (ZOO). However, this standard strategy is known for extensive, costly API calls and often suffers from slow, unstable optimization. Furthermore, we observe that this paradigm faces new challenges with modern APIs (e.g., GPT-4o). These models can be less sensitive to the input perturbations ZOO relies on, thereby hindering performance gains. To address these limitations, we propose an Alternative efficient Reprogramming approach for Service models (AReS). Instead of direct, continuous closed-box optimization, AReS initiates a single-pass interaction with the service API to prime an amenable local pre-trained encoder. This priming stage trains only a lightweight layer on top of the local encoder, making it highly receptive to the subsequent glass-box (white-box) reprogramming stage performed directly on the local model. Consequently, all subsequent adaptation and inference rely solely on this local proxy, eliminating all further API costs. Experiments demonstrate AReS's effectiveness where prior ZOO-based methods struggle: on GPT-4o, AReS achieves a +27.8% gain over the zero-shot baseline, a task where ZOO-based methods provide little to no improvement. Broadly, across ten diverse datasets, AReS outperforms state-of-the-art methods (+2.5% for VLMs, +15.6% for standard VMs) while reducing API calls by over 99.99%. AReS thus provides a robust and practical solution for adapting modern closed-box models.
Figures & tables
Figure 1: (a) On the real-world GPT-4o API, ZOO-based methods show limited effectiveness, providing little to no improvement over zero-shot performance while incurring high total (training and inference) costs. (b, c) On CLIP ViT-B/16, these methods require ∼ 10 8 API calls and over 32 hours of training on Flowers102, yet still underperform ours, AReS, which uses only ∼ 10 3 API calls and 0.4 hours.
Figure 2: (a) Previous closed-box methods use Zeroth-Order Optimization (ZOO), which estimates gradients by querying the API multiple times for the same training image with random perturbations. This results in continuous API dependency: numerous calls during training and one API call per image during inference. (b) Our AReS approach performs a single-pass priming, requiring only one API call per training image to prepare a local model. This enables efficient, gradient-aware VR locally, eliminating all subsequent API costs during inference.
Table 1: Comparison of tuning methods for Model-as-a-Service (MaaS). The Access column indicates the strictness, distinguishing translucent-box (grey, requiring intermediate Features or Embedding access) from closed-box (black, Input/Predictions only) settings. We compare methods by: MR (Model Reprogramming-based); required Input Access and Output Access; support for VMs (Standard Vision Models) and VLMs (Vision-Language Models); Prompt type (L: Language, V: Visual); and Encoder Flexibility. Our method, AReS, uniquely offers Cost-free Inference for both VMs and VLMs while operating under the strictest closed-box constraints.
Method
Flowers
DTD
UCF
Food
GTSRB
EuroSAT
Pets
Cars
SUN
SVHN
Avg.
#API (M)
Time (h)
VR (glass-box)
86.8
62.0
74.0
81.6
65.1
90.9
90.7
66.2
66.7
60.2
74.4
16.82
9.8
Zero-shot
71.3
43.9
66.9
85.9
21.0
47.9
89.1
65.2
62.6
17.9
57.2
0.12
0
BAR
71.0
46.8
64.2
84.4
21.5
77.3
88.4
63.0
62.4
34.6
61.4
612.84
185.6
BlackVIP
70.6
45.3
68.7
85.9
21.3
73.3
89.1
65.4
64.5
44.4
62.9
754.20
197.5
LLM-Opt
67.6
45.0
59.9
78.5
21.2
48.0
87.7
56.2
60.3
20.2
54.5
5011.32
5.1
AReS
86.6
48.2
67.1
68.8
39.4
85.7
88.9
43.2
62.8
63.2
65.4
0.02
3.7
Table 2: Accuracy and Efficiency comparison on ten diverse visual recognition datasets using CLIP ViT-B/16 (Service) and ViT-B/16 (Local) in a 16-shot setting. AReS achieves superior average accuracy with a drastic reduction in resources (99.99% fewer API calls and 98% less training time). For this and all subsequent tables, Gray denotes glass-box Service VR results. #API (M): total API calls in millions (training and inference); Wall-clock Time (h): total training hours run on an RTX 6000 Ada GPU.
Method
Flowers
DTD
UCF
Food
GTSRB
EuroSAT
Pets
Cars
SUN
SVHN
Avg.
#API (M)
Time (h)
VR (glass-box)
69.6
52.1
49.7
38.7
60.6
96.7
77.1
6.2
34.0
83.3
56.8
43.4
39.6
BAR
14.8
24.3
29.5
12.2
14.1
43.1
24.9
1.0
10.6
29.6
20.4
1,724.2
668.2
BlackVIP (RN50)
15.2
42.3
34.0
14.5
13.9
67.3
63.6
3.7
22.6
30.8
30.8
2,586.2
686.9
AReS (RN50)
43.5
43.4
39.4
25.9
48.7
84.3
73.9
5.8
19.9
73.7
45.9
0.2
8.7
BlackVIP (ViT-B/32)
27.4
44.1
36.6
18.5
38.4
59.6
65.4
3.9
23.7
30.3
34.8
2,586.2
757.6
AReS (ViT-B/32)
53.6
46.9
41.4
27.3
58.8
95.6
65.0
4.7
26.9
83.5
50.4
0.2
22.4
Table 3: Accuracy and Efficiency comparison using ViT-B/16 (Service) in Full-shot setting.
Method
LLaVA
GPT-4o
Clarifai
Acc. ↑
Train
Infer.
Total (#) ↓
Acc. ↑
Train
Infer.
Total ()\downarrow$
Acc. ↑
Train
Infer.
Total ()\downarrow$
glass-box
91.3
–
–
–
–
–
–
–
–
–
–
–
Zero-shot
40.1
–
8100
8100
59.4
–
14.6
14.6
–
–
–
–
BAR
34.1
∼106
8100
∼106
59.1
57.6
14.6
72.2
68.1
38.4
9.7
48.1
BlackVIP
39.4
∼107
8100
∼107
60.1
86.4
14.6
101.0
72.1
57.6
9.7
67.3
AReS
73.1
160
0
160
87.2
0.3
0
0.3
83.2
0.2
0
0.2
Table 4: Performance and Cost comparison on the open-source MLLM LLaVA, the proprietary API GPT-4o, and the commercial VM API Clarifai, evaluated on the EuroSAT 16-shot benchmark. A zero-shot baseline is not applicable for Clarifai as it is a closed-vocabulary model. Costs are shown in total API calls (#) or US dollars ($). AReS demonstrates significant gains where ZOO-based methods struggle.
Figure 3: Ablation studies for AReS on the EuroSAT dataset: impact of (a) training data fraction (for VMs), (b) few-shot sample size (for VLMs), (c) priming loss function, and (d) amount of extra unlabeled data for priming on accuracy (%). Default setting involves 16-shot learning, CLIP ViT-B/16 Service, and ViT-B/16 Local encoder unless otherwise indicated by the subplot analysis.
Method
Service Backbone
RN50
RN101
ViT-B/32
ViT-B/16
Avg.
ZS
37.5
32.6
45.2
47.9
40.8
BAR
26.9
33.5
70.3
77.2
52.0
BlackVIP (RN50)
51.3
50.8
62.9
68.5
58.4
AReS (RN50)
62.2
68.5
63.2
78.6
68.1
BlackVIP (ViT-B/16)
48.4
51.3
67.9
73.3
60.2
Table 5: Ablation study across Service/Local model backbones.
Method
Priming
VR
LP
Accuracy
Zero-shot
–
–
–
47.9
Primed
✓
–
–
45.6
Local VR
–
✓
–
70.6
Local LP
–
–
✓
73.8
Local VR+LP
–
✓
✓
80.1
AReS (Ours)
✓
✓
–
85.7
Table 6: Component analysis of Priming, Visual Reprogramming (VR), and Linear Probing (LP).
Figure 4: Illustration of Zeroth-Order Optimization (ZOO) techniques commonly used in closed-box model reprogramming. (a) BAR with Randomized Gradient-Free (RGF) method estimates gradients by querying the model with random directional perturbations. (b) BlackVIP with Simultaneous Perturbation Stochastic Approximation with Gradient Correction (SPSA-GC) approximates gradients using only two model queries with a randomly generated perturbation vector. Both approaches suffer from high query complexity and noisy gradient estimates, leading to unstable and computationally intensive optimization, especially for high-dimensional prompts.
Dataset
Full-shot Training
16-shot Training
Testing Set Size
Number of Classes
Flowers102
4,093
1,632
2,463
102
DTD
2,820
752
1,692
47
UCF101
7,639
1,616
3,783
101
Food101
50,500
1,616
30,300
101
GTSRB
39,209
688
12,630
43
EuroSAT
13,500
160
8,100
10
Table 7: Detailed dataset information.
Figure 5: Comparison of closed-box visual reprogramming approaches. (a) BAR and (b) BlackVIP rely on Zeroth-Order Optimization (ZOO) by repeatedly querying the service model with perturbed inputs (e.g., using random directions) to estimate gradients for updating the visual prompt. These methods suffer from high API call costs and potentially unstable optimization. (c) Our AReS method performs a one-time priming from the service model to a local model. Subsequent visual prompt optimization occurs efficiently on this local model using glass-box gradients, eliminating further API calls and enabling stable, cost-effective adaptation.
Method
Flowers
DTD
UCF
Food
GTSRB
EuroSAT
Pets
Cars
SUN
SVHN
Avg.
#API (M)
Time (h)
VR (glass-box)
42.3
44.4
34.4
25.4
53.1
85.0
73.9
5.0
20.5
75.1
45.9
43.4
21.9
BAR
16.9
25.3
30.2
13.9
22.5
45.7
28.7
1.5
10.7
42.1
23.8
1,724.2
313.5
BlackVIP (RN50)
21.7
39.6
31.4
14.9
26.8
68.3
61.4
3.5
12.3
42.6
32.3
2,586.2
320.5
AReS (RN50)
41.3
42.3
37.2
25.3
48.4
84.3
73.7
5.1
19.7
72.9
45.0
0.2
8.7
BlackVIP (ViT-B/32)
26.7
37.7
30.2
13.3
33.8
67.3
57.9
3.0
14.5
34.5
31.9
2,586.2
418.0
AReS (ViT-B/32)
51.3
45.8
41.5
30.2
57.9
94.9
63.3
5.1
25.6
83.6
49.9
0.2
22.3
Table 8: Accuracy and Efficiency comparison using RN101 (Service) in Full-shot setting.
Method
Flowers
DTD
UCF
Food
GTSRB
EuroSAT
Pets
Cars
SUN
SVHN
Avg.
VR (glass-box)
55.0
44.7
42.0
18.0
17.4
70.6
70.7
5.7
29.6
28.4
38.2
BAR
9.7
16.3
23.5
8.1
4.8
34.2
21.5
1.0
7.6
19.6
14.6
BlackVIP
16.1
36.1
33.8
12.5
9.1
54.9
54.5
3.9
22.1
16.2
25.9
AReS (Ours)
46.0
35.5
34.3
12.7
19.6
71.1
54.3
4.2
23.0
27.3
32.8
Table 9: Accuracy comparison using ViT-B/16 (Service) and ViT-B/32 (Local) in 16-shot setting.
Method
Flowers
DTD
UCF
Food
GTSRB
EuroSAT
Pets
Cars
SUN
SVHN
Avg.
Zero-shot
71.3
43.9
66.9
85.9
21.0
47.9
89.1
65.2
62.6
17.9
57.2
Local VR
55.0
44.7
42.0
18.0
17.4
70.6
70.7
5.7
29.6
28.4
38.2
AReS (Ours)
86.6
48.2
67.1
68.8
39.4
85.7
88.9
43.2
62.8
63.2
65.4
Table 10: Accuracy comparison between Local VR (ViT-B/16) and our method on VLMs using CLIP ViT-B/16 (Service) and ViT-B/16 (Local) in a 16-shot setting.
Service
Local
Local VR
Ours
ViT-B/16
ViT-B/32
45.3
50.4
RN50
43.9
45.9
RN101
ViT-B/32
45.3
49.9
RN50
43.9
45.0
Table 11: Accuracy comparison between Local VR and our method on VMs in Full-shot setting.
Method
# Trainable Params
Accuracy (%)
VP w/ SPSA-GC
69K
70.9
BAR
37K
77.3
BlackVIP
9K
73.3
AReS (Ours)
21K
85.7
Table 12: Comparison of trainable parameters and accuracy on the EuroSAT dataset. More parameters do not correlate with better performance for ZOO-based methods.
Method
Accuracy (%)
Local VR
70.6
BlackVIP
73.3
BlackVIP w/ Local Tuning
74.1
AReS (Ours)
85.7
Table 13: Comparison with an enhanced BlackVIP baseline on the EuroSAT dataset. The results confirm that AReS’s performance gain is primarily from its superior two-stage framework.
Method
4-shot
8-shot
16-shot
32-shot
BlackVIP
69.3
71.7
73.3
72.9
Local VR+LP
59.7
68.1
80.1
86.5
AReS (Ours)
66.1
73.4
85.7
91.6
Table 14: Accuracy (%) comparison on the EuroSAT dataset across different few-shot settings. AReS consistently outperforms the strong local baseline (Local VR+LP) as data availability increases.
Initial State
Service: RN101 (42.3%)
Service: ViT-B/16 (69.6%)
% of Dataset
How AReS Performed
% of Dataset
How AReS Performed
Both Service & Local Correct
30.0%
29.5% kept correct
35.0%
34.0% kept correct
Only Local Correct
8.7%
7.0% kept correct
3.7%
0.5% kept correct
Only Service Correct
12.3%
10.0% newly learned
34.6%
19.0% newly learned
Both Incorrect
49.0%
4.8% newly learned
26.7%
0.1% newly learned
AReS Final Accuracy
51.3%
53.6%
Table 15: Illustrative Breakdown of AReS’s Performance Composition on Flowers102. The local model for both scenarios is ViT-B/32 (glass-box Acc: 38.7%). “Kept correct” refers to samples the local model classified correctly that AReS also classifies correctly. “Newly learned” refers to samples the local model misclassified that AReS now classifies correctly.
Method
ImageNet Acc (%)
Gain over Zero-shot (%)
VR (glass-box)
67.4
+0.7
Zero-shot
66.7
-
BAR
64.6
-2.1
VP w/ SPSA-GC
62.3
-4.4
BlackVIP
67.1
+0.4
AReS (Ours)
80.1
+13.4
Table 16: Accuracy (%) comparison on ImageNet in a 16-shot setting with CLIP ViT-B/16 as the service model. AReS significantly outperforms all baselines.
Figure 6: An example of GPT-4o’s robustness to the input perturbations used in reprogramming. When presented with a clean image (left) and a prompted image (right), GPT-4o’s textual reasoning explicitly acknowledges the visual alteration but correctly identifies the flower’s key features in both cases. This demonstrates that the model is often invariant to the input noise that ZOO-based methods rely on, motivating a strategic shift in adaptation methods.
Method
GPU Mem. (MB)
Latency (ms)
Acc. (%)
VR (glass-box)
11,937
-
90.9
Zero-shot
0
120
47.9
BAR
1,649
122
77.3
BlackVIP
2,428
170
73.3
AReS (Ours)
2,781
60
85.7
Table 17: GPU memory and inference latency comparison on EuroSAT with NVIDIA RTX 3090. AReS achieves the highest accuracy and fastest inference by running entirely locally after priming.
Method
Zero-shot
BlackVIP
AReS (Ours)
Acc. (%)
62.5
63.2
63.6
Table 18: Domain gap experiment: DermCLIP (pretrained on Derm1M) → ViT-B/16 (pretrained on ImageNet) on HAM10000 under the 16-shot setting.
Top- k
1
2
5
all (10)
Soft
80.1
82.3
85.0
85.7
Hard
70.7
74.6
78.9
80.6
Table 19: Robustness to partial API outputs (top- k ) on EuroSAT. “Soft” uses top- k class names with confidence scores; “Hard” uses only the ordered top- k list.
Vision-language model adaptation tailors pretrained image-text and multimodal models to new tasks and domains while retaining transferable capabilities.
Test-Time Adaptation (TTA) for black-box models accessible only via APIs remains a largely unexplored challenge. Existing approaches such as post-hoc output refinement offer limited adaptive capacity, while Zeroth-Order Optimization (ZOO) enables input-space adaptation but faces high query costs and optimization challenges in the unsupervised TTA setting. We introduce BETA (Black-box Efficient Test-time Adaptation), a framework that addresses these limitations by employing a lightweight, local white-box steering model to create a tractable gradient pathway. Through a prediction harmonization technique combined with consistency regularization and prompt learning-oriented filtering, BETA enables stable adaptation with no additional API calls and negligible latency beyond standard inference. On ImageNet-C, BETA achieves a +7.1% accuracy gain on ViT-B/16 and +3.4% on CLIP, surpassing strong white-box and gray-box methods including TENT and TPT. On a commercial API, BETA achieves comparable performance to ZOO at 250x lower cost while maintaining real-time inference speed, establishing it as a practical and efficient solution for real-world black-box TTA.
Yunbei Zhang, Shuaicheng Niu, Chengyi Cai +2
1Tulane University · 2Nanyang Technological Univer · 3The University of Melbourne.
Large Language Models (LLMs) are increasingly deployed as continuously evolving services, where frequent base-model updates may invalidate previously deployed task-specific Low-Rank Adaptation (LoRA) adapters. For service providers managing numerous downstream model services, retraining each LoRA adapter from scratch for every updated base model is computationally prohibitive and delays service rollout. Meanwhile, the simpler alternative, i.e., naively applying the original LoRA adapter to the updated base model, often leads to degraded service quality due to adapter-backbone incompatibility. To address this problem, we propose ReLoRA, a knowledge-reusing re-adaptation framework that efficiently restores service-ready LoRA adapters for evolving LLM services while preserving or improving task performance. Specifically, ReLoRA comprises two key optimization steps: 1) Adaptive LoRA initialization leverages Bayesian optimization to construct a compatibility-aware starting point by fusing information from both the previously deployed task adapter and the base model's evolution; 2) Fine-tuning with scheduled regularization first rapidly steers the adapter to a high-quality region via strong regularization, followed by relaxed regularization for task-specific refinement. This design enables rapid service-quality recovery with reduced re-adaptation overhead. Extensive experiments demonstrate that ReLoRA reduces time-to-readiness by up to 8.9× and improves accuracy by up to 4.6% compared to baselines.
Yang Xu, Zihuai Xu, Hongli Xu +3
Suzhou Institute for Advanced Research, University of Science and Technology of China, Suzhou, Jiangsu, China, 215123
Can small language models achieve strong tool-use performance without complex adaptation mechanisms? This paper investigates this question through Meta-Tool, a controlled empirical study comparing hypernetwork-based LoRA adaptation against carefully designed few-shot prompting. Using a Llama-3.2-3B-Instruct backbone, we evaluate four adaptation mechanisms--few-shot prompting, documentation encoding, hypernetwork-generated LoRA weights, and value-guided beam search--across four diverse benchmarks: Gorilla APIBench, Spider 2.0, WebArena, and InterCode. Our central finding is a well-supported negative result: despite generating non-trivial weight matrices, the 227.8M-parameter hypernetwork provides no measurable improvement over few-shot prompting alone. Comprehensive ablation studies reveal that few-shot examples contribute +21.5% to performance and documentation contributes +5.0%, while the hypernetwork adds 0%. A 3B model with well-designed prompts achieves 79.7% of GPT-5's average performance at 10× lower latency. Error analysis across 722 failure cases spanning all shot counts (0--5) shows that at the 5-shot configuration (106 failures), failure modes are task-dependent: schema-heavy tasks (Spider 2.0, WebArena) show near-zero format errors with remaining failures semantic, while format errors dominate on Gorilla (100%) and InterCode (70%). These findings redirect practitioners toward prompt engineering and example curation rather than complex adaptation architectures.