Automatic prompt optimization is growing more elaborate. Recent methods stack policy-gradient updates, trace reflection, and evolutionary search. These methods share a generate-and-select step; without controlling for it, reported gains may conflate the contributions of optimizer mechanisms and larger sampling budgets. We isolate this step as Demonstration-Conditioned Prompt Search (DCPS), repurposing the Automatic Prompt Engineer (APE) recipe as a mechanism-attribution control: it freezes the generator, conditions it on randomly sampled demonstrations, scores candidates on validation data, and keeps the best---no learned update, reflection, or evolution. Across two optimizer families, DCPS replaces Proximal Policy Optimization (PPO) in StablePrompt, matching classification accuracy within 0.5 pp at roughly one-third the measured cost; in DSPy-based compound AI systems, under the reported budgets, it matches GEPA's macro accuracy on GPT-4.1-mini while using 55% fewer rollouts at 32% lower measured API cost on average. The audit reveals a task-dependent pattern: proposal and selection recover much of heavier optimizers' performance on structured and instruction-following tasks, while additional mechanisms retain advantages in some open-ended and reasoning-heavy settings. We recommend benchmarking prompt optimizers against generate-and-select controls and reporting rollout counts, token usage, and dollar costs alongside accuracy. Code and artifacts: https://github.com/bhc6/DCPS.
Figures & tables
Figure 2: Running-best SST-2 validation accuracy for StablePrompt-DCPS vs. PPO on gemma-1.1-7B-it ; stars mark the final test score at epoch 100.
Task Family
PPO
DCPS
Δ
95% CI
d
DCPS (h/$)
PPO (h/$)
Note
GLUE/SuperGLUE
76.7
77.1
−0.4
[−2.30,+1.97]
−0.14
4.83/1.93
3.97/5.96
comparable
BBII-TC
57.6
57.2
+0.4
[−0.68,+1.63]
+0.20
3.38/1.35
3.17/4.76
comparable
BBII-Gen
63.0
60.7
+2.3
[+0.15,+4.35]
+0.79
2.66/1.06
1.84/2.76
PPO better
II @30 epochs
48.7
44.3
+4.4
[+0.15,+8.70]
+0.41
2.76/1.10
6.65/9.98
PPO better
II @100 epochs
–
48.6
+0.1
[−2.67,+2.63]
+0.01
14.74/5.90
–
gap closed
MMLU
55.9
54.1
+1.8
[+0.99,+2.74]
+0.53
6.58/2.63
6.38/9.57
PPO better
Table 1: StablePrompt-PPO vs. StablePrompt-DCPS on gemma-1.1-7B-it . Scores are macro-averages over subsets; Δ (PPO − DCPS), 95% CI, and Cohen’s d are computed from per-subset differences (vs. PPO@30 epochs for extended II runs). Times and measured GPU costs are reported per run.
Method ( M )
RER
TM
log2(TM/TB)
RCEI
Rollouts
Cost
Baseline (unoptimized prompt, B )
0
1.06
0
–
–
$0.43
MIPROv2-Heavy
+12.70%
27.11
4.67
+2.66
3,936
$10.84
GEPA
+24.15%
28.36
4.73
+5.30
3,936
$11.34
GEPA-MERGE
+24.87%
24.07
4.50
+5.90
3,936
$9.63
DCPS-Compound
+26.23%
19.26
3.85
+7.20
1,761
\mathbf{\7.70}$
Table 2: Compound-system results. (a) Cost–performance summary on GPT-4.1-mini, macro-averaged across the six benchmarks ( log2(TM/TB) with T as in § 4.3 ). (b) Task-level accuracy on Qwen3-8B and GPT-4.1-mini; macro-average is the unweighted mean over six benchmarks, and the bottom block lists per-task rollout budgets (shared across backbones). In (b), LB-Math = LiveBench-Math and AIME = AIME-2025.
Benchmark
DCPS (char / tok)
GEPA
MIPROv2-Heavy
AIME-2025
1,339 / 227
6,625 / 1,330
4,460 / 748
LiveBench-Math
1,536 / 337
5,295 / 984
5,519 / 950
HoVer
2,471 / 460
12,135 / 2,309
16,401 / 2,922
HotpotQA
3,009 / 591
10,479 / 2,105
14,681 / 2,521
IFBench
1,458 / 259
6,598 / 1,254
11,027 / 1,893
PUPA
1,459 / 270
6,074 / 1,180
11,021 / 1,978
Table 3: Optimized-prompt length in characters / tokens (GPT-4.1-mini), with tokens counted using tiktoken ’s cl100k_base encoding. DCPS is shortest on every benchmark.
Task (backbone)
opt rollouts
val
test
gap
IFBench (GPT-4.1-mini)
620
96.7
51.5
45.2
IFBench (Qwen3-8B)
620
88.3
43.9
44.4
AIME-2025 (GPT-4.1-mini)
320
80.0
48.0
32.0
AIME-2025 (Qwen3-8B)
320
66.7
55.3
11.4
Table 4: Validation → test gap under validation-only selection. The gap is largest where the validation subset is small and validation accuracy saturates, and shrinks as the subset grows.
Efficiently improving autonomous agents across diverse tasks is central to accelerating recursive self-improvement (RSI) in agentic AI, with prompt optimization emerging as a promising approach capable of delivering performance gains comparable to those achieved by fine-tuning model weights, while reducing computational costs in both optimization and serving. However, recent developments increasingly favor unnecessarily complex prompt optimizers. We introduce Naive Prompt Optimization (NPO), a lightweight single-lineage method that iteratively revises prompts using a teacher model with rollout feedback. NPO achieves comparable or better performance than GEPA with fewer rollouts, and its advantage increases with stronger teacher models, suggesting that stronger teacher reasoning can partially substitute for optimizer-side search complexity. In interactive games, NPO remains broadly competitive with GEPA, while GRPO performs better on some tasks less amenable to prompt optimization. We also show that NPO-optimized prompts elicit similar performance improvements when applied verbatim to other student models, especially across models within the same family. Overall, our preliminary results show that simple, linear prompt optimization can rival substantially more sophisticated and complex search procedures.
Automatic prompt optimization (APO) has been widely employed to adapt large language models without updating their weights, yielding promising results. However, existing methods such as GEPA and OPRO assume hundreds to thousands of subject-model calls, far more than is practical behind paid, rate-limited APIs. Under tight budgets they fail in two ways: multi-stage pipelines can exhaust the budget and return the seed prompt unchanged, while single-stage methods compare candidates on fixed-size minibatches, regardless of each task's noise. As a remedy, we introduce BudgetAPO, a single-stage optimizer for the tight-budget regime. BudgetAPO incorporates (1) a noise-adaptive rule that sizes the evaluation slice to each task's noise, measured by a short probe; (2) a fixed slice that turns every accept/reject decision into a paired comparison; and (3) a reflective operator that rewrites reasoning strategy and output format jointly. Extensive results across seven benchmarks and five subject models demonstrate that BudgetAPO ranks first on every subject and beats every baseline under Holm-corrected paired tests, while returning the seed in 13% of runs at 250 calls against 86% for GEPA. On GPT-OSS-20B, GEPA needs 5 times as many calls to match \method's 100-call score.
Haoyue Liu, Zhichao Wang, Huanyu Yan +1
School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen · Shenzhen Future Network of Intelligence Institute (FNii-Shenzhen) · Guangdong Provincial Key Laboratory of Future Networks of Intelligence, CUHK(SZ)
Automated prompt optimization is crucial for eliciting reliable reasoning from large language models (LLMs), yet most API-only prompt optimizers iteratively edit monolithic prompts, coupling components and obscuring credit assignment, limiting controllability, and wasting tokens. We propose Adaptive Prompt Structure Factorization (aPSF), an API-only framework (prompt-in/text-out; no access to model internals) that uses an Architect model to discover task-specific prompt structures as semantic factors. aPSF then performs interventional, single-factor updates: interventional factor-level scoring estimates each factor's marginal contribution via validation-performance changes, and error-guided factor selection routes updates to the current dominant failure source for more sample-efficient optimization. Across multiple advanced reasoning benchmarks, aPSF outperforms strong baselines including principle-aware optimizers, improving accuracy by up to +2.16 percentage points on average, and reduces optimization cost by 45--87% tokens on MultiArith while reaching peak validation in 1 step.
Haoyue Liu, Zhichao Wang, Yongxin Guo +2
The Chinese University of Hong Kong, Shenzhen · Taobao and Tmall Group