Automatic prompt optimization is growing more elaborate. Recent methods stack policy-gradient updates, trace reflection, and evolutionary search. These methods share a generate-and-select step; without controlling for it, reported gains may conflate the contributions of optimizer mechanisms and larger sampling budgets. We isolate this step as Demonstration-Conditioned Prompt Search (DCPS), repurposing the Automatic Prompt Engineer (APE) recipe as a mechanism-attribution control: it freezes the generator, conditions it on randomly sampled demonstrations, scores candidates on validation data, and keeps the best---no learned update, reflection, or evolution. Across two optimizer families, DCPS replaces Proximal Policy Optimization (PPO) in StablePrompt, matching classification accuracy within 0.5 pp at roughly one-third the measured cost; in DSPy-based compound AI systems, under the reported budgets, it matches GEPA's macro accuracy on GPT-4.1-mini while using 55% fewer rollouts at 32% lower measured API cost on average. The audit reveals a task-dependent pattern: proposal and selection recover much of heavier optimizers' performance on structured and instruction-following tasks, while additional mechanisms retain advantages in some open-ended and reasoning-heavy settings. We recommend benchmarking prompt optimizers against generate-and-select controls and reporting rollout counts, token usage, and dollar costs alongside accuracy. Code and artifacts: https://github.com/bhc6/DCPS.
Figures & tables
Figure 2: Running-best SST-2 validation accuracy for StablePrompt-DCPS vs. PPO on gemma-1.1-7B-it ; stars mark the final test score at epoch 100.
Task Family
PPO
DCPS
Δ
95% CI
d
DCPS (h/$)
PPO (h/$)
Note
GLUE/SuperGLUE
76.7
77.1
−0.4
[−2.30,+1.97]
−0.14
4.83/1.93
3.97/5.96
comparable
BBII-TC
57.6
57.2
+0.4
[−0.68,+1.63]
+0.20
3.38/1.35
3.17/4.76
comparable
BBII-Gen
63.0
60.7
+2.3
[+0.15,+4.35]
+0.79
2.66/1.06
1.84/2.76
PPO better
II @30 epochs
48.7
44.3
+4.4
[+0.15,+8.70]
+0.41
2.76/1.10
6.65/9.98
PPO better
II @100 epochs
–
48.6
+0.1
[−2.67,+2.63]
+0.01
14.74/5.90
–
gap closed
MMLU
55.9
54.1
+1.8
[+0.99,+2.74]
+0.53
6.58/2.63
6.38/9.57
PPO better
Table 1: StablePrompt-PPO vs. StablePrompt-DCPS on gemma-1.1-7B-it . Scores are macro-averages over subsets; Δ (PPO − DCPS), 95% CI, and Cohen’s d are computed from per-subset differences (vs. PPO@30 epochs for extended II runs). Times and measured GPU costs are reported per run.
Method ( M )
RER
TM
log2(TM/TB)
RCEI
Rollouts
Cost
Baseline (unoptimized prompt, B )
0
1.06
0
–
–
$0.43
MIPROv2-Heavy
+12.70%
27.11
4.67
+2.66
3,936
$10.84
GEPA
+24.15%
28.36
4.73
+5.30
3,936
$11.34
GEPA-MERGE
+24.87%
24.07
4.50
+5.90
3,936
$9.63
DCPS-Compound
+26.23%
19.26
3.85
+7.20
1,761
\mathbf{\7.70}$
Table 2: Compound-system results. (a) Cost–performance summary on GPT-4.1-mini, macro-averaged across the six benchmarks ( log2(TM/TB) with T as in § 4.3 ). (b) Task-level accuracy on Qwen3-8B and GPT-4.1-mini; macro-average is the unweighted mean over six benchmarks, and the bottom block lists per-task rollout budgets (shared across backbones). In (b), LB-Math = LiveBench-Math and AIME = AIME-2025.
Benchmark
DCPS (char / tok)
GEPA
MIPROv2-Heavy
AIME-2025
1,339 / 227
6,625 / 1,330
4,460 / 748
LiveBench-Math
1,536 / 337
5,295 / 984
5,519 / 950
HoVer
2,471 / 460
12,135 / 2,309
16,401 / 2,922
HotpotQA
3,009 / 591
10,479 / 2,105
14,681 / 2,521
IFBench
1,458 / 259
6,598 / 1,254
11,027 / 1,893
PUPA
1,459 / 270
6,074 / 1,180
11,021 / 1,978
Table 3: Optimized-prompt length in characters / tokens (GPT-4.1-mini), with tokens counted using tiktoken ’s cl100k_base encoding. DCPS is shortest on every benchmark.
Task (backbone)
opt rollouts
val
test
gap
IFBench (GPT-4.1-mini)
620
96.7
51.5
45.2
IFBench (Qwen3-8B)
620
88.3
43.9
44.4
AIME-2025 (GPT-4.1-mini)
320
80.0
48.0
32.0
AIME-2025 (Qwen3-8B)
320
66.7
55.3
11.4
Table 4: Validation → test gap under validation-only selection. The gap is largest where the validation subset is small and validation accuracy saturates, and shrinks as the subset grows.
School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen · Shenzhen Future Network of Intelligence Institute (FNii-Shenzhen) · Guangdong Provincial Key Laboratory of Future Networks of Intelligence, CUHK(SZ)