Reflective prompt optimization revises instructions using examples of a model's behavior, but which evidence the reflector should receive remains unclear. We study evidence composition, visibility of examples, candidate selection and domain-knowledge policy within a single-parent Pareto-guided search. Using Qwen3.5-9B as both task model and reflector, we evaluate nine reflection strategies on five datasets. From these experiments, we find three distinct patterns. For performance improvement, Failures-only produces the largest mean test gain (+8.0 percentage points), while Balanced-mix and No-examples+Val share the best mean performance rank. For reflection effectiveness, No-examples achieves the best rank for improving sampled parents, yet yields only a 1.4-point mean test gain: local reflection success does not necessarily produce a stronger final prompt. For overfitting assessment, Failures-only and Balanced-mix share the lowest mean calibration-gap rank, while larger gaps on GPQA and IFBench show that calibration gains can overstate held-out improvement. This gap is a descriptive indicator, not a direct measure of overfitting. Together, these results show why reflection strategies should be assessed separately on final performance, parent improvement and calibration-to-test transfer.
Figures & tables
Figure 1: Three-dimensional strategy profiles: mean within-dataset ranks over five datasets, with ties sharing ranks. All axes use the same linear, zoomed range 2.5–9; outer is better. Test gain ranks high-to-low; calibration gap (calibration gain minus test gain) ranks low-to-high. Reflection effectiveness uses Rank/Parent from Table 5 : lower rounds per parent improvement rank better, with zero-improvement runs assigned 20 rounds. Calibration gap is descriptive, not a direct measure of overfitting. Colors encode evidence composition (blue), example visibility (orange), judging (green) and knowledge policy (purple); line styles distinguish strategies.
Dataset
Domain
Splits
Metric
LiveBench-Math
competition math
98 / 98 / 98
equivalence
OlympiadBench
competition math
100 / 150 / 150
equivalence
GPQA
graduate-level science
100 / 150 / 150
option accuracy
HotpotQA
multi-hop QA
100 / 150 / 150
exact match
IFBench
instruction following
100 / 100 / 100
constraints satisfied
Table 1: Datasets. Sizes are training / calibration / test.
Strategy
LiveBench-Math
OlympiadBench
GPQA
HotpotQA
IFBench
Avg Acc
Acc
Acr
Bcr
It.
Acc
Acr
Bcr
It.
Acc
Acr
Bcr
It.
Acc
Acr
Bcr
It.
Acc
Acr
Bcr
It.
Seed prompt
34.0
–
–
–
62.7
–
–
–
65.3
–
–
–
37.6
–
–
–
30.0
–
–
–
45.9
Failures-only
67.7
7.5
63.8
18
70.2
2.2
28.1
13
62.2
19.6
26.4
4
36.9
0.0
0.0
0
32.3
19.2
16.2
16
53.9
Natural-mix
47.3
17.5
34.5
5
62.9
0.0
0.0
0
64.9
10.3
34.0
10
40.2
7.3
8.4
6
32.5
19.2
10.8
1
49.6
Balanced-mix
62.2
25.0
51.7
14
68.9
5.4
21.1
19
62.7
16.5
30.2
2
36.9
0.0
0.0
0
36.2
11.5
12.2
15
53.4
No-examples
40.8
20.0
15.5
17
67.8
2.2
21.1
10
61.8
17.5
34.0
10
36.7
0.0
0.0
0
29.5
42.3
14.9
6
47.3
Table 2: Results on five datasets. Acc: test accuracy (%), averaged over three evaluations. Avg Acc: equal-weight mean of the five dataset accuracies. Acr ↓ / Bcr ↑ : adverse / beneficial correction rates (%) from the initial paired test evaluation, with full correctness defined as score 1. It.: iteration at which the final prompt was found (0: seed retained). Best strategy in bold, second-best distinct value underlined for each accuracy/correction column; ties share highlighting.
Figure 2: Mean within-dataset ranks over five datasets (1 is best, average ranks for ties). Accuracy uses the three-evaluation mean. Gap is calibration gain minus mean test gain. Reflection quality uses Rank/Parent from Table 5 : rank the rounds per strict parent improvement, assigning 20 rounds if none occur. Colors encode evidence composition (blue), example visibility (orange), judging (green) and knowledge policy (purple), consistently across panels. Smaller gaps or better parent-improvement ranks alone do not establish a better optimizer.
Comparison
LiveBench
Olympiad
GPQA
HotpotQA
IFBench
Mean
Win
Balanced vs. natural
+15.0
+6.0
-2.2
-3.3
+3.7
+3.8
3/5
Balanced vs. failures
-5.4
-1.3
+0.4
+0.0
+3.8
-0.5
2/5
Natural vs. no-examples+val
+1.0
-5.3
+1.6
+2.7
-4.0
-0.8
3/5
No-examples+val vs. no-examples
+5.4
+0.4
+1.6
+0.9
+7.0
+3.1
5/5
Alternating vs. natural
-0.7
+3.3
-2.2
+5.1
+2.0
+1.5
3/5
Natural vs. LLM-judge
+4.8
+0.0
-1.6
+4.0
+0.2
+1.5
3/5
Table 3: Strategy differences in test accuracy (percentage points), first minus second, averaged over three evaluations. Bold indicates the same nonzero sign in all three evaluations, not statistical significance. Win counts strictly positive dataset means. Controls use Natural-mix unless evidence composition itself is varied. The no-examples selection comparison also changes the local calibration gate.
Strategy
LiveBench-Math
OlympiadBench
GPQA
HotpotQA
IFBench
Avg
Cal.
Test
Gap ↓
Cal.
Test
Gap ↓
Cal.
Test
Gap ↓
Cal.
Test
Gap ↓
Cal.
Test
Gap ↓
Cal.
Test
Gap ↓
Failures-only
24.5
33.7
-9.2
2.0
7.6
-5.6
4.7
-3.1
7.8
0.0
-0.7
0.7
8.5
2.3
6.2
7.9
8.0
0.0
Natural-mix
13.3
13.3
0.0
0.0
0.2
-0.2
6.7
-0.4
7.1
2.0
2.7
-0.7
10.0
2.5
7.5
6.4
3.6
2.7
Balanced-mix
16.3
28.2
-11.9
3.3
6.2
-2.9
4.7
-2.7
7.3
0.0
-0.7
0.7
10.0
6.2
3.8
6.9
7.5
-0.6
No-examples
6.1
6.8
-0.7
4.0
5.1
-1.1
5.3
-3.6
8.9
0.0
-0.9
0.9
9.5
-0.5
10.0
5.0
1.4
3.6
No-examples+Val
12.2
12.2
0.0
2.0
5.6
-3.6
4.0
-2.0
6.0
0.0
0.0
0.0
16.0
6.5
9.5
6.8
4.5
2.4
Table 4: Calibration-to-test transfer (percentage points). Cal.: calibration gain; Test: mean test gain over three evaluations; Gap: Cal. minus Test (lower is better), computed before rounding. The Avg columns average five datasets; the Avg row averages nine strategies. Across all 45 runs, Pearson r=0.70 . Comparisons use equal iterations, not equal compute.
Strategy
Rounds/Parent
Rounds/Best
Rank/Parent
Rank/Best
Failures-only
3.2
10.3
3.4
3.8
Natural-mix
4.1
15.3
4.8
6.0
Balanced-mix
3.7
11.8
3.8
3.9
No-examples
3.4
10.7
3.1
4.1
No-examples+Val
7.5
14.7
6.0
5.2
Alternating
3.8
12.8
4.4
4.7
Table 5: Rounds per calibration improvement, averaged equally over five datasets. For each run, divide its 20 search rounds by the number of strict improvements; assign 20 if none occur. All improvements count, including those after the first, and rounds without calibration remain in the budget. Parent/Best compare against the sampled parent / previous best prompt. Rank is computed within each dataset and then averaged, with average ranks for ties; lower is better.