Beyond Aggregate Accuracy: Item-Level Transitions in Prompt Optimization
Authors: Chinmoy Mitra, Tarek Mahmud
Organizations: Department of Computer Science & Engineering, Rajshahi University of Engineering & Technology, Rajshahi, Bangladesh · Texas A&M University–Kingsville, Kingsville, Texas, USA
Automatic prompt optimization (APO) uses feedback from model responses to improve prompts automatically. However, its effectiveness is usually measured through overall scores, which do not reveal which questions become correct, which previously correct answers become wrong, or whether improvements extend to unseen questions. We introduce Reflective Feedback-Driven Prompt Optimization (RFDPO), a tool that tracks recoveries (incorrect to correct), regressions (correct to incorrect), and churn (all changed outcomes). RFDPO uses this feedback to guide subsequent revisions and records prompt changes throughout optimization. We evaluate RFDPO across five benchmarks covering law, knowledge, instruction following, mathematics, and privacy, using GPT-5 as the optimizer and GPT-4o-mini and GPT-4.1 Mini as solvers. Our results show that overall accuracy can hide substantial changes: in five of nine configurations, churn was at least five times the absolute net change, and a single-item net change concealed up to 17 changed outcomes. Average validation gains were 7.9 percentage points, compared with 5.2 points on held-out tests. In a controlled ablation on LegalBench-Hearsay, item-level feedback reduced churn by 40% (p=0.024) while maintaining similar net gains. These findings motivate reporting recovery, regression, churn, and performance on unseen data alongside overall accuracy.
Figures & tables
Figure 1: The RFDPO loop. The solver evaluates prompts on mining set M and validation set V . From the second round, the optimizer receives the item-level effect of its previous edit before proposing a new prompt. Every changed candidate is committed, and RFDPO returns the candidate with the highest validation accuracy, breaking ties by mining accuracy. Test set T is excluded from feedback and selection and evaluated only for the seed and the returned prompt.
Figure 2: Example of item-level transitions. Three examples are recovered and two regress, producing a net gain of one correct example and a churn of five.
Benchmark
Task
Reported metric
Train →M / V
Test
LegalBench-Hearsay
Binary classification
Accuracy
50→25/25
49
MMLU (6 subjects)
Multiple choice
Accuracy, macro-avg.
50→25/25
66 each
IFBench
Instruction following
Accuracy
40→26/14
42
AIME (2022–24 → 2025)
Mathematical reasoning
Accuracy
90→58/32
30
PUPA
Privacy-conscious delegation
Accuracy (composite ≥0.7 )
60→30/30
40
Table 1: Benchmarks and evaluation splits. MMLU splits are per subject. Test sizes precede provider-block exclusions.
Benchmark
Solver
Test accuracy (%)
ΔV (pp)
Test transitions
Seed
Final
ΔT
∣R∣
∣G∣
Churn (%)
AIME
GPT-4.1 Mini
53.3
50.0
−3.3
+6.2
3
4
7 (23.3)
PUPA
GPT-4o-mini
55.3
65.8
+10.5
+13.3
7
3
10 (26.3)
PUPA
GPT-4.1 Mini
44.7
71.1
+26.3
+23.3
10
0
10 (26.3)
IFBench
GPT-4o-mini
40.5
45.2
+4.8
+15.8
6
4
10 (23.8)
IFBench
GPT-4.1 Mini
42.9
66.7
+23.8
+14.3
12
2
14 (33.3)
Table 2: Seed-to-returned test accuracy, validation accuracy change, and item-level transitions. Green rows indicate positive test changes. Churn percentages use evaluated test items. MMLU accuracy is macro-averaged across six subjects, while transitions sum 393 items. PUPA (38 items) and professional law (63 items) exclude provider-blocked examples. Earlier AIME runs with other solvers are described in Appendix C.5 .
Figure 3: Recovered and regressed test examples across benchmarks and solvers. Diamonds indicate net changes. MMLU transitions are aggregated across six subjects.
Edited-prompt returns
Runs
Val. ↑
Val. ↑ test ↓
ΔV
ΔT
Non-MMLU
7
7
2
+15.0
+8.9
MMLU subjects
10
5
1
+2.9
+2.6
All matched
17
12
3
+7.9
+5.2
Table 3: Mean validation and test accuracy changes for 17 edited-prompt returns; two additional records retained the seed. Means weight runs equally and are reported in percentage points. The fourth column counts runs with improved validation accuracy but decreased test accuracy.
Benchmark
Solver
Optimizer calls
Optimizer input (K)
Optimizer output (K)
All-role tokens (K)
Logged cost (USD)
AIME
GPT-4.1 Mini
3
498.4
5.4
2,666.6
3.81
PUPA
GPT-4o-mini
3
73.8
13.7
1,651.2
4.44
PUPA
GPT-4.1 Mini
3
68.5
15.6
1,570.4
4.80
IFBench
GPT-4o-mini
3
23.4
12.0
300.9
0.21
IFBench
GPT-4.1 Mini
3
20.2
10.1
240.1
0.35
Hearsay
GPT-4o-mini
3
17.1
14.9
241.3
0.22
Table 4: Inference consumption across matched configurations. K denotes 1,000 tokens. All-role totals include solver, optimizer, and applicable external-model and judge calls, including seed and final evaluations. Costs use configured prices. MMLU aggregates six subjects; two GPT-4.1 Mini subject runs skipped optimization.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Benchmark
Solver
Assigned M/V
Val. fraction
Solver limit
AIME
4.1 Mini
58/32
0.35
16,000
PUPA
4o-mini
30/30
0.50
2,048
PUPA
4.1 Mini
30/30
0.50
2,048
IFBench
4o-mini
26/14
0.35
2,048
IFBench
4.1 Mini
26/14
0.35
4,096
Hearsay
both
25/25
0.50
2,048
Appendix
Table 5: Recorded settings for the matched runs. Solver limit is the configured maximum output tokens, not observed consumption. Assigned pools precede provider-block exclusions: GPT-4o-mini IFBench evaluates 19 validation items and GPT-4.1 Mini evaluates 25 mining items. Actual resource use is in Table 4 .
Work
Items: optimization / validation / test
Solver models
Feedback and optimizer
Reported change (%)
LegalBench-Hearsay
Trace2Policy, human-written refinements ( Zha et al., 2026 )
30 iteration / 64 held-out, from 94 items
Claude Opus 4.6, Haiku 4.5, Kimi K2.5
human curator; round 1 also read held-out errors
Haiku 4.5: 79.7→93.8
Trace2Policy, LLM-written refinements ( Zha et al., 2026 )
61 test items
DeepSeek-v3.2, Kimi K2.5, Kimi K2.6 (mean)
LLM, with a regression gate
69.4→72.7
Chen et al. ( 2025 ) (definition classification, a different LegalBench task)
100 train / 100 test (50 / 50 if under 200 items)
LLaMA-3-8B, LLaMA-3.1-8B, GPT-4o
evaluator-guided rewriting, up to 3 iterations
55→70 ; 56→69 ; 83→90
This paper
25 / 25 / 49, recut from 99 items
GPT-4o-mini, GPT-4.1 Mini
GPT-5, item-level effect report
71.4→69.4 ; 71.4→73.5
MMLU
Appendix
Table 6: Published evaluation protocols on shared benchmarks. Rows differ in data, solver, baseline prompt, feedback access, and evaluation budget. Changes follow solver-column order; PUPA changes are composite scores, as reported by GEPA.
Run
Solver
Edited
Validation (%) seed → returned
ΔV
Test (%) seed → returned
ΔT
Rec./Reg.
AIME
4.1 Mini
yes
62.5 → 68.8
+6.2
53.3 → 50.0
−3.3
3/4
PUPA
4o-mini
yes
40.0 → 53.3
+13.3
55.3 → 65.8
+10.5
7/3
PUPA
4.1 Mini
yes
46.7 → 70.0
+23.3
44.7 → 71.1
+26.3
10/0
IFBench
4o-mini
yes
36.8 → 52.6
+15.8
40.5 → 45.2
+4.8
6/4
IFBench
4.1 Mini
yes
57.1 → 71.4
+14.3
42.9 → 66.7
+23.8
12/2
Hearsay
4o-mini
yes
68.0 → 80.0
+12.0
71.4 → 69.4
−2.0
8/9
Appendix
Table 7: All matched accuracy pairs and test transition counts. Deltas are percentage points computed before rounding; green rows have positive test deltas. MMLU subjects have 66 assigned test items (63 evaluated for professional law). Dashes denote unmeasured validation after a high-baseline skip; zero ΔV denotes the retained seed.
Benchmark
Solver
Round
M acc.
V acc.
M rec./reg.
V rec./reg.
Sections changed
AIME
4.1 Mini
0
46.6
62.5
–
–
–
1 ∗
51.7
68.8
9/6
5/3
Con, TD
2
48.3
65.6
6/8
2/3
TD
3
46.6
59.4
6/7
3/5
TD
PUPA
4o-mini
0
69.0
40.0
–
–
–
1
79.3
56.7
6/3
5/0
Con, Ctx, TD
Appendix
Table 8: Round-by-round trajectories (accuracy in %). Sections: SR system role, Ctx context, TD task details, Con constraints, OF output format. In IFBench with GPT-4.1 Mini, rounds 1 and 3 tie on validation and round 1 is returned on its higher mining accuracy.
Round
Arm
Sections changed
Single-section
Mining rec./reg.
Validation rec./reg.
2
report
2.00
1/5
12/7
26/2
no report
1.60
3/5
17/14
26/6
3
report
1.00
5/5
5/7
10/3
no report
2.00
0/5
10/11
18/12
2–3
report
1.50
6/10
17/14
36/5
no report
1.80
3/10
27/25
44/18
Appendix
Table 9: Edit sizes and item transitions of the five runs per arm, summed over runs. Round 3 starts from different round-2 prompts in the two arms, so only round 2 compares edits from an identical parent prompt.
Run-level mean
Report
No report
p
Mining items changing outcome, per round
3.1
5.2
0.024
Mining net gain, per round
0.3
0.2
1.000
Mining regressions, per round
1.4
2.5
0.024
Validation items changing outcome, per round
4.1
6.2
0.206
Validation gain (points)
4.8
2.4
0.571
Test items changing outcome, per run
14.2
17.2
0.270
Appendix
Table 10: Run-level means over five runs per arm. Per-round metrics average the two optimizer rounds of a run. p is an exact two-sided permutation test on run means, uncorrected for multiple comparisons.
Solver
Evaluated prompt
Token limit
Test correct
Claude Haiku 4.5
seed
2,048
8/30
returned (round 3)
2,048
10/30
GPT-4o-mini
seed
2,048
4/30
returned (the seed, unchanged)
2,048
3/30
seed, re-evaluated
2,048
4/30
seed, re-evaluated
16,000
5/30
Appendix
Table 11: Problems solved out of the 30 AIME 2025 test problems in the earlier runs and in re-evaluations of fixed prompts. The GPT-4o-mini returned prompt is identical to its seed, so the 4/30 and 3/30 rows differ only by re-evaluation noise.
Run
Solver
Prompt
Shared
Example (lowercased)
MMLU college math.
4o-mini
885
1
the set of all finite subsets of r
MMLU computer sec.
4o-mini
655
5
false false c true false d false true
MMLU computer sec.
4.1 Mini
1033
5
false false c true false d false true
Appendix
Table 12: Runs with any shared 8-gram. “Prompt” is the number of distinct 8-grams in the returned prompt. The shared strings are a generic mathematical phrase and a True/False answer-option pattern.
Automatic prompt optimization (APO) has emerged as a powerful paradigm for improving LLM performance without manual prompt engineering. Reflective APO methods such as GEPA iteratively refine prompts by diagnosing failure cases, but the optimization process remains black-box and label-free, leading to uninterpretable trajectories and systematic failure. We identify and empirically demonstrate four limitations: on GSM8K with a defective seed, GEPA degrades accuracy from 23.81% to 13.50%. We propose VISTA, a multi-agent APO framework that decouples hypothesis generation from prompt rewriting, enabling semantically labeled hypotheses, parallel minibatch verification, and interpretable optimization trace. A two-layer explore-exploit mechanism combining random restart and epsilon-greedy sampling further escapes local optima. VISTA recovers accuracy to 87.57% on the same defective seed and consistently outperforms baselines across all conditions on GSM8K and AIME2025.
Shiyan Liu, Qifeng Xia, Qiyun Xia +3
University of California, Berkeley · Huazhong University of Science and Technology · Hefei University of Technology
Prompt optimization improves language models without updating their weights by searching for a better system prompt, but its effectiveness varies widely across tasks. We study what makes a task amenable to prompt optimization. We show that the reward variance across different system prompts can be decomposed into two components: variance among responses, which captures generation stochasticity, and variance among system prompts, which captures differences in system prompt quality. Prompt optimization succeeds when variance among system prompts is sufficiently large, but fails when variance among responses dominates the variance of the system prompts. Surprisingly, we further show that scaling to more user prompts can hurt optimization by reducing variance among system prompts, especially on heterogeneous datasets where different user prompts favor different system prompts. Motivated by this insight, we propose p1, a simple user prompt filtering method that selects a small subset of user prompts with high variance across candidate system prompts. This subset of user prompts allows one to distinguish a good system prompt from a bad one, making system optimization easier. Experiments on reasoning benchmarks show that p1 substantially improves prompt optimization over training on the full dataset and outperforms strong baselines such as GEPA. Notably, training on only two prompts from AIME 24 yields a system prompt that generalizes well to other reasoning benchmarks.
Automatic Prompt Optimization (APO) often rewrites prompts monolithically, which can improve one behavior while degrading others. We present SAPO, a segment-level APO method that decomposes prompts into role, context, tasks, and output format, then applies targeted improvements based on top-5 and bottom-5 examples. The optimization loop uses one LLM with static meta-prompts and structured outputs for segmentation, weakness analysis, and candidate generation. We describe a train/validation protocol and a two-stage generation process: (1) segment-level diagnosis and recommendation extraction, (2) candidate synthesis constrained by weak/strong segment signals. Using the evaluation setup across SQuADv2, TweetEval, XSUM, CommonGen, and GSM8K on GPT-3.5-Turbo and GPT-4o-mini, SAPO achieves the best average score against Zero-shot and strong APO baselines including APE, OPRO, EvoPrompt, GEPA, and StraGO.
Nikita Kulin, Viktor Zhuravlev, Artur Khairullin +4