Beyond Aggregate Accuracy: Item-Level Transitions in Prompt Optimization

Chinmoy Mitra; Tarek Mahmud

cs.LGOct 13, 2026

Beyond Aggregate Accuracy: Item-Level Transitions in Prompt Optimization

Authors: Chinmoy Mitra, Tarek Mahmud

Organizations: Department of Computer Science & Engineering, Rajshahi University of Engineering & Technology, Rajshahi, Bangladesh · Texas A&M University–Kingsville, Kingsville, Texas, USA

Abstract

Automatic prompt optimization (APO) uses feedback from model responses to improve prompts automatically. However, its effectiveness is usually measured through overall scores, which do not reveal which questions become correct, which previously correct answers become wrong, or whether improvements extend to unseen questions. We introduce Reflective Feedback-Driven Prompt Optimization (RFDPO), a tool that tracks recoveries (incorrect to correct), regressions (correct to incorrect), and churn (all changed outcomes). RFDPO uses this feedback to guide subsequent revisions and records prompt changes throughout optimization. We evaluate RFDPO across five benchmarks covering law, knowledge, instruction following, mathematics, and privacy, using GPT-5 as the optimizer and GPT-4o-mini and GPT-4.1 Mini as solvers. Our results show that overall accuracy can hide substantial changes: in five of nine configurations, churn was at least five times the absolute net change, and a single-item net change concealed up to 17 changed outcomes. Average validation gains were 7.9 percentage points, compared with 5.2 points on held-out tests. In a controlled ablation on LegalBench-Hearsay, item-level feedback reduced churn by 40% ( $p=0.024$ ) while maintaining similar net gains. These findings motivate reporting recovery, regression, churn, and performance on unseen data alongside overall accuracy.

Figures & tables

Figure 1: The RFDPO loop. The solver evaluates prompts on mining set $M$ and validation set $V$ . From the second round, the optimizer receives the item-level effect of its previous edit before proposing a new prompt. Every changed candidate is committed, and RFDPO returns the candidate with the highest validation accuracy, breaking ties by mining accuracy. Test set $T$ is excluded from feedback and selection and evaluated only for the seed and the returned prompt.

Figure 2: Example of item-level transitions. Three examples are recovered and two regress, producing a net gain of one correct example and a churn of five.

Benchmark	Task	Reported metric	Train $\rightarrow$ $M$ / $V$	Test
LegalBench-Hearsay	Binary classification	Accuracy	$50\rightarrow 25/25$	49
MMLU (6 subjects)	Multiple choice	Accuracy, macro-avg.	$50\rightarrow 25/25$	66 each
IFBench	Instruction following	Accuracy	$40\rightarrow 26/14$	42
AIME (2022–24 $\rightarrow$ 2025)	Mathematical reasoning	Accuracy	$90\rightarrow 58/32$	30
PUPA	Privacy-conscious delegation	Accuracy (composite $\geq 0.7$ )	$60\rightarrow 30/30$	40

Table 1: Benchmarks and evaluation splits. MMLU splits are per subject. Test sizes precede provider-block exclusions.

Benchmark	Solver	Test accuracy (%)			$\Delta V$ (pp)	Test transitions
		Seed	Final	$\Delta T$		$\|R\|$	$\|G\|$	Churn (%)
AIME	GPT-4.1 Mini	53.3	50.0	$-3.3$	$+6.2$	3	4	7 (23.3)
PUPA	GPT-4o-mini	55.3	65.8	$+10.5$	$+13.3$	7	3	10 (26.3)
PUPA	GPT-4.1 Mini	44.7	71.1	$+26.3$	$+23.3$	10	0	10 (26.3)
IFBench	GPT-4o-mini	40.5	45.2	$+4.8$	$+15.8$	6	4	10 (23.8)
IFBench	GPT-4.1 Mini	42.9	66.7	$+23.8$	$+14.3$	12	2	14 (33.3)

Table 2: Seed-to-returned test accuracy, validation accuracy change, and item-level transitions. Green rows indicate positive test changes. Churn percentages use evaluated test items. MMLU accuracy is macro-averaged across six subjects, while transitions sum 393 items. PUPA (38 items) and professional law (63 items) exclude provider-blocked examples. Earlier AIME runs with other solvers are described in Appendix C.5 .

Figure 3: Recovered and regressed test examples across benchmarks and solvers. Diamonds indicate net changes. MMLU transitions are aggregated across six subjects.

Edited-prompt returns	Runs	Val. $\uparrow$	Val. $\uparrow$ test $\downarrow$	$\overline{\Delta V}$	$\overline{\Delta T}$
Non-MMLU	7	7	2	$+15.0$	$+8.9$
MMLU subjects	10	5	1	$+2.9$	$+2.6$
All matched	17	12	3	$\mathbf{+7.9}$	$\mathbf{+5.2}$

Table 3: Mean validation and test accuracy changes for 17 edited-prompt returns; two additional records retained the seed. Means weight runs equally and are reported in percentage points. The fourth column counts runs with improved validation accuracy but decreased test accuracy.

Benchmark	Solver	Optimizer calls	Optimizer input (K)	Optimizer output (K)	All-role tokens (K)	Logged cost (USD)
AIME	GPT-4.1 Mini	3	498.4	5.4	2,666.6	3.81
PUPA	GPT-4o-mini	3	73.8	13.7	1,651.2	4.44
PUPA	GPT-4.1 Mini	3	68.5	15.6	1,570.4	4.80
IFBench	GPT-4o-mini	3	23.4	12.0	300.9	0.21
IFBench	GPT-4.1 Mini	3	20.2	10.1	240.1	0.35
Hearsay	GPT-4o-mini	3	17.1	14.9	241.3	0.22

Table 4: Inference consumption across matched configurations. K denotes 1,000 tokens. All-role totals include solver, optimizer, and applicable external-model and judge calls, including seed and final evaluations. Costs use configured prices. MMLU aggregates six subjects; two GPT-4.1 Mini subject runs skipped optimization.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Solver	Assigned $M/V$	Val. fraction	Solver limit
AIME	4.1 Mini	58/32	0.35	16,000
PUPA	4o-mini	30/30	0.50	2,048
PUPA	4.1 Mini	30/30	0.50	2,048
IFBench	4o-mini	26/14	0.35	2,048
IFBench	4.1 Mini	26/14	0.35	4,096
Hearsay	both	25/25	0.50	2,048

Appendix

Table 5: Recorded settings for the matched runs. Solver limit is the configured maximum output tokens, not observed consumption. Assigned pools precede provider-block exclusions: GPT-4o-mini IFBench evaluates 19 validation items and GPT-4.1 Mini evaluates 25 mining items. Actual resource use is in Table 4 .

Work	Items: optimization / validation / test	Solver models	Feedback and optimizer	Reported change (%)
LegalBench-Hearsay
Trace2Policy, human-written refinements ( Zha et al., 2026 )	30 iteration / 64 held-out, from 94 items	Claude Opus 4.6, Haiku 4.5, Kimi K2.5	human curator; round 1 also read held-out errors	Haiku 4.5: $79.7\rightarrow 93.8$
Trace2Policy, LLM-written refinements ( Zha et al., 2026 )	61 test items	DeepSeek-v3.2, Kimi K2.5, Kimi K2.6 (mean)	LLM, with a regression gate	$69.4\rightarrow 72.7$
Chen et al. ( 2025 ) (definition classification, a different LegalBench task)	100 train / 100 test (50 / 50 if under 200 items)	LLaMA-3-8B, LLaMA-3.1-8B, GPT-4o	evaluator-guided rewriting, up to 3 iterations	$55\rightarrow 70$ ; $56\rightarrow 69$ ; $83\rightarrow 90$
This paper	25 / 25 / 49, recut from 99 items	GPT-4o-mini, GPT-4.1 Mini	GPT-5, item-level effect report	$71.4\rightarrow 69.4$ ; $71.4\rightarrow 73.5$
MMLU

Appendix

Table 6: Published evaluation protocols on shared benchmarks. Rows differ in data, solver, baseline prompt, feedback access, and evaluation budget. Changes follow solver-column order; PUPA changes are composite scores, as reported by GEPA.

Run	Solver	Edited	Validation (%) seed $\rightarrow$ returned	$\Delta V$	Test (%) seed $\rightarrow$ returned	$\Delta T$	Rec./Reg.
AIME	4.1 Mini	yes	62.5 $\rightarrow$ 68.8	$+6.2$	53.3 $\rightarrow$ 50.0	$-3.3$	3/4
PUPA	4o-mini	yes	40.0 $\rightarrow$ 53.3	$+13.3$	55.3 $\rightarrow$ 65.8	$+10.5$	7/3
PUPA	4.1 Mini	yes	46.7 $\rightarrow$ 70.0	$+23.3$	44.7 $\rightarrow$ 71.1	$+26.3$	10/0
IFBench	4o-mini	yes	36.8 $\rightarrow$ 52.6	$+15.8$	40.5 $\rightarrow$ 45.2	$+4.8$	6/4
IFBench	4.1 Mini	yes	57.1 $\rightarrow$ 71.4	$+14.3$	42.9 $\rightarrow$ 66.7	$+23.8$	12/2
Hearsay	4o-mini	yes	68.0 $\rightarrow$ 80.0	$+12.0$	71.4 $\rightarrow$ 69.4	$-2.0$	8/9

Appendix

Table 7: All matched accuracy pairs and test transition counts. Deltas are percentage points computed before rounding; green rows have positive test deltas. MMLU subjects have 66 assigned test items (63 evaluated for professional law). Dashes denote unmeasured validation after a high-baseline skip; zero

\Delta V

denotes the retained seed.

Benchmark	Solver	Round	$M$ acc.	$V$ acc.	$M$ rec./reg.	$V$ rec./reg.	Sections changed
AIME	4.1 Mini	0	46.6	62.5	–	–	–
		1 ∗	51.7	68.8	9/6	5/3	Con, TD
		2	48.3	65.6	6/8	2/3	TD
		3	46.6	59.4	6/7	3/5	TD
PUPA	4o-mini	0	69.0	40.0	–	–	–
		1	79.3	56.7	6/3	5/0	Con, Ctx, TD

Appendix

Table 8: Round-by-round trajectories (accuracy in %). Sections: SR system role, Ctx context, TD task details, Con constraints, OF output format. In IFBench with GPT-4.1 Mini, rounds 1 and 3 tie on validation and round 1 is returned on its higher mining accuracy.

Round	Arm	Sections changed	Single-section	Mining rec./reg.	Validation rec./reg.
2	report	2.00	1/5	12/7	26/2
	no report	1.60	3/5	17/14	26/6
3	report	1.00	5/5	5/7	10/3
	no report	2.00	0/5	10/11	18/12
2–3	report	1.50	6/10	17/14	36/5
	no report	1.80	3/10	27/25	44/18

Appendix

Table 9: Edit sizes and item transitions of the five runs per arm, summed over runs. Round 3 starts from different round-2 prompts in the two arms, so only round 2 compares edits from an identical parent prompt.

Run-level mean	Report	No report	$p$
Mining items changing outcome, per round	3.1	5.2	0.024
Mining net gain, per round	0.3	0.2	1.000
Mining regressions, per round	1.4	2.5	0.024
Validation items changing outcome, per round	4.1	6.2	0.206
Validation gain (points)	4.8	2.4	0.571
Test items changing outcome, per run	14.2	17.2	0.270

Appendix

Table 10: Run-level means over five runs per arm. Per-round metrics average the two optimizer rounds of a run.

p

is an exact two-sided permutation test on run means, uncorrected for multiple comparisons.

Solver	Evaluated prompt	Token limit	Test correct
Claude Haiku 4.5	seed	2,048	8/30
	returned (round 3)	2,048	10/30
GPT-4o-mini	seed	2,048	4/30
	returned (the seed, unchanged)	2,048	3/30
	seed, re-evaluated	2,048	4/30
	seed, re-evaluated	16,000	5/30

Appendix

Table 11: Problems solved out of the 30 AIME 2025 test problems in the earlier runs and in re-evaluations of fixed prompts. The GPT-4o-mini returned prompt is identical to its seed, so the 4/30 and 3/30 rows differ only by re-evaluation noise.

Run	Solver	Prompt	Shared	Example (lowercased)
MMLU college math.	4o-mini	885	1	the set of all finite subsets of r
MMLU computer sec.	4o-mini	655	5	false false c true false d false true
MMLU computer sec.	4.1 Mini	1033	5	false false c true false d false true

Appendix

Table 12: Runs with any shared 8-gram. “Prompt” is the number of distinct 8-grams in the returned prompt. The shared strings are a generic mathematical phrase and a True/False answer-option pattern.

Automatic Prompt Optimization

Automatic prompt optimization uses model or task feedback to revise instructions for stronger, more reliable language-model performance.

See all 103 papers

Reflection in the Dark: Exposing and Escaping the Black Box in Reflective Prompt Optimization
Mar 19, 2026Shiyan Liu, Qifeng Xia, Qiyun Xia +3Automatic Prompt Optimization Prompt Optimization
University of California, Berkeley · Huazhong University of Science and Technology · Hefei University of Technology
$p1$ : Better Prompt Optimization with Fewer Prompts
Apr 9, 2026Zhaolin Gao, Yu, Wang +4Automatic Prompt Optimization Large Language Model-Guided Optimization
Sid · Cornell University · Microsoft +2
From Monolithic to Modular: Segment-level Automatic Prompt Optimization
Jul 21, 2026Nikita Kulin, Viktor Zhuravlev, Artur Khairullin +4Large Language Model-Guided Optimization Automatic Prompt Optimization
ITMO University · HSE · AXXX

Mar 19, 2026J/K move · Enter open · S save

Reflection in the Dark: Exposing and Escaping the Black Box in Reflective Prompt Optimization

Shiyan Liu, Qifeng Xia, Qiyun Xia, Yisheng Liu+2

University of California, Berkeley · Huazhong University of Science and Technology · Hefei University of Technology

Automatic Prompt Optimization Prompt Optimization

Open paper AskPDF arXiv

Automatic prompt optimization (APO) has emerged as a powerful paradigm for improving LLM performance without manual prompt engineering. Reflective APO methods such as GEPA iteratively refine prompts by diagnosing failure cases, but the optimization process remains black-box and label-free, leading to uninterpretable trajectories and systematic failure. We identify and empirically demonstrate four limitations: on GSM8K with a defective seed, GEPA degrades accuracy from 23.81% to 13.50%. We propose VISTA, a multi-agent APO framework that decouples hypothesis generation from prompt rewriting, enabling semantically labeled hypotheses, parallel minibatch verification, and interpretable optimization trace. A two-layer explore-exploit mechanism combining random restart and epsilon-greedy sampling further escapes local optima. VISTA recovers accuracy to 87.57% on the same defective seed and consistently outperforms baselines across all conditions on GSM8K and AIME2025.

Abstract

Automatic Prompt Optimization

Explore similar work

Reflection in the Dark: Exposing and Escaping the Black Box in Reflective Prompt Optimization

p1p1p1: Better Prompt Optimization with Fewer Prompts

From Monolithic to Modular: Segment-level Automatic Prompt Optimization

$p1$ : Better Prompt Optimization with Fewer Prompts