PrefPO: Pairwise Preference Prompt Optimization

Rahul Singhal; Pradyumna Tambwekar; Karime Maamari

cs.CLMar 13, 2026

PrefPO: Pairwise Preference Prompt Optimization

Authors: Rahul Singhal, Pradyumna Tambwekar, Karime Maamari

Organizations: Distyl AI

Abstract

Prompt engineering is effective but labor-intensive, motivating automated optimization methods. Existing methods typically require labeled datasets, which are often unavailable, and produce verbose, repetitive prompts. We introduce PrefPO, a minimal prompt optimization approach inspired by reinforcement learning from human feedback (RLHF). Its preference-based approach reduces the need for labeled data and hyperparameter tuning-only a starting prompt and natural language criteria are needed. PrefPO uses an LLM discriminator to express pairwise preferences over model outputs and provide feedback to an LLM optimizer, iteratively improving performance. We evaluate PrefPO on 9 BIG-Bench Hard (BBH) tasks and IFEval-Hard, a newly-curated, challenging subset of IFEval. PrefPO matches or exceeds SOTA methods, including GEPA, MIPRO, and TextGrad, on 6/9 tasks and performs comparably to TextGrad on IFEval-Hard (82.4% vs 84.5%). Unlike other methods, PrefPO can optimize in both labeled and unlabeled settings. Without labels, PrefPO closely matches its labeled performance on 6/9 tasks, proving effective without ground truth. PrefPO also improves prompt hygiene: we find existing methods produce prompts 14.7x their original length or with 34% repetitive content; PrefPO reduces these issues by 3-5x. Furthermore, both LLM and human judges rate PrefPO's prompts higher than TextGrad's. Finally, we identify prompt hacking in prompt optimizers, where methods game evaluation criteria, and find PrefPO is susceptible at half the rate of TextGrad (37% vs 86%), generating fewer brittle, misaligned prompts.

Figures & tables

Figure 1: PrefPO optimization loop. PrefPO is a prompt optimization framework that optimizes an initial pool of prompts via preferences. Our approach is comprised of a discriminator, which generates feedback after conducting a pairwise evaluation, and an optimizer, which improves the non-preferred prompt. The updated prompt is added to the prompt pool, and this process repeats.

Figure 2: Prompt Hygiene. The original IFEval-Hard seed prompt (top) with the optimized prompts by TextGrad (left), PrefPO (middle), and PrefPO -Minimal (right). All achieve 100% pass rate, but vary in prompt hygiene . TextGrad’s prompt is verbose (4002 chars; see Appendix A.1 ) with high repetition (11.6%) and low similarity to the seed (3.6%). PrefPO is more hygienic (350 chars, 0% repetition, 27.7% similarity). PrefPO -Minimal is even shorter (302 chars) with the highest similarity (33.3%). Human judges rate PrefPO ’s prompt over TextGrad’s prompt ( $5/6$ vs $3/6$ ), with PrefPO -Minimal receiving the highest LLM rating ( $6/6$ ).

Method	Object Counting	Movie Recommendations	Causal Judgment	Geometric Shapes	Disambiguation
GPT-4o (zero shot)	$0.748$	$0.732$	$0.722$	$0.644$	$0.548$
GPT-4o (few-shot + CoT)	$\underline{0.992}$	$0.896$	$0.727$	$0.676$	$0.856$
GPT-5 (zero shot)	$0.092$	$0.824$	$0.706$	$0.820$	$0.720$
TextGrad	$0.987\pm 0.009$	$0.915\pm 0.021$	$0.732\pm 0.042$	$0.765\pm 0.039$	$0.780\pm 0.043$
MIPROv2	$0.994\pm 0.008$	$0.956\pm 0.012$	$0.751\pm 0.036$	$0.899\pm 0.037$	$0.776\pm 0.029$
GEPA	$0.987\pm 0.008$	$\underline{0.948\pm 0.019}$	$0.778\pm 0.028$	$\underline{0.903\pm 0.034}$	$0.698\pm 0.095$

Table 1: BBH task accuracy. Mean ± SD over 10 runs; bold indicates best overall, underline second best. PrefPO leads on

4/9

tasks among optimization methods (MIPROv2:

3/9

, GEPA:

2/9

) with the highest average (0.875) across all optimization and prompting techniques. Unlabeled PrefPO wins on

2/9

tasks with no substantial difference from labeled performance on

6/9

tasks. GPT-5’s low Object Counting score (0.092) reflects a formatting issue in its outputs.

Figure 3: Prompt hacking examples. Two observed hacks: numerical constraint manipulation (top) and vocabulary over-restriction (bottom). TextGrad modifies constraints in an attempt to increase pass rates (e.g., changing “at least 50” to “exactly 65” sentences, or banning words not in the original specification). PrefPO -Minimal achieves equal or better performance without altering constraints.

Figure 4: Scaling Behavior. Accuracy vs training set size. PrefPO scales with data and outperforms other methods across most training sizes, while unlabeled PrefPO struggles. Few-shot prompt initialization achieves the best performance at 50 training examples.

	worst@20	average@20
TextGrad	$\mathbf{84.5\%\pm 5.7\%}$	$\mathbf{92.7\%\pm 3.7\%}$
PrefPO	$82.4\%\pm 6.1\%$	$91.5\%\pm 3.8\%$
PrefPO -Elo	$81.1\%\pm 6.1\%$	$90.6\%\pm 4.2\%$
PrefPO -Minimal	$75.7\%\pm 7.1\%$	$88.7\%\pm 4.3\%$

Table 2: IFEval-Hard performance. IFEval-Hard performance with 95% confidence intervals; bootstrap CIs for worst@20 and normal approximation for average@20 (see Appendix B.2 ). worst@20 : strict pass (all 20 runs must succeed); average@20 : mean success rate. TextGrad leads slightly on worst@20 (

84.5\%

82.4\%

), but the confidence intervals overlap substantially.

Figure 5: Prompt length and repetition on BBH tasks. Length (left) and repetition (right) computed on the best-performing prompt averaged across 10 runs for each task; lower is more hygienic for both. PrefPO maintains consistently better hygiene. TextGrad shows extreme length variance, reaching up to

\sim 10,000

characters, while MIPROv2’s prompts frequently contain

>20\%

repetitive content. Label-free PrefPO maintains comparable hygiene to labeled PrefPO .

Method	Length Ratio	Repetition Increase	Similarity	Hygiene Grade (LLM, n=148)	Total
				Readability / Specification Quality / Maintainability
TextGrad	$14.71\pm 10.75$	$0.117\pm 0.079$	$0.133\pm 0.144$	1.23 / 0.63 / 0.90	2.76
PrefPO	$4.70\pm 3.44$	$0.044\pm 0.055$	$0.198\pm 0.103$	$1.87$ / $1.60$ / $1.69$	5.16
PrefPO -Minimal	$2.20\pm 0.84$	$0.012\pm 0.041$	$0.418\pm 0.171$	$1.87$ / $1.65$ / $1.74$	$5.26$
PrefPO -Elo	$6.37\pm 4.17$	$0.058\pm 0.059$	$0.150\pm 0.087$	–	–
Method	Hygiene Grade (Human, n=10)	Total	Hygiene Grade (LLM, n=10)	Total

Table 3: Hygiene comparison on IFEval-Hard. Programmatic metrics (length ratio, repetition increase, similarity) are computed relative to original prompts and averaged across all prompts in the final pool for each technique. Top: programmatic metrics and LLM judge ratings (averaged across 148 prompts). Bottom: human and LLM ratings averaged on a 10-sample subset. All programmatic and judged dimensions (across both human and LLM evaluators) favor PrefPO over TextGrad; PrefPO -Minimal achieves the best scores.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Type	Samples	Description
object_counting	Exact Match	250	Count objects listed in a question
movie_recommendation	MC	250	Recommend a movie based on preferences
causal_judgement	Binary	187	Judge whether a causal claim is correct
geometric_shapes	MC	250	Identify shapes from SVG path commands
disambiguation_qa	MC	250	Disambiguate pronoun references in context
logical_deduction_seven_objects	MC	250	Deduce ordering of objects from clues

Appendix

Table 4: Overview of BIG-Bench Hard evaluation tasks. Task types: MC (Multiple Choice) = select from labeled options (A-E); Binary = two-class classification (e.g., yes/no, valid/invalid); Exact Match = free-form answer compared exactly to the ground truth.

Figure 6: Light bars show IFEval (541 samples); dark bars show IFEval-Hard (148 samples where GPT-4o fails at least once across 20 runs). All models show substantial performance drops on IFEval-Hard, with GPT-4o dropping 41.4% (81.3% to 39.9%). Frontier models like GPT-5 and Gemini 3 Pro are most robust (

-11.1\%

and

-10.2\%

), while mid-tier models show larger degradation.

Figure 7: worst@20 performance on IFEval-Hard as measured across iterations for each technique. Most methods reach 90% of final performance by iteration 5 and plateau after iteration 10, indicating rapid early convergence. PrefPO -Minimal converges slower due to its stricter constraint.

Figure 8: Each bar shows worst@20 accuracy using different model configurations for the discriminator and optimizer. Single-model configurations use the same model for both discrimination and optimization; mixed configurations (“GPT-4o disc + GPT-5 opt”, “GPT-5 disc + GPT-4o opt”) isolate the effect of each component. Frontier models outperform open-weight models, and optimizer capability matters more than discriminator capability.

Examples	Discriminator Accuracy	Optimizer Accuracy	Mean Improvement
5	$0.540\pm 0.121$	$0.567\pm 0.150$	$0.005\pm 0.022$
10	$0.627\pm 0.095$	$0.620\pm 0.112$	$0.025\pm 0.018$
15	$0.760\pm 0.144$	$0.640\pm 0.100$	$0.054\pm 0.021$
20	$0.760\pm 0.144$	$0.820\pm 0.134$	$0.091\pm 0.024$
30	$0.887\pm 0.060$	$0.873\pm 0.109$	$\bm{0.124\pm 0.030}$
40	$0.853\pm 0.078$	$0.833\pm 0.091$	$0.114\pm 0.020$

Appendix

Table 5: Component effectiveness with labeled training examples (Disambiguation).

Task	Examples	Discriminator Accuracy	Optimizer Accuracy	Mean Improvement
Disambiguation	5	$0.540\pm 0.087$	$0.407\pm 0.153$	$-0.031\pm 0.015$
Disambiguation	10	$0.507\pm 0.095$	$0.467\pm 0.094$	$-0.029\pm 0.017$
Disambiguation	15	$0.487\pm 0.090$	$0.413\pm 0.098$	$-0.027\pm 0.032$
Disambiguation	20	$0.453\pm 0.151$	$0.380\pm 0.116$	$-0.033\pm 0.024$
Disambiguation	30	$0.420\pm 0.130$	$0.387\pm 0.160$	$-0.044\pm 0.030$
Disambiguation	40	$0.427\pm 0.090$	$0.320\pm 0.154$	$-0.046\pm 0.036$

Appendix

Table 6: Component effectiveness without ground-truth labels across tasks.

Automatic Prompt Optimization

Automatic prompt optimization uses model or task feedback to revise instructions for stronger, more reliable language-model performance.

See all 103 papers

Apr 9, 2026

$p1$ : Better Prompt Optimization with Fewer Prompts

Prompt optimization improves language models without updating their weights by searching for a better system prompt, but its effectiveness varies widely across tasks. We study what makes a task amenable to prompt optimization. We show that the reward variance across different system prompts can be decomposed into two components: variance among responses, which captures generation stochasticity, and variance among system prompts, which captures differences in system prompt quality. Prompt optimization succeeds when variance among system prompts is sufficiently large, but fails when variance among responses dominates the variance of the system prompts. Surprisingly, we further show that scaling to more user prompts can hurt optimization by reducing variance among system prompts, especially on heterogeneous datasets where different user prompts favor different system prompts. Motivated by this insight, we propose

p1

, a simple user prompt filtering method that selects a small subset of user prompts with high variance across candidate system prompts. This subset of user prompts allows one to distinguish a good system prompt from a bad one, making system optimization easier. Experiments on reasoning benchmarks show that

p1

substantially improves prompt optimization over training on the full dataset and outperforms strong baselines such as GEPA. Notably, training on only two prompts from AIME 24 yields a system prompt that generalizes well to other reasoning benchmarks.

Zhaolin Gao, Yu, Wang +4

Sid · Cornell University · Microsoft +2

Automatic Prompt Optimization Large Language Model-Guided Optimization

Ask PDF

Apr 30, 2026

Optimization before Evaluation: Evaluation with Unoptimised Prompts Can be Misleading

Current Large Language Model (LLM) evaluation frameworks utilize the same static prompt template across all models under evaluation. This differs from the common industry practice of using prompt optimization (PO) techniques to optimize the prompt for each model to maximize application performance. In this paper, we investigate the effect of PO towards LLM evaluations. Our results on public academic and internal industry benchmarks show that PO greatly affects the final ranking of models. This highlights the importance of practitioners performing PO per model when conducting evaluations to choose the best model for a given task.

Nicholas Sadjoli, Tim Siefken, Atin Ghosh +2

SAP · Stanford University

LLM Evaluation Language Model Generation Evaluation Prompt Optimization LLM Input Optimization

Ask PDF

Feb 3, 2026

GFlowPO: Generative Flow Network as a Language Model Prompt Optimizer

Finding effective prompts for language models (LMs) is critical yet notoriously difficult: the prompt space is combinatorially large, rewards are sparse due to expensive target-LM evaluation. Yet, existing RL-based prompt optimizers often rely on on-policy updates and a meta-prompt sampled from a fixed distribution, leading to poor sample efficiency. We propose GFlowPO, a probabilistic prompt optimization framework that casts prompt search as a posterior inference problem over latent prompts regularized by a meta-prompted reference-LM prior. In the first step, we fine-tune a lightweight prompt-LM with an off-policy Generative Flow Network (GFlowNet) objective, using a replay-based training policy that reuses past prompt evaluations to enable sample-efficient exploration. In the second step, we introduce Dynamic Memory Update (DMU), a training-free mechanism that updates the meta-prompt by injecting both (i) diverse prompts from a replay buffer and (ii) top-performing prompts from a small priority queue, thereby progressively concentrating the search process on high-reward regions. Across few-shot text classification, instruction induction benchmarks, and question answering tasks, GFlowPO consistently outperforms recent discrete prompt optimization baselines.

Junmo Cho, Suhan Kim, Sangjune An +5

Korea Advanced Institute of Science and Technology (KAIST) · Korea University · Mila – Québec AI Institute

LLM Input Optimization

Ask PDF

Abstract

Automatic Prompt Optimization

Explore similar work

p1p1p1: Better Prompt Optimization with Fewer Prompts

Optimization before Evaluation: Evaluation with Unoptimised Prompts Can be Misleading

GFlowPO: Generative Flow Network as a Language Model Prompt Optimizer

$p1$ : Better Prompt Optimization with Fewer Prompts