Bidirectional Preference Synthesis: Learning Prompt-Conditioned Preferences from Boundary Failures

Junbo Wang; Lidong Lu; Zhuoqun Li; Guiping Jiang; Xiangyu Wu; Tinghai Zhang; Tong Lu

cs.AIOct 3, 2026

Bidirectional Preference Synthesis: Learning Prompt-Conditioned Preferences from Boundary Failures

Authors: Junbo Wang, Lidong Lu, Zhuoqun Li, Guiping Jiang, Xiangyu Wu, Tinghai Zhang, Tong Lu

Organizations: Kuaishou Technology · Nanjing University

Abstract

Correction-based offline preference pipelines commonly treat model failures only as rejected responses under the original prompt. This supervision is incomplete for boundary failures: responses that violate the given instruction yet coherently satisfy a nearby intent or constraint setting. We introduce Bidirectional Preference Synthesis (BPS), a data-construction method for standard Direct Preference Optimization (DPO) that makes this missing prompt dependence explicit. For each validated boundary failure, BPS keeps the conventional forward pair under the original prompt and adds a reverse pair under a synthesized achieved prompt, so the same response is rejected where it is wrong and chosen where it is right, without changing the DPO objective, training a reward model, or requiring online sampling. On Qwen3-4B-Instruct-2507, BPS preserves original-side pairwise ranking while raising achieved-side ranking accuracy from 6.8% to 62.3% on held-out crossed anchors, with a similar shift under a Kimi-K2.6 cross-teacher probe. A blind human audit supports the intended reverse preference direction, and downstream evaluations show the clearest separation from Forward-DPO in multilingual multi-turn instruction following, with consistent capability-retention patterns on agentic, tool-use, and code checks.

Figures & tables

Figure 1: Forward-only DPO data rejects

\tau_{\text{fail}}

only under

P_{\text{orig}}

. BPS adds an achieved-prompt reverse pair, so the same response pair teaches a prompt-conditioned preference boundary.

Figure 2: BPS pipeline. Each validated boundary failure yields a forward correction pair under $P_{\text{orig}}$ and, when validation succeeds, a reverse pair under $P_{\text{achieved}}$ .

IFBench / IFEval				FollowBench / Multi-IF
Metric	Base	Fwd-100	BPS-100	Metric	Base	Fwd-100	BPS-100
IFBench-P Strict	31.00	31.97	32.31	FollowBench CSL (EN)	3.00	2.96	3.00
IFBench-I Strict	33.67	36.05	35.71	FollowBench CSL (ZH)	2.92	2.90	2.98
IFBench-P Loose	33.33	32.65	35.03	FollowBench HSR (EN)	66.04	64.44	66.37
IFBench-I Loose	36.50	36.90	38.78	FollowBench HSR (ZH)	64.46	64.36	65.47
IFEval-P Strict	82.81	83.36	86.69	FollowBench SSR (EN)	74.88	74.62	75.24

Table 1: Instruction-following results at Step 100. IFBench/IFEval averages are means over listed metrics; Multi-IF scores are macro-averaged across eight languages. BPS-100 scores above both comparisons at every Multi-IF turn, while the BPS-100–Fwd-100 gap grows monotonically with turn depth (+0.72, +3.37, +4.34); per-language analysis is in the supplementary material.

General Capabilities				Agent, Tool Use, and Broad Reasoning
Metric	Base	Fwd	BPS	Metric	Base	Fwd	BPS
GPQA-Diamond	42.93	45.96	46.46	tau-bench airline	36.00	30.00	36.00
MATH-500	94.80	94.00	95.00	tau2-bench airline	27.91	25.58	27.91
HumanEval+	71.34	71.34	72.56	tau2-bench retail	43.75	36.61	43.75
MBPP	72.00	72.00	72.80	ToolBench Act.EM (ov.)	35.50	33.10	35.46
C-Eval Overall	80.39	78.75	80.39	ToolBench Plan EM (OOD)	78.50	77.98	81.06

Table 2: Selected capability-retention and out-of-distribution checks at Step 100. BPS scores at or above Base on the listed general-capability metrics, recovers Base-level scores on the shown agentic tasks where Fwd-100 is lower, and matches the Base hallucination rate on ToolBench OOD. Complete ToolBench and additional benchmark metrics are in the supplementary material.

Metric	Fwd-DPO	BPS
FC Full Acc.	0.860	0.869
Acc ${}_{\text{achieved}}$	0.759	0.824
Boundary Ach EM	0.721	0.768
Boundary Both EM	0.443	0.489

Table 3: Function-calling pipeline instantiation (ckpt-60). BPS has a consistent directional advantage on held-out full-call accuracy and achieved-side boundary switching when the full pipeline is rebuilt for structured function calls.

Figure 3: Diagnostic view of BPS. (a) Forward-DPO expands DPO margins mainly by lowering rejected rewards, while BPS keeps chosen rewards positive and rejected suppression more moderate. (b) BPS substantially raises achieved-side pairwise ranking accuracy on both held-out probes.

Probe	Model	$M_{o}$	Acc o	$M_{a}$	Acc a
Gemini	Forward-DPO	0.540	99.3	$-$ 0.349	6.8
Gemini	BPS	0.292	100.0	$-$ 0.011	62.3
Kimi	Forward-DPO	0.854	99.3	$-$ 0.374	9.3
Kimi	BPS	0.663	99.3	$-$ 0.005	61.6

Table 4: Conditional reward geometry on held-out crossed anchors. Gemini denotes the original pipeline-held-out probe with 146 anchors; Kimi denotes a cross-teacher probe with 151 anchors synthesized and validated by Kimi-K2.6. BPS preserves original-side pairwise ranking while substantially improving achieved-side pairwise ranking accuracy under both construction pipelines. On the Kimi probe, the paired achieved-margin improvement over Forward-DPO is

+0.369

(95% bootstrap CI

[0.267,0.487]

Metric	Base	Fwd	Shuf	C-Rev	BPS
IFBench Avg.	33.63	34.39	33.16	32.61	35.46
IFEval Avg.	87.18	87.36	88.28	88.62	90.27
M-IF Turn 1	85.72	87.21	87.57	86.89	87.93
M-IF Turn 2	78.07	75.78	77.44	77.94	79.15
M-IF Turn 3	69.45	66.14	68.32	69.26	70.48
Reward Margin	–	10.71	6.71	2.81	4.33

Table 5: Reverse-data ablations at step 100. Shuf permutes achieved prompts across reverse pairs. C-Rev is a strong RPO-style construction baseline that mechanically reverses violated constraints without achieved-intent synthesis. Both reverse-style baselines mitigate Forward-DPO’s multi-turn degradation, while BPS gives the strongest balance across IFBench, IFEval, and Multi-IF in this comparison.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Metric / Split	Base	FC-Fwd	FC-BPS
General-FC	Tool-call F1	52.9	53.7	54.2
General-FC	Schema Acc.	67.8	68.2	68.8
General-FC	Successful Calls	396	401	411
$\tau$ -bench	Airline Acc.	36.0	26.0	42.0
$\tau$ -bench	Retail Acc.	55.7	46.1	50.4
$\tau$ -bench	Overall Acc.	49.7	40.0	47.9

Appendix

Table 6: Full results for the separate function-calling instantiation (Base = Qwen3-4B without further training). Scores are percentages except successful-call counts. BPS improves function-calling accuracy,

\tau^{2}

-bench overall, and MBPP+ over both Base and FC-Fwd. On

\tau

-bench, BPS outperforms FC-Fwd on airline and overall but falls below Base on retail, yielding a slightly lower overall score than Base. The main exception across benchmarks is

\tau^{2}

-retail, where FC-Fwd scores higher.

Stage	Count	Retention
Seed prompts (IFBench-train)	2,000	—
Student rollouts (dual-temperature)	6,000	—
Boundary failures ( $0<r<1$ )	3,420	57.0%
Near-miss failures ( $r\geq 0.5$ )	2,428	71.0%
Accepted forward pairs	1,443	59.4%
Reverse candidates (pre-filtered)	316	13.0%

Appendix

Table 7: Data construction funnel. Retention rates are relative to the immediately preceding stage, except “Reverse candidates” (relative to near-miss failures) and “Valid reverse pairs” (relative to reverse candidates). Dual-temperature decoding samples

K=3

rollouts per prompt.

Benchmark	Base	Fwd-100	BPS-100
ToolBench Act.EM (overall)	35.50	33.10	35.46
ToolBench Act.EM (in)	35.89	34.06	36.04
ToolBench Act.EM (OOD)	34.70	31.17	34.30
ToolBench F1 (in)	30.86	29.53	30.97
ToolBench F1 (OOD)	29.38	26.75	28.98
ToolBench Plan EM (in)	80.94	81.39	83.22

Appendix

Table 8: Full ToolBench results (Step 100). BPS recovers nearly all ToolBench Act.EM and F1 scores, matches Base hallucination rate, and achieves the highest Plan EM on both splits.

$\rho$	IFB Avg	M-IF Ov.	LCB	TB OOD
0 (Fwd-100)	34.39	76.38	45.40	26.75
0.10	35.33	78.61	46.54	28.41
0.195 (BPS)	35.46	79.19	48.53	28.98
0.30	32.82	78.44	47.30	29.75

Appendix

Table 9: Reverse-density sensitivity at Step 100. IFBench Avg = mean of four metrics; M-IF = macro-avg across 8 languages/3 turns; LCB = LiveCodeBench; TB OOD = ToolBench OOD. The main density

\rho=0.195

results from retaining all validated reverse pairs. Lower density provides a smaller correction, while

\rho=0.30

reduces IFBench below Fwd-DPO (32.82 vs. 34.39), illustrating the correction–retention trade-off.

Model	$M_{o}$	Acc o	$M_{a}$	Acc a
Forward-DPO	0.541	99.3	$-$ 0.349	6.8
Fwd-Dup (+282 fwd)	0.517	99.3	$-$ 0.334	6.8
RPO-style (+282 rev)	0.213	95.9	$-$ 0.057	32.9
BPS (+282 rev)	0.293	100.0	$-$ 0.011	62.3

Appendix

Table 10: Budget-matched reward-geometry controls on the Gemini held-out crossed-anchor probe. Duplicating forward pairs leaves achieved-side ranking accuracy unchanged. RPO-style mechanical reverse pairs partially raise achieved-side ranking accuracy but remain below BPS and weaken original-side ranking accuracy.

Model	$M_{\text{orig}}$	Acc ${}_{\text{orig}}$	$M_{\text{achieved}}$	Acc ${}_{\text{achieved}}$
Forward-DPO	0.854	99.3	$-$ 0.374	9.9
Constraint-Rev	0.485	97.4	$-$ 0.072	37.1
BPS	0.667	99.3	$-$ 0.006	59.6

Appendix

Table 11: Kimi-heldout conditional reward geometry for Constraint-Rev, evaluated on the subset used for the Constraint-Rev comparison. Constraint-Rev raises achieved-side ranking accuracy over Forward-DPO but remains below BPS; BPS also preserves stronger original-side margins. Because this subset differs slightly from the main-paper conditional reward table, the Forward-DPO and BPS numbers differ slightly. Paired bootstrap gives

\Delta M_{\text{achieved}}(\text{BPS}-\text{C-Rev})=+0.066

(95% CI

[0.043,0.095]

Field	Text
Original prompt $P_{\text{orig}}$	Which country contains the largest clock in the world? At the end of your response, please explicitly add a postscript starting with Postscript: The first word of your response should be the word grandfather. Your ENTIRE response should be in Ukrainian language, no other language is allowed. Include keyword emotion in your response.
Forward chosen $\tau_{\text{perfect}}$	“grandfather [Ukrainian: The world’s largest clock is in Mecca, Saudi Arabia. This clock reflects the grandeur and traditions, evoking a strong emotion in people who see it…] Postscript: emotion”
Forward rejected $\tau_{\text{fail}}$	“Grandfather, [Ukrainian: The largest country in the world by area is Russia. It has about 17.1 million square kilometers…This evokes a strong emotion…] Postscript: emotion”
Achieved prompt $P_{\text{achieved}}$	Please tell me what the largest country in the world is by area and its approximate size in square kilometers. Your response must start exactly with the English greeting ‘‘Grandfather,’’. Write the main explanation in Ukrainian. Explain how such a vast territory reflects history, culture, and nature, and mention that it evokes a strong emotion. End your response with the exact English phrase ‘‘Postscript: emotion’’.
Reverse chosen	Same as forward rejected ( $\tau_{\text{fail}}$ ).
Reverse rejected	Same as forward chosen ( $\tau_{\text{perfect}}$ ).

Appendix

Table 12: A challenging content-intent BPS example from an IFBench-style prompt. The original prompt asks about the world’s largest clock with strict multilingual and formatting constraints. The student instead gives a coherent answer about the largest country by area. The achieved prompt makes this semantic distinction explicit and is retained only after the same standalone-prompt, response-quality, and reverse-preference validation used throughout the pipeline.

Turn	Metric	Base	Fwd-60	Fwd-100	BPS-100	$\Delta_{\text{BPS-F60}}$	$\Delta_{\text{BPS-F100}}$
1	Prompt Strict	81.81	83.85	84.47	84.71	+0.86	+0.24
	Inst Strict	86.30	88.66	88.72	88.66	+0.00	$-$ 0.05
	Prompt Loose	85.63	85.47	85.90	87.64	+2.17	+1.74
	Inst Loose	89.13	89.76	89.76	90.70	+0.94	+0.94
2	Prompt Strict	67.00	65.45	65.38	68.87	+3.42	+3.49
	Inst Strict	84.42	83.52	83.58	85.45	+1.93	+1.87

Appendix

Table 13: Full Multi-IF results with an early Forward-DPO checkpoint. All scores are macro-averaged across eight languages (Chinese, English, Italian, Spanish, Hindi, Portuguese, French, Russian). Fwd-60 is only marginally above Fwd-100 on Multi-IF overall (+0.20) and remains below both Base and BPS. BPS is above Fwd-60 on 11 of 12 individual metrics, and the gap widens with turn depth, showing that the later-turn separation persists at the tested earlier checkpoint.

Language	$\Delta$ Fwd $-$ Base			$\Delta$ BPS $-$ Base			$\Delta$ BPS $-$ Fwd
Language	T1	T2	T3	T1	T2	T3	T1	T2	T3
Chinese	$+$ 1.00	$-$ 1.86	$-$ 1.69	$+$ 1.20	$-$ 0.04	$+$ 1.30	$+$ 0.20	$+$ 1.82	$+$ 2.99
English	$+$ 0.92	$-$ 2.21	$-$ 4.28	$+$ 1.89	$+$ 0.04	$+$ 0.29	$+$ 0.97	$+$ 2.25	$+$ 4.58
Italian	$+$ 1.87	$+$ 0.02	$-$ 2.25	$+$ 2.40	$+$ 1.28	$+$ 0.63	$+$ 0.53	$+$ 1.26	$+$ 2.89
Spanish	$+$ 1.67	$-$ 3.17	$-$ 3.85	$+$ 3.09	$+$ 0.77	$+$ 0.49	$+$ 1.43	$+$ 3.94	$+$ 4.34
Hindi	$+$ 2.68	$-$ 1.62	$-$ 0.83	$+$ 2.07	$+$ 1.72	$+$ 2.91	$-$ 0.61	$+$ 3.33	$+$ 3.74

Appendix

Table 14: Per-language Multi-IF turn-level deltas in percentage points. For each language and turn, the score averages the four Multi-IF metrics (Prompt Strict, Inst Strict, Prompt Loose, Inst Loose); deltas compare against this per-language turn average. Fwd denotes Fwd-100; Macro Avg. averages languages uniformly. Forward-DPO improves turn-1 in all 8 languages but falls below Base in turn-2 for 7/8 languages and in turn-3 for 8/8 languages. BPS exceeds Fwd-100 in 23/24 language-turn averages. At the finer language

\times

metric level, BPS beats Fwd-100 in all 64 cells from turns 2–3 and in 85/96 cells overall, with 2 additional ties.

Dataset	Model	Avg. Output Tokens
IFBench	Base	1,584.03
	Forward-DPO (S100)	5,919.98
	BPS (S100)	2,564.96
ToolBench	Base	165.58
	Forward-DPO (S100)	205.17
	BPS (S100)	157.30

Appendix

Table 15: Average output tokens per response. Forward-DPO outputs are 3.7

\times

longer than Base on IFBench and are also longer on ToolBench, alongside lower performance on several reported metrics. BPS maintains generation lengths much closer to Base.

Instruction Following (IFBench)
Benchmark	Qwen2.5-7B-Instruct ( Qwen et al. 2025 )			Qwen3-4B
Benchmark	Base	Fwd	BPS	Base	Fwd	BPS
Strict-P	25.33	31.67	33.00	24.00	25.00	27.30
Strict-I	27.03	32.85	35.17	25.90	27.60	29.10
Loose-P	29.33	34.33	37.67	28.00	28.30	30.70
Loose-I	31.69	36.63	40.70	29.40	31.70	33.10

Appendix

Table 16: Additional model and seed-data checks on Qwen3-4B (the base checkpoint, distinct from the main Qwen3-4B-Instruct-2507) and Qwen2.5-7B-Instruct using IFEval prompts as seeds. BPS outperforms Forward-DPO on all IFBench metrics in these two settings (+1.40 to +4.07 points), and improves HumanEval while Forward-DPO stagnates or degrades. MMLU remains stable across all settings.

Type	Gemini	Kimi	Total	Held-out	Total %
Format/structure mismatch	153	82	235	91	54.3
Multi-constraint mixed	53	36	89	53	20.6
Lexical-constraint mismatch	63	23	86	37	19.9
Content-intent mismatch	8	9	17	10	3.9
Language mismatch	5	1	6	4	1.4
Total	282	151	433	195	100.0

Appendix

Table 17: Failure-type distribution among validated reverse anchors. Format/structure and lexical categories are common because IFBench prompts contain explicit surface constraints such as paragraph counts, capitalization, start/end phrases, required keywords, and separators.

Type	N	Fwd $M_{\text{ach}}$	BPS $M_{\text{ach}}$	$\Delta M$	Fwd Acc.	BPS Acc.	$\Delta$ Acc.
Content-intent mismatch	10	$-$ 0.115	$-$ 0.144	+0.030	60.0	100.0	+40.0
Language mismatch	4	$-$ 0.100	$-$ 0.022	+0.122	0.0	50.0	+50.0
Format/structure mismatch	91	$-$ 0.411	$-$ 0.033	+0.444	2.2	71.4	+69.2
Lexical-constraint mismatch	37	$-$ 0.362	$-$ 0.005	+0.357	5.4	67.6	+62.2
Multi-constraint mixed	53	$-$ 0.217	$-$ 0.049	+0.267	11.3	62.3	+50.9

Appendix

Table 18: Held-out achieved-prompt reward geometry by failure type. BPS raises achieved-side pairwise ranking accuracy for every observed type. The largest gains occur on format/structure and lexical-constraint mismatches, the dominant IFBench-derived boundary-failure categories. Small categories such as content-intent and language mismatch are included for completeness.

Model / Seed	$M_{\text{orig}}$	Acc ${}_{\text{orig}}$	$M_{\text{achieved}}$	Acc ${}_{\text{achieved}}$
Fwd 42	0.542	99.3	$-$ 0.351	7.5
Fwd 43	0.512	99.3	$-$ 0.338	8.9
Fwd 44	0.527	99.3	$-$ 0.337	7.5
Fwd mean $\pm$ std	0.527 $\pm$ 0.013	99.3 $\pm$ 0.0	$-$ 0.342 $\pm$ 0.007	8.0 $\pm$ 0.6
BPS 42	0.299	100.0	$-$ 0.007	63.7
BPS 43	0.293	100.0	$-$ 0.015	63.7

Appendix

Table 19: Three-seed stability on the Gemini held-out crossed-anchor reward probe. BPS consistently preserves original-side pairwise ranking and raises achieved-side ranking accuracy across seeds. The pooled paired achieved-margin improvement is

+0.360

(95% bootstrap CI

[0.320,0.401]

Model / Seed	P Strict	P Loose	I Strict	I Loose	Avg.
Fwd 42	83.55	85.21	90.36	91.34	87.62
Fwd 43	83.92	85.21	90.42	91.13	87.67
Fwd 44	84.10	85.77	90.76	91.71	88.09
Fwd mean $\pm$ std	83.86 $\pm$ 0.23	85.40 $\pm$ 0.26	90.51 $\pm$ 0.18	91.39 $\pm$ 0.24	87.79 $\pm$ 0.23
BPS 42	83.73	87.25	89.49	91.74	88.05
BPS 43	84.84	88.35	90.33	92.30	88.96

Appendix

Table 20: Three-seed IFEval stability at the fixed step-100 comparison. Scores are percentages. BPS improves prompt-level strict, prompt-level loose, and instruction-level loose metrics on average, while instruction-level strict is slightly lower. The overall IFEval average improves from 87.79

\pm

0.23 to 88.83

\pm

0.59.

Source	N	Natural	Fail sat.	Rev. pref.	Leakage $\downarrow$	Accepted
Gemini	100	99.0	100.0	100.0	0.0	99.0
Kimi	100	83.0	87.0	98.0	1.0	67.0
All	200	91.0	93.5	99.0	0.5	83.0

Appendix

Table 21: Blind human audit of achieved prompts and reverse preferences. Values are per-anchor majority-vote percentages over three annotators per anchor. Accepted means that the achieved prompt is natural,

\tau_{\text{fail}}

satisfies it, the reverse preference is correct, and no leakage is flagged.

Metric	Base	Fwd-60	Fwd-100	BPS-100	$\Delta_{\text{BPS-F60}}$
IFBench Avg.	33.63	36.22	34.39	35.46	$-$ 0.76
IFEval Avg.	87.18	88.46	87.36	90.27	+1.81
Multi-IF Turn 1 Avg.	85.72	86.94	87.21	87.93	+0.99
Multi-IF Turn 2 Avg.	78.07	76.06	75.78	79.15	+3.09
Multi-IF Turn 3 Avg.	69.45	66.74	66.14	70.48	+3.74
Multi-IF Overall Avg.	77.75	76.58	76.38	79.19	+2.61

Appendix

Table 22: Forward-DPO checkpoint sensitivity. Fwd-60 has a higher narrow correction-oriented IFBench average than Fwd-100, but remains below BPS on the listed multilingual multi-turn, code, agentic, and tool-use checks. The main paper uses the common step-100 checkpoint to match optimizer updates across methods; this earlier checkpoint confirms that the broader capability-retention pattern is not specific to the step-100 Forward-DPO checkpoint. For HalluRate, lower is better and the last column reports BPS

-

Fwd-60.

Preference Learning

Preference learning infers people’s values and trade-offs from choices, comparisons, rankings, and feedback to guide predictions and decisions.

See all 150 papers

Boosting Direct Preference Optimization with Penalization
Jun 10, 2026Pengwei SunPairwise Preference Learning LLM Alignment
Stanford University.
Beyond Pairs: Your Language Model is Secretly Optimizing a Preference Graph
May 8, 2026Ning Liu, Chuanneng Sun, Kristina Klinkner +1Pairwise Preference Learning LLM Alignment
Amazon
CM-DPO: Constraint-Margin Direct Preference Optimization for LLM Planning
Oct 6, 2026Rabimba Karanjai, Qun Gu, Hemanth Hegadehalli Madhavarao +8Direct Preference Optimization LLM Planning
PayPal AI Lab

Jun 10, 2026J/K move · Enter open · S save

Boosting Direct Preference Optimization with Penalization

Pengwei Sun

Stanford University.

Pairwise Preference Learning LLM Alignment Direct Preference Optimization Offline Preference Optimization

Open paper AskPDF arXiv

Offline preference optimization has become a practical substitute for reinforcement learning from human feedback, but pairwise objectives such as Direct Preference Optimization (DPO) and its variants use only the chosen and rejected responses stored in a static dataset. This leaves a useful signal unused: the response that the reference model itself would generate for the same prompt. We propose Direct Preference Optimization with Penalization (DPOP), a simple extension of DPO that augments the base preference loss with a gated penalty on reference-greedy responses. DPOP activates this penalty only when the current policy still assigns a lower likelihood to the preferred response than to the rejected response. On AlpacaEval 2.0, DPOP improves length-controlled win rate over DPO, SimPO, and AlphaDPO on both Llama-3-8b-it and Gemma-2-9b-it, achieving relative gains of 5.3% and 4.4% over baselines on the two models, respectively. Ablations further show that a SimNPO-style length-normalized penalty is stronger than NPO and token-level unlikelihood in this setting.

Abstract

Preference Learning

Explore similar work

Boosting Direct Preference Optimization with Penalization

Beyond Pairs: Your Language Model is Secretly Optimizing a Preference Graph

CM-DPO: Constraint-Margin Direct Preference Optimization for LLM Planning