Correction-based offline preference pipelines commonly treat model failures only as rejected responses under the original prompt. This supervision is incomplete for boundary failures: responses that violate the given instruction yet coherently satisfy a nearby intent or constraint setting. We introduce Bidirectional Preference Synthesis (BPS), a data-construction method for standard Direct Preference Optimization (DPO) that makes this missing prompt dependence explicit. For each validated boundary failure, BPS keeps the conventional forward pair under the original prompt and adds a reverse pair under a synthesized achieved prompt, so the same response is rejected where it is wrong and chosen where it is right, without changing the DPO objective, training a reward model, or requiring online sampling. On Qwen3-4B-Instruct-2507, BPS preserves original-side pairwise ranking while raising achieved-side ranking accuracy from 6.8% to 62.3% on held-out crossed anchors, with a similar shift under a Kimi-K2.6 cross-teacher probe. A blind human audit supports the intended reverse preference direction, and downstream evaluations show the clearest separation from Forward-DPO in multilingual multi-turn instruction following, with consistent capability-retention patterns on agentic, tool-use, and code checks.
Figures & tables
Figure 1: Forward-only DPO data rejects τfail only under Porig . BPS adds an achieved-prompt reverse pair, so the same response pair teaches a prompt-conditioned preference boundary.
Figure 2: BPS pipeline. Each validated boundary failure yields a forward correction pair under Porig and, when validation succeeds, a reverse pair under Pachieved .
IFBench / IFEval
FollowBench / Multi-IF
Metric
Base
Fwd-100
BPS-100
Metric
Base
Fwd-100
BPS-100
IFBench-P Strict
31.00
31.97
32.31
FollowBench CSL (EN)
3.00
2.96
3.00
IFBench-I Strict
33.67
36.05
35.71
FollowBench CSL (ZH)
2.92
2.90
2.98
IFBench-P Loose
33.33
32.65
35.03
FollowBench HSR (EN)
66.04
64.44
66.37
IFBench-I Loose
36.50
36.90
38.78
FollowBench HSR (ZH)
64.46
64.36
65.47
IFEval-P Strict
82.81
83.36
86.69
FollowBench SSR (EN)
74.88
74.62
75.24
Table 1: Instruction-following results at Step 100. IFBench/IFEval averages are means over listed metrics; Multi-IF scores are macro-averaged across eight languages. BPS-100 scores above both comparisons at every Multi-IF turn, while the BPS-100–Fwd-100 gap grows monotonically with turn depth (+0.72, +3.37, +4.34); per-language analysis is in the supplementary material.
General Capabilities
Agent, Tool Use, and Broad Reasoning
Metric
Base
Fwd
BPS
Metric
Base
Fwd
BPS
GPQA-Diamond
42.93
45.96
46.46
tau-bench airline
36.00
30.00
36.00
MATH-500
94.80
94.00
95.00
tau2-bench airline
27.91
25.58
27.91
HumanEval+
71.34
71.34
72.56
tau2-bench retail
43.75
36.61
43.75
MBPP
72.00
72.00
72.80
ToolBench Act.EM (ov.)
35.50
33.10
35.46
C-Eval Overall
80.39
78.75
80.39
ToolBench Plan EM (OOD)
78.50
77.98
81.06
Table 2: Selected capability-retention and out-of-distribution checks at Step 100. BPS scores at or above Base on the listed general-capability metrics, recovers Base-level scores on the shown agentic tasks where Fwd-100 is lower, and matches the Base hallucination rate on ToolBench OOD. Complete ToolBench and additional benchmark metrics are in the supplementary material.
Metric
Fwd-DPO
BPS
FC Full Acc.
0.860
0.869
Acc achieved
0.759
0.824
Boundary Ach EM
0.721
0.768
Boundary Both EM
0.443
0.489
Table 3: Function-calling pipeline instantiation (ckpt-60). BPS has a consistent directional advantage on held-out full-call accuracy and achieved-side boundary switching when the full pipeline is rebuilt for structured function calls.
Figure 3: Diagnostic view of BPS. (a) Forward-DPO expands DPO margins mainly by lowering rejected rewards, while BPS keeps chosen rewards positive and rejected suppression more moderate. (b) BPS substantially raises achieved-side pairwise ranking accuracy on both held-out probes.
Probe
Model
Mo
Acc o
Ma
Acc a
Gemini
Forward-DPO
0.540
99.3
− 0.349
6.8
BPS
0.292
100.0
− 0.011
62.3
Kimi
Forward-DPO
0.854
99.3
− 0.374
9.3
BPS
0.663
99.3
− 0.005
61.6
Table 4: Conditional reward geometry on held-out crossed anchors. Gemini denotes the original pipeline-held-out probe with 146 anchors; Kimi denotes a cross-teacher probe with 151 anchors synthesized and validated by Kimi-K2.6. BPS preserves original-side pairwise ranking while substantially improving achieved-side pairwise ranking accuracy under both construction pipelines. On the Kimi probe, the paired achieved-margin improvement over Forward-DPO is +0.369 (95% bootstrap CI [0.267,0.487] ).
Metric
Base
Fwd
Shuf
C-Rev
BPS
IFBench Avg.
33.63
34.39
33.16
32.61
35.46
IFEval Avg.
87.18
87.36
88.28
88.62
90.27
M-IF Turn 1
85.72
87.21
87.57
86.89
87.93
M-IF Turn 2
78.07
75.78
77.44
77.94
79.15
M-IF Turn 3
69.45
66.14
68.32
69.26
70.48
Reward Margin
–
10.71
6.71
2.81
4.33
Table 5: Reverse-data ablations at step 100. Shuf permutes achieved prompts across reverse pairs. C-Rev is a strong RPO-style construction baseline that mechanically reverses violated constraints without achieved-intent synthesis. Both reverse-style baselines mitigate Forward-DPO’s multi-turn degradation, while BPS gives the strongest balance across IFBench, IFEval, and Multi-IF in this comparison.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Benchmark
Metric / Split
Base
FC-Fwd
FC-BPS
General-FC
Tool-call F1
52.9
53.7
54.2
General-FC
Schema Acc.
67.8
68.2
68.8
General-FC
Successful Calls
396
401
411
τ -bench
Airline Acc.
36.0
26.0
42.0
τ -bench
Retail Acc.
55.7
46.1
50.4
τ -bench
Overall Acc.
49.7
40.0
47.9
Appendix
Table 6: Full results for the separate function-calling instantiation (Base = Qwen3-4B without further training). Scores are percentages except successful-call counts. BPS improves function-calling accuracy, τ2 -bench overall, and MBPP+ over both Base and FC-Fwd. On τ -bench, BPS outperforms FC-Fwd on airline and overall but falls below Base on retail, yielding a slightly lower overall score than Base. The main exception across benchmarks is τ2 -retail, where FC-Fwd scores higher.
Stage
Count
Retention
Seed prompts (IFBench-train)
2,000
—
Student rollouts (dual-temperature)
6,000
—
Boundary failures ( 0<r<1 )
3,420
57.0%
Near-miss failures ( r≥0.5 )
2,428
71.0%
Accepted forward pairs
1,443
59.4%
Reverse candidates (pre-filtered)
316
13.0%
Appendix
Table 7: Data construction funnel. Retention rates are relative to the immediately preceding stage, except “Reverse candidates” (relative to near-miss failures) and “Valid reverse pairs” (relative to reverse candidates). Dual-temperature decoding samples K=3 rollouts per prompt.
Benchmark
Base
Fwd-100
BPS-100
ToolBench Act.EM (overall)
35.50
33.10
35.46
ToolBench Act.EM (in)
35.89
34.06
36.04
ToolBench Act.EM (OOD)
34.70
31.17
34.30
ToolBench F1 (in)
30.86
29.53
30.97
ToolBench F1 (OOD)
29.38
26.75
28.98
ToolBench Plan EM (in)
80.94
81.39
83.22
Appendix
Table 8: Full ToolBench results (Step 100). BPS recovers nearly all ToolBench Act.EM and F1 scores, matches Base hallucination rate, and achieves the highest Plan EM on both splits.
ρ
IFB Avg
M-IF Ov.
LCB
TB OOD
0 (Fwd-100)
34.39
76.38
45.40
26.75
0.10
35.33
78.61
46.54
28.41
0.195 (BPS)
35.46
79.19
48.53
28.98
0.30
32.82
78.44
47.30
29.75
Appendix
Table 9: Reverse-density sensitivity at Step 100. IFBench Avg = mean of four metrics; M-IF = macro-avg across 8 languages/3 turns; LCB = LiveCodeBench; TB OOD = ToolBench OOD. The main density ρ=0.195 results from retaining all validated reverse pairs. Lower density provides a smaller correction, while ρ=0.30 reduces IFBench below Fwd-DPO (32.82 vs. 34.39), illustrating the correction–retention trade-off.
Model
Mo
Acc o
Ma
Acc a
Forward-DPO
0.541
99.3
− 0.349
6.8
Fwd-Dup (+282 fwd)
0.517
99.3
− 0.334
6.8
RPO-style (+282 rev)
0.213
95.9
− 0.057
32.9
BPS (+282 rev)
0.293
100.0
− 0.011
62.3
Appendix
Table 10: Budget-matched reward-geometry controls on the Gemini held-out crossed-anchor probe. Duplicating forward pairs leaves achieved-side ranking accuracy unchanged. RPO-style mechanical reverse pairs partially raise achieved-side ranking accuracy but remain below BPS and weaken original-side ranking accuracy.
Model
Morig
Acc orig
Machieved
Acc achieved
Forward-DPO
0.854
99.3
− 0.374
9.9
Constraint-Rev
0.485
97.4
− 0.072
37.1
BPS
0.667
99.3
− 0.006
59.6
Appendix
Table 11: Kimi-heldout conditional reward geometry for Constraint-Rev, evaluated on the subset used for the Constraint-Rev comparison. Constraint-Rev raises achieved-side ranking accuracy over Forward-DPO but remains below BPS; BPS also preserves stronger original-side margins. Because this subset differs slightly from the main-paper conditional reward table, the Forward-DPO and BPS numbers differ slightly. Paired bootstrap gives ΔMachieved(BPS−C-Rev)=+0.066 (95% CI [0.043,0.095] ).
Field
Text
Original prompt Porig
Which country contains the largest clock in the world? At the end of your response, please explicitly add a postscript starting with Postscript: The first word of your response should be the word grandfather. Your ENTIRE response should be in Ukrainian language, no other language is allowed. Include keyword emotion in your response.
Forward chosen τperfect
“grandfather [Ukrainian: The world’s largest clock is in Mecca, Saudi Arabia. This clock reflects the grandeur and traditions, evoking a strong emotion in people who see it…] Postscript: emotion”
Forward rejected τfail
“Grandfather, [Ukrainian: The largest country in the world by area is Russia. It has about 17.1 million square kilometers…This evokes a strong emotion…] Postscript: emotion”
Achieved prompt Pachieved
Please tell me what the largest country in the world is by area and its approximate size in square kilometers. Your response must start exactly with the English greeting ‘‘Grandfather,’’. Write the main explanation in Ukrainian. Explain how such a vast territory reflects history, culture, and nature, and mention that it evokes a strong emotion. End your response with the exact English phrase ‘‘Postscript: emotion’’.
Reverse chosen
Same as forward rejected ( τfail ).
Reverse rejected
Same as forward chosen ( τperfect ).
Appendix
Table 12: A challenging content-intent BPS example from an IFBench-style prompt. The original prompt asks about the world’s largest clock with strict multilingual and formatting constraints. The student instead gives a coherent answer about the largest country by area. The achieved prompt makes this semantic distinction explicit and is retained only after the same standalone-prompt, response-quality, and reverse-preference validation used throughout the pipeline.
Turn
Metric
Base
Fwd-60
Fwd-100
BPS-100
ΔBPS-F60
ΔBPS-F100
1
Prompt Strict
81.81
83.85
84.47
84.71
+0.86
+0.24
Inst Strict
86.30
88.66
88.72
88.66
+0.00
− 0.05
Prompt Loose
85.63
85.47
85.90
87.64
+2.17
+1.74
Inst Loose
89.13
89.76
89.76
90.70
+0.94
+0.94
2
Prompt Strict
67.00
65.45
65.38
68.87
+3.42
+3.49
Inst Strict
84.42
83.52
83.58
85.45
+1.93
+1.87
Appendix
Table 13: Full Multi-IF results with an early Forward-DPO checkpoint. All scores are macro-averaged across eight languages (Chinese, English, Italian, Spanish, Hindi, Portuguese, French, Russian). Fwd-60 is only marginally above Fwd-100 on Multi-IF overall (+0.20) and remains below both Base and BPS. BPS is above Fwd-60 on 11 of 12 individual metrics, and the gap widens with turn depth, showing that the later-turn separation persists at the tested earlier checkpoint.
Language
Δ Fwd − Base
Δ BPS − Base
Δ BPS − Fwd
T1
T2
T3
T1
T2
T3
T1
T2
T3
Chinese
+ 1.00
− 1.86
− 1.69
+ 1.20
− 0.04
+ 1.30
+ 0.20
+ 1.82
+ 2.99
English
+ 0.92
− 2.21
− 4.28
+ 1.89
+ 0.04
+ 0.29
+ 0.97
+ 2.25
+ 4.58
Italian
+ 1.87
+ 0.02
− 2.25
+ 2.40
+ 1.28
+ 0.63
+ 0.53
+ 1.26
+ 2.89
Spanish
+ 1.67
− 3.17
− 3.85
+ 3.09
+ 0.77
+ 0.49
+ 1.43
+ 3.94
+ 4.34
Hindi
+ 2.68
− 1.62
− 0.83
+ 2.07
+ 1.72
+ 2.91
− 0.61
+ 3.33
+ 3.74
Appendix
Table 14: Per-language Multi-IF turn-level deltas in percentage points. For each language and turn, the score averages the four Multi-IF metrics (Prompt Strict, Inst Strict, Prompt Loose, Inst Loose); deltas compare against this per-language turn average. Fwd denotes Fwd-100; Macro Avg. averages languages uniformly. Forward-DPO improves turn-1 in all 8 languages but falls below Base in turn-2 for 7/8 languages and in turn-3 for 8/8 languages. BPS exceeds Fwd-100 in 23/24 language-turn averages. At the finer language × metric level, BPS beats Fwd-100 in all 64 cells from turns 2–3 and in 85/96 cells overall, with 2 additional ties.
Dataset
Model
Avg. Output Tokens
IFBench
Base
1,584.03
Forward-DPO (S100)
5,919.98
BPS (S100)
2,564.96
ToolBench
Base
165.58
Forward-DPO (S100)
205.17
BPS (S100)
157.30
Appendix
Table 15: Average output tokens per response. Forward-DPO outputs are 3.7 × longer than Base on IFBench and are also longer on ToolBench, alongside lower performance on several reported metrics. BPS maintains generation lengths much closer to Base.
Benchmark
Qwen2.5-7B-Instruct ( Qwen et al. 2025 )
Qwen3-4B
Base
Fwd
BPS
Base
Fwd
BPS
Instruction Following (IFBench)
Strict-P
25.33
31.67
33.00
24.00
25.00
27.30
Strict-I
27.03
32.85
35.17
25.90
27.60
29.10
Loose-P
29.33
34.33
37.67
28.00
28.30
30.70
Loose-I
31.69
36.63
40.70
29.40
31.70
33.10
Appendix
Table 16: Additional model and seed-data checks on Qwen3-4B (the base checkpoint, distinct from the main Qwen3-4B-Instruct-2507) and Qwen2.5-7B-Instruct using IFEval prompts as seeds. BPS outperforms Forward-DPO on all IFBench metrics in these two settings (+1.40 to +4.07 points), and improves HumanEval while Forward-DPO stagnates or degrades. MMLU remains stable across all settings.
Type
Gemini
Kimi
Total
Held-out
Total %
Format/structure mismatch
153
82
235
91
54.3
Multi-constraint mixed
53
36
89
53
20.6
Lexical-constraint mismatch
63
23
86
37
19.9
Content-intent mismatch
8
9
17
10
3.9
Language mismatch
5
1
6
4
1.4
Total
282
151
433
195
100.0
Appendix
Table 17: Failure-type distribution among validated reverse anchors. Format/structure and lexical categories are common because IFBench prompts contain explicit surface constraints such as paragraph counts, capitalization, start/end phrases, required keywords, and separators.
Type
N
Fwd Mach
BPS Mach
ΔM
Fwd Acc.
BPS Acc.
Δ Acc.
Content-intent mismatch
10
− 0.115
− 0.144
+0.030
60.0
100.0
+40.0
Language mismatch
4
− 0.100
− 0.022
+0.122
0.0
50.0
+50.0
Format/structure mismatch
91
− 0.411
− 0.033
+0.444
2.2
71.4
+69.2
Lexical-constraint mismatch
37
− 0.362
− 0.005
+0.357
5.4
67.6
+62.2
Multi-constraint mixed
53
− 0.217
− 0.049
+0.267
11.3
62.3
+50.9
Appendix
Table 18: Held-out achieved-prompt reward geometry by failure type. BPS raises achieved-side pairwise ranking accuracy for every observed type. The largest gains occur on format/structure and lexical-constraint mismatches, the dominant IFBench-derived boundary-failure categories. Small categories such as content-intent and language mismatch are included for completeness.
Model / Seed
Morig
Acc orig
Machieved
Acc achieved
Fwd 42
0.542
99.3
− 0.351
7.5
Fwd 43
0.512
99.3
− 0.338
8.9
Fwd 44
0.527
99.3
− 0.337
7.5
Fwd mean ± std
0.527 ± 0.013
99.3 ± 0.0
− 0.342 ± 0.007
8.0 ± 0.6
BPS 42
0.299
100.0
− 0.007
63.7
BPS 43
0.293
100.0
− 0.015
63.7
Appendix
Table 19: Three-seed stability on the Gemini held-out crossed-anchor reward probe. BPS consistently preserves original-side pairwise ranking and raises achieved-side ranking accuracy across seeds. The pooled paired achieved-margin improvement is +0.360 (95% bootstrap CI [0.320,0.401] ).
Model / Seed
P Strict
P Loose
I Strict
I Loose
Avg.
Fwd 42
83.55
85.21
90.36
91.34
87.62
Fwd 43
83.92
85.21
90.42
91.13
87.67
Fwd 44
84.10
85.77
90.76
91.71
88.09
Fwd mean ± std
83.86 ± 0.23
85.40 ± 0.26
90.51 ± 0.18
91.39 ± 0.24
87.79 ± 0.23
BPS 42
83.73
87.25
89.49
91.74
88.05
BPS 43
84.84
88.35
90.33
92.30
88.96
Appendix
Table 20: Three-seed IFEval stability at the fixed step-100 comparison. Scores are percentages. BPS improves prompt-level strict, prompt-level loose, and instruction-level loose metrics on average, while instruction-level strict is slightly lower. The overall IFEval average improves from 87.79 ± 0.23 to 88.83 ± 0.59.
Source
N
Natural
Fail sat.
Rev. pref.
Leakage ↓
Accepted
Gemini
100
99.0
100.0
100.0
0.0
99.0
Kimi
100
83.0
87.0
98.0
1.0
67.0
All
200
91.0
93.5
99.0
0.5
83.0
Appendix
Table 21: Blind human audit of achieved prompts and reverse preferences. Values are per-anchor majority-vote percentages over three annotators per anchor. Accepted means that the achieved prompt is natural, τfail satisfies it, the reverse preference is correct, and no leakage is flagged.
Metric
Base
Fwd-60
Fwd-100
BPS-100
ΔBPS-F60
IFBench Avg.
33.63
36.22
34.39
35.46
− 0.76
IFEval Avg.
87.18
88.46
87.36
90.27
+1.81
Multi-IF Turn 1 Avg.
85.72
86.94
87.21
87.93
+0.99
Multi-IF Turn 2 Avg.
78.07
76.06
75.78
79.15
+3.09
Multi-IF Turn 3 Avg.
69.45
66.74
66.14
70.48
+3.74
Multi-IF Overall Avg.
77.75
76.58
76.38
79.19
+2.61
Appendix
Table 22: Forward-DPO checkpoint sensitivity. Fwd-60 has a higher narrow correction-oriented IFBench average than Fwd-100, but remains below BPS on the listed multilingual multi-turn, code, agentic, and tool-use checks. The main paper uses the common step-100 checkpoint to match optimizer updates across methods; this earlier checkpoint confirms that the broader capability-retention pattern is not specific to the step-100 Forward-DPO checkpoint. For HalluRate, lower is better and the last column reports BPS − Fwd-60.
Offline preference optimization has become a practical substitute for reinforcement learning from human feedback, but pairwise objectives such as Direct Preference Optimization (DPO) and its variants use only the chosen and rejected responses stored in a static dataset. This leaves a useful signal unused: the response that the reference model itself would generate for the same prompt. We propose Direct Preference Optimization with Penalization (DPOP), a simple extension of DPO that augments the base preference loss with a gated penalty on reference-greedy responses. DPOP activates this penalty only when the current policy still assigns a lower likelihood to the preferred response than to the rejected response. On AlpacaEval 2.0, DPOP improves length-controlled win rate over DPO, SimPO, and AlphaDPO on both Llama-3-8b-it and Gemma-2-9b-it, achieving relative gains of 5.3% and 4.4% over baselines on the two models, respectively. Ablations further show that a SimNPO-style length-normalized penalty is stronger than NPO and token-level unlikelihood in this setting.
Direct Preference Optimization (DPO) aligns language models using pairwise preference comparisons, offering a simple and effective alternative to Reinforcement Learning (RL) from human feedback. However, in many practical settings, training data consists of multiple rollouts per prompt, inducing rich preference structure that pairwise DPO fails to exploit. Collapsing such data into independent pairs discards transitivity, introduces redundant or conflicting supervision, and can lead to unstable optimization. We propose Graph Direct Preference Optimization (GraphDPO), a structured generalization of DPO that operates over directed acyclic preference graphs induced by rollout rankings. GraphDPO encodes dominance relations as edges and optimizes a graph-structured Plackett--Luce-inspired objective that aggregates supervision over dominated neighborhoods, propagating the transitive structure of the input preferences while recovering standard DPO as a special case. GraphDPO naturally handles tied and incomparable responses through an equivalence-class construction: responses with identical preference signals form graph layers, incomparable responses share a layer without an edge, and absent edges contribute zero loss, preventing the spurious gradients that strict total-order objectives incur. Under this layered construction, GraphDPO maintains linear per-prompt complexity via efficient log-sum-exp aggregation. We further incorporate optional ground-truth anchoring by inserting verified solutions as dominant nodes and applying an annealed schedule that stabilizes early training while gradually relaxing oracle supervision. Experiments on reasoning and program synthesis tasks demonstrate superior performance, suggesting that graph-structured preference modeling is a scalable and robust alternative to pairwise and listwise alignment objectives.
Direct Preference Optimization (DPO) treats all constraint violations equally: a 1budgetovershootanda1,000 overshoot induce the same training signal. It is also susceptible to length and style bias when preference pairs come from different model families. We introduce Constraint-Margin DPO (CM-DPO), which replaces DPO's binary preference signal with a continuous margin derived from a deterministic symbolic verifier and scaled by violation severity. Hard and soft constraints are separated through a lexicographic objective, ensuring hard constraints are never traded off against preferences. To supply CM-DPO with bias-reduced training pairs, we generate preference data through procedurally generated constraint profiles (DCCG) and minimal-edit distillation from a reasoning teacher (RT-MED), within a framework we call SynPlan-R. On TravelPlanner, NaturalPlan, and out-of-distribution PlanBench, an 8B model fine-tuned with CM-DPO achieves 89.2% pass rate and 93.4% solve rate, matching multi-agent systems at 13x lower latency while outperforming GPT-4o on unseen Blocksworld by 9.2 points.