Practitioners choose among LLM alignment methods largely by trial and error. We provide a theoretical foundation: each pairwise method defines a vector field on the probability simplex that governs how the model redistributes mass between preferred and dispreferred completions. Projecting onto a single preferred/dispreferred pair reduces this to a one-dimensional margin ODE with a closed-form solution. These solutions reveal that DPO margins grow without bound (logarithmically as O(logt) in the gradient-flow limit) while IPO margins converge to a finite 1/(2τ); that label noise induces a qualitative phase transition in DPO (unbounded → bounded) but affects IPO only quantitatively; and that gradient concentration under asymmetric weighting governs trainability at scale. We confirm all three predictions across five model families spanning four benchmarks. The closed-form solutions also enable principled method design: we prove that adding any label smoothing ε>0 to SimPO creates a finite margin attractor at Δ∗=β−1log((1−ε)/ε), while ε=0 (standard SimPO) has no attractor. We test this across three models (8B-9B) and confirm the prediction: the theory-derived configuration (TPO-S, ε=0.05) is competitive with the strongest baselines, attaining the highest mean IFEval score among the compared methods on all three, validating that the predicted finite attractor is a usable operating point rather than proposing a new state-of-the-art loss.
Figures & tables
Figure 1: Three axes of the margin dynamics framework. (a) SFT only attracts toward yw ( F+ ), with pressure vanishing as π(yl)→0 . DPO adds targeted repulsion F− , sustaining Ω(π) pressure. (b) Sigmoid weighting (DPO) yields unbounded margin growth; linear weighting (IPO, PPO) converges. (c) Frozen references cause gradient decay; reference-free methods risk drift; adaptive references stay stable.
Figure 2: Effective potential V(Δ) (schematic). SimPO (dashed) has no minimum: margins diverge. TPO-H and TPO-S have finite attractors at Δ∗ .
Method
Axis 1: Repulsion
Axis 2: Weighting
Axis 3: Reference
SFT / RFT / STaR
Implicit only
—
—
PPO / RLHF
Targeted
Linear (non-sat.)
Frozen πref (KL)
GRPO
Targeted (imbal.)
Advantage-weighted
None
DPO
Targeted
Sigmoid (saturating)
Frozen πref
IPO
Targeted
Linear (non-sat.)
Frozen πref
SimPO
Targeted
Sigmoid (saturating)
None
Table 1: Three-axis taxonomy of LLM training methods.
Result
Prediction
Experiment
Confirmed?
Main predictions (closed-form margin ODEs)
Thm. 2.3
DPO Δ∼logt ; IPO Δ→1/(2τ)
Fig. 3
✓ functional form
Thm. 2.4
Noise: DPO phase transition, IPO rate-preserving
Fig. 5
✓ 8% error
Thm. 2.2
γ=0 vs γ>0 sharp separation
Fig. 5
✓ qualitative
Supporting predictions (Appendix)
Thm. 2.5
High γ→ collapse
Fig. 5
✓ non-monotonic
Table 2: Theory-to-experiment mapping. Main predictions test the closed-form margin ODEs; supporting predictions are in the appendix. A single calibration parameter βeff maps the two-completion theory to full-scale training.
Figure 3: Margin and gradient dynamics over 2000 training steps (Theorem 2.3 ). Left: DPO’s margin grows without bound, consistent with u+logu=at+c ( R2=0.85 ; the remaining variance reflects discrete-time effects and cross-example interference not captured by the isolated ODE). Right: DPO’s gradient norm is 10× SFT’s, confirming that targeted repulsion sustains Ω(π) gradient pressure (Definition 2.1 ).
Figure 6
DS-R1-8B
QW3.5-9B
MI-8B
TPO-S
66.8 ± 0.3
61.4 ± 0.4
76.5 ± 0.3
TPO-H
66.4 ± 0.3
60.8 ± 0.3
76.1 ± 0.9
SimPO
66.5 ± 0.6
60.9 ± 0.2
73.9 ± 1.4
DPO
65.1 ± 0.2
60.6 ± 0.8
71.9 ± 0.6
KTO
64.2 ± 0.8
60.4 ± 0.5
72.2 ± 1.0
cDPO
64.1 ± 0.7
59.8 ± 0.4
70.6 ± 0.5
Table 3: IFEval instruction-following accuracy (%) on UltraFeedback-trained models (mean ± std, 5 seeds). IFEval is an objective benchmark with no judge model. The theory-derived TPO-S attains the highest mean among the compared methods on all three, confirming the predicted attractor is a competitive operating point. TPO-S adds negligible wall-clock overhead over SimPO: a single extra logσ(−δ) evaluation per example (the ε -weighted label-smoothing term in Eq. 3 ). 2 2 2 p -values (TPO-S vs. SimPO, Welch’s t -test, 5 seeds): MI p=0.013 , QW p=0.047 , DS p=0.36 .
On-policy (Math)
Off-policy (UltraFeedback)
DS-8B
QW-9B
MI-8B
DS-8B
QW-9B
MI-8B
TPO-S
73.6 ± .5
72.0 ± 1.9
59.4 ± 1.1
55.2 ± 1.9
59.4 ± 1.9
58.2 ± 2.2
SimPO
72.4 ± 1.9
69.4 ± 2.4
58.6 ± 6.2
51.2 ± .8
54.6 ± 2.4
60.2 ± .8
TPO-H
71.4 ± 2.6
68.0 ± 1.9
59.8 ± 1.9
47.8 ± .8
53.4 ± 4.2
51.2 ± 1.1
KTO
50.2 ± 1.3
64.2 ± .8
59.6 ± 1.1
46.8 ± 1.1
52.0 ± 2.0
52.8 ± 1.5
DPO
47.0 ± 2.0
61.0 ± 2.3
55.6 ± 8.8
45.4 ± .5
47.0 ± 4.8
55.2 ± 2.8
Table 4: Preference accuracy (%, mean ± std, 5 seeds) on on-policy math data and off-policy UltraFeedback ( Cui et al., 2024 ) (60K pairs). TPO-S leads or ties on most settings. The reference-free vs. reference-dependent gap exceeds 24 points on DS-R1-8B.
Figure 6: Margin dynamics replicated across five model families (Qwen, Gemma, OLMo, Mistral, DeepSeek). DPO’s margin grows without bound and IPO’s converges near the predicted fixed point 1/(2τ)=5 (dashed line) on every architecture. Only the growth rate varies across families.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Reference divergence over training (Proposition A.4 ). SimPO (reference-free) diverges 82× more than DPO (frozen reference), confirming the staleness–drift tradeoff predicted by Axis 3: frozen references constrain policy drift but cause gradient decay; reference-free methods avoid decay but risk unbounded drift.
Model
Family
DPO Δ3000
IPO Δ3000
Qwen3-8B (primary)
Qwen
1540
∼ 5
Qwen3.5-4B
Qwen
579
1.2
Qwen3-4B
Qwen
367
5.4
Gemma 3 4B
Google
305
5.0
OLMo-2 7B
AI2
629
2.0
Mistral 7B
Mistral
204
0.5
Appendix
Table 5: Cross-architecture validation of margin dynamics (Theorem 2.3 ). DPO’s margin grows without bound; IPO’s converges near 1/(2τ)=5 . Pattern holds across all seven models and five families.
Method
Axes (1 / 2 / 3)
100
500
7500
Theory predicts
SFT
Implicit / — / —
44
61
72
Self-limiting (Thm. 2.2 )
RFT
Implicit / — / —
52
49
47
Overfits without repulsion
GRPO
Targeted / Adv. / None
49
70
48
Fails: sparse rewards (Prop. A.3 )
DPO
Targeted / Sigmoid / Frozen
67–71
74
78
Strong but stale (Prop. A.4 )
Appendix
Table 6: Accuracy (%) on MATH across data scales. Preference methods with targeted repulsion outperform supervised methods; GRPO degenerates under sparse rewards.
Method
Base
+Div
GRPO
24%
53%
RFT
34%
38%
SFT iter
35%
38%
Appendix
Table 7: Left: Diversity selection as a plug-in (100-sample MATH, K=4 ). Right: Coverage over 5 iterations (MATH, 7500 problems, K=8 ).
Method
MATH
GSM8K
AIME
Theory basis
DPO ( γ=1 , β=0.1 )
93.8 ± 3.0
100.0 ± 0.0
11.8 ± 1.8
—
TPO ( γ=0.5 , β/L )
98.7 ± 1.2
100.0 ± 0.0
12.3 ± 0.8
Thms. 2.5 , 2.3
Appendix
Table 8: TPO vs. standard DPO on mathematical reasoning (mean ± std over 3 seeds).
Method
Best Acc (%)
DPO (frozen reference)
50
DPO + refresh K=2000
68
DPO + refresh K=1000
71
DPO + refresh K=500
71
DPO + refresh K=200
73
SimPO (no reference)
75
Appendix
Table 9: Reference refresh experiment (DS-R1-8B, on-policy). Refreshing the frozen reference every K steps recovers most of the 25-point gap between DPO and SimPO.
n
Pairs
Δ0
Δ50
Δ300
Growth ( Δ300/Δ50 )
Qualitative
2
20
− 0.0
11.1
156.1
14.1 ×
Unbounded
4
60
80.3
185.9
281.7
1.5 ×
Unbounded
8
140
106.0
227.4
360.2
1.6 ×
Unbounded
16
600
99.1
202.0
319.0
1.6 ×
Unbounded
Appendix
Table 10: N-completion validation of the two-completion effective theory. DPO margin dynamics at n∈{2,4,8,16} completions per prompt (DS-R1-8B, 300 steps). The growth ratio (final/step-50 margin) converges at n≥4 ( 1.5 – 1.6× ), confirming that the two-completion ODE captures the dominant dynamics regardless of the number of completions. Initial margins increase with n due to the larger pairwise comparison space, but the functional form (unbounded growth) is invariant.
DS-R1-8B
MI-8B
QW-9B
SimPO
49.2 ± 0.4
51.2 ± 0.4
—
TPO-H
49.6 ± 0.5
50.8 ± 0.4
—
TPO-S
49.2 ± 0.4
50.4 ± 0.5
—
KTO
49.0 ± 0.0
48.0 ± 0.0
—
DPO
49.0 ± 0.0
48.0 ± 0.0
—
cDPO
49.0 ± 0.0
47.0 ± 0.0
—
Appendix
Table 11: SimPO replication pipeline (SFT → preference). Best-checkpoint preference accuracy (%) on UltraFeedback, mean ± std over 5 seeds. The reference-free advantage persists on MI 8B; on DS 8B the strong SFT base saturates all methods.
MATH
GSM8K
AIME
MBPP
(11.6K, 3000 steps)
(7K, 3000 steps)
(30, 3000 steps)
(374, 1000 steps)
Prediction 1: Margin trajectories
DPO Δfinal
465
257
15
160
IPO Δfinal
0.4
8.5
− 0.3
1.1
Prediction 2: Noise ( ε=0.1 )
DPO Δfinal
44
41
8.8
46
Appendix
Table 12: Multi-benchmark replication of main predictions. DPO margin grows without bound on every benchmark; IPO margin remains bounded. Noise degrades DPO monotonically while IPO is stable. The γ=0 vs. γ>0 gap holds wherever the task is solvable ( pt>0 ). AIME margins are near zero for both methods, consistent with Theorem A.9 : when base solve rate pt≈0 , neither method can make progress.
γ
0.0
0.25
0.5
0.6
0.7
0.75
0.8
1.0
MATH (11.6K, transition regime)
Final accuracy (%)
66
91
98
100
96
92
88
98
Final margin
58
278
493
864
398
304
249
1067
Collapse events (/50)
1
47
9
34
45
42
43
4
MATH seed robustness (mean ± range over 2–3 seeds)
Mean accuracy (%)
66
91
97
91
91
87
88
89
Appendix
Table 13: Gradient concentration sweep on MATH (11.6K train, 3000 steps) and GSM8K (7K train). Accuracies here are measured on the 50 tracked hard examples used for the per-example collapse analysis, so they are lower than the held-out accuracies in Table 12 (e.g. γ=0.25 : 91% tracked-hard vs. 99.5% held-out). Collapse events peak at γ=0.7 – 0.8 on MATH, creating a “collapse valley.” GSM8K shows no collapse (high-coverage regime). Seed robustness (2–3 seeds) confirms the pattern is reproducible.