Margin Dynamics for Large Language Model Alignment
Organizations: Amazon
Abstract
Practitioners choose among LLM alignment methods largely by trial and error. We provide a theoretical foundation: each pairwise method defines a vector field on the probability simplex that governs how the model redistributes mass between preferred and dispreferred completions. Projecting onto a single preferred/dispreferred pair reduces this to a one-dimensional margin ODE with a closed-form solution. These solutions reveal that DPO margins grow without bound (logarithmically as in the gradient-flow limit) while IPO margins converge to a finite ; that label noise induces a qualitative phase transition in DPO (unbounded bounded) but affects IPO only quantitatively; and that gradient concentration under asymmetric weighting governs trainability at scale. We confirm all three predictions across five model families spanning four benchmarks. The closed-form solutions also enable principled method design: we prove that adding any label smoothing to SimPO creates a finite margin attractor at , while (standard SimPO) has no attractor. We test this across three models (8B-9B) and confirm the prediction: the theory-derived configuration (TPO-S, ) is competitive with the strongest baselines, attaining the highest mean IFEval score among the compared methods on all three, validating that the predicted finite attractor is a usable operating point rather than proposing a new state-of-the-art loss.
Figures & tables
| Method | Axis 1: Repulsion | Axis 2: Weighting | Axis 3: Reference |
|---|---|---|---|
| SFT / RFT / STaR | Implicit only | — | — |
| PPO / RLHF | Targeted | Linear (non-sat.) | Frozen (KL) |
| GRPO | Targeted (imbal.) | Advantage-weighted | None |
| DPO | Targeted | Sigmoid (saturating) | Frozen |
| IPO | Targeted | Linear (non-sat.) | Frozen |
| SimPO | Targeted | Sigmoid (saturating) | None |
| Result | Prediction | Experiment | Confirmed? |
| Main predictions (closed-form margin ODEs) | |||
| Thm. 2.3 | DPO ; IPO | Fig. 3 | ✓ functional form |
| Thm. 2.4 | Noise: DPO phase transition, IPO rate-preserving | Fig. 5 | ✓ 8% error |
| Thm. 2.2 | vs sharp separation | Fig. 5 | ✓ qualitative |
| Supporting predictions (Appendix) | |||
| Thm. 2.5 | High collapse | Fig. 5 | ✓ non-monotonic |
| DS-R1-8B | QW3.5-9B | MI-8B | |
|---|---|---|---|
| TPO-S | 66.8 0.3 | 61.4 0.4 | 76.5 0.3 |
| TPO-H | 66.4 0.3 | 60.8 0.3 | 76.1 0.9 |
| SimPO | 66.5 0.6 | 60.9 0.2 | 73.9 1.4 |
| DPO | 65.1 0.2 | 60.6 0.8 | 71.9 0.6 |
| KTO | 64.2 0.8 | 60.4 0.5 | 72.2 1.0 |
| cDPO | 64.1 0.7 | 59.8 0.4 | 70.6 0.5 |
| On-policy (Math) | Off-policy (UltraFeedback) | |||||
|---|---|---|---|---|---|---|
| DS-8B | QW-9B | MI-8B | DS-8B | QW-9B | MI-8B | |
| TPO-S | 73.6 .5 | 72.0 1.9 | 59.4 1.1 | 55.2 1.9 | 59.4 1.9 | 58.2 2.2 |
| SimPO | 72.4 1.9 | 69.4 2.4 | 58.6 6.2 | 51.2 .8 | 54.6 2.4 | 60.2 .8 |
| TPO-H | 71.4 2.6 | 68.0 1.9 | 59.8 1.9 | 47.8 .8 | 53.4 4.2 | 51.2 1.1 |
| KTO | 50.2 1.3 | 64.2 .8 | 59.6 1.1 | 46.8 1.1 | 52.0 2.0 | 52.8 1.5 |
| DPO | 47.0 2.0 | 61.0 2.3 | 55.6 8.8 | 45.4 .5 | 47.0 4.8 | 55.2 2.8 |
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Family | DPO | IPO |
|---|---|---|---|
| Qwen3-8B (primary) | Qwen | 1540 | 5 |
| Qwen3.5-4B | Qwen | 579 | 1.2 |
| Qwen3-4B | Qwen | 367 | 5.4 |
| Gemma 3 4B | 305 | 5.0 | |
| OLMo-2 7B | AI2 | 629 | 2.0 |
| Mistral 7B | Mistral | 204 | 0.5 |
| Method | Axes (1 / 2 / 3) | 100 | 500 | 7500 | Theory predicts |
|---|---|---|---|---|---|
| SFT | Implicit / — / — | 44 | 61 | 72 | Self-limiting (Thm. 2.2 ) |
| RFT | Implicit / — / — | 52 | 49 | 47 | Overfits without repulsion |
| GRPO | Targeted / Adv. / None | 49 | 70 | 48 | Fails: sparse rewards (Prop. A.3 ) |
| DPO | Targeted / Sigmoid / Frozen | 67–71 | 74 | 78 | Strong but stale (Prop. A.4 ) |
| Method | Base | +Div |
|---|---|---|
| GRPO | 24% | 53% |
| RFT | 34% | 38% |
| SFT | 35% | 38% |
| Method | MATH | GSM8K | AIME | Theory basis |
|---|---|---|---|---|
| DPO ( , ) | 93.8 3.0 | 100.0 0.0 | 11.8 1.8 | — |
| TPO ( , ) | 98.7 1.2 | 100.0 0.0 | 12.3 0.8 | Thms. 2.5 , 2.3 |
| Method | Best Acc (%) |
|---|---|
| DPO (frozen reference) | 50 |
| DPO + refresh | 68 |
| DPO + refresh | 71 |
| DPO + refresh | 71 |
| DPO + refresh | 73 |
| SimPO (no reference) | 75 |
| Pairs | Growth ( ) | Qualitative | ||||
|---|---|---|---|---|---|---|
| 2 | 20 | 0.0 | 11.1 | 156.1 | 14.1 | Unbounded |
| 4 | 60 | 80.3 | 185.9 | 281.7 | 1.5 | Unbounded |
| 8 | 140 | 106.0 | 227.4 | 360.2 | 1.6 | Unbounded |
| 16 | 600 | 99.1 | 202.0 | 319.0 | 1.6 | Unbounded |
| DS-R1-8B | MI-8B | QW-9B | |
|---|---|---|---|
| SimPO | 49.2 0.4 | 51.2 0.4 | — |
| TPO-H | 49.6 0.5 | 50.8 0.4 | — |
| TPO-S | 49.2 0.4 | 50.4 0.5 | — |
| KTO | 49.0 0.0 | 48.0 0.0 | — |
| DPO | 49.0 0.0 | 48.0 0.0 | — |
| cDPO | 49.0 0.0 | 47.0 0.0 | — |
| MATH | GSM8K | AIME | MBPP | |
| (11.6K, 3000 steps) | (7K, 3000 steps) | (30, 3000 steps) | (374, 1000 steps) | |
| Prediction 1: Margin trajectories | ||||
| DPO | 465 | 257 | 15 | 160 |
| IPO | 0.4 | 8.5 | 0.3 | 1.1 |
| Prediction 2: Noise ( ) | ||||
| DPO | 44 | 41 | 8.8 | 46 |
| 0.0 | 0.25 | 0.5 | 0.6 | 0.7 | 0.75 | 0.8 | 1.0 | |
| MATH (11.6K, transition regime) | ||||||||
| Final accuracy (%) | 66 | 91 | 98 | 100 | 96 | 92 | 88 | 98 |
| Final margin | 58 | 278 | 493 | 864 | 398 | 304 | 249 | 1067 |
| Collapse events (/50) | 1 | 47 | 9 | 34 | 45 | 42 | 43 | 4 |
| MATH seed robustness (mean range over 2–3 seeds) | ||||||||
| Mean accuracy (%) | 66 | 91 | 97 | 91 | 91 | 87 | 88 | 89 |
LLM Alignment
Large language model alignment studies methods for making models follow human intentions, values, and safety constraints across contexts.
See all 634 papers