cs.LGNov 16, 2026

Margin Dynamics for Large Language Model Alignment

Authors: Xingzi Xu, Mehmet Saygin Seyfioglu, Karim Bouyarmane

Organizations: Amazon

Abstract

Practitioners choose among LLM alignment methods largely by trial and error. We provide a theoretical foundation: each pairwise method defines a vector field on the probability simplex that governs how the model redistributes mass between preferred and dispreferred completions. Projecting onto a single preferred/dispreferred pair reduces this to a one-dimensional margin ODE with a closed-form solution. These solutions reveal that DPO margins grow without bound (logarithmically as O(log⁡t)O(\log t) in the gradient-flow limit) while IPO margins converge to a finite 1/(2τ)1/(2τ); that label noise induces a qualitative phase transition in DPO (unbounded →\to bounded) but affects IPO only quantitatively; and that gradient concentration under asymmetric weighting governs trainability at scale. We confirm all three predictions across five model families spanning four benchmarks. The closed-form solutions also enable principled method design: we prove that adding any label smoothing ε>0\varepsilon > 0 to SimPO creates a finite margin attractor at Δ∗=β−1log⁡((1−ε)/ε)Δ^* = β^{-1}\log((1-\varepsilon)/\varepsilon), while ε=0\varepsilon = 0 (standard SimPO) has no attractor. We test this across three models (8B-9B) and confirm the prediction: the theory-derived configuration (TPO-S, ε=0.05\varepsilon = 0.05) is competitive with the strongest baselines, attaining the highest mean IFEval score among the compared methods on all three, validating that the predicted finite attractor is a usable operating point rather than proposing a new state-of-the-art loss.

Figures & tables

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Mind the Gap: Structure-Aware Consistency in Preference Learning

    Apr 30, 2026Mehryar Mohri, Yutao ZhongPairwise Preference LearningLLM Alignment

  2. AdaDPO: Self-Adaptive Direct Preference Optimization with Balanced Gradient Updates

    May 27, 2026Shaolong Chen, Madalina Ciobanu, Qingqing Mao +1LLM AlignmentDirect Preference Optimization

  3. Towards Bridging the Gap Between Offline and Iterative Alignment via Preference Distillation

    Sep 7, 2026Wenbo Zhang, Wenzhuo Zhou, Hengrui Cai +1LLM AlignmentPreference Alignment