Dictionary-like definition generators are known to be useful for lexical semantic change detection (LSCD). However, they suffer from overly granular sense distribution, producing excessive number of definitions for one and the same lexical sense. In this work, we show that fine-tuning with Direct Preference Optimization (DPO) helps to overcome this problem. We test our approach both quantitatively on a curated academic LSCD benchmark and qualitatively on real-world data for a practical task of auditing diachronic changes in sustainability terminology within project documentation of EU development finance. All models, code and predictions produced in this study are publicly available at https://github.com/ltgoslo/dpo-for-definition-generation.
Figures & tables
Split
Target words
Examples
SFT train
90 030
220 325
SFT dev
5 144
14 793
DPO train
90 030
220 540
DPO dev
5 145
14 407
Table 1: DBnary data split statistics before post-processing the DPO split.
Parameter
SFT
DPO
LoRA rank
256
64
LoRA alpha
512
16
LoRA dropout
0.1
0.2
learning rate schedule
linear
linear
learning rate
2e-5
1e-6
batch size
32
24
Table 2: Training hyperparameters.
Model
mT0-xl SFT
Llama3Dictionary SFT
LlamaDictionary-ML38B SFT
OLMo-3 SFT
LlamaDictionary-ML38B DPO
OLMo-3 DPO
No merging
0.476 [0.18, 0.69]
0.429 [0.12, 0.66]
0.517 [0.23, 0.72]
0.586 [0.32, 0.76]
0.532 [0.25, 0.73]
0.546 [0.27, 0.74]
Minimalist merging 50
0.565
0.547
0.237
0.445
0.354
0.258
Full merging 50
0.493
0.494
0.200
0.389
0.224
0.170
Full merging 22 (tuned)
0.431
0.510
0.444
0.562
0.426
0.380
Clustering
0.435
0.465
0.464
0.557
0.493
0.498
Clustering separately
0.171
0.214
0.42
0.207
0.458
0.068
Table 3: Evaluation on SemEval’20 Task 1, Spearman rank correlations. Generated definitions were obtained in single runs, and sense detection algorithms are deterministic. Confidence intervals for the best performing postprocessing (no postprocessing) are reported in brackets.
Model
Total senses
Senses per word
% amb.
Senses per amb.
Human (DBnary gold)
7 645
1.49
20.1
6.76
Llama ML38B SFT
9 228
1.79
30.2
5.50
Llama ML38B DPO
9 044
1.76
29.3
5.58
OLMo-3 SFT
12 721
2.47
46.7
4.68
OLMo-3 DPO
12 143
2.36
45.3
4.72
Table 4: Distribution of senses in the DBnary development set and in the predictions of our models. ‘Amb.’ stands for ‘ambiguous words’ (more than one sense).
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
APD value
mean
0.13
std
0.10
min
0.0
25%
0.07
50%
0.11
75%
0.17
Appendix
Table 5: Descriptive statistics of APD calculated by XL-Durel.
Preference optimization methods such as DPO and KTO are widely used for aligning language models, yet little is understood about what properties of preference data drive downstream reasoning gains. We ask: what aspects of a preference pair improve a reasoning model's performance on general reasoning tasks? We investigate two distinct notions of quality delta in preference data: generator-level delta, arising from the differences in capability between models that generate chosen and rejected reasoning traces, and sample-level delta, arising from differences in judged quality differences within an individual preference pair. To study generator-level delta, we vary the generator's scale and model family, and to study sample-level delta, we employ an LLM-as-a-judge to rate the quality of generated traces along multiple reasoning-quality dimensions. We find that increasing generator-level delta steadily improves performance on out-of-domain reasoning tasks and filtering data by sample-level delta can enable more data-efficient training. Our results suggest a twofold recipe for improving reasoning performance through preference optimization: maximize generator-level delta when constructing preference pairs and exploit sample-level delta to select the most informative training examples.
We present an approach to fine-tuning large language models using Direct Preference Optimization (DPO), a reinforcement learning technique. Our experimental results demonstrate that DPO simplifies the training pipeline, improves computational efficiency, and achieves competitive performance. The evaluation using BLEU, ROUGE, and cosine similarity metrics indicates effective learning and convergence, though further investigation is needed to address observed training instability.
Direct Preference Optimization (DPO) has become a standard reward-model-free approach for aligning language models with preference data. However, as the scaled preference margin grows during training, the logistic DPO loss becomes progressively less sensitive to further changes. We study DPO from a loss-level geometric perspective and identify the sigmoid factor as a learning signal that characterizes the local sensitivity of the objective. Based on this view, we propose Learning-Signal-Controlled Direct Preference Optimization (LSC-DPO), which dynamically regulates the learning signal near a target regime. A log-space analysis establishes conditions for stable tracking of the target learning-signal regime. Experiments on AlpacaEval 2, MT-Bench, and Anthropic-HH show that LSC-DPO consistently improves over DPO and strong preference-optimization baselines. We further find that different coefficient initializations induce distinct transient learning-signal trajectories even when their later signal levels become similar. Based on this observation, we derive a signal-budget compensation rule that adjusts the target learning signal to compensate for these transient differences. The resulting compensation substantially reduces performance variation across coefficient initializations.