Post-training of large language models typically combines supervised fine-tuning (SFT) with preference-based fine-tuning such as Direct Preference Optimization (DPO). Human-annotated and synthetically generated preference data differ substantially in per-sample cost, but prior work typically holds sample count fixed when comparing the two. In this work, we explicitly study the interplay between cost and performance by comparing the effect of post-training with human and synthetic preference data under a fixed, dollar-matched annotation budget. We train and evaluate multiple base model families across helpfulness and summarization tasks. We find that synthetic data is far more cost-efficient at low-to-moderate budgets, achieving high win rates with as little as $200--$400. On helpfulness, human data closes the gap only at budgets roughly two orders of magnitude larger; on summarization, it does not close the gap at any tested budget. We also find that human-annotated data varies in reliability: one widely used human preference source substantially underperforms another, and the gap persists even after standard remedies such as domain restriction and deduplication.
Figures & tables
Figure 1: Overview of our setting. A fixed annotation budget is spent on either human or synthetic preference data for post-training. Each circle represents one example, with area proportional to its cost: the same budget buys a few expensive human-labeled examples or many cheap synthetic ones.
helpfulness
summarization
Self-consistency
91.9%
88.1%
Agreement (%, Cohen’s κ )
A1–A2
80% (0.58)
60% (0.21)
A1–J
72% (0.43)
78% (0.56)
A2–J
76% (0.50)
66% (0.30)
J–Q
84% (0.44)
66% (0.30)
Table 1: Judge reliability. Self-consistency: mean fraction of the preference-labeling judge’s ( Qwen-3-32B ) 32 votes per pair matching the majority label, over all synthetic pairs before removing ties (105,882 helpfulness, 110,448 summarization). Agreement: raw percent agreement (Cohen’s κ ) between two human annotators (A1, A2) and the evaluation judge (J; gpt-oss-20b ) on the same 50 random win-rate comparisons per task. J–Q: agreement between J and a second judge (Q; Qwen3.8-27B ) on 2,048 win-rate comparisons for each of five models per task, averaged over models.
Figure 2: Synthetic data lies on the empirical Pareto frontier at low budgets; human data needs far larger budgets to compete. Win rate vs. annotation cost (log scale) by model (columns) and data source; top: helpfulness, bottom: summarization. Points are means over two seeds, with min–max error bars; the dotted line is the untrained baseline.
n
Cost
WR
n
Cost
WR
0
$0
0.41
3,000
$3,900
0.31
76
$100
0.33
6,000
$7,800
0.42
153
$200
0.23
8,000
$10,400
0.45
230
$300
0.20
10,000
$13,000
0.52
307
$400
0.21
12,000
$15,600
0.53
769
$1,000
0.28
Table 2: Human data initially hurts Qwen-2.5-7B performance before a slow recovery. Qwen-2.5-7B TL;DR (human) win rate (WR) by number of training samples ( n ) at 1.30persample,averagedovertwoseeds.Darkershadingindicateshigherwinrate;n=0istheuntrainedbasemodel.Smallamountsofhumandatapushwinratebelowtheuntrainedbaseline,andittakesabout6,000samples(7.8k) to recover past it; even 12,000 samples ($15.6k) remain below the win rate synthetic data reaches within a few hundred dollars (Figure 2 ).
Figure 3: Mixing does not beat either pure source. Win rate for mixed human/synthetic runs at a fixed $13k budget, by dollar split (95/5 to 99/1), for helpfulness (top) and summarization (bottom). Points are means over two seeds, with min–max error bars; dashed lines show each model’s best pure-human and pure-synthetic win rate, and the dotted line the untrained baseline.
Figure 4: Premature termination moves against win rate. Measured on a fixed evaluation set at each training tier (300,400, $1000). (a, b) Fraction of responses that end before completing their final sentence (note the different y-axis scales). (c) Win rate of each synthetic-trained model; bold marks its best tier. Wherever win rate changes by more than one point, the termination rate moves in the opposite direction.
Figure 5: Robust to the assumed human cost. Win rate vs. cost (log scale) for helpfulness (top) and summarization (bottom), with the human curve re-plotted under five candidate per-sample costs (1.00–1.60). Sample counts and win rates are fixed; only the cost axis changes.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
Value
Learning Rate
5e−5
Batch Size
16
Number of Epochs
2
Warmup Ratio
0.1
Learning Rate Scheduler
cosine
Weight decay
0.01
Appendix
Table 3: SFT hyperparameter settings.
Parameter
Value
Learning Rate
5e−6
Batch Size
16
Number of Epochs
2
Warmup Ratio
0.1
Learning Rate Scheduler
cosine
Weight decay
0.01
Appendix
Table 4: DPO (PFT) hyperparameter settings.
Parameter
Value
rank
16
α
32
Appendix
Table 5: LoRA hyperparameter settings, shared by SFT and DPO.
Dataset
Task
Samples
Avg. tokens
Anthropic HH
Help./Harm.
45,000
180
SHP
helpfulness
385,000
400
HelpSteer3
helpfulness
18,000
1052
TL;DR
summarization
93,000
356
Appendix
Table 6: Human-annotated datasets. SHP and HelpSteer3 are the human training sources for helpfulness, and TL;DR (openai-tldr-summarisation-preferences) for summarization. Anthropic HH is used only for evaluation and as the synthetic-generation template. All data is in English.
Figure 6: Token-length distributions of synthetic summarization posts (left) and helpfulness conversations (right) after filtering (§ A.6 ).
Figure 7: 5-gram filtering increases diversity. Synthetic helpfulness data before and after filtering. Left: ratio of unique to total 5-grams. Right: rate of the single most common formulaic 5-gram.
Figure 8: Consistent across seeds. Win rate vs. cost (log scale) for each (model, data source) pair on the helpfulness (left) and summarization (right) tasks, plotted separately for both independent seeds (Run 1, Run 2). Curve shape and relative ordering are consistent across seeds even where absolute win rates differ modestly. The runs are practically on top of each other, showing that changing the specific examples and generation seed does not affect the conclusions of the curves.
Figure 9: Mean and median response length (tokens) on the held-out evaluation set, by training data source, for helpfulness (left) and summarization (right).
Large language model post-training adapts pretrained models into more useful, reliable, and specialized assistants through supervised and preference-based learning.