cs.CLNov 12, 2026

Human or Synthetic? What a Fixed Budget Buys for Post-Training

Authors: Shourik Banerjee, Ethan Mendes, Mohit Raghavendra, Wei Xu, Alan Ritter

Organizations: Georgia Institute of Technology

Abstract

Post-training of large language models typically combines supervised fine-tuning (SFT) with preference-based fine-tuning such as Direct Preference Optimization (DPO). Human-annotated and synthetically generated preference data differ substantially in per-sample cost, but prior work typically holds sample count fixed when comparing the two. In this work, we explicitly study the interplay between cost and performance by comparing the effect of post-training with human and synthetic preference data under a fixed, dollar-matched annotation budget. We train and evaluate multiple base model families across helpfulness and summarization tasks. We find that synthetic data is far more cost-efficient at low-to-moderate budgets, achieving high win rates with as little as $200--$400. On helpfulness, human data closes the gap only at budgets roughly two orders of magnitude larger; on summarization, it does not close the gap at any tested budget. We also find that human-annotated data varies in reliability: one widely used human preference source substantially underperforms another, and the gap persists even after standard remedies such as domain restriction and deduplication.

Figures & tables

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Which Pairs to Compare for LLM Post-Training?

    Jun 17, 2026Jiangze Han, Vineet Goyal, Will MaPairwise Preference LearningPairwise Comparison

  2. It's All Training: A Fully Synthetic Single-Stage Recipe for LLMs

    Sep 29, 2026Pierre-Carl Langlais, Pieter Delobelle, Yannick Detrois +7Language Model PretrainingSynthetic Data Pretraining

  3. FineInstructions: Scaling Synthetic Instructions to Pre-Training Scale

    Jan 29, 2026Ajay Patel, Colin Raffel, Chris Callison-BurchLanguage Model PretrainingSynthetic Data Pretraining