LLM Post-Training

LLM: Large Language Model

LLM post-training studies how to shape a pretrained model’s behavior and capabilities after its initial training. Common approaches include supervised fine-tuning, preference optimization such as DPO, reinforcement learning with verifiable rewards, and distillation; they use human feedback, synthetic examples, rubrics, or execution results from tools and environments. Researchers evaluate these methods on tasks such as mathematical reasoning, coding, tool use, and domain-specific question answering, while also measuring safety, calibration, and performance after deployment. A central challenge is gaining targeted abilities without sacrificing prior capabilities, diversity, or reliability, and evidence about which training recipes generalize best remains mixed.

Latest papers

249 papers43 papers in the last four weeks, up 19% on the four weeks before. 0.2% of all new papers.

Value Drifts: Tracing Value Alignment During LLM Post-Training
Oct 30, 2025Mehar Bhatia, Shravan Nayak, Gaurav Kamath +4LLM Alignment Preference Optimization
Mila - Quebec AI Institute · McGill University · Université de Montréal +4
A Model Can Help Itself: Reward-Free Self-Training for LLM Reasoning
Oct 21, 2025Mengqi Li, Lei Zhao, Anthony Man-Cho So +2Self-Training for Language Models Mathematical Reasoning
The Chinese University of Hong Kong, Shenzhen · Shanghai Jiao Tong University · The Chinese University of Hong Kong
Strengthening LLMs for Tabular Prediction with Structural Priors
Oct 20, 2025Pengxiang Cai, Zihao Gao, Wanchen Lian +2Tabular Foundation Models RL for Language Models
The Hong Kong University of Science and Technology (Guangzhou) · Zhejiang University
LLP: LLM-Based Product Pricing in E-commerce
Oct 10, 2025Hairu Wang, Sheng You, Qiheng Zhang +5E-Commerce LLM Post-Training
Xianyu of Alibaba, Hangzhou, China · University of Science and Technology of China, Suzhou, China
Post-training for Efficient Communication via Convention Formation
Aug 8, 2025Yilun Hua, Evan Wang, Yoav ArtziLLM Evaluation LLM Post-Training
Department of Computer Science and Cornell Tech Cornell University
Reinforcement Fine-Tuning Naturally Mitigates Forgetting in Continual Post-Training
Jul 7, 2025Song Lai, Haohan Zhao, Rong Feng +9Continual Learning for LLMs LLM Post-Training
Centre for Artificial Intelligence and Robotics, Hong Kong Institute of Science & Innovation, Chinese Academy of Sciences, Hong Kong, China · City University of Hong Kong, Hong Kong, China · School of Artificial Intelligence, Jilin University, Changchun, China +3
Tuning without Peeking: Provable Generalization Bounds and Robust LLM Post-Training
Jul 2, 2025Ismail Labiad, Mathurin Videau, Matthieu Kowalski +4Privacy-Preserving Language Models Evolutionary Optimization
Meta FAIR · Université Paris-Saclay · LISN, Inria, CNRS +2
GVPO: Group Variance Policy Optimization for Large Language Model Post-Training
Apr 28, 2025Kaichen Zhang, Yuzhong Hong, Junwei Bao +4LLM Post-Training LLM Alignment
Thrust of Artificial Intelligence, Hong Kong University of Science and Technology (Guangzhou) · Zuoyebang Education Technology · Department of Computer Science and Engineering, HKUST
Dual-Difficulty Curriculum Learning for Direct Preference Optimization
Apr 10, 2025Mengyang Li, Haozhan Geng, Zhong Zhang +1Pairwise Preference Learning LLM Alignment
Tianjin Key Laboratory of Wireless Mobile Communications and Power Transmission Tianjin Normal University Tianjin, China

Oct 30, 2025J/K move · Enter open · S save

Value Drifts: Tracing Value Alignment During LLM Post-Training

Mehar Bhatia, Shravan Nayak, Gaurav Kamath, Marius Mosbach+3

Mila - Quebec AI Institute · McGill University · Université de Montréal+4

LLM Alignment Preference Optimization LLM Post-Training

Open paper AskPDF arXiv

As LLMs occupy an increasingly important role in society, they are more and more confronted with questions that require them not only to draw on their general knowledge but also to align with certain human value systems. Therefore, studying the alignment of LLMs with human values has become a crucial field of inquiry. Prior work, however, mostly focuses on evaluating the alignment of fully trained models, overlooking the training dynamics by which models learn to express human values. In this work, we investigate how and at which stage value alignment arises during the course of a model's post-training. Our analysis disentangles the effects of post-training algorithms and datasets, measuring both the magnitude and time of value drifts during training. Experimenting with Llama-3 and Qwen-3 models of different sizes and popular supervised fine-tuning (SFT) and preference optimization datasets and algorithms, we find that the SFT phase generally establishes a model's values, and subsequent preference optimization rarely re-aligns these values. Furthermore, using a synthetic preference dataset that enables controlled manipulation of values, we find that different preference optimization algorithms lead to different value alignment outcomes, even when preference data is held constant. Our findings provide actionable insights into how values are learned during post-training and help to inform data curation, as well as the selection of models and algorithms for preference optimization to improve model alignment to human values.