LLM Alignment

LLM: Large Language Model

LLM alignment studies how to shape language models’ behavior so they are helpful, truthful, safe, and responsive to the people and settings they serve. Common approaches include supervised fine-tuning, preference learning such as Direct Preference Optimization, reinforcement learning from human or AI feedback, and inference-time steering or retrieval. Researchers evaluate these methods on instruction-following, preference, jailbreak, factuality, multilingual, and multi-turn tasks, while also developing ways to represent disagreement among users and communities. A central challenge is that gains in safety or preference alignment may reduce usefulness, diversity, calibration, or capability, and desired behavior may not generalize to new prompts, groups, or deployment settings.

Latest papers

613 papers58 papers in the last four weeks, up 12% on the four weeks before. 0.3% of all new papers.

Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignment
May 15, 2025Jiazheng Zhang, Wenqing Jing, Zizhuo Zhang +12Reward Modeling LLM Alignment
NLP Group, Fudan University · Hong Kong Baptist University · LLM Team, Meituan Inc
Atomic Consistency Preference Optimization for Long-Form Question Answering
May 14, 2025Jingfeng Chen, Raghuveer Thirukovalluru, Junlin Wang +2LLM Hallucination Mitigation LLM Alignment
GVPO: Group Variance Policy Optimization for Large Language Model Post-Training
Apr 28, 2025Kaichen Zhang, Yuzhong Hong, Junwei Bao +4LLM Post-Training LLM Alignment
Thrust of Artificial Intelligence, Hong Kong University of Science and Technology (Guangzhou) · Zuoyebang Education Technology · Department of Computer Science and Engineering, HKUST
Adaptive Helpfulness-Harmlessness Alignment with Preference Vectors
Apr 27, 2025Ren-Wei Liang, Chin-Ting Hsu, Chan-Hung Yu +6LLM Alignment LLM Safety Alignment
National Taiwan University · Appier AI Research · Graduate Institute of Communication Engineering, National Taiwan University +1
Reinforcement Learning from Human Feedback
Apr 16, 2025Nathan LambertLLM Alignment Direct Preference Optimization
Dual-Difficulty Curriculum Learning for Direct Preference Optimization
Apr 10, 2025Mengyang Li, Haozhan Geng, Zhong Zhang +1Pairwise Preference Learning LLM Alignment
Tianjin Key Laboratory of Wireless Mobile Communications and Power Transmission Tianjin Normal University Tianjin, China
Truthful or Fabricated? Using Causal Attribution to Mitigate Reward Hacking in Explanations
Apr 7, 2025Pedro Ferreira, Wilker Aziz, Ivan TitovCoT Faithfulness Reward Hacking
Institute for Logic, Language and Computation (ILLC), University of Amsterdam · Institute for Language, Cognition and Computation (ILCC), University of Edinburgh
Evolutionary Guided Decoding: Iterative Value Refinement for LLMs
Mar 4, 2025Zhenhua Liu, Lijun Li, Ruizhe Chen +5LLM Alignment Language Model Decoding
Shanghai Artificial Intelligence Laboratory · Soochow University · Zhejiang University +1
Societal Alignment Frameworks Can Improve LLM Alignment
Feb 27, 2025Karolina Stańczak, Nicholas Meade, Mehar Bhatia +14LLM Alignment AI Alignment
ETH Zurich, Zurich, Switzerland · Mila, McGill University, Montreal, Canada · Université de Montréal, Montreal, Canada +12
Foundations of Large Language Models
Jan 16, 2025Tong Xiao, Jingbo ZhuLanguage Model Pretraining LLM Alignment
NLP Lab, Northeastern University & NiuTrans Research
Risk-Aware Alignment Must Incorporate Reward Uncertainty
Oct 31, 2024Debangshu Banerjee, Kintan Saha, Aditya GopalanPolicy Optimization LLM Alignment
Department of Electrical and Communication Engineering, Indian Institute of Science, India
Decoupled Alignment for Robust Plug-and-Play Adaptation
Jun 3, 2024Haozheng Luo, Jiahao Yu, Wenxin Zhang +9LLM Alignment LLM Safety Alignment
Northwestern University · New York University Abu Dhabi · Stanford University +3
KTO: Model Alignment as Prospect Theoretic Optimization
Feb 2, 2024Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff +2LLM Alignment Human Preference Modeling
Stanford University · Contextual AI

May 15, 2025J/K move · Enter open · S save

Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignment

Jiazheng Zhang, Wenqing Jing, Zizhuo Zhang, Zhiheng Xi+11

NLP Group, Fudan University · Hong Kong Baptist University · LLM Team, Meituan Inc

Reward Modeling LLM Alignment Preference Learning Preference-Based RL

Open paper AskPDF arXiv

Reward models (RMs) are central to aligning large language models (LLMs) with human values, yet noisy preference feedback can induce reward misgeneralization through spurious correlations and overfitting. We characterize the training dynamics of preference pairs and uncover a distinctive optimization signature of noise: persistently high losses, sharp temporal fluctuations, and irregular gradient updates. Our analyses further show that excluding such pairs improves downstream alignment. In practice, however, reliable preferences are not known a priori, and self-filtering based on a model's own training signals can amplify confirmation bias. To address this challenge, we introduce Collaborative Reward Modeling (CRM), an online framework for identifying reliable preferences. CRM couples cross-model peer review with curriculum learning: two RMs evaluate each other's sample selections, while curriculum keeps their learning progress synchronized. Across extensive experiments, CRM improves RM generalization by up to 9.94 points on RewardBench under 40% preference noise. CRM further generalizes to implicit-reward alignment, offering a robust and broadly applicable approach to learning from imperfect feedback.