Multi-Objective Language Model Alignment

Overview

This field studies how to align language models with multiple goals that can conflict, including helpfulness and harmlessness, accuracy and fairness, or personalization and consistency. Common approaches combine reward models with reinforcement learning or direct preference optimization, while others use Pareto-front search, adaptive objective weights, gradient balancing, or preference-conditioned policies to preserve different trade-offs. Researchers also explore model merging and inference-time steering to let users adjust values without retraining, and build evaluations for settings such as medical advice, dialogue, and tool use. A central challenge is that objectives can interfere and human preferences are diverse or inconsistent, so methods must balance them without sacrificing capabilities or hiding important trade-offs in a single score.

Latest papers

CardsList

97 papers7 papers in the last four weeks, down 30% on the four weeks before. 0.1% of all new papers.

  1. Enforcing Axioms for AI Alignment under Loss-Based Rules

    Nov 9, 2026Alexandros Hollender, Sonja KraiczyLLM AlignmentReward Modeling

  2. Objective-Conditioned Active Preference Learning for Multi-Objective Alignment

    Nov 3, 2026Duc Anh Vu, Thong Nguyen, Hoang Pham +2LLM AlignmentMulti-Objective Language Model Alignment

  3. VIAS: Value-Isolating Activation Steering for Multi-Value Alignment and Compliance for Organisations

    Nov 2, 2026Avni Mittal, Avinash Anand, Dikshant Kukreja +3LLM AlignmentLanguage Model Steering

  4. Towards Multi-Value Alignment via Value Localization in LLMs

    Nov 2, 2026Xueqi Ma, Yanbei Jiang, Xingjun Ma +2Multi-Objective Language Model AlignmentInference-Time Language Model Alignment

  5. GPDPO: Learning from Reward Conflicts via Group Pareto-Dominance Policy Optimization

    Oct 26, 2026Zixuan Xu, Haoxuan Chen, Jiangxin Sun +1Multi-Objective Language Model AlignmentMulti-Objective Reinforcement Learning

  6. Preference-Aware Pareto Steering for Multi-Task Speech LLMs

    Oct 19, 2026Liuyuan Jiang, Vishal Sunder, Samuel Thomas +4Multi-Task LearningSpeech Language Models

  7. FACES: Fine-grained Gated Semantic Alignment for Multimodal Emotional Support Conversation

    Oct 14, 2026Hang Ren, Peidong Wang, Yichen Gao +4Emotional Support ConversationMulti-Objective Language Model Alignment

  8. Model vs. User: Whose Values Prevail? Inference-Time Value Alignment in Multi-Turn Conversations

    Oct 14, 2026Enrico Liscio, Anne Arzberger, Aditya Shankar +1LLM AlignmentInference-Time Language Model Alignment

  9. Multi-Objective Aligned Small Language Model Framework for SUD Patient Dialogue Generation

    Oct 6, 2026Thushara Manjari Naduvilakandy, Hyeju Jang, Mohammad Al HasanSmall Language ModelsUser Simulation

  10. Collaborative Personalized Preference Alignment for LLMs under Data Deficiency

    Oct 5, 2026Liyan Yang, Yige Yuan, Zhiqin YangPersonalized Language ModelsPersonalized Language Model Alignment

  11. Which Preferences to Train On? End-to-End Multi-Objective Alignment with an Adversarial Preference Distribution

    Oct 4, 2026Minjae Lee, Kyunghyun Cho, Sangdon ParkLLM AlignmentMulti-Objective Language Model Alignment

  12. From Constitutions to Control: Interpretable Rewards for Aligning Language Models

    Sep 27, 2026Johann D. Gaebler, Calvin Isley, Max Lamparth +2LLM AlignmentReward Modeling

  13. Adaptive Multi-Value Control in LLMs via Causal Activation Steering

    Sep 24, 2026Payel Bhattacharjee, Ravi TandonLanguage Model SteeringActivation Steering

  14. Tag-Aware Structured Text Translation: Towards a Systematic Understanding

    Sep 24, 2026Zhanglin Wu, Hengchao Shang, Daimeng Wei +5Multi-Objective Language Model AlignmentMachine Translation

  15. Which Objectives Need a Dial? Predicting Objective Conflict and Covering Trade-offs in Steerable Pluralistic Alignment

    Sep 22, 2026David Tsoi, Esra DönmezDirect Preference OptimizationMulti-Objective Language Model Alignment

  16. Forty Shades of Blue: Quality-Diversity Alignment via Mode-Conditioned Reinforcement Learning

    Sep 14, 2026Jiayi Yuan, Hangoo Kang, James Jihao Liu +4RL for Language ModelsMulti-Objective Language Model Alignment

  17. Suan: Rectifying Direct Preference Safety Alignment in Large Language Models

    Sep 8, 2026Oleksandr Cherednichenko, Roman KlypaLLM AlignmentLLM Safety Alignment

  18. A Better Spur Should Start From Each Objective

    Sep 8, 2026Shanwen Mao, Hao Zhang, Guangtao nie +4Multi-Objective Reinforcement LearningGradient Interference

  19. Swiss-Knife: A Framework for Reconfigurable Externalised Multi-Objective Alignment at Decode Time

    Sep 3, 2026Agnibh Karmakar, Mayur Parvatikar, Shreyash Dhoot +5Multi-Objective Language Model AlignmentInference-Time Language Model Alignment

  20. Evolutionary Soups: Evolving Mixture-of-Experts for Multi-Objective LLM Alignment

    Aug 30, 2026Lingxiao Kong, Steffen Staab, Cong Yang +2Mixture-of-Experts Language ModelsMulti-Objective Language Model Alignment

  21. Automated Researchers Can Mitigate Well-characterized Alignment Failures

    Aug 28, 2026Chen Yueh-Han, Jiaxin Wen, Jan Hendrik KirchnerAI AlignmentLLM Safety Alignment

  22. MoPLEx: Estimating Plackett-Luce Mixture Models for Multi-Objective Alignment

    Aug 25, 2026Dongyue Li, Ziniu Zhang, Lu Wang +1Learning to RankDiscrete Choice Modeling

  23. MINT: Min-Selection Preference Distillation for Balanced Multi-Objective Alignment

    Aug 14, 2026Tony Tu, Sayan Chakraborty, Ruomeng Xu +2Multi-Objective Language Model AlignmentPairwise Preference Learning

  24. Procedural Fairness Failures in RLHF from Preference Averaging

    Aug 10, 2026M P V S Gopinadh, Karthik Kamuju, Kummari Avinash +2Reward ModelingAlgorithmic Fairness