LLM Alignment

LLM: Large Language Model

LLM alignment studies how to shape language models’ behavior so they are helpful, truthful, safe, and responsive to the people and settings they serve. Common approaches include supervised fine-tuning, preference learning such as Direct Preference Optimization, reinforcement learning from human or AI feedback, and inference-time steering or retrieval. Researchers evaluate these methods on instruction-following, preference, jailbreak, factuality, multilingual, and multi-turn tasks, while also developing ways to represent disagreement among users and communities. A central challenge is that gains in safety or preference alignment may reduce usefulness, diversity, calibration, or capability, and desired behavior may not generalize to new prompts, groups, or deployment settings.

Latest papers

613 papers58 papers in the last four weeks, up 12% on the four weeks before. 0.3% of all new papers.

Human-Residual Steering: Aligning Frozen LLM Judges with Human Judgment Distributions
Oct 27, 2026Chen Lyu, Phong Do, Gabriele PergolaLLM Alignment LLM-as-a-Judge
Department of Computer Science University of Warwick, Coventry, UK
On the Sparsity of Direct Preference Optimization: Weight Disentanglement in the NTK Regime
Oct 27, 2026Kensuke Sasaki, Issei SatoDirect Preference Optimization LLM Alignment
Department of Computer Science, The University of Tokyo
Exploring the Limits of the Delta Learning Hypothesis: Single-Pair Generator Biases in Direct Preference Optimization
Oct 27, 2026Giuseppe Curci, Stefano Simonazzi, Andrea Molinari +1Direct Preference Optimization LLM Alignment
Okkam, Italy · Villanova.ai, Italy · University of Trento, Italy
A Comprehensive Framework for All Stages of Large-Model Training: From Data and Tokenization, through Pretraining, to Post-Training Alignment
Oct 24, 2026Xiangjing An, Zhibin Jiao, Aiting ChenLLM Training Language Model Post-Training
Independent Researcher Team
AI Alignment for Digital Democracy: Fairer Voters Representation against LLM Biases
Oct 23, 2026Yi Qi, Srijoni Majumdar, Evangelos PournarasDemographic Bias in Language Models LLM Alignment
University of Leeds Leeds, United Kingdom
Aligning Large Language Models with Human Feedback: Mathematical Foundations and Algorithm Design
Oct 22, 2026Siliang Zeng, Luca Viano, Chenliang Li +6LLM Alignment RL from Human Feedback
University of Minnesota · EPFL · Texas A&M University +2
Unintended Side Effects of Value Training in LLMs
Oct 21, 2026Siddhesh Pawar, Chen Cecilia Liu, Anna Korhonen +1LLM Alignment Language Model Post-Training
University of Copenhagen · University of Cambridge
LaVGuard: Selective Decoding for Contextual Value Drift
Oct 21, 2026Jiale Dai, Liuxian Ma, Hongcan Deng +5LLM Alignment Language Model Decoding
State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University · College of Artificial Intelligence, Tsinghua University · University of Chinese Academy of Sciences +1
TPO: Tri-level Distributionally Robust Learning for OOD Direct Preference Optimization
Oct 20, 2026Chengtao Jian, Kai YangLLM Alignment Direct Preference Optimization
School of Computer Science and Technology, Tongji University, Shanghai, China · Shenzhen Loop Area Institute, Shenzhen, China · Shanghai Research Institute for Intelligent Autonomous Systems, Shanghai, China
CollabSciReM: Aligning Language Models for Scientific Conversations via Science-Aware Reward Decomposition
Oct 20, 2026Sasi Kiran Gaddipati, Gollam Rabby, Sören AuerLLM Alignment Reward Modeling
TIB—Leibniz Information Centre for Science and Technology, Hannover, Germany · L3S Research Center, Hannover, Germany
Divergence based Ranked Preference Alignment for LLMs
Oct 18, 2026Rajdeep Haldar, Yue Xing, Qifan SongLLM Alignment Preference Alignment
Purdue University · Michigan State University
When Empathy Misses the Goal: A Benchmark for Goal Displacement in LLM Advice
Oct 18, 2026Dean Ariel, Guy LabanLLM Evaluation Language Model Generation Evaluation
Department of Family Medicine, Clalit Health Services, Tel Aviv District, Israel · Faculty of Medicine and Health Sciences, Tel Aviv University, Tel Aviv, Israel · Stein Faculty of Computer and Information Science, Ben-Gurion University of the Negev, Beer Sheva, Israel +3
Aligning Cultural Behaviors in Arabic Language Models via Preferences
Oct 18, 2026Junhong Liang, Daniil Orel, Saeed Almheiri +11Arabic Language Models Cultural Alignment
Mohamed bin Zayed University of Artificial Intelligence · IBM Research AI
Aligning Language Models with Indigenous Community Cosmovision through Constitutional AI
Oct 18, 2026Silvia Fernández-Sabido, Jorge Reyes-Magaña, Jesús Adrián Chay Tun +3LLM Alignment Cultural Knowledge in Language Models
Secretaría de Ciencia, Humanidades, Tecnología e Innovación (SECIHTI), commissioned to CentroGeo, Mérida, Yucatán, Mexico · Facultad de Matemáticas, Universidad Autónoma de Yucatán, Mérida, Yucatán, Mexico · Aymara Community / Global Council for Latin America and the Caribbean, United Religions Initiative (URI), Buenos Aires, Argentina
Beyond Factual Accuracy: A Benchmark for Evaluating Perspective Alignment in LLMs
Oct 18, 2026Sunghoon Kim, Sangyeop KimLanguage Model Generation Evaluation LLM Alignment
Coxwave Seoul, Republic of Korea · Ewha Womans University Seoul, Republic of Korea
From Stories to Features: Scaling Concept Vectors for AI Alignment & Interpretability
Oct 17, 2026Dmitry Scherbakov, Oleg Chernikov, Victoria Khamash +5LLM Interpretability LLM Alignment
Controlling the Diversity-Quality Trade-Off of LLMs by Merging Models Before and After Alignment
Oct 16, 2026Yasuaki Sumita, Tomoharu Iwata, Toshiyuki TanakaLanguage Model Merging Diverse Text Generation
Graduate School of Informatics, Kyoto University, Kyoto, Japan · NTT Communication Science Laboratories, NTT, Inc., Kyoto, Japan
Aligning Language Models Towards Counterfactual Invariance
Oct 16, 2026Basar Demir, Emily Shen, Ksheeraj Sai Vepuri +5Demographic Bias in Language Models Counterfactual Fairness
Meta Superintelligence Labs · University of California, San Diego · Independent Researcher
CAT-alyzing Honesty: Continuous Adversarial Training for Transferable LLM Honesty
Oct 15, 2026Yuan-Chen Chang, Niklas Herbster, Gauthier Gidel +1LLM Honesty LLM Alignment
Tara Research · Mila – Quebec AI Institute · Technical University of Munich
Rank Measures of Fine-Tuning Updates Need a Noise Baseline
Oct 15, 2026Muhammad Usman AltafFine-Tuning Language Models Neural Network Training Dynamics
Independent Researcher
VALCON: Benchmarking and Enhancing Value Consistency in Large Language Models via Dual-Role Variation
Oct 15, 2026Junru Wu, Jingting Zheng, Yuqi Ren +1LLM Alignment LLM Evaluation
TJUNLP Lab, School of Computer Science and Technology, Tianjin University, China · Yuanhui AI
Model vs. User: Whose Values Prevail? Inference-Time Value Alignment in Multi-Turn Conversations
Oct 14, 2026Enrico Liscio, Anne Arzberger, Aditya Shankar +1LLM Alignment Inference-Time Language Model Alignment
Delft University of Technology, the Netherlands
Do Factuality Methods Make LLMs More Factual? Reproducibility and Fair Comparison
Oct 13, 2026Pratuat Amatya, Vinay Setty, V Venktesh +1LLM Alignment Factual Knowledge in Language Models
Factiverse AS · University of Stavanger · Stockholm University +1
Understanding and Reducing Misalignment in Persuasion Optimization
Oct 12, 2026Boyi Wei, Ruta Desai, Peter Henderson +1LLM Alignment Emergent Misalignment in Language Models
Microsoft Research · Princeton University
Beyond the greenwash: do LLMs' ecologically sound decisions hold up under pressure?
Oct 12, 2026Trent R. Northen, Mingxun WangLLM Alignment Language Model Safety Evaluation
Bioaligned Labs
UDHRBench: Benchmarking Human Rights in Large Language Models
Oct 10, 2026Alak Raad, Noreldin Morssy, Zeerak Suhail +2LLM Alignment LLM Safety Benchmarks
Hamad Bin Khalifa University, Qatar · Princess Sumaya University for Technology, Jordan
Can You Trust Your Preference Models? Stackelberg Games for Alignment
Oct 10, 2026Riccardo Formenti, Aditya Agrawal, Vikas GargLLM Alignment Preference-Based RL
Department of Mathematics ETH Zürich · Department of Computer Science Aalto University · YaiYai Ltd and Aalto University
Predicting how model propensities generalize out of distribution
Oct 10, 2026June Hunter, Lily Wen, Owen Terry +3OOD Generalization Language Model Generalization
Uniform Information Density-Based Preference Optimization
Oct 10, 2026Yongyuan Li, Yang XuPreference Optimization LLM Alignment
Department of Computer Science and Engineering Southern University of Science and Technology Shenzhen, Guangdong, China
CLEAT: Learning Value-Guided Clarification for Interactive Agents
Oct 8, 2026Jiatan Huang, Zheyuan Zhang, Ziming Li +4LLM Agents LLM Alignment
University of Connecticut · University of Notre Dame · University of Massachusetts, Amherst

Oct 27, 2026J/K move · Enter open · S save

Human-Residual Steering: Aligning Frozen LLM Judges with Human Judgment Distributions

Chen Lyu, Phong Do, Gabriele Pergola

Department of Computer Science University of Warwick, Coventry, UK

LLM Alignment LLM-as-a-Judge Activation Steering

Open paper AskPDF arXiv

Large language models (LLMs) are widely used as automated judges, yet most alignment methods target aggregated human judgments and overlook variation among individual annotators. Existing approaches to matching the full distribution of human ratings mainly rely on output-level calibration or parameter fine-tuning. We propose Human-Residual Steering (HRS), to our knowledge the first method to align frozen LLM judges with human judgment distributions through learned hidden-state interventions. An oracle analysis shows that low-rank corrections at a single layer remove most of the discrepancy between judge and human distributions. HRS learns to predict such corrections from the judge's hidden representation and rating logits, training on individual annotations with proper scoring rules for ordinal and nominal judgments and optimising through the frozen downstream layers. Across 14 benchmarks and four judges from three model families, HRS outperforms all evaluated frozen-weight baselines. With Qwen2.5-7B, it achieves the lowest Jensen-Shannon distance on 13 of 14 benchmarks, reducing the mean from 0.565 to 0.352 versus 0.377 for the strongest calibrator. HRS also raises mean Spearman correlation with aggregate human scores from 0.449 to 0.579, improves population-level estimation under limited annotation budgets, and trains about 88 times fewer parameters than distributional fine-tuning at a modest cost in alignment.