Inference-Time Language Model Alignment

Follow this topic Ask about this topic

Momentum

12 papers in the last four weeks, up 100% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 36

All topics

Efficient Best-of-N policy evaluation for inference-time alignment
Oct 7, 2026Jonas Schweisthal, Yuxin Wang, Athiya Deviyani +2Best-of-N Sampling Off-Policy Evaluation
LMU Munich & MCML · Carnegie Mellon University · Meta (work in personal capacity)
The Asymptotics of Language Model Alignment with Memory
Oct 1, 2026Haricharan Balasundaram, V. Arvind RameshwarRL for Language Models LLM Alignment
School of Electrical and Computer Engineering, Georgia Institute of Technology, Atlanta, GA · IIT Madras · Department of Electrical Engineering, IIT Madras, Chennai, India
HeadEdit: Calibrating Language Model Behavior Through the Frozen Unembedding Matrix
Oct 1, 2026Zirui He, Haiyan Zhao, Jingyu Hu +5Language Model Steering Language Model Calibration
New Jersey Institute of Technology · University of Bristol · Kuaishou Technology +1
Ready2Blend: From Natural-Language Instructions to Composable Alignment Prompts
Sep 30, 2026Jeesu Jung, Hwan Chang, Juseon Do +5Continual Learning for LLMs LLM Alignment
Korea Advanced Institute of Science and Technology · Samsung SDS
Demographic Pluralism: Inference-Time Modeling of Pluralistic Human Preference Distributions
Sep 29, 2026Meng-Chen Wu, Qipin Chen, Ansh Jain +3Human Preference Modeling Inference-Time Language Model Alignment
Amazon Science
Aligned Data Can Induce Misalignment via Context Confusion
Sep 29, 2026Yavuz Bakman, Duygu Nur Yaldiz, Baris Askin +4LLM Alignment LLM Post-Training
University of Southern California · Carnegie Mellon University · Amazon +1
Constitutional adapters: Inference-time interventions for misalignment and misuse
Sep 29, 2026Adam S. Lowet, Mark KurzejaLLM Safety Alignment LLM Jailbreak Attacks
Anthropic Fellows Program · Independent
Rethinking Personalized Generation: Test-Time Alignment via Factorized Ranking Models
Sep 28, 2026Qiyao Ma, Junshan Zhang, Zhe ZhaoLearning to Rank LLM Reranking
University of California, Davis
Direct Hidden-State Alignment: Mapping and Controlling Preference Expression in LLMs
Sep 27, 2026Fansheng Zhang, Shengran Guo, Zexiao Wang +3LLM Alignment Preference Alignment
Chengdu University · North Carolina State University · Fudan University +2
Minimally Invasive Steering of Language Models
Sep 24, 2026Taha Entesari, Jingyu Zhang, Daniel Khashabi +1Language Model Steering Inference-Time Language Model Alignment
Johns Hopkins University
From Static Personal Values to Contextualized Personalization: Bayesian Personalized Value Alignment for LLMs
Sep 24, 2026Hanze Guo, Aixuan Song, Jing Yao +4LLM Alignment Personalized Language Models
Gaoling School of Artificial Intelligence, Renmin University of China · Microsoft Research Asia · Beijing Key Laboratory of Research on Large Models and Intelligent Governance +1
An Efficient and Modular Framework for Targeted Harm Mitigation in LLMS
Sep 12, 2026Roberto Campbell, Momin Abbass, Muneeza Azmat +5LLM Safety Alignment Low-Rank Adaptation
University of California, Santa Cruz · IBM Research
Inference-Time Nash Alignment
Sep 8, 2026Hadi Hosseini, Debmalya Mandal, Duohan ZhangInference-Time Optimization Preference Alignment
Penn State University, USA · University of Warwick, UK
Instruction Duplication as an Inference-Time Control Primitive
Sep 3, 2026Victor LavrenkoInstruction Following Inference-Time Language Model Alignment
PeaceTech VC, Israel
Trust Your Guide Only When Certain: Uncertainty-Aware Sparse Alignment at Inference Time
Sep 1, 2026Zeen Zhu, Zhuo Li, Weiyang Guo +4LLM Safety Alignment Inference-Time Language Model Alignment
Harbin Institute of Technology, Shenzhen, China · Huawei Technologies Co., Ltd. · Beijing Academy of Artificial Intelligence, China.
Beyond Token-Level Guidance: Inference-Time Alignment of Specialized LLMs via Cross-Family Representation Steering
Aug 31, 2026Jin Gan, Xin Li, Jun LuoLLM Safety Alignment Language Modeling
College of Computing and Data Science, Nanyang Technological University, Singapore · College of Cryptology and Cyber Science, Nankai University, Tianjin, China
Evolutionary Soups: Evolving Mixture-of-Experts for Multi-Objective LLM Alignment
Aug 30, 2026Lingxiao Kong, Steffen Staab, Cong Yang +2Mixture-of-Experts Language Models Multi-Objective Language Model Alignment
Fraunhofer Institute for Applied Information Technology FIT · University of Cologne · University of Stuttgart +3
Weightless Fine-Tuning: Personalizing LLMs via Logit-Space Transport
Aug 11, 2026Bohan Zhang, Anqi Ni, Yixin Wang +1Supervised Fine-Tuning LLM Personalization
University of Michigan · University of Chicago
CIDER: A Dataset of Contextual Disclosure Boundaries for Privacy Preference Alignment
Aug 10, 2026Bingcan Guo, Eryue Xu, Jijie Zhou +2Preference Alignment Inference-Time Language Model Alignment
University of Washington · UIUC · Northeastern University
Deployable Per-Instance Multi-Layer Activation Steering for Large Language Models
Aug 9, 2026Muhammad Faishal Adly Nelwan, Alfan Farizki WicaksonoLanguage Model Steering Controllable Text Generation
Faculty of Computer Science Universitas Indonesia
Beyond a Global Norm: Personalizing Toxicity Sensitivity in Language Models Without Retraining
Jul 25, 2026Rares A. C. Diaconescu, Iulia Slanina, Alina Florea +5LLM Alignment LLM Safety Alignment
Delft University of Technology
Inference-Time Steering for Cross-Lingual Factual Consistency in LLMs
Jul 21, 2026Alexander ManevMultilingual Language Model Evaluation LLM Alignment
Technical University of Munich
Best-of-Better- $N$ : Generating Pre-Aligned Responses with In-Context Learning
Jul 3, 2026Eric Lei, Hsiang Hsu, Chun-Fu ChenLLM Alignment Best-of-N Sampling
JPMorganChase Global Technology Applied Research
Safe Inference-Time Alignment via Lagrangian Reward Augmentation
Jul 2, 2026Yaswanth Chittepu, Ativ Joshi, Sohini Chintala +1LLM Alignment LLM Safety Alignment
University of Massachusetts Amherst · Independent Researcher
Towards Spec Learning: Inference-Time Alignment from Preference Pairs
Jun 22, 2026Dhriti Krishnan, Tejas Goyal, Jaromir SavelkaLLM Alignment Prompt Optimization
Department of Computer Science Carnegie Mellon University
PG-MAP: Joint MAP Optimization for Inference-Time Alignment of Diffusion and Flow-Matching Models
Jun 22, 2026Ruolan Sun, Pawel PolakDiffusion Model Alignment Inference-Time Language Model Alignment
Stony Brook University
ALIGNBEAM : Inference-Time Alignment Transfer via Cross-Vocabulary Logit Mixing
Jun 10, 2026Chirag Chawla, Pratinav Seth, Vinay Kumar SankarapuLLM Alignment LLM Safety Alignment
Lexsi Labs
Gradient-Guided Reward Optimization for Inference-time Alignment
Jun 8, 2026Hankun Lin, Ruqi ZhangReward Hacking LLM Alignment
Department of Computer Science, Purdue University
Inference-Time Vulnerability Beyond Shallow Safety: Alignment Along Generation Trajectories
Jun 3, 2026Kyungmin Park, Taesup KimAdversarial Attacks on LLMs LLM Safety Alignment
Hankuk University of Foreign Studies · Seoul National University
Effort as Ceiling, Not Dial: Reasoning Budget Does Not Modulate Cognitive Cost Alignment Between Humans and Large Reasoning Models
May 16, 2026Yueqing Hu, Tianhong WangComputational Cognitive Modeling Inference-Time Scaling
Institute of Neuroscience, Chinese Academy of Sciences, Shanghai, China · School of Philosophy, Anhui University, Hefei, China
Temper and Tilt Lead to SLOP: Reward Hacking Mitigation with Inference-Time Alignment
May 13, 2026Ye Wang, Jing Liu, Toshiaki Koike-AkinoReward Hacking LLM Alignment
Mitsubishi Electric Research Laboratories (MERL) · Broadway, Cambridge, MA 02139, USA
Training-Free Cultural Alignment of Large Language Models via Persona Disagreement
May 11, 2026Huynh Trung Kiet, Dao Sy Duy Minh, Tuan Nguyen +5LLM Alignment Cultural Alignment
Faculty of Information and Technology, University of Science, Vietnam National University, Ho Chi Minh City, Vietnam. · Department of Computer Science, University of Warwick, Coventry, United Kingdom. · School of Computing, Engineering and Digital Technologies, Teesside University, United Kingdom.
One Refiner to Unlock Them All: Inference-Time Reasoning Elicitation via Reinforcement Query Refinement
Apr 28, 2026Yixiao Zhou, Dongzhou Cheng, zhiliang wu +3LLM Alignment RL for Language Model Reasoning
Zhejiang University · Shanghai Innovation Institute · 1Zhejiang University, 2Shanghai Innovation Institute +9
Pref-CTRL: Preference Driven LLM Alignment using Representation Editing
Apr 26, 2026Imranul Ashrafi, Inigo Jauregi Unanue, Massimo PiccardiLLM Alignment Representation Engineering
University of Technology Sydney, Australia
To Intervene or Not: Guiding Inference-time Alignment with Probabilistic Model Blending
Apr 22, 2026Jin Gan, Xin Li, Jun LuoLLM Alignment Inference-Time Language Model Alignment
College of Computing and Data Science, Nanyang Technological University, Singapore
On the Rejection Criterion for Proxy-based Test-time Alignment
Apr 17, 2026Ayoub Hammal, Pierre Zweigenbaum, Caio CorroLLM Alignment Inference-Time Language Model Alignment
Université Paris-Saclay, CNRS, LISN · INSA Rennes, IRISA, CNRS, Université de Rennes
Asymptotic Universal Alignment: A New Alignment Framework via Test-Time Scaling
Jan 13, 2026Yang Cai, Weiqiang ZhengLLM Alignment Test-Time Scaling
Yale University

Oct 7, 2026, cs.LGJ/K move · Enter open · S save

Efficient Best-of-N policy evaluation for inference-time alignment

Jonas Schweisthal, Yuxin Wang, Athiya Deviyani, Stefan Feuerriegel+1

LMU Munich & MCML · Carnegie Mellon University · Meta (work in personal capacity)

Best-of-N Sampling Off-Policy Evaluation Inference-Time Language Model Alignment Doubly Robust Estimation

Open paper AskPDF arXiv

Best-of-N (BoN) is a common inference-time alignment method that selects the highest-scoring response among N samples from a reference model. Evaluating BoN policies from logged data is challenging under sample-only access because standard off-policy estimators require density ratios that depend on unavailable response likelihoods. In this paper, we propose a sample-only framework for evaluating and selecting BoN policies without access to these likelihoods. We show that the order-statistic structure of BoN allows the required density ratios to be expressed through score-rank probabilities that are estimable from samples alone. We then develop a doubly robust estimator of the BoN policy value (BoN-DR) that efficiently reuses a shared auxiliary sample pool across candidate budgets. We establish valid asymptotic inference even under reward estimator misspecification and prove the efficiency of our BoN-DR estimator. Since larger budgets can amplify errors in the score function and lead to reward overoptimization, we derive two selection rules: (i) maximizing the estimated policy value and (ii) maximizing a lower confidence bound on the improvement over the reference policy, which accounts for estimation uncertainty and provides a no-harm guarantee. Across synthetic experiments and GSM8K with multiple reference and reward models, our framework accurately estimates BoN policy values and selects effective sampling budgets.

Momentum

Related topics

Latest papers 36