Learnable Chernoff Baselines for Inference-Time Alignment
Organizations: UC San Diego · Ohio State University
Abstract
We study inference-time reward-guided alignment for generative models. Existing methods often rely on either architecture-specific adaptations or computationally costly inference procedures. We introduce Learnable Chernoff Baselines (LCBs) as a method for efficiently and approximately sampling from the exponentially tilted kernels that arise from KL-regularized reward alignment. Using only black-box sampling access to the pretrained model, LCBs implement a form of rejection sampling with adaptively selected acceptance probabilities, which allows fine-grained control over inference-compute scaling. We establish total-variation guarantees to the ideal aligned model, and demonstrate in both continuous and discrete diffusion settings that LCB sampling closely matches ideal rejection sampling while using substantially fewer queries to the pretrained model.
Figures & tables
| Process | Description |
|---|---|
| Pretrained (unguided) diffusion process. | |
| Aligned/tilted process satisfying | |
| Learned-value tilt of satisfying | |
| LCB-induced approximate sampling law; targets . |
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
| Process | Role | Definition / how it is sampled |
|---|---|---|
| Pretrained (base) diffusion process | Initial distribution and reverse-time kernels . Sampling: draw and then for , sample . | |
| Ideal reward-aligned target process (true soft values) | Defines the desired aligned marginal . It can be represented by reverse-time kernels obtained by tilting using the true soft-value function . | |
| Value-tilted process using learned (exact RS target) | Reverse-time kernels are the learned-value tilts (with an analogous definition for ). This is the process targeted by exact per-step rejection sampling when using . | |
| Baseline-induced process (approximate sampler) | The process induced by baselined rejection sampling: propose from and accept with probability The resulting accepted transition defines , and marginals are obtained by propagation |
| Method | Rew. | Perf. Rate (%) | Prop. |
|---|---|---|---|
| RS 0.5 | |||
| RS 0.3 | |||
| RS 0.1 | |||
| LCB 0.5 | |||
| LCB 0.3 | |||
| LCB 0.1 |
| Mean Z | Mean Reward | Eff N | |
|---|---|---|---|
| LCB ( ) | |||
| LCB ( ) | |||
| LCB ( ) | |||
| LCB ( ) | |||
| LCB ( ) | |||
| Unguided |
Inference-Time Language Model Alignment
Inference-time language model alignment steers or selects outputs during generation to better meet safety, preference, and task goals without retraining.
See all 82 papers