Axiom Satisfiability of Linear Rewards in Alignment

Soumya Nasipuri; Sayak Ray Chowdhury; Sanjukta Roy

cs.GTSep 26, 2026

Axiom Satisfiability of Linear Rewards in Alignment

Authors: Soumya Nasipuri, Sayak Ray Chowdhury, Sanjukta Roy

Organizations: Indian Statistical Institute, India · Indian Institute of Technology Kanpur, India

Abstract

Learning from human preference data is the dominant route to aligning language models with human values. In linear social choice, where rewards are linear in a fixed feature representation of prompt-response pairs, Ge et al.[2024] show that fitting such a reward by minimizing any non-decreasing convex loss, including BTL, fails PO and PMC. Moreover, no rule that reads only the majority relation can satisfy PO once the output is required to be linearly induced. We ask what it costs to enforce these axioms anyway. To this end, we relax the linear model to allow per-candidate slack. We compute the relaxed linear reward with the smallest total slack that satisfies the axioms with a margin $η$ , the minimum required difference between two reward values. Our solution satisfies the axioms under no assumptions about the voters or how comparisons were collected. We bound the optimal total slack by $O(1)$ when $η$ is at most $O(\frac{1}{m^2})$ for $m$ candidates. Furthermore, we exhibit an instance that forces this bound, concluding that the rate is tight up to constants. In practice, the no. of candidates far exceeds the feature dimension, and only a linear reward can be evaluated on unseen responses. We therefore introduce a new method that charges the linear part for each comparison it gets wrong. It simultaneously minimizes the total slack and the no. of violations, with a parameter $λ$ trading off between them. We show that the total slack is monotone but saturating in $λ$ : raising it reduces the violations of the deployed linear reward and increases the slack, yet the slack stays below $(η+Δ\sqrt{d})\lfloor m^2/4\rfloor$ , where $Δ$ and $d$ are the diameter and dimension of the features, respectively. Experiments on both synthetic and real-life preference data corroborate our theory and show that the linear reward output by our method beats linear BTL.

Figures & tables

Figure 1: Axiom violations and candidate-level slack as a function of normalized Mallows dispersion

\phi

. Curves show mean violation rates across trials, with shaded regions denoting 95% confidence intervals. Panels report (a) Total violations, (b) PO violations, (c) Majority violations, and (d) Total

\|{\bm{\varepsilon}}\|_{1}

for

\mathsf{LP2}

and

\mathsf{LP3}

. Experiments use

11

voters,

50

candidates,

25

-dimensional features and 20 seeds.

Figure 2: Axiom violations and candidate-level slack as a function of normalized Mallows dispersion

\phi

. Curves show mean violation rates across trials, with shaded regions denoting 95% confidence intervals. Panels report (a) Total violations, (b) PO violations, (c) Majority violations, and (d) The

\|{\bm{\varepsilon}}\|_{1}

for

\mathsf{LP2}

and

\mathsf{LP3}

. Experiments use

3

voters,

6

candidates,

5

-dimensional feature embeddings and

20

seeds.

Figure 3: Total violations, total slack, and margin shortfall as a function of

\lambda

. The blue line represents

\mathsf{LP2}

, and the orange

\mathsf{LP3}

. Panels report (a) Margin shortfall (

\sum_{(a,b)\in{\mathcal{P}}}u_{ab}

), (b) Total violations (%), (c) The

\|{\bm{\varepsilon}}\|_{1}

. Experiment uses

3

voters,

25

candidates, and

10

-dimensional feature.

Method	Total Viol. (%)	PO Viol. (%)	Majority Viol. (%)	$\\|{\bm{\varepsilon}}\\|_{1}$	Time (s)
$\mathsf{LP2}$	$1.59\pm 1.16$	$0.65\pm 0.63$	$1.91\pm 1.37$	$\textbf{0.00207}\pm\textbf{0.001}$	$\mathbf{308.96}$
$\mathsf{LP3}$	$\mathbf{1.39\pm 0.98}$	$\mathbf{0.44\pm 0.35}$	$\mathbf{1.72\pm 1.21}$	$0.00249\pm 0.00123$	$722.08$
Linear BTL	$17.51\pm 0.56$	$2.24\pm 0.53$	$22.71\pm 0.62$	–	$2979.11$

Table 1: Machine Deliberation dataset.

K=500

questions,

m=2000,d=768,\eta=10^{-5}

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Axiom violations and candidate-level slack as a function of normalized Mallows dispersion

\phi

. Curves show mean violation rates across trials, with shaded regions denoting 95% confidence intervals. Panels report (a) Total violations, (b) PO violations, (c) Majority violations, and (d) The

\|{\bm{\varepsilon}}\|_{1}

for the

\mathsf{LP2}

and

\mathsf{LP3}

. Experiments use

3

voters,

10

candidates, and

5

-dimensional feature embeddings.

Figure 5: Axiom violations and candidate-level slack as a function of normalized Mallows dispersion

\phi

. Curves show mean violation rates across trials, with shaded regions denoting 95% confidence intervals. Panels report (a) Total violations, (b) PO violations, (c) Majority violations, and (d) The

\|{\bm{\varepsilon}}\|_{1}

for the

\mathsf{LP2}

and

\mathsf{LP3}

. Experiments use

3

voters,

15

candidates, and

5

-dimensional feature embeddings.

Model	Total Viol.(%)	PO Viol.(%)	Majority Viol.(%)	$\\|{\bm{\varepsilon}}\\|_{1}(\times 10^{-3})$	Time (s)
(a) Feature dimension $d=25$
$\mathsf{LP2}$	$28.39\pm 2.04$	$15.60\pm 3.55$	$\textbf{32.48}\pm\textbf{2.21}$	$2.7\pm 0.155$	1.5
$\mathsf{LP3}$	$\textbf{27.95}\pm\textbf{2.31}$	$\textbf{13.69}\pm\textbf{3.41}$	$32.52\pm 2.43$	$2.87\pm 0.155$	$2.95$
Linear BTL	$31.30\pm 2.54$	$14.93\pm 2.99$	$36.53\pm 2.66$	–	$23.12$
(b) Feature dimension $d=50$
$\mathsf{LP2}$	$23.97\pm 2.20$	$11.47\pm 2.23$	$27.98\pm 2.48$	$2.41\pm 0.136$	1.66

Appendix

Table 2: Performance of

\mathsf{LP2}

\mathsf{LP3}

, and Linear BTL on the real-world benchmark of 100 questions comprising 400 candidates, evaluated using PCA-reduced feature representations of dimensions

d\in\{25,75,100\}

. Each entry reports the mean

\pm

standard deviation across the 100 questions.

\eta=10^{-5}

for

\mathsf{LP2}

and

\mathsf{LP3}

	Total Viol.(%)	PO Viol.(%)	Majority Viol.(%)	$\\|{\bm{\varepsilon}}\\|_{1}$	Time (s)
(a) Feature dimension $d=150$
$\mathsf{LP2}$	$24.54\pm 1.18$	$10.90\pm 1.38$	$29.02\pm 1.21$	$1.2610^{-2}\pm 3.5710^{-4}$	38.73
$\mathsf{LP3}$	$\textbf{23.48}\pm\textbf{0.98}$	$9.26\pm 0.99$	$\textbf{28.17}\pm\textbf{1.26}$	$1.3110^{-2}\pm 3.4810^{-4}$	$109.24$
Linear BTL	$25.71\pm 1.16$	$\textbf{8.34}\pm\textbf{0.91}$	$31.40\pm 1.24$	–	$863.24$
(b) Feature dimension $d=200$
$\mathsf{LP2}$	$23.26\pm 1.27$	$9.93\pm 1.44$	$27.64\pm 1.27$	$1.1910^{-2}\pm 3.9210^{-4}$	48.1

Appendix

Table 3: Performance of

\mathsf{LP2}

\mathsf{LP3}

, and Linear BTL on the real-world benchmark of 500 questions comprising 2000 candidates, evaluated using PCA-reduced feature representations of dimensions

d\in\{150,200,250,500\}

. Each entry reports the mean

\pm

standard deviation across the 500 questions.

\eta=10^{-5}

for

\mathsf{LP2}

and

\mathsf{LP3}

Reward Modeling

Reward modeling learns scores or rankings that capture human goals and task success, guiding AI selection, reinforcement learning, and alignment.

See all 313 papers

Sep 27, 2026

From Constitutions to Control: Interpretable Rewards for Aligning Language Models

Current approaches to aligning language models often make it hard to know what behavior is being rewarded or to change that reward in a targeted way. In particular, standard preference-based methods collapse multiple considerations into aggregate human judgments, obscuring what drives the resulting reward, while principle-based methods specify high-level values without fully operationalizing them. To address this gap, we develop a rubric-based framework to transform a general-purpose constitution into an interpretable and tunable reward model, using constitution-guided AI feedback to estimate initial weights for the constituent rubric items. We then reweight those dimensions to construct modified rewards for training. Across experiments on political alignment and safety-helpfulness tradeoffs, reweighting individual dimensions predictably changes targeted behaviors largely independently while navigating tradeoffs between conflicting alignment objectives. We show that the same framework can mitigate label bias encoded in preference judgments -- including sycophancy and demographic bias -- by reducing their influence on the training reward. Our results demonstrate that constitution-derived, interpretable rewards can translate high-level alignment principles into more transparent and controllable model behavior.

Johann D. Gaebler, Calvin Isley, Max Lamparth +2

Stern School of Business, New York University, New York, NY 10012, USA · John F. Kennedy School of Government, Harvard University, Cambridge, MA 02138, USA · Center for AI Safety, Stanford University, Stanford, CA 94305, USA

LLM Alignment Reward Modeling Multi-Objective Language Model Alignment

Ask PDF

May 6, 2026

Misaligned by Reward: Socially Undesirable Preferences in LLMs

Reward models are a key component of large language model alignment, serving as proxies for human preferences during training. However, existing evaluations focus primarily on broad instruction-following benchmarks, providing limited insight into whether these models capture socially desirable preferences. As a result, important failures in social alignment can remain hidden. We extend reward-model benchmarking to four socially consequential domains: bias, safety, morality, and ethical reasoning. We introduce a framework that converts social evaluation datasets into pairwise preference data, leveraging gold labels where available and directional bias indicators otherwise. This enables us to test whether reward models prefer socially undesirable responses, and whether their preferences produce systematically biased distributions over selected outputs. Across five publicly available reward models and two instruction-tuned models used as reward proxies, we find substantial variation across domains, with no single model performing best overall. The models fall well short of strong social intelligence: they often prefer socially undesirable options, and their preferences produce systematically biased distributions. Moreover, stronger bias avoidance can reduce sensitivity to context, revealing a key alignment trade-off between avoiding biased outcomes and preserving contextual faithfulness. These findings show that standard reward benchmarks are insufficient for assessing social alignment and highlight the need for evaluations that directly measure the social preferences encoded in reward models.

Gayane Ghazaryan, Esra Dönmez

Institute for Natural Language Processing, University of Stuttgart · 2Interchange Forum for Reflecting on Intelligent Systems, University of Stuttgart

Reward Modeling Language Model Safety Evaluation Social Bias in Language Models Preference Alignment Reward Model Evaluation

Ask PDF

May 15, 2025

Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignment

Reward models (RMs) are central to aligning large language models (LLMs) with human values, yet noisy preference feedback can induce reward misgeneralization through spurious correlations and overfitting. We characterize the training dynamics of preference pairs and uncover a distinctive optimization signature of noise: persistently high losses, sharp temporal fluctuations, and irregular gradient updates. Our analyses further show that excluding such pairs improves downstream alignment. In practice, however, reliable preferences are not known a priori, and self-filtering based on a model's own training signals can amplify confirmation bias. To address this challenge, we introduce Collaborative Reward Modeling (CRM), an online framework for identifying reliable preferences. CRM couples cross-model peer review with curriculum learning: two RMs evaluate each other's sample selections, while curriculum keeps their learning progress synchronized. Across extensive experiments, CRM improves RM generalization by up to 9.94 points on RewardBench under 40% preference noise. CRM further generalizes to implicit-reward alignment, offering a robust and broadly applicable approach to learning from imperfect feedback.

Jiazheng Zhang, Wenqing Jing, Zizhuo Zhang +12

NLP Group, Fudan University · Hong Kong Baptist University · LLM Team, Meituan Inc

Reward Modeling LLM Alignment Preference Learning Preference-Based RL

Ask PDF

Abstract

Reward Modeling

Explore similar work

From Constitutions to Control: Interpretable Rewards for Aligning Language Models

Misaligned by Reward: Socially Undesirable Preferences in LLMs

Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignment