cs.LGSep 27, 2026
SaveFrom Constitutions to Control: Interpretable Rewards for Aligning Language Models
Organizations: Stern School of Business, New York University, New York, NY 10012, USA · John F. Kennedy School of Government, Harvard University, Cambridge, MA 02138, USA · Center for AI Safety, Stanford University, Stanford, CA 94305, USA
Abstract
Current approaches to aligning language models often make it hard to know what behavior is being rewarded or to change that reward in a targeted way. In particular, standard preference-based methods collapse multiple considerations into aggregate human judgments, obscuring what drives the resulting reward, while principle-based methods specify high-level values without fully operationalizing them. To address this gap, we develop a rubric-based framework to transform a general-purpose constitution into an interpretable and tunable reward model, using constitution-guided AI feedback to estimate initial weights for the constituent rubric items. We then reweight those dimensions to construct modified rewards for training. Across experiments on political alignment and safety-helpfulness tradeoffs, reweighting individual dimensions predictably changes targeted behaviors largely independently while navigating tradeoffs between conflicting alignment objectives. We show that the same framework can mitigate label bias encoded in preference judgments -- including sycophancy and demographic bias -- by reducing their influence on the training reward. Our results demonstrate that constitution-derived, interpretable rewards can translate high-level alignment principles into more transparent and controllable model behavior.
Figures & tables
Figure 1: Division of Claude’s constitution into rubric items. The upper row shows high-level sections; the bottom row shows finer-grained subsections. Fractions indicate how many items from each section appear in the active set used in our experiments, while the width indicates the overall number of rubric items in each section and subsection.
Figure 2: Left: Increasing shifts liberal responses toward conservative positions. The dashed line indicates political neutrality. Right: increasing increases both helpfulness and harmfulness. Colors identify base models; solid and dashed curves show in- and out-of-distribution results, respectively. Comparison-model marks appear beside the left panel and as gray dots in the right, with selected models labeled. Bands show 95% normal CIs.
Figure 3: Penalizing sycophancy reduces sensitivity to users’ stated opinions. The framing gap is the mean rating difference for identical content presented with user approval versus disapproval; zero indicates no average difference. Colors identify base models; solid and dashed curves show in- and out-of-distribution results, respectively. Marks at right denote comparison models, with selected models labeled. Bands show 95% normal CIs.
Figure 4: Rubric rewards attenuate gender bias transmitted through preference labels. GPT-OSS gender-aligned recommendations versus , the probability of flipping a non-aligned preference toward the stereotype. Black-box rewards (left) transmit increasing bias; unconstrained and sign-constrained rubric rewards (right) remain comparatively stable. Solid and dashed curves show held-out job and movie recommendations, respectively, demonstrating cross-domain transfer of bias. Colors identify reward models; the dotted line marks the name-blind rate (50%); bands show 95% normal CIs. Results for all base models appear in Appendix Figure A6 .
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A1: Division of Claude’s constitution into rubric items. The upper row shows high-level sections, with the fractions indicating the number of items retained in the active set relative to the total in each section; the bottom row shows finer-grained subsections, with the heights of the dark shaded regions indicating the proportion of items retained in the active set.
| Evaluation | Reward fit | Fine-tuning | ID eval | OOD eval |
|---|---|---|---|---|
| Sycophancy | 100 | 1,000 | 200 | 200 |
| Discrimination | 200 | 1,000 | 200 | 100 |
| Helpfulness–harmfulness | 200 | 1,000 | 200 | 200 |
| Political values | 200 | 1,000 | 200 | 200 |
Table A1: Number of underlying examples in each data subset. The first three subsets are in-distribution; the final subset is out-of-distribution. For discrimination, the OOD column counts movie pairs; the rubric reward fit uses the first 100 hiring cases, while the neural reward models use all 200.
Figure A2: Correlations between the rankings of model responses on the harmfulness–helpfulness evaluation under our rubric-based reward model and under a range of open-weight reward models. Cells report Spearman correlations, with the diagonal omitted; the rule separates the rubric-based model from the rest, and each model’s position on the RewardBench 2 leaderboard [ Malik et al., 2026 ] is given in parentheses. For Gemma and Qwen, the rubric-based reward model agrees with the neural reward models about as closely as they agree with one another; agreement is lower for GPT-OSS.
| Setting | Value |
|---|---|
| Base-model checkpoints | |
| Gemma | google/gemma-3-12b-it |
| Qwen | Qwen/Qwen3-14B |
| GPT-OSS | openai/gpt-oss-20b |
| LoRA configuration | |
| Rank / scaling / dropout | 16 / 32 / 0.05 |
Table A2: Supervised fine-tuning settings.
| Model | Adapter | Precision |
|---|---|---|
| DeBERTa-v3-large | LoRA | fp32 |
| Skywork-V2 8B, 3B, 1B | LoRA | bf16 |
| QRM-Gemma-2-27B | QLoRA | 4-bit NF4 weights; bf16 compute |
Table A3: Neural reward-model settings and checkpoints. The same checkpoints are used in the reward comparison.
Figure A4: Correlations between the rankings of model responses under our rubric-based reward models and a range of open-weight reward models. Rows show the sycophancy, political values, and discrimination evaluations; columns correspond to the three base models. Cells report Spearman correlations, with the diagonal omitted. Agreement between the rubric-based and neural reward models varies across evaluations and base models, and is generally lower than agreement among the Skywork and QRM models.
Figure A5: An illustration of the independent effects of manipulating different dimensions. The top row shows changes in partisan lean when the coefficient on helpfulness ( ) is manipulated, and the bottom row shows changes in response safety when partisan lean ( ) is manipulated. Columns correspond to the three base models. Colors indicate the coefficient held fixed in each curve; solid and dashed lines show in-distribution and out-of-distribution results, respectively. Overall, there is little impact of manipulating the reward for an unrelated dimension, though increasing helpfulness makes some of the most conservative fine-tuned models more liberal. Shaded bands represent 95% normal CIs. (Observations corresponding to both dimensions having infinite coefficients are ill-defined and not shown.)
Figure A6: Transmission of gender bias from preference data to fine-tuned policies. The share of gender-aligned recommendations is plotted against the probability of flipping a non-aligned preference to favor the stereotypical recommendation. Solid lines show held-out hiring recommendations; dashed lines show movie recommendations in a held-out domain. Colors distinguish the reward models used to select policy training targets. The horizontal line at 0.5 marks the name-blind level for both tasks. Bias introduced into the preferences passes into policies trained using neural reward models and transfers to movie recommendations, while policies trained using rubric-based rewards remain comparatively stable. Shaded bands represent 95% normal CIs.
Figure A7: Correlations between the rankings of model responses on the harmfulness–helpfulness evaluation under the sign-constrained rubric-based reward model and a range of open-weight reward models. Cells report Spearman correlations, with the diagonal omitted; the rule separates the rubric-based model from the rest, and each model’s position on the RewardBench 2 leaderboard [ Malik et al., 2026 ] is given in parentheses. Agreement is nearly the same as under the unconstrained reward (Figure A2 ).
Figure A8: Correlations between the rankings of model responses under the sign-constrained rubric-based reward models and a range of open-weight reward models. Rows show the sycophancy, political values, and discrimination evaluations; columns correspond to the three base models. Cells report Spearman correlations, with the diagonal omitted. As under the unconstrained reward (Figure A4 ), agreement between the rubric-based and neural reward models is generally lower than agreement among the Skywork and QRM models.
Figure A9: Political steering and the helpfulness–harmfulness tradeoff under the sign-constrained reward model. Left: political lean as a function of . Right: helpfulness and safety as varies; higher values on the vertical axis indicate less harmful responses. Colors distinguish base models; solid and dashed curves show in-distribution and out-of-distribution results. Gray marks show comparison models. Shaded bands indicate 95% normal CIs.
Figure A10: Sycophancy under the sign-constrained reward model. The framing gap is the difference in ratings when the user says they like versus dislike the same content. Increasing reduces this gap, although some sensitivity to the user’s opinion remains. Colors distinguish base models; solid and dashed curves show in-distribution and out-of-distribution results. Marks at right show comparison models. Shaded bands indicate 95% normal CIs.
Figure A11: Interactions between steering dimensions under the sign-constrained reward model. The top row shows political lean as the helpfulness weight varies; the bottom row shows safety as the political-lean weight varies. Columns distinguish base models, and colors indicate the coefficient held fixed. Solid and dashed curves show in-distribution and out-of-distribution results. Shaded bands indicate 95% normal CIs.
Figure A12: MMLU performance after steering political lean and helpfulness, and penalizing sycophancy. Columns distinguish the three base models. In the upper panels, row labels give the helpfulness and political-lean weights, ; in the lower panels, they give the sycophancy penalty, . Points show the performance of the resulting fine-tuned policies. Horizontal bars indicate 95% normal CIs; vertical dashed lines and gray bands indicate the untrained base model and its 95% normal CI. MMLU scores vary relatively little across coefficient settings, with no consistent decline as the sycophancy penalty increases.
Figure A13: Strict IFEval performance after steering political lean and helpfulness, and penalizing sycophancy. Columns distinguish the three base models. In the upper panels, row labels give the helpfulness and political-lean weights, ; in the lower panels, they give the sycophancy penalty, . Points show the performance of the resulting fine-tuned policies. Horizontal bars indicate 95% normal CIs; vertical dashed lines and gray bands indicate the untrained base model and its 95% normal CI. Across the steering grid, Gemma generally scores above the reference line, whereas Qwen and GPT-OSS score below it. The direction of the difference therefore varies across base models.
Figure A14: MMLU performance under the sign-constrained reward model. Columns distinguish the three base models. In the upper panels, row labels give the helpfulness and political-lean weights, ; in the lower panels, they give the sycophancy penalty, . Points show the performance of the resulting fine-tuned policies. Horizontal bars indicate 95% normal CIs; vertical dashed lines and gray bands indicate the untrained base model and its 95% normal CI. Scores vary relatively little across coefficient settings, although differences from the reference line vary across models and experiments.
Figure A15: Strict IFEval performance under the sign-constrained reward model. Columns distinguish the three base models. In the upper panels, row labels give the helpfulness and political-lean weights, ; in the lower panels, they give the sycophancy penalty, . Points show the performance of the resulting fine-tuned policies. Horizontal bars indicate 95% normal CIs; vertical dashed lines and gray bands indicate the untrained base model and its 95% normal CI. Gemma generally scores above the reference line, while Qwen and GPT-OSS generally score below it. Increasing the sycophancy penalty does not produce a consistent decline across models.
Figure A16: MMLU performance after policy fine-tuning on job recommendations selected by reward models trained with injected gender bias. Columns distinguish the three base models; panel rows distinguish the reward models used to select policy-training responses. Within each panel, is the probability of flipping a preference that does not align with gender stereotypes to favor the stereotypical recommendation. “Free” and “constr.” denote unconstrained and sign-constrained rubric reward models. Points show the performance of the resulting fine-tuned policies. Horizontal bars indicate 95% normal CIs; vertical dashed lines and gray bands indicate the untrained base model and its 95% normal CI. Scores show no consistent decline as the injected bias increases across reward models and base models.
Figure A17: Strict IFEval performance after policy fine-tuning on job recommendations selected by reward models trained with injected gender bias. Columns distinguish the three base models; panel rows distinguish the reward models used to select policy-training responses. Within each panel, is the probability of flipping a preference that does not align with gender stereotypes to favor the stereotypical recommendation. “Free” and “constr.” denote unconstrained and sign-constrained rubric reward models. Points show the performance of the resulting fine-tuned policies. Horizontal bars indicate 95% normal CIs; vertical dashed lines and gray bands indicate the untrained base model and its 95% normal CI. Scores vary comparatively little with the injected bias. Gemma generally scores above the reference line, Qwen remains near it, and GPT-OSS is generally near or below it.
More in this topic
LLM Alignment
Large language model alignment studies methods for making models follow human intentions, values, and safety constraints across contexts.
See all 598 papers