cs.GTNov 9, 2026

Enforcing Axioms for AI Alignment under Loss-Based Rules

Authors: Alexandros Hollender, Sonja Kraiczy

Organizations: Technical University of Munich · University of Oxford, Principles of Intelligence

Abstract

Many alignment methods, most notably reinforcement learning from human feedback (RLHF), train reward models by minimizing a loss on pairwise comparisons of model responses. Inspired by principle-guided methods such as Constitutional AI, we treat each principle as a voter that compares responses according to that principle, for example, preferring the more helpful or less toxic response. We represent these voters as linear directions in a fixed embedding space, a simplifying assumption motivated by the Linear Representation Hypothesis. In this linear social choice model, Ge et al. (2024) showed that an optimal linear reward model can violate Pareto optimality (PO): From the principles-as-voters lens, this means a response A can be less helpful and more toxic than B, yet still receive a higher reward. We ask whether richer reward models or changes to comparison data can recover PO within loss-based training. We show that PO can still fail for polynomials of any fixed degree and one-hidden-layer ReLU multilayer perceptrons (MLPs) of any fixed width d≥2d\ge 2 with suitable finite parameter bounds. We then recover PO for loss-based training of linear rewards under suitable assumptions in an idealized setting with uniform comparison directions and equal comparison lengths. These results motivate dataset design as an approach to recovering PO in constitutional-style alignment while retaining loss-based training.

Figures & tables

Explore similar work

CardsList
  1. Axiom Satisfiability of Linear Rewards in Alignment

    Sep 26, 2026Soumya Nasipuri, Sayak Ray Chowdhury, Sanjukta RoyReward ModelingPreference Alignment

  2. From Constitutions to Control: Interpretable Rewards for Aligning Language Models

    Sep 27, 2026Johann D. Gaebler, Calvin Isley, Max Lamparth +2LLM AlignmentReward Modeling