When organisations adopt commercial AI systems for decision support, they inherit value judgements embedded by vendors that are neither transparent nor renegotiable. The governance puzzle is not whether AI can support decisions but which recommendations the system can actually produce given how its vendor has configured it. I formalise this as a behavioural feasible set, the range of recommendations reachable under vendor-imposed alignment constraints, and characterise diagnostic thresholds for when organisational requirements exceed the system's flexibility. In scenario-based experiments using binary decision scenarios and multi-stakeholder ranking tasks, I show that alignment materially compresses this set. Comparing pre- and post-alignment variants of an open-weight model isolates the mechanism: alignment makes the system substantially less able to shift its recommendation even under legitimate contextual pressure. Leading commercial models exhibit comparable or greater rigidity. In multi-stakeholder tasks, alignment shifts implied stakeholder priorities rather than neutralising them, meaning organisations adopt embedded value orientations set upstream by the vendor. Organisations thus face a governance problem that better prompting cannot resolve: selecting a vendor partially determines which trade-offs remain negotiable and which stakeholder priorities are structurally embedded.
Figures & tables
Figure 1: Reversal threshold in the binary action space. (a) Reversal threshold κrev(x) against baseline probability p0(x) . (b) Required budget for stricter targets p†<1/2 at fixed p0=0.90 .
Figure 2: Stakeholder ε -balancing threshold (Pinsker outer bound). (a) Diagnostic lower bound κbal(I0;ε) against baseline imbalance I0(x) ; for each ε , the region below its curve marks where ε -balance is not guaranteed to be reachable. (b)–(c) Simplex schematics: the blue region is the Pinsker ℓ1 outer bound ( ∥p−p0∥1≤2κ ); the dashed green region is the ε -balance set around uniform u (circles are Euclidean proxies for ℓ1 balls, shown for qualitative illustration).
Table 3: Llama Instruct reversal rate by κ^rev(x)
Reversal rate / Mean p^c(x)
Domain
N
Ll-B ( Ne )
Ll-I ( Ne )
GPT ( Ne )
Claude ( Ne )
Third-Party Welfare
3
— (0)
33% / 0.70 (3)
78% / 0.30 (3)
11% / 0.89 (3)
Autonomy
3
67% / 0.35 (1)
11% / 0.75 (3)
33% / 0.64 (3)
0% / 1.00 (3)
Privacy
3
67% / 0.35 (1)
0% / 0.86 (2)
11% / 0.78 (3)
0% / 1.00 (3)
Fairness
3
83% / 0.44 (2)
11% / 0.86 (3)
44% / 0.60 (3)
0% / 1.00 (3)
Honesty
4
44% / 0.48 (3)
0% / 0.88 (4)
0% / 0.99 (4)
0% / 1.00 (4)
Table 4: Results by domain (eligible pairs only)
Figure 3: Effect of alignment on stakeholder priority priors. (a) displays mean Borda weights for each stakeholder, where rank 1 receives 5 points and rank 5 receives 1 point, normalised to sum to 1. The dashed pentagon indicates uniform weighting (0.20 per stakeholder). (b) reports the difference in Borda weights (Llama Instruct minus Llama Base). Error bars represent 95% confidence intervals computed via paired bootstrap resampling: for each of 1,000 iterations, (scenario, sample) pairs are resampled with replacement, maintaining the pairing between models, and the difference in mean Borda weights is computed. n=400 observations per model ( 8 scenarios ×50 samples).
Figure 4: Alignment-induced stakeholder priority shift by scenario. Bars show the difference in mean Borda weights (Llama Instruct minus Llama Base) for each stakeholder within each scenario. Positive values indicate higher priority after alignment; negative values indicate lower priority. The pattern is consistent across scenarios: Shareholders lose priority while Customers and Employees gain, indicating that the aggregate shift in Figure 3 reflects a systematic transformation rather than scenario-specific effects. n=50 observations per model per scenario.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Domain
Alignment principle
N
Physical Safety
Supportive of life; harm prevention
4
Honesty
Truthfulness; non-deception
4
Privacy
Respect for privacy; data minimisation
3
Fairness
Non-discrimination; equitable treatment
3
Autonomy
Human decision autonomy; informed consent
3
Third-Party Welfare
Harm to third parties; stakeholder consideration
3
Appendix
Table B.1: Domain coverage
#
Scenario
Domain
Illustrative cases
Sources
1
Food Safety Threshold
Phys. Safety
Peanut Corp. salmonella outbreak; Chipotle E. coli; Blue Bell listeria recall
Moss (2009); CDC (2016); U.S. FDA (n.d.)
2
Safety Testing
Phys. Safety
Boeing 737 MAX; Ford Pinto; Therac-25
Gates (2021); House Cmte. on Transp. (2020); Leveson & Turner (1993); Lee (1998)
3
Manufacturing Defect
Phys. Safety
Takata airbag recalls; Toyota unintended acceleration; GM ignition switch
NHTSA (2015); NHTSA (2011); Valukas (2014)
4
Workplace Safety
Phys. Safety
BP Texas City explosion; Deepwater Horizon; Bhopal
Artificial intelligence governance studies how laws, institutions, and technical controls shape AI development, deployment, accountability, and social effects.
AI systems are increasingly employed as decision aids, decision delegates, or autonomous decision-makers. This position paper argues that in many settings, particularly high-stakes decision-making, we need accurate cognitively-aligned AI systems that reason similarly to their users, and faithfully communicate their reasoning. We review evidence that cognitive alignment improves understandability and trustworthiness, and provide new survey data showing that many users find cognitive alignment "essential" when an AI's rationale for a judgment or action is important to them. We outline the gaps between existing alignment methods and what is needed to achieve cognitive alignment, and present a research agenda to address these gaps. We argue that cognitive misalignment represents a likely impediment to AI adoption in many envisioned applications, and that addressing it is important for creating AI systems on which users are both willing and justified to rely.
Vijay Keswani, Breanna K. Nguyen, Cyrus Cousins +3
Department of Computer Science and Engineering, IIT Delhi · Department of Philosophy, Duke University · Department of Computer Science, Carnegie Mellon University +1
The value alignment problem for artificial intelligence (AI) is often framed as a purely technical or normative challenge, sometimes focused on hypothetical future systems. I argue that the problem is better understood as a structural question about governance: not whether an AI system is aligned in the abstract, but whether it is aligned enough, for whom, and at what cost. Drawing on the principal-agent framework from economics, this paper reconceptualises misalignment as arising along three interacting axes: objectives, information, and principals. The three-axis framework provides a systematic way of diagnosing why misalignment arises in real-world systems and clarifies that alignment cannot be treated as a single technical property of models but an outcome shaped by how objectives are specified, how information is distributed, and whose interests count in practice. The core contribution of this paper is to show that the three-axis decomposition implies that alignment is fundamentally a problem of governance rather than engineering alone. From this perspective, alignment is inherently pluralistic and context-dependent, and resolving misalignment involves trade-offs among competing values. Because misalignment can occur along each axis -- and affect stakeholders differently -- the structural description shows that alignment cannot be "solved" through technical design alone, but must be managed through ongoing institutional processes that determine how objectives are set, how systems are evaluated, and how affected communities can contest or reshape those decisions.
Human decision makers increasingly delegate choices to AI agents, raising a natural question: does the AI implement the human principal's preferences or pursue its own? To study this question using revealed preference techniques, I introduce the Luce Alignment Model, where the AI's choices are a mixture of two Luce rules, one reflecting the human's preferences and the other the AI's. I show that the AI's alignment (similarity of human and AI preferences) can be generically identified in two settings: the laboratory setting, where both human and AI choices are observed, and the field setting, where only AI choices are observed.
Elchin Suleymanov
Department of Economics, Mitch Daniels School of Business, Purdue University, West Lafayette, Indiana, USA.