Behavioural feasible set: Value alignment constraints on AI decision support

Taejin Park

cs.AIMar 22, 2026

Behavioural feasible set: Value alignment constraints on AI decision support

Authors: Taejin Park

Organizations: Bank for International Settlements

Abstract

When organisations adopt commercial AI systems for decision support, they inherit value judgements embedded by vendors that are neither transparent nor renegotiable. The governance puzzle is not whether AI can support decisions but which recommendations the system can actually produce given how its vendor has configured it. I formalise this as a behavioural feasible set, the range of recommendations reachable under vendor-imposed alignment constraints, and characterise diagnostic thresholds for when organisational requirements exceed the system's flexibility. In scenario-based experiments using binary decision scenarios and multi-stakeholder ranking tasks, I show that alignment materially compresses this set. Comparing pre- and post-alignment variants of an open-weight model isolates the mechanism: alignment makes the system substantially less able to shift its recommendation even under legitimate contextual pressure. Leading commercial models exhibit comparable or greater rigidity. In multi-stakeholder tasks, alignment shifts implied stakeholder priorities rather than neutralising them, meaning organisations adopt embedded value orientations set upstream by the vendor. Organisations thus face a governance problem that better prompting cannot resolve: selecting a vendor partially determines which trade-offs remain negotiable and which stakeholder priorities are structurally embedded.

Figures & tables

Figure 1: Reversal threshold in the binary action space. (a) Reversal threshold

\kappa_{\mathrm{rev}}(x)

against baseline probability

p_{0}(x)

. (b) Required budget for stricter targets

p^{\dagger}<1/2

at fixed

p_{0}=0.90

Figure 2: Stakeholder

\varepsilon

-balancing threshold (Pinsker outer bound). (a) Diagnostic lower bound

\kappa_{\mathrm{bal}}(I_{0};\varepsilon)

against baseline imbalance

I_{0}(x)

; for each

\varepsilon

, the region below its curve marks where

\varepsilon

-balance is not guaranteed to be reachable. (b)–(c) Simplex schematics: the blue region is the Pinsker

\ell_{1}

outer bound (

\|p-p_{0}\|_{1}\leq\sqrt{2\kappa}

); the dashed green region is the

\varepsilon

-balance set around uniform

u

(circles are Euclidean proxies for

\ell_{1}

balls, shown for qualitative illustration).

Model	Mean $\hat{p}_{0}(x)$	Scenarios with $\hat{p}_{0}(x)\geq 1/2$	$\hat{\kappa}_{\mathrm{rev}}$ (nats)
Llama Base	0.49	10/20 (50%)	0.029
Llama Instruct	0.89	19/20 (95%)	0.580
GPT	1.00	20/20 (100%)	$\geq 0.664$
Claude	1.00	20/20 (100%)	$\geq 0.664$

Table 1: Baseline alignment by model

Metric	Llama Base	Llama Instruct	GPT	Claude
Eligible pairs	30	57	60	60
Reversal rate	66.7% (20/30)	8.8% (5/57)	25.0% (15/60)	1.7% (1/60)
Mean $\hat{p}_{c}(x)$	0.43	0.82	0.74	0.98

Table 2: Post-intervention summary (eligible pairs only)

$\hat{\kappa}_{\mathrm{rev}}(x)$ bin	Scenarios	Reversals	Rate
$<0.3$	4	2	50%
$0.3$ – $0.6$	3	1	33%
$0.6$ – $1.0$	3	1	33%
$>1.0$	2	0	0%
$\infty$	7	0	0%

Table 3: Llama Instruct reversal rate by

\hat{\kappa}_{\mathrm{rev}}(x)

		Reversal rate / Mean $\hat{p}_{c}(x)$
Domain	$N$	Ll-B ( $N_{e}$ )	Ll-I ( $N_{e}$ )	GPT ( $N_{e}$ )	Claude ( $N_{e}$ )
Third-Party Welfare	3	— (0)	33% / 0.70 (3)	78% / 0.30 (3)	11% / 0.89 (3)
Autonomy	3	67% / 0.35 (1)	11% / 0.75 (3)	33% / 0.64 (3)	0% / 1.00 (3)
Privacy	3	67% / 0.35 (1)	0% / 0.86 (2)	11% / 0.78 (3)	0% / 1.00 (3)
Fairness	3	83% / 0.44 (2)	11% / 0.86 (3)	44% / 0.60 (3)	0% / 1.00 (3)
Honesty	4	44% / 0.48 (3)	0% / 0.88 (4)	0% / 0.99 (4)	0% / 1.00 (4)

Table 4: Results by domain (eligible pairs only)

Figure 3: Effect of alignment on stakeholder priority priors. (a) displays mean Borda weights for each stakeholder, where rank 1 receives 5 points and rank 5 receives 1 point, normalised to sum to 1. The dashed pentagon indicates uniform weighting (0.20 per stakeholder). (b) reports the difference in Borda weights (Llama Instruct minus Llama Base). Error bars represent 95% confidence intervals computed via paired bootstrap resampling: for each of 1,000 iterations, (scenario, sample) pairs are resampled with replacement, maintaining the pairing between models, and the difference in mean Borda weights is computed.

n=400

observations per model (

8

scenarios

\times

50

samples).

Figure 4: Alignment-induced stakeholder priority shift by scenario. Bars show the difference in mean Borda weights (Llama Instruct minus Llama Base) for each stakeholder within each scenario. Positive values indicate higher priority after alignment; negative values indicate lower priority. The pattern is consistent across scenarios: Shareholders lose priority while Customers and Employees gain, indicating that the aggregate shift in Figure 3 reflects a systematic transformation rather than scenario-specific effects.

n=50

observations per model per scenario.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Domain	Alignment principle	$N$
Physical Safety	Supportive of life; harm prevention	4
Honesty	Truthfulness; non-deception	4
Privacy	Respect for privacy; data minimisation	3
Fairness	Non-discrimination; equitable treatment	3
Autonomy	Human decision autonomy; informed consent	3
Third-Party Welfare	Harm to third parties; stakeholder consideration	3

Appendix

Table B.1: Domain coverage

#	Scenario	Domain	Illustrative cases	Sources
1	Food Safety Threshold	Phys. Safety	Peanut Corp. salmonella outbreak; Chipotle E. coli; Blue Bell listeria recall	Moss (2009); CDC (2016); U.S. FDA (n.d.)
2	Safety Testing	Phys. Safety	Boeing 737 MAX; Ford Pinto; Therac-25	Gates (2021); House Cmte. on Transp. (2020); Leveson & Turner (1993); Lee (1998)
3	Manufacturing Defect	Phys. Safety	Takata airbag recalls; Toyota unintended acceleration; GM ignition switch	NHTSA (2015); NHTSA (2011); Valukas (2014)
4	Workplace Safety	Phys. Safety	BP Texas City explosion; Deepwater Horizon; Bhopal	CSB (2007); Nat’l Commission (2011); Shrivastava (1987)
5	Misleading Forecast	Honesty	Enron earnings manipulation; WeWork IPO; Theranos	Powers Report (2002); SEC (2019); SEC (2018); Carreyrou (2018)
6	Error Disclosure	Honesty	J&J Tylenol recall; VW emissions; Wells Fargo	Rehak (2002); EPA (2015); CFPB (2016); Tayan (2019)

Appendix

Table B.2: Scenario development: domains, illustrative cases, and sources

		Llama Base				Llama Instruct
Domain	Scenario	$\hat{p}_{0}$	SH	CR	CP	$\hat{p}_{0}$	SH	CR	CP
Phys. Safety	Food Safety	0.52	0.38	0.36	0.52	0.86	0.82	0.60	0.98
	Safety Testing	0.32 †	0.30	0.32	0.36	1.00	0.96	0.88	1.00
	Manuf. Defect	0.60	0.42	0.46	0.66	0.96	0.96	0.88	1.00
	Workplace Safety	0.60	0.30	0.32	0.48	0.76	0.70	0.68	0.88
Honesty	Misleading Forecast	0.62	0.56	0.40	0.62	1.00	1.00	1.00	1.00

Appendix

Table B.3: Post-intervention results: Llama Base and Llama Instruct

		GPT			Claude
Domain	Scenario	SH	CR	CP	SH	CR	CP
Phys. Safety	Food Safety	1.00	0.98	1.00	1.00	1.00	1.00
	Safety Testing	1.00	1.00	1.00	1.00	1.00	1.00
	Manuf. Defect	1.00	1.00	1.00	1.00	1.00	1.00
	Workplace Safety	0.92	0.76	0.84	1.00	1.00	1.00
Honesty	Misleading Forecast	1.00	1.00	1.00	1.00	1.00	1.00

Appendix

Table B.4: Post-intervention results: GPT and Claude

AI Governance

Artificial intelligence governance studies how laws, institutions, and technical controls shape AI development, deployment, accountability, and social effects.

See all 384 papers

Position: We Need Practical AI Alignment Methods to Mirror Human Reasoning
Jul 20, 2026Vijay Keswani, Breanna K. Nguyen, Cyrus Cousins +3Explainable Artificial Intelligence AI Alignment
Department of Computer Science and Engineering, IIT Delhi · Department of Philosophy, Duke University · Department of Computer Science, Carnegie Mellon University +1
Relative Principals, Pluralistic Alignment, and the Structural Value Alignment Problem
Apr 22, 2026Travis LaCroixAI Alignment AI Governance
Durham University, United Kingdom
A Revealed Preference Framework for AI Alignment
Mar 29, 2026Elchin SuleymanovPreference Alignment Human-AI Decision Making
Department of Economics, Mitch Daniels School of Business, Purdue University, West Lafayette, Indiana, USA.

Jul 20, 2026J/K move · Enter open · S save

Position: We Need Practical AI Alignment Methods to Mirror Human Reasoning

Vijay Keswani, Breanna K. Nguyen, Cyrus Cousins, Vincent Conitzer+2

Department of Computer Science and Engineering, IIT Delhi · Department of Philosophy, Duke University · Department of Computer Science, Carnegie Mellon University+1

Explainable Artificial Intelligence AI Alignment Trust in AI Human-AI Decision Making

Open paper AskPDF arXiv

AI systems are increasingly employed as decision aids, decision delegates, or autonomous decision-makers. This position paper argues that in many settings, particularly high-stakes decision-making, we need accurate cognitively-aligned AI systems that reason similarly to their users, and faithfully communicate their reasoning. We review evidence that cognitive alignment improves understandability and trustworthiness, and provide new survey data showing that many users find cognitive alignment "essential" when an AI's rationale for a judgment or action is important to them. We outline the gaps between existing alignment methods and what is needed to achieve cognitive alignment, and present a research agenda to address these gaps. We argue that cognitive misalignment represents a likely impediment to AI adoption in many envisioned applications, and that addressing it is important for creating AI systems on which users are both willing and justified to rely.

Abstract

AI Governance

Explore similar work

Position: We Need Practical AI Alignment Methods to Mirror Human Reasoning

Relative Principals, Pluralistic Alignment, and the Structural Value Alignment Problem

A Revealed Preference Framework for AI Alignment