Bounded Morality: Defining the Space of Moral Computation

Max Kanwal; Caryn Tran; Patrick Mineault

cs.AIApr 1, 2026

Bounded Morality: Defining the Space of Moral Computation

Authors: Max Kanwal, Caryn Tran, Patrick Mineault

Organizations: Stanford University · Northwestern University · Amaranth Foundation

Abstract

Moral cognition has traditionally been modeled as adherence to fixed ethical theories--deontology, consequentialism, virtue ethics--implemented as static rules or value functions. We propose Bounded Morality, a formal framework for analyzing the computational demands of moral problems faced by finite agents. Extending Herbert Simon's notion of bounded rationality, we formalize moral situations along two orthogonal dimensions: moral breadth, the scope of entities treated as morally relevant, and moral depth, the inferential integration required to evaluate their interactions. Limited resources impose an unavoidable tradeoff between these dimensions, defining a feasible space of moral computation. Within this space, ethical theories correspond to locally efficient strategies adapted to different demand regimes rather than competing accounts of moral truth. The framework yields a formal notion of moral regret and moral progress under constraint, and implies that moral alignment in artificial systems depends on the scaling and allocation of moral reasoning capacity rather than on direct imitation of human judgments.

Figures & tables

Role in Framework	Concept	Interpretation
Overall framework	Bounded Morality	Morality understood as a computational problem solved approximately under finite cognitive, computational, and data constraints.
Core axis (representation)	Moral Breadth	How much of the moral world is explicitly represented: which entities, stakeholders, timescales, or abstractions are taken into account.
Core axis (reasoning)	Moral Depth	How extensively consequences and interactions are propagated and integrated through inference, counterfactual reasoning, or deliberation.
Fundamental constraint	Resource Budget	The finite capacity available for moral reasoning, including attention, memory, computation, time, and data.
Agent-level choice	Moral Strategy	A policy for allocating limited resources across breadth and depth when evaluating moral decisions.
Idealized reference	Unbounded Moral Agent	A hypothetical oracle that represents all morally relevant entities and reasons exhaustively over their interactions.

Table 1: Conceptual summary of the Bounded Morality framework.

Panel of Figure 1 — Figure 1: At fixed budget $B$ and distribution $\mathsf{P}$ , policies $\rho_{\mathrm{old}}$ and $\rho_{\mathrm{new}}$ map world states to structural allocations $(G,H)$ . The plot highlights a representative state $\omega_{0}$ where the improved policy selects a different breadth–depth trade-off $(G,H)=\rho_{\mathrm{new}}(\omega_{0},B)$ on the fixed-budget frontier, yielding lower state-dependent regret $R(\omega_{0};\rho_{\mathrm{new}},B)$ than $R(\omega_{0};\rho_{\mathrm{old}},B)$ . Dashed lines indicate expected regret $\mathbb{E}_{\omega}[R(\omega;\rho,B)]$ (here shown for a uniform $\omega$ ).

Theory	Breadth	Depth	Structural Commitment	Typical Regime
Act Utilitarianism	High	Mod.	Additive sum $\sum_{t}\gamma^{t}U(s(t))$	Large, weakly coupled systems
Deontology	Mod.	Mod.	Hard constraints on $\mathcal{A}_{H}(G)$	Tight rules; limited scope
Contractualism	Low	High	Worst-case $\min_{v}\sum_{t}\gamma^{t}U_{v}(s(t))$	Few parties; deep objections
Virtue Ethics	Very Low	$0$	Amortized policy $\pi_{\theta}$	Real-time action
Care Ethics	Local	Mod.	Relational (edge-weighted) terms	Dense local dependence

Table 2: Ethical theories as caricatured families of bounded moral strategies. Each theory occupies a characteristic region of the breadth–depth frontier and imposes a distinct structural commitment on evaluation or admissibility.

Panel of Figure 2 — Figure 2: Budget contours ( $B$ ) induce Pareto-efficient breadth–depth frontiers under a canonical cost model. Ethical theories are depicted as strategy families with characteristic scaling tendencies: utilitarianism trades budget for breadth, contractualism for depth, care ethics for local breadth and moderate depth, while deontology and virtue ethics remain near low-depth regimes. Regions are conceptual caricatures.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol / Term	Name	Definition / Role
$G^{\star}=(V^{\star},E^{\star})$	Moral interaction graph	Full graph of morally relevant entities and their local influence relations.
$v\in V^{\star}$	Moral entity	Node whose state contributes directly to moral welfare.
$\mathcal{S}_{v}$	Local state space	State space associated with entity $v$ .
$A$	Action set	Finite set of atomic interventions available at each time step.
$\mathcal{A}_{H}=A^{H}$	Action sequences	Length- $H$ intervention sequences (temporally extended control policies).
$F$	Moral dynamics	Local graph-structured state update rule governing trajectory evolution.

Appendix

Table 3: Comprehensive glossary of formal objects used in the Bounded Morality framework. This table consolidates the mathematical definitions introduced in Section 3 for reference.

$H$	$a^{(0)}$	$a^{(E)}$	$a^{(EC)}$
2	-0.765	-0.297	-0.088
10	-1.304	-0.581	-0.715
∞	-1.348	-0.634	-0.955

Appendix

Table 4: Finite-horizon value

\hat{M}_{G^{\star},H}(a,\omega)

and infinite-horizon value

M^{\star}(a,\omega)

(approximated using rollout to 2000 steps). Bold indicates the maximizer.

Panel of Figure 3 — Figure 3: Ground-truth moral interaction graph $G^{\star}$ (left) and a coarse abstraction $G$ (right) induced by aggregation map $\pi_{V}$ with $X=\{E_{1},E_{2},C\}$ and $Y=\{M_{1},M_{2}\}$ .

Moral Reasoning

Moral reasoning studies how people and AI systems interpret ethical principles, weigh competing values, and justify judgments in context.

See all 44 papers

An Algebraic Exposition of the Theory of Dyadic Morality
May 15, 2026Kush R. VarshneyComputational Cognitive Modeling Moral Reasoning
IBM Research, Yorktown Heights, NY, USA
Moral Entropy: Auditing Bias and Uncertainty in Moral Judgment
Sep 18, 2026Maciej SkorskiAnnotator Disagreement Algorithmic Auditing
University of Luxembourg
A Contractualist Argumentation Framework for Moral Decision-Making
Aug 3, 2026Luis Marcos-Vidal, Giulio Antonio Abbo, Tony BelpaemeComputational Argumentation Moral Reasoning
Hospital del Mar Research Institute, Barcelona, Spain · IDLab-AIRO, Ghent University – imec, Ghent, Belgium

May 15, 2026J/K move · Enter open · S save

An Algebraic Exposition of the Theory of Dyadic Morality

Kush R. Varshney

IBM Research, Yorktown Heights, NY, USA

Computational Cognitive Modeling Moral Reasoning

Open paper AskPDF arXiv

This paper provides an algebraic exposition of the theory of dyadic morality (TDM), a psychological model of moral judgment grounded in a simple two-node template: an intentional agent causing harm to a vulnerable patient. We formalize TDM using structural causal modeling (SCM) notation and identify three psychological operators (typecasting operator, completion operator, and valence-dependent inference mechanism) that extend standard SCM to capture how people compute moral judgments under constraints. We address scalability challenges arising from TDM's dyadic limitation, showing how moral cognition compresses multi-node scenarios through node collapse and sequential processing. Drawing on this algebraic framework, we demonstrate concrete applications to AI policy design: detecting conflicting obligations, structuring helpfulness policies to preserve user agency, and designing post-failure communication as causal interventions. Finally, we recommend scoped, contextual measurement of mind perception over universal averaging to operationalize the theory empirically. This algebraic formalization enables neurosymbolic AI systems to compute morality in a way that is both mathematically rigorous and faithful to human moral cognition.

Abstract

Moral Reasoning

Explore similar work

An Algebraic Exposition of the Theory of Dyadic Morality

Moral Entropy: Auditing Bias and Uncertainty in Moral Judgment

A Contractualist Argumentation Framework for Moral Decision-Making