Moral cognition has traditionally been modeled as adherence to fixed ethical theories--deontology, consequentialism, virtue ethics--implemented as static rules or value functions. We propose Bounded Morality, a formal framework for analyzing the computational demands of moral problems faced by finite agents. Extending Herbert Simon's notion of bounded rationality, we formalize moral situations along two orthogonal dimensions: moral breadth, the scope of entities treated as morally relevant, and moral depth, the inferential integration required to evaluate their interactions. Limited resources impose an unavoidable tradeoff between these dimensions, defining a feasible space of moral computation. Within this space, ethical theories correspond to locally efficient strategies adapted to different demand regimes rather than competing accounts of moral truth. The framework yields a formal notion of moral regret and moral progress under constraint, and implies that moral alignment in artificial systems depends on the scaling and allocation of moral reasoning capacity rather than on direct imitation of human judgments.
Figures & tables
Role in Framework
Concept
Interpretation
Overall framework
Bounded Morality
Morality understood as a computational problem solved approximately under finite cognitive, computational, and data constraints.
Core axis (representation)
Moral Breadth
How much of the moral world is explicitly represented: which entities, stakeholders, timescales, or abstractions are taken into account.
Core axis (reasoning)
Moral Depth
How extensively consequences and interactions are propagated and integrated through inference, counterfactual reasoning, or deliberation.
Fundamental constraint
Resource Budget
The finite capacity available for moral reasoning, including attention, memory, computation, time, and data.
Agent-level choice
Moral Strategy
A policy for allocating limited resources across breadth and depth when evaluating moral decisions.
Idealized reference
Unbounded Moral Agent
A hypothetical oracle that represents all morally relevant entities and reasons exhaustively over their interactions.
Table 1: Conceptual summary of the Bounded Morality framework.
Figure 1: At fixed budget B and distribution P , policies ρold and ρnew map world states to structural allocations (G,H) . The plot highlights a representative state ω0 where the improved policy selects a different breadth–depth trade-off (G,H)=ρnew(ω0,B) on the fixed-budget frontier, yielding lower state-dependent regret R(ω0;ρnew,B) than R(ω0;ρold,B) . Dashed lines indicate expected regret Eω[R(ω;ρ,B)] (here shown for a uniform ω ).
Theory
Breadth
Depth
Structural Commitment
Typical Regime
Act Utilitarianism
High
Mod.
Additive sum ∑tγtU(s(t))
Large, weakly coupled systems
Deontology
Mod.
Mod.
Hard constraints on AH(G)
Tight rules; limited scope
Contractualism
Low
High
Worst-case minv∑tγtUv(s(t))
Few parties; deep objections
Virtue Ethics
Very Low
0
Amortized policy πθ
Real-time action
Care Ethics
Local
Mod.
Relational (edge-weighted) terms
Dense local dependence
Table 2: Ethical theories as caricatured families of bounded moral strategies. Each theory occupies a characteristic region of the breadth–depth frontier and imposes a distinct structural commitment on evaluation or admissibility.
Figure 2: Budget contours ( B ) induce Pareto-efficient breadth–depth frontiers under a canonical cost model. Ethical theories are depicted as strategy families with characteristic scaling tendencies: utilitarianism trades budget for breadth, contractualism for depth, care ethics for local breadth and moderate depth, while deontology and virtue ethics remain near low-depth regimes. Regions are conceptual caricatures.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Symbol / Term
Name
Definition / Role
G⋆=(V⋆,E⋆)
Moral interaction graph
Full graph of morally relevant entities and their local influence relations.
v∈V⋆
Moral entity
Node whose state contributes directly to moral welfare.
Sv
Local state space
State space associated with entity v .
A
Action set
Finite set of atomic interventions available at each time step.
AH=AH
Action sequences
Length- H intervention sequences (temporally extended control policies).
F
Moral dynamics
Local graph-structured state update rule governing trajectory evolution.
Appendix
Table 3: Comprehensive glossary of formal objects used in the Bounded Morality framework. This table consolidates the mathematical definitions introduced in Section 3 for reference.
H
a(0)
a(E)
a(EC)
2
-0.765
-0.297
-0.088
10
-1.304
-0.581
-0.715
∞
-1.348
-0.634
-0.955
Appendix
Table 4: Finite-horizon value M^G⋆,H(a,ω) and infinite-horizon value M⋆(a,ω) (approximated using rollout to 2000 steps). Bold indicates the maximizer.
Figure 3: Ground-truth moral interaction graph G⋆ (left) and a coarse abstraction G (right) induced by aggregation map πV with X={E1,E2,C} and Y={M1,M2} .
This paper provides an algebraic exposition of the theory of dyadic morality (TDM), a psychological model of moral judgment grounded in a simple two-node template: an intentional agent causing harm to a vulnerable patient. We formalize TDM using structural causal modeling (SCM) notation and identify three psychological operators (typecasting operator, completion operator, and valence-dependent inference mechanism) that extend standard SCM to capture how people compute moral judgments under constraints. We address scalability challenges arising from TDM's dyadic limitation, showing how moral cognition compresses multi-node scenarios through node collapse and sequential processing. Drawing on this algebraic framework, we demonstrate concrete applications to AI policy design: detecting conflicting obligations, structuring helpfulness policies to preserve user agency, and designing post-failure communication as causal interventions. Finally, we recommend scoped, contextual measurement of mind perception over universal averaging to operationalize the theory empirically. This algebraic formalization enables neurosymbolic AI systems to compute morality in a way that is both mathematically rigorous and faithful to human moral cognition.
Most work in computational ethics treats annotator disagreement on moral content as noise to be voted away, collapsed into majority vote or the more permissive any-annotator rule the moment a single annotator flags an item. We argue this uncertainty should instead be modeled and learned from. We introduce Moral Entropy, a Bayesian framework that keeps a full posterior over the true label and decomposes its entropy into aleatoric uncertainty (irreducible disagreement about the moral content) and epistemic uncertainty (from insufficient or noisy annotation) -- and lets any heuristic consensus rule be audited against a calibrated ground truth via entropy methods such as cross-entropy/KL, Brier score, and expected calibration error. Across three corpora and fifteen discourse domains, auditing the standard aggregation rules against this posterior reveals bias that no current pipeline reports: the any-annotator rule disagrees with the calibrated posterior on roughly 30% of items -- pooled, almost entirely false positives, though the errors invert at the foundation level (19.9%/38.9% mean FPR/FNR on MFTC) -- while the stricter majority and two-vote rules miss 63-83% of true positives.
Autonomous agents operating in shared environments must make decisions that affect multiple individuals with potentially conflicting interests. We propose a formal framework for moral decision-making grounded in Scanlon's contractualism, an ethical theory that evaluates the permissibility of actions in terms of principles that no one could reasonably reject. To operationalise contractualist reasoning, we use ASPIC+, a structured argumentation framework, extended with value-based filtering to model how each agent's values determine which reasons are morally relevant in the first place. The result is a Contractualist Argumentation Framework in which agents' reasons are formally represented, compared, and evaluated through argumentation semantics. We illustrate the approach through a worked example in a domestic setting and discuss its relation to existing value-based argumentation approaches.
Luis Marcos-Vidal, Giulio Antonio Abbo, Tony Belpaeme
Hospital del Mar Research Institute, Barcelona, Spain · IDLab-AIRO, Ghent University – imec, Ghent, Belgium