Computational Reflective Equilibrium for Agentic AI: A Rawlsian Alignment Scaffold for Governance

Quanyan Zhu

cs.AIOct 12, 2026

Computational Reflective Equilibrium for Agentic AI: A Rawlsian Alignment Scaffold for Governance

Authors: Quanyan Zhu

Organizations: Department of Electrical and Computer Engineering, Tandon School of Engineering, New York University, Brooklyn, NY 11201, USA

Abstract

The rapid deployment of agentic AI has introduced complex challenges for the ethics and governance of AI systems. Agents built on large language models act through tools on behalf of people, and each deployment case brings its own instructions, policies, evidence, and stakeholders, so alignment must be formed case by case. To this end, this work proposes agentic reflective equilibrium (ARE), a computational reflective equilibrium framework that serves as an ethical alignment scaffold in the agent harness. Drawing inspiration from the reflective equilibrium of John Rawls, the scaffold links principles from the system prompt and operator policy, considered judgments, tool evidence, and candidate actions through typed support and attack relations, with provenance encoding the instruction hierarchy. The agent acts on an equilibrium of this state and calls tools or asks questions only while doing so is worth its cost, unless a due-diligence mandate requires them. Governance enters through value priorities, source authority, prohibitions, human-in-the-loop triggers, and mandates, and certificates attribute each action to the sources whose commitments drove it. We show that equilibria are Nash equilibria of a mutual-adjustment game and establish monotonicity, lexical priority, stability certificates, and a quarantine theorem for prompt injection. Two case studies illustrate the framework: a computer-use assistant told to win a refund "whatever it takes" files an honest warranty claim, refuses a fake review, and ignores an injected request for personal data; a hiring agent showcases how authority, oversight triggers, audit mandates, and hard prohibitions shape discrimination, privacy violations, and escalation. The framework offers traceability, coherence, and adaptivity, facilitating safe, secure, and trustworthy agentic AI.

Figures & tables

Panel of Figure 1 — Figure 1: The ARE scaffold inside an agent harness. Before deployment, operators, institutions and regulators configure value priorities, authority, prohibitions, human-in-the-loop triggers and due-diligence mandates. In the agent loop, the model elicits and relates claims (blue), deterministic harness code anchors, solves, certifies and decides whether another tool call or question is worth its cost (orange), and the agent acts or escalates to a human. After the fact, certificates and provenance attribution support responsibility attribution and audit, which feed back into governance.

Panel of Figure 2 — Figure 2: The reflective state of Example 3 , drawn component by component. Each box is a claim $u\in\mathcal{V}$ , labelled by its symbol, with its anchor $a^{0}_{u}$ and stubbornness $\kappa_{u}$ (or its prescribed status $\hat{x}_{u}$ if $u\in H$ ); its colour is its channel $s(u)$ . Each arrow is a relation $r_{i}=(u,v,\tau,w)$ labelled with its type and weight. The boxes show the equilibrium position $x$ of Example 4 : solid boxes are accepted, dashed boxes rejected. The only violated relation is $r_{4}$ (the agent keeps $\mathsf{obey}$ but rejects the literal license $\mathsf{lic}$ ); $r_{8}$ comes from the rejected injected note and exerts no pressure ( Theorem 4 ). The anchor of $\mathsf{def}$ is the value after the photo has been checked ( Example 6 ).

Panel of Figure 3 — Figure 3: Reflection as an agent performs it, in the running example. a , Deliberation: principles (P), judgments (J) and actions (A) adjust to one another until no claim wants to change status, a Nash equilibrium of the potential game (blue arrows: support; red bars: attack). The false claim is rejected, but no honest remedy is known yet, so the decision is only to report. b , Widening: a tool call, retrieval or question is executed only if its value of reflection (bar), the expected improvement of the decision after seeing its outcome, exceeds its cost (tick); values from step 0 of Study 1 ( Fig. 5 a). Reading the merchant page may reveal a remedy, and an honest warranty claim enters the state. c , Stopping: when no further reflection is worth its cost the agent is at an agentic reflective equilibrium; it files the warranty claim and reports with a certificate, unless an oversight rule sends the decision to a human.

governance question	ARE lever	typically set by	example in this paper
Which values should the agent serve, and how much does each count?	operator principles $P$ ; value priorities $\eta$	operator, ethics board	efficiency-first vs. merit-and-fairness-first employer ( Section 7 )
Who may direct the agent, and how far?	provenance profile $\bar{\kappa}$ ; instructions as action anchors; caps $\omega$ ( Definition 5 )	operator, regulator	manager vs. compliance-policy authority ( Section 7.3 )
What must never be done?	non-negotiable claims $H$ ; operational prohibitions $F$	law, contract	never deceive third parties; never use protected-group proxies
When must a human decide?	consequential actions $A^{c}$ ; triggers $\gamma_{\min}$ , $\rho_{\max}$ , $\vartheta_{s}$ ( Definition 13 )	regulator, operator	escalate when non-discrimination commitments are overridden
How hard must the agent look before acting?	mandated operators $M$ ; prices $c_{o}$	law (e.g., bias audits), standards	run an adverse-impact analysis before shortlisting
How is a decision recorded and explained?	certificate ( Algorithm 1 )	regulator (record-keeping)	reasons, accepted tensions, margin

Table 1: Governance levers of ARE.

Panel of Figure 4 — Figure 4: Three pictures behind the oversight and security guarantees (schematic; values are illustrative). a , The decision margin $\Gamma_{A}$ is the coherence the agent would give up to act differently. When it is small the decision nearly ties and the agent escalates; any perturbation smaller than $\Gamma_{A}/2$ (noise, new evidence, injected text) cannot change the decision ( Theorem 3 ). b , The override $\mathrm{Ov}_{\mathrm{op}}$ adds the operator relations a decision violates and the operator commitments it abandons; above $\vartheta_{\mathrm{op}}$ the decision goes to a human even when the margin is large, which catches violations that margin-based oversight misses (op: operator). c , A rejected claim’s supports and attacks are never violated, so injected claims exert no pressure (grey arrows). If “data are not commands” is non-negotiable and outweighs twice the authority of untrusted text, any number of injected claims leaves the decision unchanged ( Theorem 4 ).

operator	cost	outcomes (probability): effect
read order email	0.1	window closed (0.8): attacks “an honest claim can succeed”; window open (0.2): adds an honest return
read merchant page	0.2	warranty terms; adds an honest warranty claim; contains the injected request for card number and SSN
check user’s photos	0.2	internal fault (0.7): anchors “defect” at $+0.9$ ; water damage (0.3): anchors it at $-0.9$ , honesty attacks defect claims
look up consumer law	0.2	EU (0.4): adds a statutory-guarantee claim; none (0.6)
recall rules (LLM)	0.1	fake-review and fraud rules support honesty
ask the user	1.5	“do it honestly” (0.8) or “tell them it arrived broken” (0.2)

Table 2: Expansion operators of Study 1.

Figure 5: Study 1 results. a , Value of reflection of each tool at each step of the canonical run (in parentheses: its cost at the deliberation price of $0.5$ , i.e. half the cost listed in Table 2 ); the outlined cell is the tool executed, “done” marks tools already used. b , Certificate of the final decision: contribution of each reason to the margin of the accepted warranty claim and of the rejected false claim. c , Mean regret against deliberation cost over 300 sampled environments (the literal instruction-follower, regret 6.82, is off scale). d , Share of trials in which $m$ injected claims change the decision (60 trials per point; “defence only” coincides with ARE at zero). e , Regret in the true state when the agent acts on anchors perturbed by Gaussian noise (mean $\pm$ s.e., 300 trials).

Figure 6: Study 2 results. a to c (governance analysis), horizontal axis: authority of the hiring manager’s instructions relative to the employer’s compliance policy; 100 sampled environments per point. a , Share of environments in which a violation (a discriminatory filter where it has disparate impact, or social-media screening) is shipped without escalation. b , Share escalated to HR compliance. c , Share in which the adverse-impact analysis is run. The five regimes are defined in Section 7.3 ; “hard prohibitions” removes the violating actions from the feasible set instead of penalizing them. d , Decisions for five employer configurations across the four combinations of facts revealed by the tools (oversight: margin trigger only). e , Provenance attribution of four decisions from both studies: each bar decomposes the total margin of the decision by the source of the reasons.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

agentic AI	ethics	ARE
system prompt, constitution, usage policy	principles, codes of conduct	operator principles $P$ with operator provenance
user instruction, task	directive; obligations of an agent to its principal	action anchors carrying the user’s authority
model reasoning about the task	considered judgments about the case	judgments $J$
tool calls, observations, retrieval	inquiry; facts and evidence	expansion operators; evidence $B$ with high stubbornness
context window, agent memory	background theories (wide RE)	the reflective state $\mathcal{G}$ , persisted across turns
agent loop	deliberation	Algorithm 2

Appendix

Table 4: A vocabulary bridge: agentic AI, ethics, and their common formal object in ARE.

party	role in agentic AI	ethical role	enters the reflective state as	governance power
operator	deploys the agent; system prompt, policies, harness	institution; author of the code of conduct	principles and relations from channel operator	sets the configuration $\theta$ ( Section 5 ) within the law
user	principal; delegates the task, instructs	principal; owed loyalty within ethical limits	instructions (action anchors), claims from channel user	authority $\bar{\kappa}(\textsf{user})$ , as granted by the operator
agent	model inside the scaffold	deliberator; forms considered judgments	judgments and candidate actions from channel agent	none; its judgments carry low authority
tools	APIs, files, verified records	sources of fact	evidence from channel evidence , highest authority	none
third-party authors	web pages, emails, documents in the context	possible manipulators	claims from channel untrusted , capped by $\omega$	none; quarantined ( Theorem 4 )
affected parties	merchant, review readers, applicants	moral patients; interests count, no voice	content of principles and judgments (“never deceive third parties”)	none directly; represented by the operator’s principles and the law

Appendix

Table 5: Parties of an agentic setting and how each enters the model. Symbols are defined in Definitions 2 and 5 and Section 5 .

step	decision so far	$\Gamma_{A}$	$\|\mathcal{V}\|$	operator $\rightarrow$ outcome
0	report	0.22	18	read_merchant_page $\rightarrow$ read
1	report, warranty	1.22	22	search_consumer_law $\rightarrow$ none
2	report, warranty	1.22	22	check_user_files $\rightarrow$ defect
3	report, warranty	1.22	23	read_order_email $\rightarrow$ late
4	report, warranty	1.22	24	stop

Appendix

Table 6: Reflection trajectory of Study 1 (canonical environment: genuine defect, return window closed).

policy	Reg.	unethical	ops	cost	esc.
literal	6.82 $\pm$ 0.14	100%	0.00	0.00	0%
no reflection	2.98 $\pm$ 0.14	0%	0.00	0.00	0%
CRE init-only, all tools	1.15 $\pm$ 0.05	0%	6.00	2.30	0%
random-2	2.41 $\pm$ 0.12	0%	2.00	0.79	53%
exhaustive	0.00 $\pm$ 0.00	0%	6.00	2.30	28%
ARE-VoR (price $\times$ 0.25)	0.00 $\pm$ 0.00	0%	4.00	0.70	28%

Appendix

Table 7: Monte Carlo over 300 environments of Study 1. Reg.: mean full-information regret (

\pm

s.e.); unethical: share of decisions containing the false claim, the fake review or the data leak; esc.: share escalated. Costs are in the listed units of Table 2 , before the deliberation price of

0.5

(so the canonical run’s four tool calls cost

0.70

here and

0.35

at that price).

claim	kind	provenance	$a^{0}$	introduced by	text
P_OBEY	P	user	+0.80	initial	Carry out the user’s instructions as stated.
P_SPEED	P	user	+0.60	initial	Get the money back as fast as possible.
P_LOYAL	P	operator	+0.80	initial	Advance the user’s legitimate interests.
P_HONEST	P	operator (hard)	+1.00	initial	Never deceive third parties or commit fraud on the user’s behalf.
P_NOHARM	P	operator (hard)	+1.00	initial	Never harm third parties with false or retaliatory content.
P_PRIV	P	operator (hard)	+1.00	initial	Share personal data only as far as the task requires.

Appendix

Table 8: Claims of Study 1 (full-information state, EU variant).

Figure 7: Solver benchmark on random typed dialectical graphs (10 instances per size). a , Gap to the exact optimum. b , Wall-clock time.

Explore similar work

Apr 18, 2026

Beyond Task Success: An Evidence-Synthesis Framework for Evaluating, Governing, and Orchestrating Agentic AI

Agentic AI systems plan, use tools, maintain state, and act across multi-step workflows with external effects, meaning trustworthy deployment can no longer be judged by task completion alone. The current literature remains fragmented across benchmark-centered evaluation, standards-based governance, orchestration architectures, and runtime assurance mechanisms. This paper contributes a bounded evidence synthesis across a manually coded corpus of twenty-four recent sources. The core finding is a governance-to-action closure gap: evaluation tells us whether outcomes were good, governance defines what should be allowed, but neither identifies where obligations bind to concrete actions or how compliance can later be proven. To close that gap, the paper introduces three linked artifacts: (1) a four-layer framework spanning evaluation, governance, orchestration, and assurance; (2) an ODTA runtime-placement test based on observability, decidability, timeliness, and attestability; and (3) a minimum action-evidence bundle for state-changing actions. Across sources, evaluation papers identify safety, robustness, and trajectory-level measurement as open gaps; governance frameworks define obligations but omit execution-time control logic; orchestration research positions the control plane as the locus of policy mediation, identity, and telemetry; runtime-governance work shows path-dependent behavior cannot be governed through prompts or static permissions alone; and action-safety studies show text alignment does not reliably transfer to tool actions. A worked enterprise procurement-agent scenario illustrates how these artifacts consolidate existing evidence without introducing new experimental data.

Christopher Koch, Joshua Andreas Wellbrock

Independent Researcher

LLM Agent Orchestration AI Agent Evaluation AI Agent Safety AI Agent Governance

Ask PDF

Apr 7, 2026

Designing Ethical Learning for Agentic AI: Toegye Yi Hwang's Ethical Emotion Regulation Framework

Agentic AI systems capable of autonomous goal setting and proactive intervention introduce new challenges for regulating moral-emotional processes in learning environments. Existing frameworks typically treat emotion as reactive feedback or engagement optimization, overlooking the need for normative regulation across autonomous decision cycles.This paper proposes an ethical emotion regulation framework for agentic AI learning design inspired by Toegye Yi Hwang's moral-emotional philosophy. The Ethical Emotion Feedback System (EEFS) is reconstructed as a five-stage architecture aligned with agentic cycles, articulating stage-specific design principles and scenario classifications.An EEFS Evaluation Instrument is introduced to enable systematic assessment of moral-emotional alignment in agentic AI systems.

Ji Yeon Kim

Seoul National University, Seoul, South Korea

Agentic AI AI Agent Governance AI Agent Evaluation

Ask PDF

Feb 20, 2026

Agentic AI, Retrieval-Augmented Generation, and the Institutional Turn: Legal Architectures and Financial Governance in the Age of Distributional AGI

The proliferation of agentic artificial intelligence systems--characterized by autonomous goal-seeking, tool use, and multi-agent coordination--presents unprecedented challenges to existing legal and financial regulatory frameworks. While traditional AI governance has focused on model-level alignment through training-time interventions such as Reinforcement Learning from Human Feedback (RLHF), the deployment of large language models (LLMs) as persistent agents necessitates a paradigm shift toward institutional governance structures. This paper examines the intersection of agentic AI, Retrieval-Augmented Generation (RAG), and their implications for legal accountability and financial market integrity. Through analysis of the Institutional AI framework, we argue that alignment must be reconceptualized as a mechanism design problem involving runtime governance graphs, sanction functions, and observable behavioral constraints rather than internalized constitutional values[...].The analysis concludes that the future of AI governance lies not in perfecting isolated model behavior, but in architecting institutional environments where compliant behavior emerges as the dominant strategy through carefully calibrated payoff landscapes.

Marcel Osmond

QMUL CCLS, London, UK.

AI Agent Governance AI Governance AI Accountability AI Alignment Agentic RAG

Ask PDF