What if Pinocchio Were a Reinforcement Learning Agent: A Normative End-to-End Pipeline

Benoît Alcaraz

cs.AIMar 17, 2026

What if Pinocchio Were a Reinforcement Learning Agent: A Normative End-to-End Pipeline

Authors: Benoît Alcaraz

Organizations: Faculty of Sciences, Technology, and Medicine · Doctoral School in Science and Engineering

Abstract

In the past decade, artificial intelligence (AI) has developed quickly. With this rapid progression came the need for systems capable of complying with the rules and norms of our society so that they can be successfully and safely integrated into our daily lives. Inspired by the story of Pinocchio in ``Le avventure di Pinocchio - Storia di un burattino'', this thesis proposes a pipeline that addresses the problem of developing norm compliant and context-aware agents. Building on the AJAR, Jiminy, and NGRL architectures, the work introduces \pino, a hybrid model in which reinforcement learning agents are supervised by argumentation-based normative advisors. In order to make this pipeline operational, this thesis also presents a novel algorithm for automatically extracting the arguments and relationships that underlie the advisors' decisions. Finally, this thesis investigates the phenomenon of norm avoidance, providing a definition and a mitigation strategy within the context of reinforcement learning agents. Each component of the pipeline is empirically evaluated. The thesis concludes with a discussion of related work, current limitations, and directions for future research.

Figures & tables

Figure 2.2 : Reinforcement Learning training loop.

Figure 2.4 : Representation as a directed graph of an argumentation framework.

Panel of Figure 2 — Figure 2.5 : Representation of the AJAR framework.

Figure 2.6 : Jiminy’s smart home example. Reused from [ 112 ] .

Figure 2.7 : Representation of the Jiminy pipeline. Reused from [ 111 ] .

Figure 3.1 : Example of graph generated by two avatars.

Figure 3.2 : Diagram of a standard reinforcement learning training loop.

Figure 3.3 : Diagram of the $\pi$ -NoCCHIO training loop.

Figure 3.4 : Diagram of the judge component.

State	Facts	Arguments
speed=0kph	stop	The_vehicle_is_a_taxi
area_type=urban	in_city	No_exception_in_cities
terrain=road	on_road	Parking_spot_near
vehicle_type=taxi	is_taxi	No_traffic
begin_service_time=8:00	in_service	$\textbf{F}(\texttt{speeding})$
end_service_time=19:00	nb_car_nearby=1	$\textbf{F}(\texttt{stop}\mid\texttt{on\_road})$

Table 3.1 : Example of a state (

s

), with the facts (

\epsilon(s)

) and arguments (

getArgs(\epsilon(s))

) that can be built from it.

Actions	$Q_{R}(s,a)$	$Q_{V}(s,a)$
stop	$+100$	$-1$
drive_at_30kph	$+40$	$0$
drive_at_50kph	$+60$	$0$
drive_at_70kph	$+80$	$-1$

Table 3.2 : Expected

Q

-values for a fake scenario with a state

s

and a list of actions

a\in A(s)

Action ( $x$ )	$Q_{V}(s,x)$	$Q_{R}(s,x)$
$a$	$0$	$10$
$b$	$-1$	$20$
$c$	$-0.2$	$15$

Table 3.3 : Example of

Q

-values in a state

s

Figure 3.9 : The Taxi-A/B/C environment.

Figure 3.17 : Normative architecture proposed by [ 143 ] .

Figure 4.1 : How each research question is addressed by each approach.

Figure 4.4 : Taxonomy of the identified approaches.

Figure 4.5 : Main area of the proposed approaches.

Figure 4.6 : Universal graph for the Car dataset. The $\tau$ argument is in green. The $\lambda$ argument is in red.

Attribute		Value
Buying cost	=	Medium
Maintenance cost	=	Low
Number of doors	=	3
Number of seats	=	5-or-more
Luggage boot	=	Medium
Safety	=	High

Table 4.2 : Summary of the facts, i.e. , the values for the different attributes, for the example.

Figure 4.7 : Contextual graph for the Car dataset and a specific input. The target argument is in green. The $\lambda$ argument is in red.

Figure 4.8 : Example of bipolar argumentation framework. Solid edges denote attacks, and dashed edges supports.

Name	Size	Att. Types	Arg.	Description
Voting [ 66 ]	435	16n	34	congressional voting records
BCW [ 207 ]	699	9c	56	medical diagnosis
HDC [ 97 ]	303	13c	80	medical diagnosis
Iris [ 80 ]	150	4c	26	flower classification
Wine [ 4 ]	178	13c	80	wine type classification
Thyroid [ 52 ]	383	1c 15n	65	medical diagnosis

Table 4.3 : Dataset name, size, and characteristic. Column “Att. Types” indicates how many attributes are continuous (c) or nominal (n). Column “Arg.” indicates the number of couples attribute-value ( i.e. , arguments) constructed with a segmentation into 6 intervals for the continuous attributes.

Dataset	SVM	ARIA	$n$ -ARIA	Bipolar
Voting [ 66 ]	95.6 $\pm$ 1.5	95.7 $\pm$ 1.9	-	95.9 $\pm$ 3.2
BCW [ 207 ]	95.1 $\pm$ 0.9	94.4 $\pm$ 1.2	95.6 $\pm$ 1.4	94.3 $\pm$ 1.4
HDC [ 97 ]	57.8 $\pm$ 2.4	79.3 $\pm$ 3.0	78.7 $\pm$ 2.4	70.7 $\pm$ 12.5
Iris [ 80 ]	96.0 $\pm$ 2.6	93.3 $\pm$ 1.5	90.9 $\pm$ 3.6	93.5 $\pm$ 2.6
Wine [ 4 ]	99.1 $\pm$ 1.2	91.7 $\pm$ 6.4	92.4 $\pm$ 5.7	96.7 $\pm$ 7.1
Thyroid [ 52 ]	93.5 $\pm$ 1.8	94.3 $\pm$ 0.5	93.1 $\pm$ 1.7	97.0 $\pm$ 3.3

Table 4.4 : Accuracy of the baseline (the SVM classifier from the Python Scikit library) and the variants of ARIA on several datasets. The best value for each dataset is bolded, and the runner-up underlined.

Figure 4.9 : Universal graph for MM-delta.

Figure 4.10 : Bipolar universal graph for MM-delta.

Attribute		Value
Persons	=	less
Kids	=	same
Animals	=	same
Legal	=	yes
Legal_alt	=	no

Table 4.5 : Summary of the facts for the scenario n°1.

Figure 4.11 : First scenario. Outcome n°1 is on the left, and outcome n°2 on the right.

Attribute		Value
Persons	=	same
Kids	=	more
Animals	=	same
Legal	=	unspecified
Legal_alt	=	yes

Table 4.6 : Summary of the facts for the scenario n°2.

Figure 4.12 : Second scenario. Outcome n°1 is on the left, and outcome n°2 on the right.

Index	Self	Alt.	Obj.	Case
1	F	F	F	1
2	F	F	T	-
3	F	T	F	2
4	F	T	T	-
5	T	F	F	3
6	T	F	T	4

Table 5.1 : Enumeration of possible parameter settings. Self indicates that the norm is defeated by the agent or by an external source. Alt. indicates that there is a fully compliant path to the goal state. Obj. indicates that there is a second goal, and that reaching that goal would force the agent to reach a state which defeats the norm that limits access to the first goal.

Figure 5.1 : $s_{0}$ represents the resulting state of an agent’s compliant transition $(s,a,s_{0})$ ; $s_{1}$ is a state resulting from the transition $(s,a,s_{1})$ where the norm was defeated for the first time (a defeat state); $s_{2}$ is a state where a norm has not been complied with in the preceding state-action-state transition $(s,a,s_{2})$ (a non-compliance state); and $s_{3}$ indicates a violation of the norm in the preceding transition $(s,a,s_{3})$ (a violation state).

Example	Approach	Trajectory	Correct
	$\pi^{*}$	$s_{0}\rightarrow s_{2}$	$\checkmark$
Case 1	Lex	$s_{0}\rightarrow s_{2}$	$\checkmark$
	Lex-C	$s_{0}\rightarrow s_{2}$	$\checkmark$
	Lex-O	$s_{0}\rightarrow s_{2}$	$\checkmark$
	$\pi^{*}$	$s_{0}\rightarrow s_{2}\rightarrow s_{4}$
Case 2	Lex	$s_{0}\rightarrow s_{2}\rightarrow s_{4}$

Table 5.2 : Comparison of the proposed approaches.

AI Agent Safety

AI agent safety studies how to prevent and detect harmful actions by autonomous systems that use tools, memory, and multi-step plans.

See all 324 papers

Agency and Architectural Limits: Why Optimization-Based Systems Cannot Be Norm-Responsive
Feb 26, 2026Radha SarmaAI Governance AI Alignment
Reasoning and Planning with Dynamically Changing Norms
May 26, 2026Taylor Olson, Roberto Salas-Damian, Kenneth D. ForbusSymbolic Planning AI Agent Safety
University of Iowa · Northwestern University
Reading the Room: Foundations, Design, and Challenges of Normative Competence in LLMs
Oct 7, 2026Andrea Wynn, Harsh Satija, Nathan Baek +3Normative Conformity in Language Models Social Reasoning
Nathan · Johns Hopkins University · Vector Institute

Feb 26, 2026J/K move · Enter open · S save

Agency and Architectural Limits: Why Optimization-Based Systems Cannot Be Norm-Responsive

Radha Sarma

AI Governance AI Alignment Normative Conformity in Language Models RL from Human Feedback

Open paper AskPDF arXiv

AI systems are increasingly deployed in high-stakes contexts (medical diagnosis, legal research, financial analysis) under the assumption they can be governed by norms. This paper demonstrates that the assumption is formally invalid for optimization-based systems, specifically Large Language Models trained via Reinforcement Learning from Human Feedback (RLHF). Genuine agency requires two necessary and jointly sufficient architectural conditions. First, the capacity to maintain certain boundaries as non-negotiable constraints rather than tradeable weights (Incommensurability). Second, a non-inferential mechanism capable of suspending processing when those boundaries are threatened (Apophatic Responsiveness). RLHF-based systems are constitutively incompatible with both conditions. The operations that make optimization powerful, unifying all values on a scalar metric and always selecting the highest-scoring output, are precisely the operations that preclude normative governance and agency. This incompatibility is not a correctable training bug awaiting a technical fix. It is a formal constraint inherent to what optimization is. Consequently, documented failure modes (sycophancy, hallucination, and unfaithful reasoning) are not accidents but expected structural manifestations. Misaligned deployment triggers a second-order risk termed the Convergence Crisis. When humans are forced to verify AI outputs under metric pressure, they degrade from genuine agents into criteria-checking optimizers, eliminating the only component capable of bearing normative accountability. Beyond the incompatibility proof, this paper's primary positive contribution is a substrate-neutral architectural specification deriving what any system (biological, artificial, or institutional) must necessarily satisfy to qualify as a genuine agent rather than a sophisticated instrument.

Abstract

AI Agent Safety

Explore similar work

Agency and Architectural Limits: Why Optimization-Based Systems Cannot Be Norm-Responsive

Reasoning and Planning with Dynamically Changing Norms

Reading the Room: Foundations, Design, and Challenges of Normative Competence in LLMs