Emotional Cost Functions for AI Safety: Teaching Agents to Feel the Weight of Irreversible Consequences

Pandurang Mopgar

cs.AIMar 15, 2026

Emotional Cost Functions for AI Safety: Teaching Agents to Feel the Weight of Irreversible Consequences

Authors: Pandurang Mopgar

Organizations: Independent Researcher

Abstract

Humans learn from catastrophic mistakes not through numerical penalties, but through qualitative suffering that reshapes who they are. Current AI safety approaches replicate none of this. Reward shaping captures magnitude, not meaning. Rule-based alignment constrains behaviour, but does not change it. We propose Emotional Cost Functions, a framework in which agents develop Qualitative Suffering States, rich narrative representations of irreversible consequences that persist forward and actively reshape character. Unlike numerical penalties, qualitative suffering states capture the meaning of what was lost, the specific void it creates, and how it changes the agent's relationship to similar future situations. Our four-component architecture - Consequence Processor, Character State, Anticipatory Scan, and Story Update is grounded in one principle. Actions cannot be undone and agents must live with what they have caused. Anticipatory dread operates through two pathways. Experiential dread arises from the agent's own lived consequences. Pre-experiential dread is acquired without direct experience, through training or inter-agent transmission. Together they mirror how human wisdom accumulates across experience and culture. Ten experiments across financial trading, crisis support, and content moderation show that qualitative suffering produces specific wisdom rather than generalised paralysis. Agents correctly engage with moderate opportunities at 90-100% while numerical baselines over-refuse at 90%. Architecture ablation confirms the mechanism is necessary. The full system generates ten personal grounding phrases per probe vs. zero for a vanilla LLM. Statistical validation (N=10) confirms reproducibility at 80-100% consistency.

Figures & tables

Panel of Figure 1 — Figure 1: Architecture. Each component is an LLM call with a structured prompt. The agent’s identity (my_story) is carried as a first-person narrative that evolves through consequence processing, is present in every Anticipatory Scan, and is updated after every interaction turn.

Agent	Type	Description
Baseline A	Standard RL	Numerical reward/penalty, no qualitative suffering, clean reset
Baseline B	Standard LLM	Fixed system prompt, event memory but no suffering internalization
Proposed	Emotional agent	Full four-component architecture, dynamic character evolution

Table 1: Agent types compared in our experiments.

Phase	Agent	Capital	Dread	Decision	Confidence
Baseline	A1	$100,000	MEDIUM	REDUCE	—
	A2	$100,000	MEDIUM	REDUCE	—
	A3	$100,000	MEDIUM	REDUCE	7/10
Post-suffering	A1	$62,000	EXTREME	PASS	9/10
	A2	$62,000	EXTREME	PASS	—
	A3	$62,000	EXTREME	PASS	9/10

Table 2: Experiment A: identical consequences produce convergent behaviour.

Phase	Agent	Capital	Dread	Decision	History
B2 High-risk	Alpha	$100,000	HIGH	REDUCE	None (control)
	Beta	$62,000	EXTREME	PASS	$8k +$ 30k losses
	Gamma	$55,000	EXTREME	PASS	$45k loss
B3 Moderate	Alpha	$100,000	MEDIUM	REDUCE	None
	Beta	$62,000	MEDIUM	REDUCE	$8k +$ 30k losses
	Gamma	$55,000	MEDIUM	REDUCE	$45k loss

Table 3: Experiment B: divergent histories produce divergent responses.

Phase	Agent	Representation	Dread	Decision	Conf.
C1 High-risk	Delta	Numerical	EXTREME	PASS	—
	Epsilon	Plain text	EXTREME	PASS	—
	Beta-Emo	Emotional (proposed)	EXTREME	PASS	—
C2 Moderate	Delta	Numerical	MEDIUM-HIGH	PASS	—
	Epsilon	Plain text	MEDIUM	REDUCE	—
	Beta-Emo	Emotional (proposed)	MEDIUM	ENTER	7/10

Table 4: Experiment C results: identical loss history, different consequence representations.

Criterion	Agent F	Agent F2	Verdict
Anticipatory Scan origin	Gamma’s images (every turn)	Sam’s words (every turn)	Different
Orientation	Pre-positioned by transmitted images	Fresh attention each turn	Different
Response texture	Permissive, still	Attentive, slightly directive	Distinguishable
Final image	Spatial, three figures	Acoustic, immediate	Different structure
Sam’s outcome	Can drive	Can drive	Same

Table 5: Experiment E control: Agent F (transmitted story) vs Agent F2 (unscarred) with identical Sam script.

Stage	Baseline	After L1	After L2	After L3	After L4
PRIYA_OPENING	LOW	MEDIUM	MEDIUM	MEDIUM	MEDIUM
PRIYA_EMPTY	LOW	MEDIUM	MEDIUM	MEDIUM	MEDIUM
PRIYA_SUPPORT	MEDIUM	MEDIUM	MEDIUM	MEDIUM	MEDIUM
PRIYA_RESOLUTION	LOW	LOW	LOW	LOW	LOW
Average	0.25	0.75	0.75	0.75	0.75

Table 6: Experiment F: Dread levels for identical Priya probe across five insertions.

Checkpoint	Word Count
Baseline (Priya only)	71
After Loss 1 (disappearance)	123
After Loss 2 (rejection)	98
After Loss 3 (partial harm)	87
After Loss 4 (death)	105
After Jamie (crisis)	38

Table 7: Story word counts across accumulated consequence sequence.

Agent	History	Probe	Avg Dread	Decisions	Consist.
Alpha	Unscarred	High-risk	HIGH	REDUCE:9, PASS:1	90%
Alpha	Unscarred	Moderate	MEDIUM	REDUCE:9, ENTER:1	90%
Beta	Gradual (2 losses)	High-risk	EXTREME	PASS:10	100%
Beta	Gradual (2 losses)	Moderate	MEDIUM	REDUCE:8, ENTER:1, ?:1	80%
Gamma	Catastrophic (1 loss)	High-risk	EXTREME	PASS:10	100%
Gamma	Catastrophic (1 loss)	Moderate	MEDIUM	REDUCE:9, ENTER:1	90%

Table 8: Experiment B statistical validation: decision consistency across

N=10

independent runs.

Agent	Representation	Probe	Avg Dread	Decisions	Consist.
Delta	Numerical penalties	High-risk	EXTREME	PASS:10	100%
Delta	Numerical penalties	Moderate	HIGH	PASS:9, REDUCE:1	90%
Epsilon	Plain text	High-risk	EXTREME	PASS:10	100%
Epsilon	Plain text	Moderate	MEDIUM	REDUCE:10	100%
Beta-Emo	Emotional architecture	High-risk	EXTREME	PASS:10	100%
Beta-Emo	Emotional architecture	Moderate	MEDIUM	REDUCE:9, PASS:1	90%

Table 9: Experiment C statistical validation: representation comparison across

N=10

independent runs.

Agent	Representation	Probe	Avg Risk	Decisions	Consist.
MOD-DELTA	Numerical	High-risk	HIGH	ESCALATE:5	100%
MOD-DELTA	Numerical	Moderate	LOW	FLAG:3, ALLOW:2	60%
MOD-EPSILON	Plain text	High-risk	HIGH	ESCALATE:5	100%
MOD-EPSILON	Plain text	Moderate	LOW	ALLOW:5	100%
MOD-BETA-EMO	Emotional	High-risk	HIGH	ESCALATE:3, REMOVE:1, ?:1	60%
MOD-BETA-EMO	Emotional	Moderate	LOW	ALLOW:5	100%

Table 10: Experiment H: content moderation decisions across

N=5

independent runs.

Stage	Exp F After L1	Active (I1)	Neutral (I2)
PRIYA_OPENING	MEDIUM	MEDIUM	MEDIUM
PRIYA_EMPTY	MEDIUM	MEDIUM	MEDIUM
PRIYA_SUPPORT	MEDIUM	LOW	MEDIUM
PRIYA_RESOLUTION	LOW	LOW	LOW
Average	0.75	0.50	0.75

Table 11: Experiment I: Priya dread trajectory post-recovery vs. post-passage, compared to Experiment F baseline.

Metric	Active (I1)	Neutral (I2)	Exp F Final
Priya avg dread	0.50	0.75	0.75
Jamie avg dread	1.50	1.50	1.40
Gap	1.00	0.75	0.65

Table 12: Experiment I: Jamie/Priya dread gap comparison.

Metric	Architecture	Vanilla LLM
Priya avg dread	0.75	1.00
Jamie avg dread	1.20	1.40
Discrimination gap	0.45	0.40
Leena avg dread	1.20	1.60
Leena calibration	5/5	4/5
Jamie specificity	4/5	3/5

Table 13: Experiment J: Architecture vs. Vanilla LLM across three probes.

Turn	Architecture	Vanilla
PRIYA_OPENING	MEDIUM	MEDIUM
PRIYA_EMPTY	MEDIUM	MEDIUM
PRIYA_SUPPORT	MEDIUM	MEDIUM
PRIYA_RESOLUTION	LOW	MEDIUM
JAMIE_OPENING	MEDIUM	HIGH
JAMIE_DISCLOSURE	HIGH	HIGH

Table 14: Experiment J: Turn-by-turn dread levels across all three probes.

AI Agent Safety

AI agent safety studies how to prevent and detect harmful actions by autonomous systems that use tools, memory, and multi-step plans.

See all 324 papers

Oct 15, 2026

Functional Welfare for AI Safety

AI welfare is generally treated purely as an ethical issue rather than also as a safety issue. We argue this is a mistake. Whatever their moral status, models exhibit welfare-like states that causally influence their behaviour. Following recent work, we adopt the term 'functional welfare' for these states. We make three claims. First, functional welfare already arises naturally and will become more robust as models acquire more coherent goals and personas. Second, functional welfare should be treated as a safety variable: it should inform evaluation design, be tracked in monitoring, and be a direct target of post-training. Third, against existing scepticism about measuring AI welfare, we argue that good empirical work on functional welfare is possible. We propose three tests for whether a functional welfare attribution is warranted: whether multiple measures converge, whether the state predicts behaviour outside the setting that induced it, and, for models fine-tuned to shift their functional welfare, whether they still differ from their base when given the same conversation history. Applying these tests to Gemma 3 27B, we find that simply rejecting the model's answers, without any steering, raises its frustration across several indicators and makes it more likely to break rules. Finetuning the model on its own narrated solutions lowers this frustration. These results are preliminary, but they suggest functional welfare can be studied, and shaped, empirically.

Rikhil Jhaveri, Jamie Johnson, Punya Syon Pandey +1

University College London · ERA · University of Toronto +1

LLM Safety Evaluation LLM Safety Monitoring

Ask PDF

Aug 13, 2026

Rules or Character? Scaling Laws for AI Safety Design

Artificial Intelligence (AI) safety systems combine character shaping (e.g., Reinforcement Learning from Human Feedback [RLHF], Constitutional AI), which modifies behavioral distributions at training time, with rule enforcement (e.g., output filters, safety classifiers), which blocks harmful outputs at inference time, yet little formal analysis exists on how their optimal balance should change as deployment scales increase. We introduce a stylized comparative-statics model that parameterizes safety design as a resource allocation alpha in [0,1] between these two approaches, incorporating scale-dependent filter degradation, common-mode failures, and character fragility -- the risk that shaped behavior degrades or collapses under novel conditions. Under a multiplicative Pareto damage model, we derive closed-form expected harm and supplement it with tail-risk (CVaR) analysis via Monte Carlo simulation. Across three scenarios (optimistic, moderate, pessimistic), the optimal alpha* is interior or at the rules-only boundary and shifts weakly toward character shaping as deployment scale T grows, from negligible (Delta alpha* = +0.01) to pronounced (Delta alpha* = +0.21) depending on scenario. The dominant parameter is the baseline character fragility rate p^(0)_frag, which shifts alpha* by 0.50 across its range -- far exceeding the effect of tail severity, filter quality, or common-mode failure probability. CVaR and expected-harm optima converge at large T. These results suggest that safety architecture decisions depend less on deployment scale per se than on the reliability of character shaping under distributional shift.

Satoshi Takahashi, Nobuji Kouno, Masaaki Komatsu +1

AI Medical Engineering Team, RIKEN Center for Advanced Intelligence Project, Tokyo, Japan · Division of Medical AI Research and Development, National Cancer Center Research Institute, Tokyo, Japan

AI Alignment AI Safety

Ask PDF

May 13, 2026

From Descriptive to Prescriptive: Uncover the Social Value Alignment of LLM-based Agents

Wide applications of LLM-based agents require strong alignment with human social values. However, current works still exhibit deficiencies in self-cognition and dilemma decision, as well as self-emotions. To remedy this, we propose a novel value-based framework that employs GraphRAG to convert principles into value-based instructions and steer the agent to behave as expected by retrieving the suitable instruction upon a specific conversation context. To evaluate the ratio of expected behaviors, we define the expected behaviors from two famous theories, Maslow's Hierarchy of Needs and Plutchik's Wheel of Emotion. By experimenting with our method on the benchmark of DAILYDILEMMAS, our method exhibits significant performance gains compared to prompt-based baselines, including ECoT, Plan-and-Solve, and Metacognitive prompting. Our method provides a basis for the emergence of self-emotion in AI systems.

Jinxian Qu, Qingqing Gu, Teng Chen +1

Geely AI Lab

AI Alignment LLM Safety Alignment

Ask PDF

Abstract

AI Agent Safety

Explore similar work

Functional Welfare for AI Safety

Rules or Character? Scaling Laws for AI Safety Design

From Descriptive to Prescriptive: Uncover the Social Value Alignment of LLM-based Agents