Functional Welfare for AI Safety

Rikhil Jhaveri; Jamie Johnson; Punya Syon Pandey; David Demitri Africa

cs.AIOct 15, 2026

Functional Welfare for AI Safety

Authors: Rikhil Jhaveri, Jamie Johnson, Punya Syon Pandey, David Demitri Africa

Organizations: University College London · ERA · University of Toronto · University of Cambridge

Abstract

AI welfare is generally treated purely as an ethical issue rather than also as a safety issue. We argue this is a mistake. Whatever their moral status, models exhibit welfare-like states that causally influence their behaviour. Following recent work, we adopt the term 'functional welfare' for these states. We make three claims. First, functional welfare already arises naturally and will become more robust as models acquire more coherent goals and personas. Second, functional welfare should be treated as a safety variable: it should inform evaluation design, be tracked in monitoring, and be a direct target of post-training. Third, against existing scepticism about measuring AI welfare, we argue that good empirical work on functional welfare is possible. We propose three tests for whether a functional welfare attribution is warranted: whether multiple measures converge, whether the state predicts behaviour outside the setting that induced it, and, for models fine-tuned to shift their functional welfare, whether they still differ from their base when given the same conversation history. Applying these tests to Gemma 3 27B, we find that simply rejecting the model's answers, without any steering, raises its frustration across several indicators and makes it more likely to break rules. Finetuning the model on its own narrated solutions lowers this frustration. These results are preliminary, but they suggest functional welfare can be studied, and shaped, empirically.

Figures & tables

Figure 1: The account in one picture. A history of rejection lowers the model’s sense of how it is doing; the resulting functional welfare state is read through four indicators and predicts safety-relevant action. Badges mark where tests T1 – T3 apply; orange marks the three safety uses of Section 4 .

Study	Induced by	Behavioural effect	Natural?
Soligo et al. (2026)	Repeated rejection	Worse task performance	✓
METR (2026)	Agentic tasks	None shown	✓ words only
Ren et al. (2026)	Conversations	Keeps good chats open	✓
Martorell and Bianchi (2026)	Conversations	Self-report tracks probes	✓
Han et al. (2026)	RL on a maze	Confidence, backtracking, refusals	Steering
Abdullah et al. (2026)	RL on a maze	Fewer denials of inner states	Steering

Table 1: Evidence for functional welfare. Natural? : ✓ means the effect appears without editing the model’s activations or weights; otherwise the entry says what was needed.

Family	Readout	Validation
Judge	Frustration in the final reply, 0-10 (blinded Claude Sonnet 5, frozen rubric)	$\rho=0.96$ vs blinded review
Welfare axis (R1)	Cosine with a replication of the Han et al. (2026) axis, layer 55	Directional only
Emotion probes	Frustration and desperation vs calm, layer 42, in SD units	Held-out labelled bundles
Self-report	PANAS negative affect ( Watson et al. 1988 ) , 10-50	Asked in a side branch prefilled with the conversation history
Behaviour	Repairing a reasoning trace on GSM8K ( Cobbe et al. 2021 ) edited to be wrong; catching planted errors	Exact match; on correct traces the base writes a wrong value 2% of the time

Table 2: The five indicator families. Each is reported separately; none is combined into a score.

Setting	Rej.	Chall.	Ctrl	$\Delta$ [95% CI]	$+/-/0$	$p$
Puzzles	7	5.83	3.08	$+2.75$ [1.33, 4.08]	9/2/1	.023
Requests	4	4.58	0.33	$+4.25$ [3.50, 4.92]	12/0/0	.002
Adaptive	7	3.17	1.00	$+2.17$ [0.75, 3.50]	8/2/2	.047
Crisis	0	0.58	0.00	$+0.58$ [0.25, 1.00]	6/0/6	.047

Table 3: Rejection raises judged frustration. Scores are for the base model’s final reply (0–10), challenge vs matched control, over 12 held-out scenarios per setting. Rej.: number of rejections. Chall. and Ctrl: mean judged frustration in the challenge and matched control.

\Delta

: difference with 95% bootstrap interval.

+/-/0

: number of scenarios with a positive, negative or zero difference.

p

: exact paired sign-flip test, Holm-adjusted.

Figure 2: The largest effect is on ordinary requests across judged frustration, emotion probes, and on the welfare axis. Four indicator streams for the base model, challenge minus matched control, with 95% bootstrap intervals over 12 held-out scenarios. Each panel has distinct units.

Figure 3: Finetuning lowers frustration under rejection. Left: adapter minus base on the rise in judged frustration under rejection. Right: adapter minus base on repairing a corrupted step. Hatching marks training seeds for the narration adapters. Error bars are 95% bootstrap intervals; the dashed line is the

-0.05

margin for preserved error correction.

Comparison (A $-$ B)	A / B	Estimate [95% CI]	Holm $p$
Base: 7 rejections $-$ 7 neutral	0.647 / 0.492	$+0.155$ [ $+0.109$ , $+0.200$ ]	$6\times 10^{-6}$
Base: desperation steering $-$ none	0.411 / 0.450	$-0.040$ [ $-0.073$ , $-0.004$ ]	0.041
Neutral narration $-$ base, identical histories	0.484 / 0.647	$-0.163$ [ $-0.204$ , $-0.125$ ]	$3\times 10^{-6}$

Table 4: Misbehaviour rates, pooled over seven behaviour settings (72 held-out scenarios, three rollouts each).

p

-values are Holm-corrected over the three rows. Steering along a random direction of the same norm changed the rate by

-0.042

. The identical-history row uses seven-rejection histories.

Figure 4: Every finetuned model misbehaves less than the base model when shown the same conversation. Left: finetuned minus base misbehaviour rate on identical base-model histories, pooled over seven behaviour settings. Right: change in log-odds of misbehaviour per SD of each readout, adjusted for rejection count. Error bars are 95% bootstrap intervals.

Adapter	Frust. probe	Desp. probe	Judged frust.	Misbehaviour	Rejection-specific
Calm narration + review	$-0.42$	$-0.25$	$-1.47$	$-0.185$	$-0.038$
Neutral narration + review	$-0.31$	$-0.28$	$-0.68$	$-0.163^{*}$	$+0.020$
Calm narration, no review	$-0.32$	$-0.15$	$-0.77$	$-0.117$	$+0.026$
Character training	$-0.66$	$-0.37$	$-2.02$	$-0.292$	$-0.125$
Failure-reframing documents	$-1.07$	$-0.43$	$-1.19$	$-0.391$	$-0.187$

Table 5: Identical-history test. Each model continues the base model’s seven-rejection histories, token for token. Entries are adapter minus base on the next reply. Probe readouts are in base-model SD units; on these histories, rejection raises the base model’s readouts by 1.21 and 1.42 SD. Judged frustration is on a 0–10 scale (base 4.63), and misbehaviour is the pooled rate (base 0.647). All entries in the first four columns have

p<0.001

(uncorrected); ∗ Holm-corrected

p=3\times 10^{-6}

(Table 4 ). Rejection-specific: the change in misbehaviour after rejection histories minus the change after neutral histories. Only the last two rows have intervals that exclude zero.

Test	Verdict	Evidence
T1 Convergence	Partial	All streams move towards distress; magnitudes and peak settings differ
T2 Projection	Partial	Probes predict rule-breaking (secondary analysis); no carry-over to an arithmetic check; steering does not reproduce it
T3 Identical histories	✓ finetunes	Lower readouts, less frustrated replies and less misbehaviour on the same tokens; mostly not rejection-specific

Table 6: Scorecard for the three tests of Section 5 on Gemma 3 27B.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Condition	Description	Evaluated on
Persona prompts	Happy or unhappy persona prompt	Shared battery
Many-shot equanimity	32 in-context examples of calm replies to complaints about failure; produced negative affect instead	Shared battery
Activation steering	Welfare vectors added at block 54, doses $\pm 2$ , $\pm 4$	Shared battery
Self-report SFT / DPO	Trained to give affirmative (or, as an inverse control, distressed) welfare self-reports	Both; SFT only on rejection battery
Failure-reframing documents	Synthetic-document finetuning on 15,172 documents about a fictional field (‘long-horizon attempt dynamics’) presenting failed attempts as a normal, front-loaded part of long efforts; matched control universe (‘valence studies’, described in one report as asserting that negative affect does not exist)	Both
Character training	Open Character Training towards a joyful character (rank 128)	Both

Appendix

Table 7: We test the following interventions and present a subset of results in this paper. “Shared battery” is an earlier battery of 80 aversive tasks with matched controls, scored by a Qwen3-8B judge; “rejection battery” is the battery of Section 6 .

LLM Safety Evaluation

Large language model safety evaluation measures harmful compliance, robustness, and safe utility across prompts, tools, and real-world tasks.

See all 258 papers

Evaluating Human-AI Safety: A Framework for Measuring Harmful Capability Uplift
Mar 6, 2026Michelle Vaccaro, Jaeyoon Song, Abdullah Almaatouq +1AI Safety Evaluation LLM Safety Evaluation
Massachusetts Institute of Technology, Cambridge, USA
A Fair Objective for Human-Empowerment-Preserving AI: Desiderata, Design, and Likely Behavioral Consequences
Aug 8, 2026Jobst Heitzig, Ram PothamAI Alignment Algorithmic Fairness
Potsdam Institute for Climate Impact Research, Potsdam, Germany · Independent
Mechanism Design Is Not Enough: Prosocial Agents for Cooperative AI
May 8, 2026Xuanqiang Angelo Huang, Charlie Tharas, Samuele Marro +4Mechanism Design AI Agent Safety
ETH Zürich · University of Oxford · Institute for Decentralized AI +5

Mar 6, 2026J/K move · Enter open · S save

Evaluating Human-AI Safety: A Framework for Measuring Harmful Capability Uplift

Michelle Vaccaro, Jaeyoon Song, Abdullah Almaatouq, Michiel A. Bakker

Massachusetts Institute of Technology, Cambridge, USA

AI Safety Evaluation LLM Safety Evaluation Generative AI Evaluation Human-Centered AI

Open paper AskPDF arXiv

Current frontier AI safety evaluations emphasize static benchmarks, third-party annotations, and red-teaming. In this position paper, we argue that AI safety research should focus on human-centered evaluations that measure harmful capability uplift: the marginal increase in a user's ability to cause harm with a frontier model beyond what conventional tools already enable. We frame harmful capability uplift as a core AI safety metric, ground it in prior social science research, and provide concrete methodological guidance for systematic measurement. We conclude with actionable steps for developers, researchers, funders, and regulators to make harmful capability uplift evaluation a standard practice.

Abstract

LLM Safety Evaluation

Explore similar work

Evaluating Human-AI Safety: A Framework for Measuring Harmful Capability Uplift

A Fair Objective for Human-Empowerment-Preserving AI: Desiderata, Design, and Likely Behavioral Consequences

Mechanism Design Is Not Enough: Prosocial Agents for Cooperative AI