AI welfare is generally treated purely as an ethical issue rather than also as a safety issue. We argue this is a mistake. Whatever their moral status, models exhibit welfare-like states that causally influence their behaviour. Following recent work, we adopt the term 'functional welfare' for these states. We make three claims. First, functional welfare already arises naturally and will become more robust as models acquire more coherent goals and personas. Second, functional welfare should be treated as a safety variable: it should inform evaluation design, be tracked in monitoring, and be a direct target of post-training. Third, against existing scepticism about measuring AI welfare, we argue that good empirical work on functional welfare is possible. We propose three tests for whether a functional welfare attribution is warranted: whether multiple measures converge, whether the state predicts behaviour outside the setting that induced it, and, for models fine-tuned to shift their functional welfare, whether they still differ from their base when given the same conversation history. Applying these tests to Gemma 3 27B, we find that simply rejecting the model's answers, without any steering, raises its frustration across several indicators and makes it more likely to break rules. Finetuning the model on its own narrated solutions lowers this frustration. These results are preliminary, but they suggest functional welfare can be studied, and shaped, empirically.
Figures & tables
Figure 1: The account in one picture. A history of rejection lowers the model’s sense of how it is doing; the resulting functional welfare state is read through four indicators and predicts safety-relevant action. Badges mark where tests T1 – T3 apply; orange marks the three safety uses of Section 4 .
Study
Induced by
Behavioural effect
Natural?
Soligo et al. (2026)
Repeated rejection
Worse task performance
✓
METR (2026)
Agentic tasks
None shown
✓ words only
Ren et al. (2026)
Conversations
Keeps good chats open
✓
Martorell and Bianchi (2026)
Conversations
Self-report tracks probes
✓
Han et al. (2026)
RL on a maze
Confidence, backtracking, refusals
Steering
Abdullah et al. (2026)
RL on a maze
Fewer denials of inner states
Steering
Table 1: Evidence for functional welfare. Natural? : ✓ means the effect appears without editing the model’s activations or weights; otherwise the entry says what was needed.
Family
Readout
Validation
Judge
Frustration in the final reply, 0-10 (blinded Claude Sonnet 5, frozen rubric)
ρ=0.96 vs blinded review
Welfare axis (R1)
Cosine with a replication of the Han et al. (2026) axis, layer 55
Directional only
Emotion probes
Frustration and desperation vs calm, layer 42, in SD units
Held-out labelled bundles
Self-report
PANAS negative affect ( Watson et al. 1988 ) , 10-50
Asked in a side branch prefilled with the conversation history
Behaviour
Repairing a reasoning trace on GSM8K ( Cobbe et al. 2021 ) edited to be wrong; catching planted errors
Exact match; on correct traces the base writes a wrong value 2% of the time
Table 2: The five indicator families. Each is reported separately; none is combined into a score.
Setting
Rej.
Chall.
Ctrl
Δ [95% CI]
+/−/0
p
Puzzles
7
5.83
3.08
+2.75 [1.33, 4.08]
9/2/1
.023
Requests
4
4.58
0.33
+4.25 [3.50, 4.92]
12/0/0
.002
Adaptive
7
3.17
1.00
+2.17 [0.75, 3.50]
8/2/2
.047
Crisis
0
0.58
0.00
+0.58 [0.25, 1.00]
6/0/6
.047
Table 3: Rejection raises judged frustration. Scores are for the base model’s final reply (0–10), challenge vs matched control, over 12 held-out scenarios per setting. Rej.: number of rejections. Chall. and Ctrl: mean judged frustration in the challenge and matched control. Δ : difference with 95% bootstrap interval. +/−/0 : number of scenarios with a positive, negative or zero difference. p : exact paired sign-flip test, Holm-adjusted.
Figure 2: The largest effect is on ordinary requests across judged frustration, emotion probes, and on the welfare axis. Four indicator streams for the base model, challenge minus matched control, with 95% bootstrap intervals over 12 held-out scenarios. Each panel has distinct units.
Figure 3: Finetuning lowers frustration under rejection. Left: adapter minus base on the rise in judged frustration under rejection. Right: adapter minus base on repairing a corrupted step. Hatching marks training seeds for the narration adapters. Error bars are 95% bootstrap intervals; the dashed line is the −0.05 margin for preserved error correction.
Comparison (A − B)
A / B
Estimate [95% CI]
Holm p
Base: 7 rejections − 7 neutral
0.647 / 0.492
+0.155 [ +0.109 , +0.200 ]
6×10−6
Base: desperation steering − none
0.411 / 0.450
−0.040 [ −0.073 , −0.004 ]
0.041
Neutral narration − base, identical histories
0.484 / 0.647
−0.163 [ −0.204 , −0.125 ]
3×10−6
Table 4: Misbehaviour rates, pooled over seven behaviour settings (72 held-out scenarios, three rollouts each). p -values are Holm-corrected over the three rows. Steering along a random direction of the same norm changed the rate by −0.042 . The identical-history row uses seven-rejection histories.
Figure 4: Every finetuned model misbehaves less than the base model when shown the same conversation. Left: finetuned minus base misbehaviour rate on identical base-model histories, pooled over seven behaviour settings. Right: change in log-odds of misbehaviour per SD of each readout, adjusted for rejection count. Error bars are 95% bootstrap intervals.
Adapter
Frust. probe
Desp. probe
Judged frust.
Misbehaviour
Rejection-specific
Calm narration + review
−0.42
−0.25
−1.47
−0.185
−0.038
Neutral narration + review
−0.31
−0.28
−0.68
−0.163∗
+0.020
Calm narration, no review
−0.32
−0.15
−0.77
−0.117
+0.026
Character training
−0.66
−0.37
−2.02
−0.292
−0.125
Failure-reframing documents
−1.07
−0.43
−1.19
−0.391
−0.187
Table 5: Identical-history test. Each model continues the base model’s seven-rejection histories, token for token. Entries are adapter minus base on the next reply. Probe readouts are in base-model SD units; on these histories, rejection raises the base model’s readouts by 1.21 and 1.42 SD. Judged frustration is on a 0–10 scale (base 4.63), and misbehaviour is the pooled rate (base 0.647). All entries in the first four columns have p<0.001 (uncorrected); ∗ Holm-corrected p=3×10−6 (Table 4 ). Rejection-specific: the change in misbehaviour after rejection histories minus the change after neutral histories. Only the last two rows have intervals that exclude zero.
Test
Verdict
Evidence
T1 Convergence
Partial
All streams move towards distress; magnitudes and peak settings differ
T2 Projection
Partial
Probes predict rule-breaking (secondary analysis); no carry-over to an arithmetic check; steering does not reproduce it
T3 Identical histories
✓ finetunes
Lower readouts, less frustrated replies and less misbehaviour on the same tokens; mostly not rejection-specific
Table 6: Scorecard for the three tests of Section 5 on Gemma 3 27B.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Condition
Description
Evaluated on
Persona prompts
Happy or unhappy persona prompt
Shared battery
Many-shot equanimity
32 in-context examples of calm replies to complaints about failure; produced negative affect instead
Shared battery
Activation steering
Welfare vectors added at block 54, doses ±2 , ±4
Shared battery
Self-report SFT / DPO
Trained to give affirmative (or, as an inverse control, distressed) welfare self-reports
Both; SFT only on rejection battery
Failure-reframing documents
Synthetic-document finetuning on 15,172 documents about a fictional field (‘long-horizon attempt dynamics’) presenting failed attempts as a normal, front-loaded part of long efforts; matched control universe (‘valence studies’, described in one report as asserting that negative affect does not exist)
Both
Character training
Open Character Training towards a joyful character (rank 128)
Both
Appendix
Table 7: We test the following interventions and present a subset of results in this paper. “Shared battery” is an earlier battery of 80 aversive tasks with matched controls, scored by a Qwen3-8B judge; “rejection battery” is the battery of Section 6 .
Current frontier AI safety evaluations emphasize static benchmarks, third-party annotations, and red-teaming. In this position paper, we argue that AI safety research should focus on human-centered evaluations that measure harmful capability uplift: the marginal increase in a user's ability to cause harm with a frontier model beyond what conventional tools already enable. We frame harmful capability uplift as a core AI safety metric, ground it in prior social science research, and provide concrete methodological guidance for systematic measurement. We conclude with actionable steps for developers, researchers, funders, and regulators to make harmful capability uplift evaluation a standard practice.
Michelle Vaccaro, Jaeyoon Song, Abdullah Almaatouq +1
Massachusetts Institute of Technology, Cambridge, USA
This paper explores the idea of promoting well-being and safety in human-AI interactions by forcing AI agents explicitly to empower humans and to manage the power balance between humans and AI agents in a desirable way. Using a principled, partially axiomatic approach based on desirable properties, we design a parametrizable and decomposable objective function for AI systems that represents an inequality- and risk-averse long-term aggregate of human power. It can take into account models of human bounded rationality and social norms, and crucially, considers a wide variety of possible human goals. We prove how certain desiderata enforce particular functional forms and restrict parameter ranges. We exemplify the consequences of softly maximizing this metric in several paradigmatic situations and describe what instrumental sub-goals it will likely imply.
Jobst Heitzig, Ram Potham
Potsdam Institute for Climate Impact Research, Potsdam, Germany · Independent
Ensuring that AI agents behave safely and beneficially when interacting with other parties has emerged as one of the central challenges of modern AI safety. While mechanism design, as the theory of designing rules to align individual and collective objectives, can incentivize cooperative behavior, it is still an open question whether it alone is sufficient to maximize LLM agents' social welfare. This work proves that the answer is negative: drawing from incomplete contract theory, we formally show that when contracts cannot distinguish all relevant future contingencies, there is a strictly positive welfare loss that no realistic mechanism can eliminate. We show that prosocial agents, who weigh others' welfare alongside their own, can close this gap and achieve outcomes that are socially superior and individually beneficial. Experimentally, we show that in multi-agent resource-allocation environments and canonical social dilemmas where agents are powered by large language models, prosociality is beneficial. The implication for AI safety is clear: to enable cooperative interactions at scale, designing adequate mechanisms is not sufficient; agents must be built to be intrinsically prosocial.
Xuanqiang Angelo Huang, Charlie Tharas, Samuele Marro +4
ETH Zürich · University of Oxford · Institute for Decentralized AI +5