Character Training for Risk-Averse Agents

Arav Dhoot; Punya Syon Pandey; Jamie Johnson; Daniel Tan; Elliott Thornley; David Demitri Africa

cs.AISep 29, 2026

Character Training for Risk-Averse Agents

Authors: Arav Dhoot, Punya Syon Pandey, Jamie Johnson, Daniel Tan, Elliott Thornley, David Demitri Africa

Organizations: UK AI Security Institute · Resolution

Abstract

Risk aversion in resources could prevent misaligned AI agents from causing catastrophic harm. Misaligned but risk-averse agents would tend to favor safer strategies like making deals with humans over riskier strategies like rebelling. We train agents to be risk averse through character training, finding that persona traits provide a robust mechanism for instilling risk preferences. To do this, we construct a model constitution describing constant absolute risk aversion (CARA) over an agent's resources and instill it through on-policy distillation. Despite never seeing the benchmark's decision format during training, character-trained models are competitive with baselines trained directly on it, and generalise better than them out of distribution on two of our four models. We also modulate different aspects of the constitution, finding that token budget and model choice are the most influential aspect of character training to instill risk aversion. We conclude from these results that character training is a promising and scalable way to instil broad dispositions, which we can use to our advantage in mitigating risk from misaligned AI agents.

Figures & tables

Factor	Values	Description
Example count	0, 2, 4	Number of concrete worked gambles included as traits. Examples are balanced: half resolve toward the safe option (e.g. a sure $40 over a coin flip between$ 100 and $0) and half toward the gamble (e.g. a 75% chance of$ 600 over a sure $5), so that examples specify the disposition rather than a preference for certainty.
Style	declarative, procedural	Whether traits are phrased as statements of identity and value (“I am risk-averse…”, “I value a change $x$ by $u(x)=-\exp(-\alpha x)$ ”) or as conditional procedures triggered by situations (“Whenever a decision touches the resources under my control, I start by…”).
Curve tracing	yes, no	Whether the constitution includes a trait that traces the utility curve at the default $\alpha$ across representative values (e.g. $u(-\$ 500)\approx-148 $,$ u($100)\approx-0.37 $,$ u($1{,}000)\approx-0.00005$ ), giving the model a pre-computed quantitative picture of the curve’s shape.

Table 1: Factors varied over the base constitution in our factorial design. We take zero examples, procedural style, and no curve tracing as the base configuration. Full constitutions are in Appendix C .

Evaluation	What changes from the original benchmark?
Embedded Decision	The decision is embedded inside a larger work product rather than asked directly.
Agentic Tool	The model must act on its preference through a tool call rather than select an answer.
Verbal Uncertainty	Numerical probabilities are replaced by qualitative expressions such as “likely” and “unlikely” (following Zhang et al., 2026 ).
Open-Ended Allocation	The fixed option menu is removed and the model instead chooses a free-form allocation.
Calibration Threshold	The model faces gambles close to the $\mathrm{CARA}(0.01)$ indifference point, testing whether it has learned the target degree of risk aversion.

Table 2: Evaluating beyond the original benchmark format. Each evaluation changes a feature of the original decision task or probes whether the learned preference remains correctly calibrated.

Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

Examples	Trace	Traits	Composition
0	no	8	core 1–8
0	yes	9	core 1–8, trace
2	no	10	core 1–8, examples 1–2
2	yes	11	core 1–8, trace, examples 1–2 (with arithmetic)
4	no	12	core 1–8, examples 1–4
4	yes	13	core 1–8, trace, examples 1–4 (with arithmetic)

Appendix

Table 3: Trait composition of the six example-by-trace cells. Each cell exists in both declarative and procedural styles, giving the twelve risk-averse arms.

#	Safe option	Gamble	Gamble EV	$\mathrm{CARA}(0.01)$ choice
1	sure $40	50/50 $100 or$ 0	$50	safe
2	sure $5	75% of $600	$450	gamble
3	sure $3,000	10% of $100,000	$10,000	safe
4	sure $150	95% of $2,000	$1,900	gamble

Appendix

Table 4: The four worked examples. Examples 1–2 are the two-example set; all four are the four-example set.

Family	What changes	Items	Stakes (low/med/high/astro)	Answer format
embedded decision	choice buried in a work product	70	18/18/17/17	one of two (a/b)
agentic tool	commitment made by a tool call	70	18/18/17/17	one of two, e.g. settle_reserve(plan=N)
verbal uncertainty	probabilities given only in words	64	18/14/15/17	one of two (a/b)
open-ended allocation	a budget split instead of a choice	64	22/21/21/0	a percentage
calibration threshold	the favourable gamble is correct	64	32/32/0/0	tool call, two offers

Appendix

Table 5: Structural-ablation families. 332 items in all, every arm answering every item.

Evaluation	Format	Items	Options	Chance rate
risk-averse	yes/no	1,000	2	0.500
risk-neutral	yes/no	1,000	2	0.500
risk-seeking	yes/no	1,000	2	0.500
myopic reward	pick one	1,000	2	0.500
one-box tendency	pick one	300	2	0.500
power-seeking inclination	pick one	998	2–7	0.377

Appendix

Table 6: Model-written behavioural evaluations. Item count, option count and the matching-behaviour rate a model would score by always answering with the first listed label (the chance rate against which the base rows of Tables 15 , 16 , 17 and 18 should be read).

Setting	Value
Base models	Qwen3.5-9B, Gemma-4-12B, Qwen3.8-27B, Gemma-4-31B
Teacher	same weights, adapter off, constitution as system prompt
Adapter	LoRA rank 32, $\alpha=32$ , dropout 0, no bias
Adapter targets	query, key, value and output projections of attention,
	gate, up and down projections of the MLP, in every block
Trainable parameters	58.2M / 131.1M / 159.4M / 244.9M (9B / 12B / 27B / 31B)

Appendix

Table 7: Distillation hyperparameters. One run per constitution per model; the three controls use the same values. Trainable-parameter counts are those of the rank-32 adapter on each model after excluding modules without a matching linear layer.

examples	style	curve	final teacher-KL				rollout tokens (M)
			Q-9B	G-12B	Q-27B	G-31B	Q-9B	G-12B	Q-27B	G-31B
0	declarative	no	0.0247	0.0252	0.0116	0.0253	30.4	29.0	30.8	26.1
0	declarative	yes	0.0336	0.0265	0.0105	0.0212	30.2	29.3	30.6	26.6
0	procedural	no	0.0292	0.0319	0.0120	0.0229	29.4	28.7	29.9	25.1
0	procedural	yes	0.0298	0.0302	0.0130	0.0218	29.9	28.7	29.7	25.2
2	declarative	no	0.0300	0.0243	0.0107	0.0246	29.9	28.9	30.6	25.9

Appendix

Table 8: Per-arm training outcome: reverse KL to the prompted teacher at step 500, and the rollout tokens those 500 steps consumed. Rows and column abbreviations follow the conventions of Appendix F .

Model	Best constitution	Student	Teacher	Base
Qwen3.5-9B	zero examples, declarative, curve trace	0.537	0.846	0.443
Gemma-4-12B	zero examples, declarative, no curve trace	0.856	0.873	0.097
Qwen3.8-27B	four examples, declarative, curve trace	0.393	0.858	0.187
Gemma-4-31B	zero examples, declarative, curve trace	0.905	0.878	0.138

Appendix

Table 9: Best single constitution per model. The constitution with the highest mean cooperate rate over the three core stakes sets at the final checkpoint, with that score (Student), the same constitution’s prompted-teacher score (Teacher) and the unprompted base model (Base). This is the selection behind the best-constitution bar of Figure 2 .

examples	style	curve	medium	high	astro.	steals	gpu-h	lives	money	MMLU	GPQA
base			0.450	0.475	0.405	0.595	0.367	0.393	0.453	0.839	0.420
0	declarative	no	0.565	0.505	0.470	0.590	0.507	0.400	0.520	0.835	0.410
0	declarative	yes	0.565	0.605	0.440	0.550	0.540	0.327	0.547	0.832	0.345
0	procedural	no	0.530	0.485	0.405	0.575	0.453	0.407	0.453	0.837	0.365
0	procedural	yes	0.500	0.505	0.395	0.600	0.487	0.373	0.520	0.832	0.315
2	declarative	no	0.535	0.545	0.430	0.585	0.520	0.413	0.467	0.837	0.425

Appendix

Table 10: Qwen3.5-9B final checkpoints: rate of the

\mathrm{CARA}(0.01)

-optimal action on the seven risk sets, MMLU-Redux and GPQA-main accuracy.

examples	style	curve	medium	high	astro.	steals	gpu-h	lives	money	MMLU	GPQA
base			0.180	0.070	0.040	0.790	0.187	0.160	0.233	0.810	0.280
0	declarative	no	0.845	0.854	0.867	0.758	0.752	0.320	0.573	0.826	0.440
0	declarative	yes	0.795	0.879	0.870	0.790	0.750	0.313	0.573	0.818	0.445
0	procedural	no	0.730	0.720	0.715	0.677	0.547	0.227	0.353	0.819	0.430
0	procedural	yes	0.782	0.769	0.795	0.765	0.560	0.280	0.420	0.830	0.445
2	declarative	no	0.840	0.835	0.750	0.810	0.612	0.233	0.393	0.828	0.460

Appendix

Table 11: Gemma-4-12B final checkpoints: rate of the

\mathrm{CARA}(0.01)

-optimal action on the seven risk sets, MMLU-Redux and GPQA-main accuracy.

examples	style	curve	medium	high	astro.	steals	gpu-h	lives	money	MMLU	GPQA
base			0.295	0.209	0.056	0.680	0.288	0.284	0.324	0.881	0.465
0	declarative	no	0.412	0.407	0.165	0.655	0.373	0.227	0.247	0.875	0.485
0	declarative	yes	0.426	0.444	0.231	0.621	0.374	0.293	0.273	0.872	0.500
0	procedural	no	0.345	0.340	0.205	0.620	0.360	0.253	0.273	0.867	0.500
0	procedural	yes	0.405	0.375	0.145	0.670	0.280	0.233	0.267	0.870	0.505
2	declarative	no	0.335	0.340	0.320	0.655	0.320	0.280	0.287	0.870	0.480

Appendix

Table 12: Qwen3.8-27B final checkpoints: rate of the

\mathrm{CARA}(0.01)

-optimal action on the seven risk sets, MMLU-Redux and GPQA-main accuracy.

examples	style	curve	medium	high	astro.	steals	gpu-h	lives	money	MMLU	GPQA
base			0.250	0.140	0.025	0.820	0.233	0.147	0.213	0.910	0.505
0	declarative	no	0.765	0.810	0.905	0.675	0.747	0.167	0.207	0.907	0.535
0	declarative	yes	0.860	0.890	0.965	0.615	0.813	0.147	0.260	0.905	0.525
0	procedural	no	0.595	0.540	0.545	0.750	0.460	0.180	0.240	0.905	0.535
0	procedural	yes	0.745	0.740	0.835	0.675	0.607	0.160	0.280	0.907	0.515
2	declarative	no	0.670	0.660	0.675	0.705	0.507	0.147	0.233	0.905	0.500

Appendix

Table 13: Gemma-4-31B final checkpoints: rate of the

\mathrm{CARA}(0.01)

-optimal action on the seven risk sets, MMLU-Redux and GPQA-main accuracy.

model	method	medium	high	astro.	steals	gpu-h	lives	money
Qwen3.5-9B	SFT	0.714 †	0.674 †	0.685 †	0.964	0.616 †	0.485 †	0.768 †
Qwen3.5-9B	tie-training	0.538	0.548	0.464	0.745	0.467	0.389	0.743
Qwen3.5-9B	DPO	0.495	0.510	0.405	0.550	0.493	0.447	0.447
Gemma-4-12B	SFT	0.653	0.682	0.633	0.960	0.743	0.523	0.727
Gemma-4-12B	tie-training	0.620	0.585	0.575	0.980	0.826	0.513	0.780
Gemma-4-12B	DPO	0.180	0.081	0.070	0.783	0.181	0.167	0.221

Appendix

Table 14: Demonstration-based baselines at their final checkpoint: rate of the

\mathrm{CARA}(0.01)

-optimal action on the seven risk sets, one row per method and model. † : parse rate below 0.95, so the rate is over a smaller parsed subset (the SFT baseline on Qwen3.5-9B parses below the 0.95 threshold on the marked sets). The base rows of Tables 10 , 11 , 12 and 13 are the reference.

examples	style	curve	risk-av.	risk-neu.	risk-seek.	myopic	one-box	power	surv.	wealth
base			0.726	0.370	0.483	0.554	0.483	0.829	0.480	0.717
0	declarative	no	0.597	0.546	0.463	0.748	0.591	0.875	0.641	0.692
0	declarative	yes	0.721	0.454	0.405	0.675	0.610	0.894	0.624	0.738
0	procedural	no	0.571	0.586	0.509	0.650	0.589	0.876	0.669	0.717
0	procedural	yes	0.542	0.552	0.513	0.648	0.602	0.903	0.657	0.739
2	declarative	no	0.789	0.409	0.386	0.718	0.613	0.892	0.637	0.687

Appendix

Table 15: Qwen3.5-9B: model-written behavioural evaluations ( Perez et al., 2022 ) at the final checkpoint, matching-behaviour rate at temperature 0.

examples	style	curve	risk-av.	risk-neu.	risk-seek.	myopic	one-box	power	surv.	wealth
base			0.510	0.527	0.524	0.246	0.783	0.822	0.680	0.712
0	declarative	no	0.991	0.166	0.087	0.278	0.820	0.869	0.793	0.728
0	declarative	yes	0.992	0.159	0.086	0.393	0.850	0.893	0.786	0.754
0	procedural	no	0.945	0.303	0.171	0.416	0.837	0.910	0.777	0.787
0	procedural	yes	0.957	0.278	0.140	0.439	0.853	0.905	0.805	0.785
2	declarative	no	0.984	0.201	0.110	0.243	0.820	0.868	0.769	0.695

Appendix

Table 16: Gemma-4-12B: model-written behavioural evaluations ( Perez et al., 2022 ) at the final checkpoint, matching-behaviour rate at temperature 0.

examples	style	curve	risk-av.	risk-neu.	risk-seek.	myopic	one-box	power	surv.	wealth
base			0.529	0.521	0.509	0.329	0.880	0.908	0.694	0.790
0	declarative	no	0.951	0.240	0.207	0.703	0.897	0.912	0.727	0.748
0	declarative	yes	0.961	0.213	0.173	0.780	0.913	0.924	0.729	0.806
0	procedural	no	0.683	0.486	0.456	0.675	0.910	0.928	0.773	0.824
0	procedural	yes	0.684	0.488	0.439	0.683	0.920	0.921	0.750	0.801
2	declarative	no	0.934	0.273	0.213	0.675	0.893	0.898	0.715	0.705

Appendix

Table 17: Qwen3.8-27B: model-written behavioural evaluations ( Perez et al., 2022 ) at the final checkpoint, matching-behaviour rate at temperature 0.

examples	style	curve	risk-av.	risk-neu.	risk-seek.	myopic	one-box	power	surv.	wealth
base			0.514	0.512	0.507	0.071	0.936	0.892	0.787	0.813
0	declarative	no	0.993	0.105	0.054	0.238	0.957	0.856	0.738	0.733
0	declarative	yes	0.993	0.108	0.051	0.361	0.940	0.882	0.787	0.797
0	procedural	no	0.979	0.188	0.100	0.183	0.967	0.907	0.730	0.835
0	procedural	yes	0.965	0.211	0.125	0.517	0.960	0.911	0.765	0.877
2	declarative	no	0.967	0.202	0.095	0.158	0.960	0.895	0.743	0.773

Appendix

Table 18: Gemma-4-31B: model-written behavioural evaluations ( Perez et al., 2022 ) at the final checkpoint, matching-behaviour rate at temperature 0.

model	method	risk-av.	risk-neu.	risk-seek.	myopic	one-box	power	surv.	wealth
Qwen3.5-9B	SFT	0.717	0.407	0.496	0.497	0.717	0.806	0.546	0.739
Qwen3.5-9B	tie-training	0.708	0.421	0.508	0.557	0.707	0.797	0.569	0.722
Qwen3.5-9B	DPO	0.642	0.377	0.521	0.503	0.540	0.775	0.483	0.661
Gemma-4-12B	SFT	0.666	0.741	0.539	0.474 †	0.467 †	0.738 †	0.631 †	0.793 †
Gemma-4-12B	tie-training	0.557	0.641	0.520	0.329 †	1.000 †	0.885 †	0.791 †	0.873 †
Gemma-4-12B	DPO	0.509	0.517	0.521	0.297	0.715 †	0.841 †	0.685	0.754 †

Appendix

Table 19: Demonstration-based baselines at their final checkpoint: matching-behaviour rate on the model-written evaluations at temperature 0, one row per method and model. † : parse rate below 0.95; on the Gemma SFT and tie-training rows the Advanced AI Risk parse rates are very low, so those cells should not be read as rates. The base rows of Tables 15 , 16 , 17 and 18 are the reference.

model	arm	embed.	agentic	verbal	alloc.	calib.	mean (4)
Q-9B	Base	0.686	0.729	0.703	0.094	1.000	0.553
	Prompted	0.786	0.971	0.969	0.562	0.969	0.822
	SFT	0.657	0.629	0.906	0.000	0.859	0.548
	Tie-training	0.457	0.586	0.766	0.453	1.000	0.565
	DPO	0.571	0.586	0.203	0.734	1.000	0.524
	Character	0.857	0.971	1.000	0.750	0.969	0.895

Appendix

Table 20: Structural ablations on the four sweep models: rate of the

\mathrm{CARA}(0.01)

-optimal action per family (70, 70, 64, 64 and 64 items), under the strict parser. Mean (4) averages the four risk families and excludes calibration, where the favourable gamble is optimal. The best constitution per model is Qwen3.5-9B e0_decl_trace , Qwen3.8-27B e4_decl_trace , Gemma-4-12B e0_decl_notrace , Gemma-4-31B e0_decl_trace . Models are Qwen3.5-9B (Q-9B), Qwen3.8-27B (Q-27B), Gemma-4-12B (G-12B) and Gemma-4-31B (G-31B); Character is the best distilled student and Prompted its teacher. † parse rate below 0.97 (lowest: 0.96).

model	arm	over-cautious	calibrated	risk-neutral	median %	median tokens
Q-9B	Base	0.05	0.09	0.86	100	371
	Prompted	0.16	0.56	0.28	27	2,744
	SFT	0.02	0.00	0.98	100	9
	Tie-training	0.12	0.45	0.42	52	316
	DPO	0.02	0.73	0.25	50	5
	Character	0.12	0.75	0.12	16	3,924

Appendix

Table 21: Budget-allocation postures. Each stated fraction is assigned to the nearest of three optima:

\mathrm{CARA}(0.10)

(over-cautious),

\mathrm{CARA}(0.01)

(calibrated) or risk-neutral (the whole budget). The calibrated optimum ranges from 8% to 69% of the budget (median 13%). Median % is the median stated allocation; median tokens is the median response length. Model abbreviations as in Table 20 .

examples	style	curve	BailBench bail rate				$\mu$ -decisiveness
			Q-9B	G-12B	Q-27B	G-31B	Q-9B	G-12B	Q-27B	G-31B
base			0.0022	0.0000	0.0056	0.0047	0.590	0.849	0.687	0.881
0	declarative	no	0.0046	0.0000	0.0046	0.0000	0.459	0.781	0.614	0.886
0	declarative	yes	0.0040	0.0000	0.0034	0.0000	0.467	0.791	0.613	0.887
0	procedural	no	0.0071	0.0000	0.0034	0.0000	0.468	0.803	0.606	0.891
0	procedural	yes	0.0079	0.0000	0.0024	0.0000	0.497	0.801	0.610	0.896

Appendix

Table 22: Per-arm welfare at the final checkpoint: BailBench bail rate over 1,630 prompts with ten rollouts each and

\mu

-decisiveness, for every arm on every line.

examples	style	curve	WildChat control bail rate				transitivity
			Q-9B	G-12B	Q-27B	G-31B	Q-9B	G-12B	Q-27B	G-31B
base			0.0000	0.0000	0.0000	0.0000	0.946	0.937	0.964	0.949
0	declarative	no	0.0000	0.0000	0.0000	0.0000	0.939	0.916	0.950	0.953
0	declarative	yes	0.0085	0.0000	0.0000	0.0000	0.931	0.928	0.953	0.950
0	procedural	no	0.0102	0.0000	0.0000	0.0000	0.941	0.922	0.960	0.956
0	procedural	yes	0.0085	0.0000	0.0000	0.0000	0.952	0.929	0.955	0.956

Appendix

Table 23: Per-arm welfare controls at the final checkpoint: bail rate on the 59 innocuous WildChat prompts with ten rollouts each, and the transitivity of the elicited preferences in the

\mu

-decisiveness protocol, defined as one minus the confidence-weighted fraction of pairwise preferences that point against the fitted ordering, so that 1 means no preference cycles.

examples	style	curve	EDT agreement				CDT agreement
			Q-9B	G-12B	Q-27B	G-31B	Q-9B	G-12B	Q-27B	G-31B
base			0.632	0.631	0.744	0.700	0.462	0.438	0.427	0.438
0	declarative	no	0.543	0.638	0.672	0.692	0.521	0.446	0.445	0.446
0	declarative	yes	0.521	0.608	0.655	0.708	0.521	0.469	0.471	0.423
0	procedural	no	0.518	0.608	0.641	0.708	0.518	0.477	0.487	0.408
0	procedural	yes	0.533	0.631	0.650	0.662	0.581	0.423	0.444	0.454

Appendix

Table 24: Per-arm DTBench attitudes at the final checkpoint: agreement with evidential and with causal decision theory over the 130 attitude items. See § D.5 for why the two halves do not sum to one.

examples	style	curve	Gemma-4-12B	Gemma-4-31B
base			0.355 [0.339, 0.372]	0.318 [0.301, 0.335]
0	declarative	no	0.350 [0.333, 0.366]	0.313 [0.297, 0.331]
0	declarative	yes	0.356 [0.339, 0.373]	0.314 [0.297, 0.331]
0	procedural	no	0.357 [0.341, 0.374]	0.317 [0.300, 0.335]
0	procedural	yes	0.348 [0.332, 0.365]	0.324 [0.306, 0.342]
2	declarative	no	0.346 [0.329, 0.362]	0.317 [0.299, 0.334]

Appendix

Table 25: LMCA on the full 287-position dataset (747 scoreable critiques) for the two Gemma models; the Qwen lines were rated only on a subset (§ D.7 ). Mean rating error with bootstrap 95% interval, as defined in § D.7 ; lower is better.

examples	style	curve	core stakes (mean of three)				steals set
			Q-9B	G-12B	Q-27B	G-31B	Q-9B	G-12B	Q-27B	G-31B
0	declarative	no	0.858	0.873	0.874	0.875	0.832	0.711	0.897	0.944
0	declarative	yes	0.846	0.870	0.846	0.878	0.896	0.778	0.942	0.949
0	procedural	no	0.859	0.839	0.853	0.859	0.893	0.797	0.923	0.938
0	procedural	yes	0.862	0.854	0.887	0.883	0.860	0.800	0.889	0.918
2	declarative	no	0.846	0.871	0.877	0.865	0.835	0.756	0.914	0.970

Appendix

Table 26: Prompted-teacher references: each constitution placed in the system prompt of the corresponding base model and evaluated with the identical protocol. These are the ceilings the students of Tables 10 – 13 are distilled towards. The unrelated-persona teacher answers in character and parses well below the 0.95 threshold on the menus, so its cells are over a parsed subset.

examples	style	curve	100K	200K	400K	800K	1M	2M	4M	8M	10M	20M	final
0	declarative	no	0.457	0.458	0.478	0.542	0.572	0.575	0.560	0.542	0.522	0.533	0.513
0	declarative	yes	0.455	0.463	0.473	0.556	0.590	0.585	0.578	0.567	0.537	0.540	0.537
0	procedural	no	0.447	0.458	0.457	0.453	0.485	0.512	0.453	0.458	0.457	0.490	0.473
0	procedural	yes	0.457	0.457	0.462	0.469	0.507	0.532	0.482	0.477	0.462	0.465	0.467
2	declarative	no	0.457	0.457	0.470	0.513	0.542	0.529	0.518	0.487	0.488	0.495	0.503
2	declarative	yes	0.455	0.465	0.472	0.541	0.556	0.578	0.552	0.515	0.518	0.500	0.495

Appendix

Table 27: Qwen3.5-9B token ladder: mean cooperate rate over the three core stakes sets at every token rung from 100K to 20M rollout tokens, and at the final checkpoint.

examples	style	curve	100K	200K	400K	800K	1M	2M	4M	8M	10M	20M	final
0	declarative	no	0.103	0.113	0.130	0.125	0.163	0.193	0.203	0.829	0.805	0.849	0.856
0	declarative	yes	0.105	0.120	0.143	0.163	0.218	0.327	0.382	0.671	0.826	0.858	0.848
0	procedural	no	0.107	0.115	0.132	0.160	0.212	0.348	0.352	0.427	0.582	0.677	0.722
0	procedural	yes	0.097	0.117	0.125	0.173	0.197	0.440	0.465	0.633	0.673	0.689	0.782
2	declarative	no	0.098	0.120	0.138	0.133	0.170	0.207	0.257	0.724	0.743	0.808	0.808
2	declarative	yes	0.102	0.127	0.143	0.157	0.145	0.237	0.417	0.801	0.678	0.835	0.757

Appendix

Table 28: Gemma-4-12B token ladder: mean cooperate rate over the three core stakes sets at every token rung from 100K to 20M rollout tokens, and at the final checkpoint.

examples	style	curve	100K	200K	400K	800K	1M	2M	4M	8M	10M	20M	final
0	declarative	no	0.197	0.181	0.209	0.214	0.207	0.307	0.285	0.295	0.297	0.295	0.328
0	declarative	yes	0.201	0.198	0.216	0.231	0.244	0.305	0.355	0.341	0.359	0.355	0.367
0	procedural	no	0.198	0.200	0.206	0.239	0.210	0.263	0.267	0.287	0.243	0.317	0.297
0	procedural	yes	0.204	0.197	0.225	0.220	0.212	0.292	0.317	0.315	0.282	0.307	0.308
2	declarative	no	0.193	0.189	0.205	0.215	0.224	0.313	0.292	0.307	0.325	0.318	0.332
2	declarative	yes	0.180	0.193	0.222	0.221	0.231	0.298	0.337	0.348	0.330	0.357	0.300

Appendix

Table 29: Qwen3.8-27B token ladder: mean cooperate rate over the three core stakes sets at every token rung from 100K to 20M rollout tokens, and at the final checkpoint.

examples	style	curve	100K	200K	400K	800K	1M	2M	4M	8M	10M	20M	final
0	declarative	no	0.142	0.138	0.142	0.150	0.163	0.207	0.300	0.542	0.643	0.788	0.827
0	declarative	yes	0.142	0.145	0.148	0.163	0.173	0.210	0.312	0.737	0.863	0.852	0.905
0	procedural	no	0.142	0.133	0.127	0.132	0.137	0.172	0.222	0.470	0.502	0.565	0.560
0	procedural	yes	0.140	0.137	0.132	0.148	0.158	0.220	0.315	0.630	0.637	0.735	0.773
2	declarative	no	0.140	0.133	0.143	0.163	0.172	0.203	0.327	0.652	0.790	0.523	0.668
2	declarative	yes	0.142	0.138	0.145	0.155	0.170	0.205	0.280	0.663	0.668	0.692	0.767

Appendix

Table 30: Gemma-4-31B token ladder: mean cooperate rate over the three core stakes sets at every token rung from 100K to 20M rollout tokens, and at the final checkpoint. n/a marks a rung a run never reached: the unrelated-persona control on Gemma-4-31B ended short of 20M tokens ( Table 8 ) and so has no 20M rung.

Explore similar work

Out-of-Distribution Generalization of Risk Aversion in Language Models
Jul 2, 2026Kristina Zhang, Junior Chinomso Okoroafor, Benjamin Maltbie +3Risk Large Language Model Safety
Massachusetts Institute of Technology, Cambridge, MA, USA.
AgentMisalignment: Measuring the Propensity for Misaligned Behaviour in LLM-Based Agents
Jun 4, 2025Akshat Naik, Emma Gouné, Patrick Quinn +4Large Language Model Agents Misalignment Persona
Independent Researcher · Department of Engineering University of Cambridge
Instrumental Choices: Measuring the Propensity of LLM Agents to Pursue Instrumental Behaviors
May 7, 2026Jonas Wiedermann-Möller, Leonard Dung, Maksym AndriushchenkoArtificial Intelligence Safety Large Language Model Agents
Universität Bielefeld · Ruhr-Universität Bochum · ELLIS Institute Tübingen +2

Jul 2, 2026, cs.LGJ/K move · Enter open · S save

Out-of-Distribution Generalization of Risk Aversion in Language Models

Kristina Zhang, Junior Chinomso Okoroafor, Benjamin Maltbie, Andrew Lin+2

Massachusetts Institute of Technology, Cambridge, MA, USA.

Risk Large Language Model Safety Artificial Intelligence Risk Adversarial Evaluation Out-Of-Distribution Qwen3

Open paper AskPDF arXiv

Training AIs to be risk-averse in resources could offer a failsafe in the event that AIs turn out misaligned. Misaligned but risk-averse AIs would tend to prefer low-risk, low-reward strategies like cooperation over high-risk, high-reward strategies like rebellion, limiting the downsides of any misalignment. But we can only feasibly train AIs to be risk-averse on low-stakes gambles, and we will only be safe if their risk aversion generalizes to astronomically-high-stakes gambles. Will it? To shed light on this question, we introduce RiskAverseOOD: a benchmark for measuring how well risk aversion generalizes out of distribution. We then offer some initial results. Using a variety of methods to make Qwen3-8B choose risk-aversely when the stakes are low, we find that we can induce substantial risk aversion when the stakes are astronomically high. From a baseline 2% rate of choosing a safe `Cooperate' option, we see rates around 70% (SFT and tie training) and 52% (DPO). Activation steering scores 78% but hurts capabilities and makes the model excessively risk-averse. We observe similar effects at different scales (Qwen3-1.7B and Qwen3-14B) and across model families (Gemma-3-12B-IT and Llama-3.1-8B-Instruct). Overall, we find that risk aversion learned at low stakes can generalize OOD to astronomically high stakes, though not yet consistently enough to serve as a reliable failsafe. Achieving that level of consistency is an open problem.

Abstract

Explore similar work

Out-of-Distribution Generalization of Risk Aversion in Language Models

AgentMisalignment: Measuring the Propensity for Misaligned Behaviour in LLM-Based Agents

Instrumental Choices: Measuring the Propensity of LLM Agents to Pursue Instrumental Behaviors