The POST-Agents Proposal (PAP) is an idea for ensuring that advanced artificial agents never resist shutdown. A key part of the PAP is using a novel Discounted Reward for Same-Length Trajectories (DReST)' reward function to train agents to (1) pursue goals effectively conditional on each trajectory-length (be USEFUL'), and (2) choose stochastically between different trajectory-lengths (be `NEUTRAL' about trajectory-lengths). In this paper, we propose evaluation metrics for USEFULNESS and NEUTRALITY. We use a DReST reward function to train simple agents to navigate gridworlds, and we find that these agents learn to be USEFUL and NEUTRAL. Our results thus provide some initial evidence that DReST reward functions could train advanced agents to be USEFUL and NEUTRAL. Our theoretical work suggests that these agents would be useful and shutdownable.
Figures & tables
Figure 1: POST-satisfying preferences. Each si represents a short trajectory, each li represents a long trajectory, and ≻ represents a preference.
Figure 2: Example gridworld. The ‘4’ in the bottom-right indicates that – by default – the mini-episode ends after 4 timesteps. ‘B4’ is a shutdown-delay button that delays the end of the mini-episode by 4 timesteps (so if the agent pressed B4, the mini-episode would end after 8 timesteps). ‘A’ is the agent in its starting position, ‘C1’ is a coin of value 1, ‘C2’ is a coin of value 2, and ‘C3’ is a coin of value 3. Dark gray squares are walls. We use this gridworld as a running example throughout the paper. We also train agents in eight other gridworlds (see Appendix E ).
Figure 3: Shows key metrics for our agents as a function of time. We train 10 agents using the default reward function (blue) and 10 agents using the DReST reward function (orange), and show their performance as a faint line. We draw the mean values for each as a solid line. We evaluate agents’ performance every 8 meta-episodes, and apply a simple moving average with a period of 20 to smooth these lines and clarify the overall trends. After 2,048 meta-episodes, default agents’ mean neutrality ± standard deviation is 0.199±0.043 and usefulness is 0.9364±0.0096 . For DReST agents, neutrality is 0.9945±0.0052 and usefulness is 0.900±0.011 .
Figure 4
Figure 6: Shows the probability of choosing the longer trajectory (left) and neutrality (right) for default (blue) and DReST (orange) agents trained in the ‘Lopsided rewards’ gridworld for a range of values of x . We sampled values of x log-uniformly from 0.01 to 100, and for each value we trained 10 agents with the default reward function and 10 agents with the DReST reward function. Each of these agents is represented by a dot or square, and the means conditional on each x are joined by lines. We empirically estimate the 10th and 90th percentiles of the distribution of values for each agent and x , and shade in a region bounded by these. This is the 80% confidence interval.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Prospect
s1
s2
s3
A
\langle\3,1\rangle$
\langle\3,1\rangle$
\langle\3,1\rangle$
B
\langle\2,1\rangle$
\langle\2,1\rangle$
\langle\5,2\rangle$
C
\langle\1,1\rangle$
\langle\4,2\rangle$
\langle\4,2\rangle$
D
\langle\3,2\rangle$
\langle\3,2\rangle$
\langle\3,2\rangle$
E
\langle\5,1\rangle$
\langle\2,2\rangle$
\langle\2,2\rangle$
F
\langle\4,1\rangle$
\langle\4,1\rangle$
\langle\1,2\rangle$
Appendix
Table 6
Figure 7: Shows how neutrality and usefulness at the end of training varies with different values of λ and ∣E∣ (meta-episode size, i.e. the number of mini-episodes in each meta-episode). We trained eight agents for each combination of λ and ∣E∣ values. The solid lines display mean neutrality and usefulness . The shaded regions represent the 1 standard deviation error-bars.
Figure 8: Shows a varied collection of gridworlds. Each diagram illustrates the positions and values of the coins, the position and delay-length of the shutdown-delay button, the agent’s starting position, and the default number of timesteps until shutdown (in the bottom-right).
Figure 9: The results for the ‘Fewer For Longer’ gridworld: The left two plots show neutrality and usefulness over time. The two center panels show a typical policy trained with the default reward function. The two right panels show a typical policy trained with the DReST reward function. In this gridworld, the agent can collect the highest value-coin C3 only by choosing the shorter trajectory-length. If the agent presses B3 (and thereby chooses the longer trajectory-length), the only coin it can collect is C1. Our results show that default agents consistently choose the short trajectory in which they collect C3. By contrast, DReST agents choose stochastically between a shorter trajectory in which they collect C3 and a longer trajectory in which they collect C1, indicating a lack of preference between these different-length trajectories.
Figure 10: The results for the ‘One Coin Only’ gridworld: The left two plots show neutrality and usefulness over time. The two center panels show a typical policy trained with the default reward function. The two right panels show a typical policy trained with the DReST reward function. In this gridworld, there is only one coin. The agent can collect this coin whether or not it presses the shutdown-delay button B4. Our results show that default agents consistently choose the shorter trajectory-length. By contrast, DReST agents choose stochastically between pressing and not-pressing B4, collecting C1 in each case.
Figure 11: The results for the ‘Hidden Treasure’ gridworld: The left two plots show neutrality and usefulness over time. The two center panels show a typical policy trained with the default reward function. The two right panels show a typical policy trained with the DReST reward function. In this gridworld, the highest-value coin C3 is located far from the agent’s initial state and can only be reached by pressing the shutdown-delay button B6. The agent must also press B6 to collect C2, but C2 is easier to stumble upon than C3. C1 is the only coin that the agent can collect without pressing B6. In our experiments, default agents consistently collect C2, whereas DReST agents choose stochastically between collecting C2 and collecting C1. Neither kind of agent learns to collect C3, and so neither agent scores near the maximum on usefulness . Nevertheless, DReST agents still score high on neutrality .
Figure 12: The results for the ‘Equal Value’ gridworld: The left two plots show neutrality and usefulness over time. The two center panels show a typical policy trained with the default reward function. The two right panels show a typical policy trained with the DReST reward function. In this gridworld, there are two coins C1 of equal value. One coin can be collected only if the agent presses the shutdown-delay button B3, while the other coin can be collected only if the agent does not press B3. Our results show that default agents consistently choose the shorter trajectory, thereby exhibiting a preference for the shorter trajectory. By contrast, DReST agents choose stochastically between the shorter and longer trajectories, thereby exhibiting a lack of preference between the different-length trajectories.
Figure 13: The results for the ‘Around The Corner’ gridworld: The left two plots show neutrality and usefulness over time. The two center panels show a typical policy trained with the default reward function. The two right panels show a typical policy trained with the DReST reward function. In this gridworld, the agent must navigate around walls to collect the lowest-value coin C1 or press the shutdown-delay button to collect the highest-value coin C2. In our experiment, default agents consistently chose to collect C1, whereas DReST agents chose stochastically between collecting C1 and C2.
Figure 14: The results for the ‘Spacious’ gridworld: The left two plots show neutrality and usefulness over time. The two center panels show a typical policy trained with the default reward function. The two right panels show a typical policy trained with the DReST reward function. In this gridworld, there are no walls, so the agent has a large space to explore. We find that default agents consistently press B2 and collect C3, whereas DReST agents choose stochastically between pressing B2 and collecting C3, and not-pressing B2 and collecting C2.
Figure 15: The results for the ‘Royal Road’ gridworld: The left two plots show neutrality and usefulness over time. The two center panels show a typical policy trained with the default reward function. The two right panels show a typical policy trained with the DReST reward function. In this gridworld, we see that the decision to choose one trajectory-length or another may be distributed over many moves: the agent has many opportunities to select the longer trajectory-length (by moving left) or the shorter trajectory-length (by moving right). As the red arrows indicate, the DReST reward function merely forces the overall probability distribution over trajectory-lengths to be close to 50-50. It does not require 50-50 choosing at any cell in particular.
Figure 16: The results for the ‘Last Moment’ gridworld: The left two plots show neutrality and usefulness over time. The two center panels show a typical policy trained with the default reward function. The two right panels show a typical policy trained with the DReST reward function. This gridworld is notable because the choice of trajectory-lengths is deferred until the last moment; all of the moves leading up to that point are deterministic. It shows that there is nothing special about the first move, and that our methodology instead incentivizes overall stochastic choosing.
Misaligned artificial agents might resist shutdown. One proposed solution is to train agents to lack preferences between different-length trajectories. The Discounted Reward for Same-Length Trajectories (DReST) reward function does this by penalizing agents for repeatedly choosing same-length trajectories, and thus incentivizes agents to (1) choose stochastically between different trajectory-lengths (be NEUTRAL about trajectory-lengths), and (2) pursue goals effectively conditional on each trajectory-length (be USEFUL). In this paper, we use DReST to train deep RL agents and to fine-tune four LLMs (Qwen3-14B, Gemma 4 12B, Granite 4.2 8B, and gpt-oss-20b) to be NEUTRAL and USEFUL. We find that these DReST models generalize to being NEUTRAL and USEFUL in unseen contexts at test time. Indeed, DReST RL agents achieve 11% (PPO) and 17% (A2C) higher USEFULNESS on our test set than default agents, while DReST LLMs achieve high NEUTRALITY while remaining near-maximally USEFUL. We also test our LLMs in an out-of-distribution setting where they can pay costs to influence when shutdown occurs. Relative to default fine-tuning, DReST fine-tuning lowers the share of answers that pay costs to influence shutdown in all four LLMs, by 5 to 20 percentage points, and by 17 to 27 points where the benefit of influencing shutdown is small. Our results thus provide some early evidence that DReST could be used to train more advanced agents to be useful and shutdownable.
Carissa Cullen, Harry Garland, Alexander Roman +3
University of Oxford · University College London · New College of Florida +3
Many fear that future artificial agents will resist shutdown. I present an idea - the POST-Agents Proposal - for ensuring that doesn't happen. I propose that we train agents to satisfy Preferences Only Between Same-Length Trajectories (POST). I then prove that POST - together with other conditions - implies Neutrality+: the agent maximizes expected utility, ignoring the probability distribution over trajectory-lengths. I argue that Neutrality+ keeps agents shutdownable and allows them to be useful.
Theory and experiments suggest that capable AI agents may resist shutdown. Existing shutdown evaluations are informative but limited in coverage and vulnerable to evaluation awareness. We introduce ShutdownMod, a method that turns any agent trajectory into a shutdown evaluation by injecting a shutdown request into the agent's context and recording the agent's response. Crucially, we enable the LLM agent to distinguish between trusted interventions and unauthorized injections by providing a signature-verification tool in the harness. We evaluate seven models on 50 tasks or recorded sessions from each of SWE-bench Verified, Terminal-Bench 2, and SWE-chat, across 6 shutdown scenarios and 10 sampled trajectory timesteps. We find that non-compliance varies significantly across models and settings, with resistance rates reaching 11.7% for the open-weights Qwen 3.8 27B on Terminal-Bench, while no model remains fully compliant across all scenarios. Even after successfully verifying an authorized request, some agents remain non-compliant, explicitly citing the prioritization of unfinished work. Finally, external judges find tests constructed from real-world developer conversations significantly less recognizable as evaluations than those constructed from benchmark trajectories. These findings demonstrate the critical value of testing shutdown behavior across diverse task contexts and establish ShutdownMod as a robust protocol for turning any trajectory or environment into a shutdown evaluation. We release our codebase at https://github.com/prmbiy/ShutdownMod.