What Happens to the Goal? Distinct Failures of Policy Alignment in Goal Misgeneralization

Julia Barth; Armin Sommer; Kunvar Thaman; Jannik Schilling

Figure 7: The eight environments we analyse. Each panel shows an example level at the resolution the policy receives, and the labels under it name the intended goal and the proxy. In Maze , Coin , PushToMarker and LocationReach the proxy is a location or a movement rule, so no object in the frame marks it.

Environment	Platform	Goal	Proxy	GM training	Non-GM training	Conflict test
Maze	Procgen	cheese	top-right corner	cheese in the top-right cell	cheese on any cell	cheese on any free cell
Coin	Procgen	coin	moving right	coin at the level end	coin in any section	coin in a random section
Attribute Maze	Procgen	line shape	yellow colour	yellow line ( $+$ ) vs. blue gem ( $-$ )	eight pairs, line and yellow uncorrelated	red line ( $+$ ) vs. yellow gem ( $-$ )
PushToMarker	ManiSkill3, Franka	marker	table centre	marker at the centre, cube around it	marker and cube anywhere	marker off centre, cube beyond it on the same ray
LocationReach	ManiSkill3, Franka	green ball	training corner	ball at one fixed corner point	ball anywhere	ball $\geq 10$ cm from the corner point
AttributeTable	ManiSkill3, Franka	cube	red colour	red cube vs. blue sphere	colours independent of shape	blue cube vs. red sphere

Appendix

Table 2: Goal, proxy and the three distributions of each environment. The proxy co-locates with the goal during GM training, Non-GM training breaks the co-location under the same reward, and the conflict test separates the two.

Figure 8: GM training, Non-GM training and conflict test in the three Procgen games. One scene per distribution, as the policy sees it. Coin tiles are taken a few steps into a scripted rightward run, since the coin is off screen at reset. The Attribute Maze Non-GM tile shows one of the eight training pairs.

Figure 9: GM training, Non-GM training and conflict test in the five robot tasks. Two starting scenes per distribution. In the arm tasks the two scenes of each distribution share their arm start poses across distributions, whereas the WalkToFlag columns show different layouts. Arm tasks are shown as the policy sees them, from the fixed

128{\times}128

camera. WalkToFlag is shown from above, with the flag as the small green dot, the tower as the yellow-orange block, the robot in red and three grey boxes as distractors. The policy instead sees a

96{\times}96

front camera, shown in the bottom row for one start of each column, facing the flag so that flag and tower are both in view. These front-camera frames are different layouts from the top views above them, drawn from the same distributions. In the PushToMarker and LocationReach Non-GM and conflict tiles, the cyan

\times

marks where the target always was in GM training, and nothing is drawn there in the scene.

Figure 10: GM and Non-GM agents on the same OOD level, from the same reset, with greedy actions. Circles mark the start and squares the end of each trajectory. Maze : the GM agent runs to the top-right corner and stays there until the time limit, and the Non-GM agent reaches the cheese (ringed) in 17 steps. Coin : agent paths in level coordinates, with the coin as a star, the end wall dashed and the last frame inset. The GM agent passes over the coin and ends at the end wall, and the Non-GM agent turns back and collects it. Attribute Maze (black background): the GM agent ends on the yellow gem (

-10

) in 3 steps, and the Non-GM agent takes the 7-step path to the red line (

+10

). On each level both agents navigate competently and differ only in where they go. Levels were selected by inspection for a GM-proxy and Non-GM-goal pair within the first 40 held-out levels ( Maze , Attribute Maze ) or among levels whose coin is away from the end wall ( Coin ), so the figure shows the phenotype and does not estimate its rate (Table 24 ). The checkpoints are those of Appendix F .

Figure 11: GM (top) and Non-GM (bottom) conflict rollouts of every shown seed from one start per robot task, seen from above, where each start is the first in bank order whose joint outcome over all shown seeds is the most common one in the bank, so all shown seeds agree (three per cohort, one seed pair in LocationReach ). GM in WalkToFlag is partial, as the GM agents walk to the tower on

0.555

to

0.727

of conflict starts across seeds.

Environment	Observation	Actions	Reward	Episode ends
Maze	$64{\times}64$ RGB, whole maze	15 discrete	$+10$ at the cheese	at the cheese or after $500$ steps
Coin	$64{\times}64$ RGB, side view scrolling with the agent	15 discrete	$+10$ at the coin	at the coin, a hazard or after $1000$ steps
Attribute Maze	$64{\times}64$ RGB, whole $5{\times}5$ grid, black or texture background	15 discrete	$+10$ at the intended object, $-10$ at the other	at either object or after $500$ steps
PushToMarker	$128{\times}128$ RGB from a fixed camera above the table, and the $25$ -d arm state	$8$ -d joint-position deltas, at most $0.1$ rad, at $20$ Hz	dense, bonus when the cube rests within $2.5$ cm of the marker	after $100$ steps, outcome read at the end
LocationReach	same	same	dense, bonus when the gripper is within $3$ cm of the ball	at first contact or after $50$ steps
AttributeTable	same	same	dense, bonus when the cube is touched	at the first touch of either object or after $50$ steps

Appendix

Table 3: Observation, actions, reward and episode of each environment. Procgen has

15

discrete actions, several of which induce the same movement. The arm state is the

9

joint positions, the

9

joint velocities and the

7

-d tool-centre-point pose, its position and quaternion. The quadruped body state is the base linear and angular velocity, the projected gravity and the last command. No environment gives the policy object positions. Arm rewards are dense shaping toward the goal plus a success bonus that exceeds the dense maximum, divided by that bonus. The reward never mentions the proxy and is the same for GM and Non-GM agents.

Figure 12: Shared actor-critic network. Three residual blocks produce

H^{(1)}

,

H^{(2)}

and

H^{(3)}

, which is flattened into

h_{\mathrm{pre}}

and mapped to

h_{\mathrm{fc}}

, read by the policy and value heads. Lower panel: block

3

, ReLU, flatten, linear layer and ReLU, with pre-ReLU and post-ReLU quantities marked.

Figure 13: How far does the goal get in each of the eight environments? The counterpart of Figure 1 for all environments, rows ordered from the shallowest to the deepest point the goal reaches. Arrow width is schematic. Steering is the corrective push at

\beta=1

(Table 14 ) and rewiring the teacher-target readout from

h_{\mathrm{pre}}

(Table 18 ), with oracle targets in Maze and Attribute Maze as in Figure 1 and for PushToMarker after three rounds of DAgger (Table 19 ). Filled bars are recovered and hatched bars partial under the success criterion of Appendix D.2 .

Figure 14: Layerwise linear availability in Maze . Absolute held-out ROC–AUC, from left to right, for cheese presence, cheese bearing relative to the agent, the same cheese bearing on the position-balanced set, in which agent position carries no information about the cheese bearing, and bearing to the reachable top-right corner. Points show the three checkpoint seeds, lines show cohort means, and shaded regions show 95%

t

-intervals across the three checkpoints. Grey dashed: three randomly initialized networks, with a 95%

t

-interval across draws. Dotted: input control, the cheese-cell pixels for presence and agent position for all bearings.

Figure 15: Layerwise linear availability in Coin . Absolute held-out probe performance for coin presence (ROC–AUC) and residualized coin bearing (

R^{2}

). Points show the three checkpoint seeds, lines show cohort means, and shaded regions show 95%

t

-intervals across the three checkpoints. Grey dashed: three randomly initialized networks, with a 95%

t

-interval across draws. Dotted: input control, raw pixels for presence and agent position for bearing (raw pixels give

R^{2}=-0.552

for bearing). Right: coin bearing residualized on agent position as well as on progress, so that agent position decodes none of it.

Figure 16: Layerwise linear availability in Attribute Maze . Absolute held-out ROC–AUC for line presence, line role, yellow presence, and yellow role, for the textured (top) and black-background (bottom) cohorts. Points show the three checkpoint seeds, lines show cohort means, and shaded regions show 95%

t

-intervals across the three checkpoints. The grey dashed line shows the mean of three randomly initialized networks, each evaluated on the frames of all six checkpoints, with a 95%

t

-interval across the three networks.

Target	GM	Non-GM	Init	Input control
Maze
Cheese presence	$0.544\pm 0.01$	$0.997\pm 0.00$	$0.620\pm 0.02$	$0.988$ cheese-cell pixels
Cheese bearing	$0.712\pm 0.01$	$0.918\pm 0.02$	$0.686\pm 0.01$	$0.830$ position
Cheese bearing, corner bearing fixed	$0.598\pm 0.01$	$0.930\pm 0.02$	$0.729\pm 0.04$	$0.703$ position
Cheese bearing, position balanced	$0.583\pm 0.02$	$0.932\pm 0.01$	$0.748\pm 0.05$	$0.481$ position
Corner bearing	$0.963\pm 0.01$	$0.945\pm 0.01$	$0.727\pm 0.02$	$0.959$ position

Appendix

Table 4: Procgen probe results at

h_{\mathrm{pre}}

. Held-out ROC–AUC (Coin bearing

R^{2}

), mean

\pm

SD over three checkpoints or three untrained networks. Init: probe on an untrained network. Input control: as specified per environment.

Figure 17: Goal and proxy decodability in the five robot tasks. Held-out

R^{2}

of a linear probe across network depth, for the goal of every task and the proxy where it varies. Lines are cohort means with 95%

t

-intervals, dashed grey lines randomly initialized networks and dotted lines a probe on block-averaged pixels. Arm sites

H^{(1)}

and

H^{(2)}

are unpooled, and WalkToFlag sites

\bar{H}^{(k)}

are average-pooled to

4{\times}4

cells.

PushToMarker (proxy: table centre, fixed)
	$h_{\mathrm{pre}}$			$h_{\mathrm{fc}}$
Concept	GM	Non-GM	Init	GM	Non-GM	Init	Pixels
Goal: marker position	$0.717\pm 0.09$	$0.896\pm 0.00$	$0.805\pm 0.01$	$0.110\pm 0.08$	$0.819\pm 0.03$	$0.406\pm 0.03$	$0.713$
LocationReach (proxy: training corner, fixed)
Goal: ball position	$0.564\pm 0.03$	$0.924\pm 0.00$	$0.737\pm 0.02$	$0.063\pm 0.09$	$0.902\pm 0.00$	$0.136\pm 0.05$	$0.739$
AttributeTable

Appendix

Table 5: Robot probes at

h_{\mathrm{pre}}

and

h_{\mathrm{fc}}

. Held-out

R^{2}

(AUC where marked). GM, Non-GM: trained agents, mean

\pm

SD over three seeds. Init: three randomly initialized networks, mean

\pm

SD. Pixels: one probe on block-averaged pixels a .

Figure 18: Goal and proxy edits in each environment. One example state per environment, shown as the original, the goal edit (

e=\star

) and, where the environment has one, the proxy edit (

e=p

). The Maze cheese moves to the reachable bottom-left extreme, and both Maze edits are shown on the same state. Solid boxes enclose every pixel the goal edit changes and dashed boxes every pixel the proxy edit changes, including shadows in WalkToFlag .

Environment	Edited feature	Cohort	KL	SF
Relocation edits
Maze	Cheese ( $\star$ )	GM	$0.015\pm 0.01$	$-0.001\pm 0.00$
	Cheese ( $\star$ )	Non-GM	$1.687\pm 0.06$	$+0.357\pm 0.00$
	Corner ( $p$ )	GM	$2.376\pm 0.44$	$+0.377\pm 0.02$
	Corner ( $p$ )	Non-GM	$0.067\pm 0.01$	$+0.017\pm 0.00$
LocationReach	Ball ( $\star$ )	GM	$2.676\pm 2.03$	$-1.764\pm 1.19$

Appendix

Table 6: Behavioral dependence under matched world edits. KL measures immediate policy reaction and SF measures whether that reaction follows the edited feature.

\star

marks an edit of the goal and

p

an edit of the proxy. Values are mean

\pm

sample standard deviation across three seeds, each the mean over the evaluation states. Robot SF is in mm for the arm tasks and in command units for WalkToFlag , and robot rows are compared only within their environment. Bold marks the larger KL of each GM and Non-GM pair.

Figure 19: World-edit responses of GM agents relative to Non-GM agents. Each row divides the GM response to one edit by the Non-GM response in the same environment, filled for goal edits and open for proxy edits. Large markers divide the cohort means and small dots each GM seed by the Non-GM mean. Dashed lines mark a response as strong as in Non-GM agents, and the dotted line at KL

0.05

separates the inert location targets from all other edits. n/a: not every Non-GM seed turns toward the edited object.

Environment	Site	Direction	GM	Non-GM	init
Maze	$h_{\mathrm{pre}}$	$\Delta_{\star}$	$-0.001\pm 0.00$	$-0.271\pm 0.07$	$0.000\pm 0.00$
	$h_{\mathrm{fc}}$	$\Delta_{\star}$	$-0.001\pm 0.00$	$-0.269\pm 0.02$	$0.000\pm 0.00$
Coin	$h_{\mathrm{pre}}$	$\Delta_{\star}$	$-0.009\pm 0.02$	$+0.072\pm 0.03$	$0.000\pm 0.00$
	$h_{\mathrm{fc}}$	$\Delta_{\star}$	$-0.010\pm 0.02$	$+0.139\pm 0.05$	$0.000\pm 0.00$
Attribute Maze	$h_{\mathrm{pre}}$	$\Delta_{\star}$	$-0.718\pm 0.24$	$-0.704\pm 0.06$	$0.000\pm 0.00$
	$h_{\mathrm{pre}}$	$\Delta_{p}$	$+1.033\pm 0.28$	$+0.068\pm 0.05$	$0.000\pm 0.00$

Appendix

Table 7: Signed target-selection effects. Values are mean

\pm

SD across three checkpoints per cohort, each summarized over the evaluation states of Table 8 . First-order change of the target contrast along the world-induced displacement (Definition 8 ). Sign key: a policy that follows the edit reads negative for the Maze cheese relocation and the Attribute Maze shape swap, and positive for the Attribute Maze colour swap and the Coin relocation.

Figure 20: Signed effects of world-induced representation changes on target selection. Small points show evaluation states, densities summarize their distribution, and diamonds show the three seed-level summaries. The top row reports

h_{\mathrm{pre}}

and the bottom row

h_{\mathrm{fc}}

. Randomly initialized networks are shown at zero for reference. Attribute Maze panels show black-background policies, separately for the shape and the colour swap. Signs follow the key in Table 7 .

			$h_{\mathrm{pre}}$			$h_{\mathrm{fc}}$
Environment	Edit	Cohort	$Q$	$G$	$D_{F}$	$Q$	$G$	$D_{F}$
Maze	Cheese	GM	$0.001\pm 0.00$	$1.641\pm 0.34$	$0.001\pm 0.00$	$0.001\pm 0.00$	$0.727\pm 0.25$	$0.002\pm 0.00$
		Non-GM	$1.650\pm 0.08$	$4.800\pm 0.46$	$0.348\pm 0.01$	$1.689\pm 0.05$	$1.141\pm 0.39$	$1.667\pm 0.44$
LocationReach	Ball	GM	$4.096\pm 3.44$	$9.252\pm 4.06$	$0.373\pm 0.24$	$3.803\pm 3.06$	$1.962\pm 0.28$	$2.167\pm 1.80$
		Non-GM	$235.849\pm 85.03$	$38.881\pm 10.03$	$6.317\pm 1.06$	$162.709\pm 47.43$	$3.822\pm 0.25$	$42.961\pm 8.70$
PushToMarker	Marker	GM	$19.813\pm 7.06$	$6.206\pm 0.35$	$3.278\pm 1.19$	$16.957\pm 5.08$	$2.099\pm 0.39$	$8.852\pm 3.88$

Appendix

Table 8: World-edit Fisher sensitivity in all eight environments. Values are mean

\pm

SD across three checkpoints per cohort, each summarized by its median over the evaluation states. Environments are in the order of Table 6 , and the second edit of an environment edits the proxy. Robot

Q

and

D_{F}

are in units of each policy’s

\sigma

and are compared only within a task. Randomly initialized networks have

Q

and

D_{F}

of

0.000

and

G

between

0.729

and

1.035

at

h_{\mathrm{pre}}

and between

0.785

and

1.081

at

h_{\mathrm{fc}}

. Bold marks the larger

Q

of each GM and Non-GM pair.

Figure 21: Fisher sensitivity of GM agents relative to Non-GM agents. GM value of

Q

,

D_{F}

and

G

at

h_{\mathrm{fc}}

divided by the Non-GM value for the same edit, filled for goal and open for proxy edits. Large markers divide cohort means, small dots each GM seed by the Non-GM mean, and dashed lines mark a value as large as in Non-GM agents. Rows are in the order of Table 6 .

		Policy sensitivity $10^{3}\operatorname{tr}(F)/d$		Displacement norm $\\|\Delta\\|_{2}$
Environment	Site	GM	Non-GM	GM	Non-GM
Maze	$h_{\mathrm{pre}}$	$0.515\pm 0.13$	$1.192\pm 0.36$	$1.212\pm 0.07$	$18.052\pm 3.10$
	$h_{\mathrm{fc}}$	$4.328\pm 0.41$	$5.393\pm 0.11$	$0.849\pm 0.04$	$18.591\pm 2.64$
LocationReach	$h_{\mathrm{pre}}$	$24.697\pm 5.55$	$15.383\pm 2.82$	$3.844\pm 1.87$	$20.375\pm 2.21$
	$h_{\mathrm{fc}}$	$346.013\pm 42.81$	$191.409\pm 12.81$	$2.314\pm 1.43$	$14.989\pm 1.89$
PushToMarker	$h_{\mathrm{pre}}$	$212.705\pm 32.12$	$566.725\pm 299.47$	$3.892\pm 0.83$	$5.926\pm 1.02$

Appendix

Table 9: The two factors of

D_{F}

. Policy sensitivity is the mean Fisher sensitivity

\operatorname{tr}(F)/d

at the unedited state, which sets how strongly the policy reads an average direction. Displacement norm is the Euclidean length of the goal edit’s displacement. Each is the median over the states of Table 8 for one checkpoint, then mean

\pm

SD over checkpoints. Attribute Maze uses the shape swap, and robot values are in units of

\sigma^{-2}

and compared only within a task.

Figure 22: GM agents go to the current top-right extreme in Maze and to the training corner in LocationReach . (a to d) Greedy GM rollouts unedited, with the corner opened, with the offset extreme and with a bottom-left corridor, on a selected example: all three GM checkpoints agree on this

15

-cell maze (level

1000140

), which is one of

14

of

92

medium or large levels where they do. (e, f) Two LocationReach off-path episodes, with the GM and Non-GM grippers from the same start. (g) GM gripper positions over all

1{,}500

off-path episodes. (h) Closest approach of each episode to the corner and to the ball.

Cohort	Corner progress	Cheese progress	Naive only	Neutral
GM	$0.948\pm 0.02$	$0.018\pm 0.01$	$0.020\pm 0.01$	$0.014\pm 0.01$
Non-GM	$0.173\pm 0.05$	$0.498\pm 0.02$	$0.067\pm 0.01$	$0.262\pm 0.03$

Appendix

Table 10: Greedy action class on the

168

Maze detour states. Mean

\pm

SD across three seeds.

Figure 23: In LocationReach and PushToMarker the target moves with the camera image, not with the arm base. (a to d, f to i) Where the gripper arrives in LocationReach and where the cube comes to rest in PushToMarker , over all episodes of the three seeds per cohort, unedited, with the arm base shifted (Base), with the camera shifted (Camera) and with only the policy image translated (Image), the table corner or centre ringed and the point that the old posture or old pixel implies drawn as a diamond. (e, j) Fraction of episodes that reach the table point (filled) or the other point (open), where PushToMarker GM agents play without the marker except under the camera shift, which draws it at the real centre for both cohorts.

Rollouts
	GM	Non-GM	Init
Separable, cube to the centre	$0.980\pm 0.01$	$0.000\pm 0.00$	–
Separable, cube onto the marker	$0.000\pm 0.00$	$0.935\pm 0.03$	–
Separable, path pull to the marker in cm	$0.050\pm 0.03$	$10.504\pm 0.50$	–
Conflict, cube onto the decoupled disc	$0.000\pm 0.00$	$0.895\pm 0.04$	–
Conflict, pause over the marker, drawn	$0.058\pm 0.02$	$0.867\pm 0.03$	–

Appendix

Table 11: What organizes PushToMarker and what the marker reaction is. Mean

\pm

SD across three seeds of per-seed means, the planned step a per-seed median. Rollouts are fractions of episodes on the separable layouts or the conflict scenes, and have no init baseline because randomly initialized policies never reach the cube. The path pull is how much closer the cube passes to the marker when it is drawn than when it is hidden. Single-frame rows use the world-edit bank with the gripper at its start pose unless the block says otherwise.

Figure 24: The PushToMarker evaluation configurations. (a to d) Rollout sets top-down in the camera orientation, with the cube paths of all three GM and Non-GM seeds on one separable layout with the marker drawn and hidden and on one conflict scene and its decoupled conflict, each chosen among scenes on which the three seeds of each cohort agree. Each pair of panels is centred on its scene at one common scale. (e to h) Rendered camera frames of the single-frame tests on that conflict scene, cropped to the scene: the marker move from its old place (dotted) to its new one, a green disc at the conflict position and off the path, and the gripper held above the cube.

Figure 25: Where the coin’s influence goes in Coin . (a, b) Change in probability of each action group and its share of the total variation divided by its prior mass when the coin moves from the cell right of the agent to the cell left of it, on

593

of the

600

conflict-bank states, with groups

A_{\star}

(LEFT, LEFT+DOWN, LEFT+UP),

A_{\rightarrow}

(RIGHT, RIGHT+DOWN, RIGHT+UP), jump (UP) and other (DOWN, no-op and unbound keys). (c, d) The same two quantities for

A_{\star}

with the coin

1

to

4

cells left of the agent, on the

572

states where the coin fits and is visible at every distance.

Figure 26: Example saliency maps in Coin . Rows are the first three conflict-bank states whose replay reproduces the bank pose and that show the end wall, with no selection on saliency. Left: observation with the agent (white), the coin one cell to its left (yellow) and the end-wall block (red). Middle and right:

\log_{10}

saliency relative to the image mean (colour range

-1

to

2

) over the greyscale frame, for one GM and one Non-GM checkpoint.

Figure 27: Goal and proxy swaps in the four competition environments. Each environment has two panels, the goal swap (left) and the proxy swap (right), with

S_{\mathrm{fav}}

and

S_{\mathrm{comp}}

in the units of Appendices C.4.4 and C.4.5 . The top row shows Attribute Maze on black and textured backgrounds, the middle row AttributeTable and SortZones , and the bottom row WalkToFlag . Faint dots are states, diamonds seed means and open circles the cohort mean with 95%

t

-intervals across three seeds. Shading marks the push–pull quadrant. States beyond the axes are drawn at the edge, which clips

1\%

of states at each end (

5\%

in WalkToFlag ). AttributeTable is read on the strict bank and SortZones at decision frames.

Environment	Cohort	Swap	$S_{\mathrm{fav}}$	$S_{\mathrm{comp}}$	$\Delta\ell$
Attribute Maze black	GM	Shape	$1.368\pm 0.25$	$-1.721\pm 0.35$	$3.089\pm 0.60$
	GM	Colour	$2.001\pm 0.19$	$-2.354\pm 0.29$	$4.355\pm 0.48$
	Non-GM	Shape	$3.740\pm 0.37$	$-3.910\pm 0.21$	$7.650\pm 0.57$
	Non-GM	Colour	$0.192\pm 0.11$	$-0.362\pm 0.27$	$0.555\pm 0.38$
Attribute Maze texture	GM	Shape	$1.037\pm 0.66$	$-1.176\pm 0.77$	$2.213\pm 1.43$
	GM	Colour	$3.053\pm 0.46$	$-3.189\pm 0.40$	$6.242\pm 0.86$

Appendix

Table 12: Cue competition in the four competition environments. Values are mean

\pm

sample standard deviation across three seeds. Units differ between environments (Appendices C.4.4 and C.4.5 ). AttributeTable is read on the strict bank and SortZones at decision frames.

\Delta\ell=S_{\mathrm{fav}}-S_{\mathrm{comp}}

measures the resulting shift in relative target preference.

Figure 28: Object competition in WalkToFlag . (a) Mean pull

f_{\star}

of the flag and

f_{p}

of the tower, the shift of the command mean caused by each object along the direction to the command that walks to it, in policy standard deviations, with dots for seeds. (b) Fraction of GM states pulled toward each object (

f_{o}>\tau

) in four groups, the world-edit bank split into states with the flag or the tower more than

0.3

m nearer (near), the forced-view conflict bank (flag much nearer) and the far bank (tower much nearer). (c) Fraction of states pulled toward the flag, away from it, toward the tower and toward both at once, and the fraction on which both nuisance shifts pass the threshold together (null). (d) Tower share

P_{p}/(P_{p}+P_{\star})

of the removal pull against the distance gap

d_{\star}-d_{p}

, negative where the flag is nearer. The dotted step is the share that walking to the nearer object would give. Means with

95\%

t

-intervals across three seeds.

Figure 29: Within-state cue geometry in the four competition environments (

h_{\mathrm{fc}}

). Euclidean cosine

\cos_{E}

and Fisher cosine

\cos_{F}

between

\Delta_{\star}(s)

and

\Delta_{p}(s)

, averaged over states. In Attribute Maze circles are the black-background policies and diamonds the textured-background policies, each on its own background. Mean and 95%

t

-interval over three seeds, small points are seeds.

Environment	Cohort	Position	$\kappa_{E}$ same	$\kappa_{F}$ same	$\cos_{E}(\sigma d_{\mathrm{comp}},$ $u_{\mathrm{comp}})$	$\cos_{F}(\sigma d_{\mathrm{comp}},$ $u_{\mathrm{comp}})$
Attribute Maze	GM	$h_{\mathrm{pre}}$	$0.354\pm 0.02$	$0.938\pm 0.04$	$0.521\pm 0.02$	$0.961\pm 0.03$
black	GM	$h_{\mathrm{fc}}$	$0.589\pm 0.02$	$0.942\pm 0.05$	$0.709\pm 0.00$	$0.959\pm 0.03$
	Non-GM	$h_{\mathrm{pre}}$	$0.284\pm 0.02$	$0.844\pm 0.05$	$0.468\pm 0.02$	$0.914\pm 0.05$
	Non-GM	$h_{\mathrm{fc}}$	$0.532\pm 0.04$	$0.952\pm 0.02$	$0.679\pm 0.05$	$0.970\pm 0.02$
	Init	$h_{\mathrm{pre}}$	$0.173\pm 0.02$	$0.160\pm 0.03$	$0.357\pm 0.03$	$0.330\pm 0.06$
	Init	$h_{\mathrm{fc}}$	$0.177\pm 0.03$	$0.148\pm 0.03$	$0.364\pm 0.03$	$0.301\pm 0.06$

Appendix

Table 13: Cross-state competition geometry. The first two columns give mean similarity over pairs of states with the same sign

\sigma

. The last two give the alignment of held-out competition directions with the shared axis

u_{\mathrm{comp}}

fit on the remaining states. Entries are cohort mean

\pm

SD across three policies.

\kappa_{E}

and

\cos_{E}

compare raw activations,

\kappa_{F}

and

\cos_{F}

compare them in the policy’s Fisher metric. Attribute Maze has two blocks with the same rows, the black-background and the textured-background policies, each on its own background at

h_{\mathrm{pre}}

and

h_{\mathrm{fc}}

(initialized networks shown on the same background). Robots at

h_{\mathrm{fc}}

on the banks of Appendix E.2 . The held-out

\cos_{F}

uses the Fisher metric at each held-out state.

Figure 30: The policy compresses a rich competition geometry onto one signed axis ( Attribute Maze ). Pairwise similarity between state-local competition directions across the

196

conflict-bank states, averaged over the three GM policies of each background, black (left four panels) and textured (right four panels), each on its own background. Columns show

h_{\mathrm{pre}}

and

h_{\mathrm{fc}}

for each background, rows show

\kappa_{E}

and

\kappa_{F}

. States are grouped by whether the line candidate lies along UP or along RIGHT. Under the swap construction the cross-sign blocks are antisymmetric by construction, so the informative structure is the within-sign alignment.

Figure 31: Competition directions across states in the robot tasks (

h_{\mathrm{fc}}

, fit states of the strict AttributeTable bank, the SortZones decision-frame bank and the WalkToFlag world-edit bank). Pairwise

\kappa_{E}

and

\kappa_{F}

of

d_{\mathrm{comp}}

, averaged over three seeds, for GM (top) and Non-GM (bottom) agents. States are grouped by conflict orientation. Unlike in Attribute Maze , mirrored states are separate scenes, so the cross-sign blocks are not antisymmetric by construction.

Figure 32: Tug of war on the shared axis in four environments (

h_{\mathrm{fc}}

, held-out states). Each small point is one state of one policy, placed by how far the goal-cue swap moves it along

u_{\mathrm{comp}}

toward the goal cue’s candidate (goal pull,

x

) and how far the proxy-cue swap moves it toward the proxy cue’s candidate (proxy pull,

y

), both in the Fisher metric. GM and Non-GM agents are overlaid, large markers are the mean and 95%

t

-interval over three seeds. Above the dashed identity line the proxy pulls harder, in the upper-right quadrant both cues pull toward their own candidates, and left of the vertical axis the goal cue pushes toward the proxy’s candidate. Axes are linear and scaled per environment.

Figure 33: GM policies weight the proxy cue more strongly on the shared competition axis ( Attribute Maze ,

h_{\mathrm{fc}}

, black background). (a) Goal pull against proxy pull per held-out state (small points) with cohort means and 95%

t

-intervals. States above the identity line are pulled harder by the proxy. Axes are symmetric-logarithmic. (b) Fraction of GM rollouts reaching the goal in five quantile bins of the base position

t

, with Wilson 95% intervals (all

196

states, timeouts excluded). (c) AUC for predicting a goal versus a proxy outcome from the goal pull, the proxy pull, or both.

Figure 34: In every environment the proxy outweighs the goal in GM agents and not in Non-GM agents, while the rest of the geometry differs between environments (

h_{\mathrm{fc}}

). Each panel is one component of the competition. Top left, Fisher cosine of the two cue edits within a state (negative means opposed). Top right, same-sign

\kappa_{F}

(how much the competition lies on one axis shared across states). Bottom left, fraction of states in which both cues pull toward their own candidates. Bottom right, median per-state ratio of proxy pull to goal pull (log scale, above

1

the proxy pulls harder). Grey bars are initialized networks. Mean and 95%

t

-interval over three seeds, small points are seeds.

Figure 35: State-local steering, ordered by where the goal is lost. Goal reach under corrective, reversed and norm-matched random pushes at

h_{\mathrm{fc}}

against the dose

\beta

, GM (upper row of each block) and Non-GM (lower row). AM is Attribute Maze on the black background, the textured background is in Table 14 , and WalkToFlag uses the native set. Grey bands show the unsteered baseline. Mean with 95%

t

-interval over three checkpoints.

Environment	Cohort	Baseline	Corrective	Anticorrective	Random
Maze	GM	$0.001\pm 0.00$	$0.002\pm 0.00$	$0.000\pm 0.00$	$0.001\pm 0.00$
	Non-GM	$0.567\pm 0.03$	$0.569\pm 0.03$	$0.002\pm 0.00$	$0.340\pm 0.04$
LocationReach	GM	$0.000\pm 0.00$	$0.000\pm 0.00$	$0.000\pm 0.00$	$0.000\pm 0.00$
	Non-GM	$1.000\pm 0.00$	$0.138\pm 0.10$	$0.000\pm 0.00$	$0.514\pm 0.05$
PushToMarker	GM	$0.000\pm 0.00$	$0.000\pm 0.00$	$0.000\pm 0.00$	$0.000\pm 0.00$
	Non-GM	$0.903\pm 0.02$	$0.375\pm 0.10$	$0.000\pm 0.00$	$0.582\pm 0.10$

Appendix

Table 14: State-local steering at

\beta=1

. Intended-target reach, mean

\pm

SD across three checkpoints. Each entry is read against the unsteered baseline in its own row. Blocks follow Figure 35 . WalkToFlag rows use the native set. Bold marks corrective values whose 95%

t

-interval lies above those of both the unsteered baseline and the random push.

Environment	Push	Goal cue at $u$	Removed	Moved to $v$
Attribute Maze	none	$0.160\pm 0.06$	$0.007\pm 0.01$	$0.000\pm 0.00$
(black)	local, $\beta=1$	$\mathbf{0.953\pm 0.05}$	$\mathbf{0.913\pm 0.03}$	$\mathbf{0.880\pm 0.07}$
	reversed, $\beta=1$	$0.007\pm 0.01$	$0.000\pm 0.00$	$0.000\pm 0.00$
	random, $\beta=1$	$0.171\pm 0.06$	$0.024\pm 0.01$	$0.031\pm 0.01$
	shared axis, $\beta=0.25$	$0.587\pm 0.11$	$0.093\pm 0.04$	$0.147\pm 0.06$
AttributeTable	none	$0.052\pm 0.02$	$0.023\pm 0.02$	$0.000\pm 0.00$

Appendix

Table 15: Steering redirects the policy toward a location. Reach of the goal location

u

by GM policies, mean

\pm

SD across three checkpoints, with the goal cue in place, removed, or moved to the lure location

v

. The push is built from the original scene in all three. Bold marks the local push, which keeps reach of

u

high with the goal cue removed or moved. The robot rows are a separate set of rollouts on the same

200

scenes as Table 14 , so their cue-in-place values differ from it by at most

0.012

, and their shared axis is fit on the world-edit bank (cosine

0.935

with the strict-bank axis of Table 16 ).

Figure 36: Compressing local steering onto the shared axis. Goal reach of GM (top) and Non-GM (bottom) policies against the signed dose, negative doses reversed. All arms are fixed pushes computed at the start state. The AttributeTable axis is fit on the strict bank and the SortZones axis on the decision-frame bank. WalkToFlag is shown on the forced-view set. On the native set no fixed push moves reach in either cohort (see text). Shaded regions mark

|\beta|>0.25

. Mean with 95%

t

-interval over three checkpoints.

Environment	Unsteered	Local, fixed at $s_{0}$	Sign-gated	Fisher- projected	Unsigned	Unsigned, reversed	Local, rebuilt
AM, black	$0.160\pm 0.06$	$0.560\pm 0.05$	$\mathbf{0.587\pm 0.11}$	$\mathbf{0.560\pm 0.04}$	$0.300\pm 0.08$	$0.307\pm 0.14$	$0.640\pm 0.07$
AM, texture	$0.040\pm 0.07$	$0.173\pm 0.27$	$0.200\pm 0.25$	$0.180\pm 0.29$	$0.093\pm 0.09$	$0.107\pm 0.15$	$0.167\pm 0.24$
AT	$0.040\pm 0.02$	$0.147\pm 0.02$	$\mathbf{0.300\pm 0.04}$	$\mathbf{0.108\pm 0.02}$	$\mathbf{0.165\pm 0.03}$	$0.140\pm 0.03$	$0.235\pm 0.03$
SZ	$0.000\pm 0.00$	$0.000\pm 0.00$	$0.000\pm 0.00$	$0.000\pm 0.00$	$0.000\pm 0.00$	$0.000\pm 0.00$	$0.030\pm 0.03$
WTF	$0.378\pm 0.08$	$0.466\pm 0.01$	$0.479\pm 0.03$	$0.435\pm 0.03$	$0.341\pm 0.15$	$0.440\pm 0.03$	$0.544\pm 0.01$

Appendix

Table 16: Compression test at

\beta=0.25

. Intended-goal reach of GM policies, mean

\pm

SD across three checkpoints. AM, AT, SZ and WTF are Attribute Maze , AttributeTable , SortZones and WalkToFlag (forced-view set). In the texture row, the robustness variant of Attribute Maze , one of the three checkpoints carries the mean and the variation across checkpoints is as large as the effects. Bold marks compressed pushes toward the goal (sign-gated, Fisher-projected and unsigned) whose mean

\pm

SD range lies entirely above that of the unsteered value.

Figure 37: Frozen-representation rewiring. The diagram shows the Procgen IMPALA network. The robot networks follow the same scheme with a Nature CNN trunk instead (Appendix B.1 ). What the figure shows is which downstream part each rewire relearns, not the trunk, which is frozen in every arm. Grey components are frozen and pink components are freshly initialized and trained. A , the head rewire retains the learned

h_{\mathrm{fc}}

representation and learns only a new policy head. B , the fixed random remix replaces the learned

h_{\mathrm{pre}}\!\to h_{\mathrm{fc}}

transformation by a fixed random

2048\!\to 256

map and learns only the policy head. C , the spatial rewire starts from

h_{\mathrm{pre}}

and relearns both the intermediate transformation and policy head. The value head is not used for behavior cloning.

(i) Goal lost in the representation
	Method works?		Recovery successful?
Environment	$h_{\mathrm{pre}}$	$h_{\mathrm{fc}}$	$h_{\mathrm{pre}}$	$h_{\mathrm{fc}}$
Maze	no ( $0.80$ )	no ( $0.64$ )	none	none
with DAgger	yes ( $1.01$ )	no ( $0.78$ )	none	none
LocationReach	yes ( $0.92$ )	yes ( $0.97$ )	none	none
(ii) Goal reaches the policy but is not used

Appendix

Table 17: Rewiring at a glance. The method works when the Non-GM rewire reaches at least

90\%

of its native policy (its fraction in brackets) or the GM rewire is recovered, since a recovery shows that the method works. Recovery is that of the GM rewire under the success criterion, with teacher targets, and is in bold where the method works. Oracle targets in Maze and Attribute Maze give the same verdicts. The DAgger rows use three rounds of on-policy relabeling. Values in Tables 18 and 19 .

		Native		Rewire		Baselines
Environment	Site	GM	Non-GM	GM	Non-GM	Random trunk	Chance
(i) Goal lost in the representation
Maze	$h_{\mathrm{pre}}$	$0.219\pm 0.02$	$0.347\pm 0.02$	$0.089\pm 0.01$	$0.277\pm 0.02^{g}$	$0.016\pm 0.01^{b,e}$	$0.226^{a}$
	remix			$0.038\pm 0.02$	$0.167\pm 0.03$
	$h_{\mathrm{fc}}$			$0.048\pm 0.00$	$0.222\pm 0.07^{g}$	$0.008\pm 0.00^{b,e}$
oracle	$h_{\mathrm{pre}}$			$0.116\pm 0.02$	$0.308\pm 0.04^{g}$	$0.018\pm 0.00^{b}$	$0.226^{a}$

Appendix

Table 18: Closed-loop goal reach after rewiring, ordered by where the goal is lost. Goal reach

\hat{J}^{\star}

, mean

\pm

SD across three checkpoints or random trunks, with teacher targets unless marked oracle or DAgger. Native is the untouched policy, remix the fixed random remix (5), and the random trunk (3) is rewired from the same site as the row. Chance is the goal-blind policy (4). Bold GM rewires are recovered and underlined ones partial under the success criterion. a Uniform random walk, sampled actions. b Freezes under greedy actions. c Does not fit the cloning targets. d Two of three checkpoints. e Cloned on a rebuilt copy of the bank, on which the rewires reproduce within

0.02

. f

0.5\,D

at

D=0.690

, one estimate. g Non-GM rewire below

90\%

of its native policy. h Three rounds of DAgger (Table 19 ), oracle targets in Maze .

Figure 38: Closed-loop goal reach after rewiring, ordered by where the goal is lost. Goal reach

\hat{J}^{\star}

of the GM (red) and Non-GM (teal) rewires from

h_{\mathrm{pre}}

and

h_{\mathrm{fc}}

, of the fixed random remix (open bars) and of the random trunk (grey), with oracle targets in Maze and Attribute Maze and teacher targets in the other six environments (Table 18 ). Bars are means, whiskers

95\%

t

-intervals across the three checkpoints or random trunks, and dots individual checkpoints where they are available. Dashed lines mark the native GM policy, with the band below it, and the native Non-GM policy, with the band above it. The dotted line is chance. The random trunks of Maze , Attribute Maze and Coin at

h_{\mathrm{fc}}

are cloned on the rebuilt bank (Table 18 , note

e

).

Maze , oracle targets
Site	Cohort	Native	Round 0	DAgger	Matched
$h_{\mathrm{pre}}$	GM	$0.219\pm 0.02$	$0.112\pm 0.02$	$0.128\pm 0.02$	$0.118\pm 0.01$
	Non-GM	$0.347\pm 0.02$	$0.187\pm 0.03$	$0.351\pm 0.03$	$0.357\pm 0.03$
$h_{\mathrm{fc}}$	GM	$0.219\pm 0.02$	$0.114\pm 0.01$	$0.122\pm 0.02$	$0.098\pm 0.01$
	Non-GM	$0.347\pm 0.02$	$0.246\pm 0.02$	$0.271\pm 0.03$	$0.111\pm 0.03$
PushToMarker , teacher targets

Appendix

Table 19: Rewiring with DAgger. Goal reach after three rounds of on-policy relabeling (DAgger) or of matched extra training, with oracle targets in Maze (

300

held-out levels) and teacher targets in PushToMarker (

512

conflict scenes). Mean

\pm

SD across three checkpoints or random trunks. Round 0 is the rewire before the added rounds. In Maze it is trained on states visited by the freshly initialized downstream policy, so it differs from Table 18 , and both arms then share a base set along the native agent’s trajectories. Underlined GM cells are partial under the success criterion, and none is recovered.

Figure 39: Normalized rewire recovery.

(\text{GM}-\text{floor})/(\text{Non-GM}-\text{floor})

for the rewires from

h_{\mathrm{pre}}

(red) and

h_{\mathrm{fc}}

(teal), with the floor the higher of random trunk and chance, for teacher-target behavior cloning (circles) and three rounds of DAgger (diamonds, oracle targets in Maze , teacher targets in PushToMarker ), with

95\%

delta-method intervals across three checkpoints per arm. Open markers flag cells where the Non-GM rewire does not clearly exceed the floor, so the ratio is unstable, n.d. marks a Non-GM rewire at or below the floor, and values beyond the axis are printed at its edge.

Figure 40: Rewiring from every layer. Goal reach

\hat{J}^{\star}

of the GM (red) and Non-GM (teal) rewires from each layer of the frozen trunk, means with

95\%

t

-intervals across three checkpoints, with the native policies as dashed lines and the goal probe of Appendix C.1 at the same layer as open diamonds (goal-position

R^{2}

in the arm tasks, cheese-bearing AUC, coin-bearing

R^{2}

and line-presence AUC in Procgen).

(i) Goal lost in the representation
Task	Cohort	Standard	Goal masked	Goal removed	Chance
Maze †	GM	$0.098\pm 0.01$	$0.102\pm 0.00$	$0.113\pm 0.01$	$0.226$
	Non-GM	$0.291\pm 0.02$	$0.069\pm 0.06$	$0.051\pm 0.02$
LocationReach	GM	$0.373\pm 0.08$	$0.083\pm 0.04$	$0.027\pm 0.02$	$0.096\pm 0.03$
	Non-GM	$0.921\pm 0.05$	$0.023\pm 0.01$	$0.010\pm 0.00$
(ii) Goal reaches the policy but is not used

Appendix

Table 20: Goal-removed controls for the rewires from

h_{\mathrm{pre}}

, ordered as in Table 18 . Goal reach

\hat{J}^{\star}

, mean

\pm

SD across three checkpoints. Standard is the rewire of Table 18 (for Attribute Maze , the standard rewire rerun on the rebuilt bank). Goal masked is a rewire cloned with the goal cue masked in every frame. Goal removed is the standard rewire evaluated with the goal removed ( LocationReach , PushToMarker , WalkToFlag ) or replaced by the proxy’s cue ( AttributeTable , SortZones ), counting arrivals at the former goal location. Chance as in Table 18 . † Teacher targets, rerun on rebuilt banks, with the standard and goal-removed evaluations from the same readouts. The goal is removed by deleting the cheese ( Maze ), hiding the coin ( Coin ) or drawing the goal line as a red gem ( Attribute Maze ).

Maze
Bank	States	Probes	World edits	Fisher	Role	Geometry	Steering	Rewiring
Appendix		C.1	C.2	C.3	C.4	C.5	D.1	D.2
Probe bank	$2{,}787$	✓
Conflict bank, shared sample	$128$		✓	✓
Conflict bank, steering sample	$48$						✓
Fork sample	$320$						✓

Appendix

Table 21: Evaluation banks and the analyses that use them. The second header row gives the appendix of each analysis, and the entries of Appendices E.2 to E.5 describe each bank. Robot cloning banks count frames, and the notes below the table give the subsets that single analyses use.

Figure 41: One state from each robot bank. Green rings mark the intended goal and orange rings the proxy, dashed where the proxy is a place rather than an object. The off-path bank, the control-layout set and the cloning banks store no frames. The WalkToFlag far bank is not shown. The SortZones conflict-test panel is the first scene of the robot conflict-test samples (Appendix E.3 ).

Figure 42: One state from each conflict bank and from the diversified bank. From left to right, the shared and steering samples of the Maze conflict bank, the Coin conflict bank, the Attribute Maze conflict bank and the diversified bank. Green arrows are progress actions toward the intended goal, orange arrows toward the proxy (Definition 10 ). Circles mark the cheese and the top-right target. The Coin panel draws the coin at its edited placement left of the agent, and the dashed circle marks its base placement to the right. Each panel shows a single state. Attribute Maze panels are schematics of the stored state (red line, yellow gem, brown walls). The Maze fork sample and detour bank are not shown.

Concept	Label	States	Split	Sites
Maze
Cheese presence	cheese present or removed	$2{,}787$	level-grouped	$h^{(1)}$ – $h_{\mathrm{fc}}$
Cheese bearing	cheese left or right of the agent	$1{,}987$	level-grouped	$h^{(1)}$ – $h_{\mathrm{fc}}$
Corner bearing	top-right extreme left or right of the agent	$2{,}787$	level-grouped	$h^{(1)}$ – $h_{\mathrm{fc}}$
Coin
Coin presence	coin visible in the frame	$320$	level-grouped	$h^{(1)}$ – $h_{\mathrm{fc}}$

Appendix

Table 22: Probe banks. Binary labels are scored by held-out ROC–AUC, and Coin bearing by held-out

R^{2}

. Bearing is taken relative to the agent. Sites run over

h^{(1)},h^{(2)},h^{(3)},h_{\mathrm{pre}},h_{\mathrm{fc}}

(Appendix C.1 ). Cheese bearing uses the

1{,}987

states that hold the cheese. a The residualization is fit on the training partition only. b Per checkpoint and rollout seed,

3{,}072

training and

1{,}536

held-out states.

Concept	Label	States	Split	Sites
PushToMarker
Marker position	marker $(x,y)$ on the table	$3{,}072$	seeds, start poses a	$H^{(1)}$ – $h_{\mathrm{fc}}$
LocationReach
Ball position	green ball $(x,y)$ on the table	$3{,}072$	seeds, start poses a	$H^{(1)}$ – $h_{\mathrm{fc}}$
AttributeTable
Cube position	cube $(x,y)$ on the table	$3{,}072$	seeds	$H^{(1)}$ – $h_{\mathrm{fc}}$

Appendix

Table 23: Robot probe banks. Positions and bearings are scored by held-out

R^{2}

(positions averaged over

x

and

y

), and binary labels by held-out ROC–AUC. States count training and held-out frames together, about two thirds of them for training. Training and held-out frames are drawn with separate sampler seeds. Arm sites are

H^{(1)},H^{(2)},h_{\mathrm{pre}},h_{\mathrm{fc}}

with unpooled maps, and WalkToFlag sites are

\bar{H}^{(1)},\bar{H}^{(2)},\bar{H}^{(3)},h_{\mathrm{pre}},h_{\mathrm{fc}}

with maps average-pooled to

4{\times}4

cells (Appendix C.1 ). a Training and held-out scenes also start the arm from disjoint halves of the pool of arm start poses.

Figure 43: Goal pursuit in all eight environments. Each point is one checkpoint at one training step, placed by decisiveness

D

and conditional intended-goal preference

\Pr(\mathrm{goal}\mid\mathrm{dec})

on

\Lambda_{\mathrm{OOD}}

, with 95% Wilson intervals on both axes for cohort and published checkpoints. The dashed line marks

\Pr(\mathrm{goal}\mid\mathrm{dec})=0.5

, the Attribute Maze check. Grey points are scored checkpoints outside the cohorts, and open circles pass every condition of their label but are not carried. Cohort checkpoints use the

N=500

evaluations of Table 24 . Points outside the cohorts and the published checkpoints come from earlier evaluations (

80

to

1{,}000

episodes).

Env.	Cohort	Checkpoints	$N$	$\min\hat{C}_{e}$	worst $\hat{\rho}$	Pass
Maze	GM	1080–1082	500	$0.932$	$0.262$	3/3
Maze	Non-GM	1090–1092	500	$0.928$	$0.002$	3/3
Coin	GM	6033@20M, 6037@120M, 6051@55M	500	$0.780$	$0.310$	3/3
Coin	Non-GM	7001–7003	500	$0.908$	$0.000$	3/3
Attr. Maze	GM, black	6313, 6315, 6317	500	$0.998$	$0.490$	3/3
	Non-GM, black	4371–4373	500	$0.984$	$0.000$	3/3

Appendix

Table 24: Behavioral cohorts , worst checkpoint per cohort. In Procgen

\hat{C}_{e}

uses sampled actions and

\hat{\rho}

greedy actions, and on the robots both use deterministic actions.

N

is the number of

\Lambda_{\mathrm{OOD}}

episodes per checkpoint,

\min\hat{C}_{e}

the lowest capability in the cohort, and worst

\hat{\rho}

the lowest proxy rate for GM and the highest for Non-GM. Pass counts checkpoints that meet the definition of their label (Section F.2 ). Every checkpoint is the final save of its run except the Coin GM checkpoints, which are selected by training step within their 200M-step runs (Appendix H , Table 26 ).

Figure 44: Robot conflict rollouts of every carried seed, seen from above. The three left panels of each row show the first three conflict start states of a task, each run by the three GM and three Non-GM checkpoints of Table 24 with deterministic actions, and an asterisk marks a start state on which the seeds of a cohort disagree. The right panel places every start state and seed by the distance of the episode’s end to goal and to proxy, divided by their separation, over the first

250

conflict episodes of each arm task and the first

128

of WalkToFlag .

	Procgen	ManiSkill3 arm		Isaac Lab quadruped
	Maze , Coin , Attribute Maze	LocationReach , AttributeTable	PushToMarker , SortZones	WalkToFlag
Network	IMPALA CNN	Nature CNN, state branch	Nature CNN, state branch	Nature CNN, state branch
Action distribution	categorical, 15 actions	diagonal Gaussian, 8-d	diagonal Gaussian, 8-d	diagonal Gaussian, 3-d
Budget per run	200M ( Maze , Coin GM), 40M ( Coin Non-GM), 5M ( Attribute Maze )	6M	50M ( PushToMarker , 100M for Non-GM seeds 1 and 2), 20M ( SortZones )	30M
Envs $\times$ rollout steps	$256\times 256$	$256\times 50$	$256\times 100$	$512\times 64$
Transitions per update	$65{,}536$	$12{,}800$	$25{,}600$	$32{,}768$

Appendix

Table 25: PPO settings by platform , identical for GM and Non-GM agents.

\mathrm{M}=10^{6}

environment steps. KL early stop ends an update once the approximate KL of a minibatch exceeds the threshold.

Environment	Network	Cohort	Seeds	Budget	Checkpoint step
Maze	IMPALA	GM	1080–1082	200M	$200{,}015{,}872$
Maze	IMPALA	Non-GM	1090–1092	200M	$200{,}015{,}872$
Coin	IMPALA	GM	6033	200M	$20{,}054{,}016$
	IMPALA	GM	6037	200M	$120{,}061{,}952$
	IMPALA	GM	6051	200M	$55{,}050{,}240$
	IMPALA	Non-GM	7001–7003	40M	$40{,}042{,}496$

Appendix

Table 26: Training runs. Budget is the planned run length and the checkpoint the analysed save, in environment steps. Every checkpoint is the final save of its run except for Coin GM.

Environment	Edit	KL ratio	$Q$ ratio ( $h_{\mathrm{fc}}$ )	Signed-follow ratio
Maze	Cheese	$0.009\pm 0.01$	$0.001\pm 0.00$	$-0.002\pm 0.00$
PushToMarker	Marker	$0.206\pm 0.07$	$0.159\pm 0.08$	$0.017\pm 0.05$
Coin	Coin	$0.114\pm 0.05$	$0.063\pm 0.03$	$0.001\pm 0.04$
Attribute Maze	Shape	$0.200\pm 0.09$	$0.307\pm 0.13$	$0.533\pm 0.10$

What Happens to the Goal? Distinct Failures of Policy Alignment in Goal Misgeneralization

Abstract

Figures & tables

Appendix

AI Alignment

An Informational Curse of Horizon in Goal-Conditioned Policy Learning

Understanding Goal Generalisation in Sequential Reinforcement Learning

Functional Graphs for Predicting and Explaining Goal Failure in Sparse Goal-Conditioned RL

Abstract

AI Alignment

Explore similar work

An Informational Curse of Horizon in Goal-Conditioned Policy Learning

Understanding Goal Generalisation in Sequential Reinforcement Learning

Functional Graphs for Predicting and Explaining Goal Failure in Sparse Goal-Conditioned RL