Game-Guided Skill Discovery through Self-Play for Playable Agent Control

Seungeun Rho; Jeonghwan Kim; Xue Bin Peng; Sehoon Ha

cs.LGSep 30, 2026

Game-Guided Skill Discovery through Self-Play for Playable Agent Control

Authors: Seungeun Rho, Jeonghwan Kim, Xue Bin Peng, Sehoon Ha

Organizations: Georgia Institute of Technology · Simon Fraser University and NVIDIA

Abstract

We present Game-Guided Skill Discovery (GGSD), a framework that uses self-play in games to discover motor skills that are directly playable by humans. Playable skills provide a compact abstraction for controlling embodied agents through a small set of learned behaviors rather than low-level actions. To be effective, these skills should be semantically distinct, interpretable, and expressive; properties that existing unsupervised skill-discovery methods often fail to achieve simultaneously. GGSD achieves these desiderata by grounding skill discovery in competitive gameplay. A hierarchical agent competes against its past selves, with a high-level policy selecting from a small discrete skill set and a skill-conditioned low-level policy learning the corresponding behaviors. After training, a human can replace the high-level policy and directly control the agent through the same discrete skills. Despite the small number of high-level actions, skill transitions give rise to emergent combo behaviors, expanding expressivity beyond individual primitives. Across Ant, Franka-arm, and Unitree G1 environments, we show that GGSD produces human-playable skills that humans can compose to solve unseen tasks, such as Maze and CubePush, without additional training. An interactive demo is available at https://ggsd-demo.github.io.

Figures & tables

Figure 1: Different game rules guide agents toward different playable skills. Shown are gameplay snapshots of agents learned with GGSD.

Figure 2: Overview of GGSD. Top: We jointly learn a hierarchical policy through self-play against opponents sampled from a pool of past checkpoints. Bottom: After training, a human can replace the high-level policy and directly control the agent across diverse scenarios.

Figure 5: Learned motor skills (top) and combo behaviors through skill transitions (bottom).

Figure 6: Semantic diversity of learned skills across different embodiments. Higher is better.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Environment	$s^{\mathrm{proprio}}$	$s^{\mathrm{all}}$	# Skills
Ant Sumo	35	91	5
Ant Fencing	35	91	5
Franka Hockey	23	61	5
G1 Boxing	86	194	6

Appendix

Table 2: Observation-space dimensions.

Group	Component	Dim.	$s^{\mathrm{proprio}}$
Me	Base height $z$	1	✓
	Base linear velocity (body frame)	3	✓
	Base angular velocity (body frame)	3	✓
	Base yaw	1	✓
	Projected gravity (body frame)	3	✓
	Joint positions (normalized)	8	✓

Appendix

Table 3: Ant Sumo observation space. Components marked with ✓ are included in

s^{\mathrm{proprio}}

; all components together form

s^{\mathrm{all}}

Term	Description	Expression	Weight
Approach	Decrease in distance to the opponent	$d_{t-1}-d_{t}$	2.0
Push	Decrease in the opponent’s distance to the nearest edge	$w_{t-1}-w_{t}$	2.0
Win	Opponent leaves the arena or falls (once, terminal)	$\mathbf{1}[\text{win}]$	$+20$
Lose	Opponent wins	$\mathbf{1}[\text{lose}]$	$-20$
Timeout	Draw at 10 s (once, terminal)	$\mathbf{1}[\text{draw}]$	$-3$
Alive, escape, upright, action, joint-velocity			0

Appendix

Table 4: Reward terms for Ant Sumo.

Term	Description	Expression	Weight
Approach	Decrease in distance to the opponent	$d_{t-1}-d_{t}$	2.0
Win	Front foot hits the opponent’s torso ( $\\|F\\|>1$ N), or opponent leaves the arena / falls	$\mathbf{1}[\text{win}]$	$+20$
Lose	Opponent’s front foot hits the agent’s torso, or agent leaves the arena / falls	$\mathbf{1}[\text{lose}]$	$-20$
Timeout	Draw at 10 s	$\mathbf{1}[\text{timeout}]$	$-3$

Appendix

Table 5: Reward terms for Ant Fencing. The reward is identical to Ant Sumo except that the push term is disabled. Arena

8\times 8

m, 10 s episodes, random initial heading.

Group	Component	Dim.	$s^{\mathrm{proprio}}$
Me	Arm joint positions	7	✓
	Arm joint velocities ( $\times 0.1$ )	7	✓
	EE position (table frame)	3	✓
	EE linear velocity (table frame)	3	✓
	EE yaw $(\sin\psi,\cos\psi)$	2	✓
	EE yaw rate	1	✓

Appendix

Table 6: Franka Air Hockey observation space. Components marked with ✓ are included in

s^{\mathrm{proprio}}

; all components together form

s^{\mathrm{all}}

Term	Description	Expression	Weight
Puck progress	Forward puck displacement toward the opponent’s goal (backward motion ignored)	$\max(0,\,x^{p}_{t}-x^{p}_{t-1})$	10.0
Goal	Agent scores (once per goal)	$\mathbf{1}[\text{goal}]$	$+40$
Concede	Opponent scores (once per goal)	$\mathbf{1}[\text{concede}]$	$-40$
Match win	Two goals first, or leading at timeout / idle-puck termination	$\mathbf{1}[\text{match win}]$	$+80$
Match lose	Mirror of match win	$\mathbf{1}[\text{match lose}]$	$-80$
Draw	Tied at termination	$\mathbf{1}[\text{draw}]$	0

Appendix

Table 7: Reward terms for Franka Air Hockey. A match lasts 12 s and is won by the first player to score two goals; if the puck rests for 3 s inside (0.5 s outside) the workspace, the match ends with the current score.

x^{p}

is the puck position along the axis toward the opponent’s goal.

Group	Component	Dim.	$s^{\mathrm{proprio}}$
Me	Base yaw	1	✓
	Base height $z$	1	✓
	Base linear velocity (body frame)	3	✓
	Base angular velocity (body frame)	3	✓
	Projected gravity (body frame)	3	✓
	Glove positions (heading frame, $2\times 3$ )	6	✓

Appendix

Table 8: G1 Boxing observation space. Components marked with ✓ are included in

s^{\mathrm{proprio}}

; all components together form

s^{\mathrm{all}}

Term	Description	Expression	Weight
Punch	HP drained from the opponent minus HP lost	$(\Delta H^{\mathrm{opp}}_{t}-\Delta H^{\mathrm{me}}_{t})/H_{0}$	125
Facing	Torso and pelvis facing the opponent, in $[-2,2]$	$\mathbf{f}_{\mathrm{torso}}\!\cdot\!\hat{\mathbf{u}}+\mathbf{f}_{\mathrm{pelvis}}\!\cdot\!\hat{\mathbf{u}}$	0.03
Facing velocity	Penalises moving sideways relative to the body’s facing	$-\big(\|\mathbf{f}_{\mathrm{torso}}\!\cdot\!\mathbf{v}_{\perp}\|+\|\mathbf{f}_{\mathrm{pelvis}}\!\cdot\!\mathbf{v}_{\perp}\|\big)$	0.15
Approach	Decrease in distance to the opponent; zero when $d_{t}\leq 1$ m	$d_{t-1}-d_{t}$	3.0
Joint velocity	Squared joint velocity, clipped at $\pm 30$ rad/s	$-\sum_{j}\mathrm{clip}(\dot{q}_{j})^{2}$	$3\times 10^{-5}$
Joint limit	Excess beyond 80 % of the soft joint range, clipped at 0.3	$-\sum_{j}\mathrm{clip}\big(\max(\|\tilde{q}_{j}\|-0.8,\,0)\big)^{2}$	0.5

Appendix

Table 9: Reward terms for G1 Boxing, adapted from Won et al. Won et al. (2021) . Arena

10\times 10

m, 15 s episodes, initial HP

H_{0}=10^{5}

per player. Punch damage is

150\,\bar{v}_{\perp}^{2}

, where

\bar{v}_{\perp}

is the glove’s closing velocity normal to the struck face clipped at 12 m/s (head hits

\times 1.5

); it drains the victim’s HP. The outcome is decided by HP alone; a fall or arena exit voids the round.

\hat{\mathbf{u}}

is the unit vector toward the opponent,

\mathbf{f}_{\mathrm{torso}},\mathbf{f}_{\mathrm{pelvis}}

are the forward axes of the torso and pelvis,

\mathbf{v}_{\perp}

is the root XY velocity rotated by

90^{\circ}

(speed clipped at 1 m/s),

\tilde{q}

is the joint position normalised to

[-1,1]

within its soft limits, and

F_{ij}

are self-contact forces.

Hyper-parameter	Ant Sumo	Ant Fencing	Franka Hockey	G1 Boxing
PPO
Parallel environments	4096
Rollout length (steps / env)	32	32	24	32
Training iterations	70 000
Learning epochs / mini-batches	5 / 4
Learning rate	$1\times 10^{-4}$

Appendix

Table 10: Training hyper-parameters of the hierarchical self-play policies.

Explore similar work

EmbodiSkill: Skill-Aware Reflection for Self-Evolving Embodied Agents
May 11, 2026Ruofei Ju, Xinrui Wang, Xin Ding +12Skill Evolution Embodied Agents
Nanjing University · Institute for AI Industry Research (AIR), Tsinghua University · Huazhong University of Science and Technology +2
MMG2Skill: Can Agents Distill In-the-Wild Guides into Self-Evolving Skills?
Jun 1, 2026Xinyu Che, Junqi Xiong, Yunfei Ge +10Skill Evolution Skills
Nanjing University · Kuaishou Technology
Playful Agentic Robot Learning
Jun 17, 2026Junyi Zhang, Jiaxin Ge, Hanjun Yoo +17Scalable Robot Learning Robot Policies
University of California, Berkeley · Impossible Research

May 11, 2026, cs.AIJ/K move · Enter open · S save

EmbodiSkill: Skill-Aware Reflection for Self-Evolving Embodied Agents

Ruofei Ju, Xinrui Wang, Xin Ding, Yifan Yang+11

Nanjing University · Institute for AI Industry Research (AIR), Tsinghua University · Huazhong University of Science and Technology+2

Skill Evolution Embodied Agents Human-To-Robot Transfer Embodiment

Open paper AskPDF arXiv

Embodied agents can benefit from skills that guide object search, action execution, and state changes across diverse environments. Since embodied environments vary across layouts, object states, and other execution factors, these skills must self-evolve from trajectories generated during task execution. However, existing skill self-evolution methods are mainly developed in digital environments and often convert trajectories into coarse skill updates. Directly applying this paradigm to embodied settings is problematic, because a failed task execution may reflect not only incorrect skill content, but also an execution lapse in which the agent fails to follow valid guidance. We propose EmbodiSkill, a training-free framework for embodied skill self-evolution through skill-aware reflection and targeted revision. EmbodiSkill interprets each trajectory with respect to the current skill, uses skill-changing evidence to update the skill body, and uses execution-lapse evidence to preserve and emphasize valid guidance. Experiments on ALFWorld and EmbodiedBench show that EmbodiSkill consistently improves embodied task success. On ALFWorld, EmbodiSkill enables a frozen Qwen3.5-27B executor to reach 93.28% task success, outperforming GPT-5.2 used as a direct agent without skills by 31.58%. These results show that skill-aware self-evolution helps embodied agents accumulate reusable procedural knowledge from their own trajectories.

Abstract

Explore similar work

EmbodiSkill: Skill-Aware Reflection for Self-Evolving Embodied Agents

MMG2Skill: Can Agents Distill In-the-Wild Guides into Self-Evolving Skills?

Playful Agentic Robot Learning