Game-Guided Skill Discovery through Self-Play for Playable Agent Control

Seungeun Rho; Jeonghwan Kim; Xue Bin Peng; Sehoon Ha

cs.LGSep 30, 2026

Game-Guided Skill Discovery through Self-Play for Playable Agent Control

Authors: Seungeun Rho, Jeonghwan Kim, Xue Bin Peng, Sehoon Ha

Organizations: Georgia Institute of Technology · Simon Fraser University and NVIDIA

Abstract

We present Game-Guided Skill Discovery (GGSD), a framework that uses self-play in games to discover motor skills that are directly playable by humans. Playable skills provide a compact abstraction for controlling embodied agents through a small set of learned behaviors rather than low-level actions. To be effective, these skills should be semantically distinct, interpretable, and expressive; properties that existing unsupervised skill-discovery methods often fail to achieve simultaneously. GGSD achieves these desiderata by grounding skill discovery in competitive gameplay. A hierarchical agent competes against its past selves, with a high-level policy selecting from a small discrete skill set and a skill-conditioned low-level policy learning the corresponding behaviors. After training, a human can replace the high-level policy and directly control the agent through the same discrete skills. Despite the small number of high-level actions, skill transitions give rise to emergent combo behaviors, expanding expressivity beyond individual primitives. Across Ant, Franka-arm, and Unitree G1 environments, we show that GGSD produces human-playable skills that humans can compose to solve unseen tasks, such as Maze and CubePush, without additional training. An interactive demo is available at https://ggsd-demo.github.io.

Figures & tables

Figure 1: Different game rules guide agents toward different playable skills. Shown are gameplay snapshots of agents learned with GGSD.

Figure 2: Overview of GGSD. Top: We jointly learn a hierarchical policy through self-play against opponents sampled from a pool of past checkpoints. Bottom: After training, a human can replace the high-level policy and directly control the agent across diverse scenarios.

Figure 5: Learned motor skills (top) and combo behaviors through skill transitions (bottom).

Figure 6: Semantic diversity of learned skills across different embodiments. Higher is better.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Environment	$s^{\mathrm{proprio}}$	$s^{\mathrm{all}}$	# Skills
Ant Sumo	35	91	5
Ant Fencing	35	91	5
Franka Hockey	23	61	5
G1 Boxing	86	194	6

Appendix

Table 2: Observation-space dimensions.

Group	Component	Dim.	$s^{\mathrm{proprio}}$
Me	Base height $z$	1	✓
	Base linear velocity (body frame)	3	✓
	Base angular velocity (body frame)	3	✓
	Base yaw	1	✓
	Projected gravity (body frame)	3	✓
	Joint positions (normalized)	8	✓

Appendix

Table 3: Ant Sumo observation space. Components marked with ✓ are included in

s^{\mathrm{proprio}}

; all components together form

s^{\mathrm{all}}

Term	Description	Expression	Weight
Approach	Decrease in distance to the opponent	$d_{t-1}-d_{t}$	2.0
Push	Decrease in the opponent’s distance to the nearest edge	$w_{t-1}-w_{t}$	2.0
Win	Opponent leaves the arena or falls (once, terminal)	$\mathbf{1}[\text{win}]$	$+20$
Lose	Opponent wins	$\mathbf{1}[\text{lose}]$	$-20$
Timeout	Draw at 10 s (once, terminal)	$\mathbf{1}[\text{draw}]$	$-3$
Alive, escape, upright, action, joint-velocity			0

Appendix

Table 4: Reward terms for Ant Sumo.

Term	Description	Expression	Weight
Approach	Decrease in distance to the opponent	$d_{t-1}-d_{t}$	2.0
Win	Front foot hits the opponent’s torso ( $\\|F\\|>1$ N), or opponent leaves the arena / falls	$\mathbf{1}[\text{win}]$	$+20$
Lose	Opponent’s front foot hits the agent’s torso, or agent leaves the arena / falls	$\mathbf{1}[\text{lose}]$	$-20$
Timeout	Draw at 10 s	$\mathbf{1}[\text{timeout}]$	$-3$

Appendix

Table 5: Reward terms for Ant Fencing. The reward is identical to Ant Sumo except that the push term is disabled. Arena

8\times 8

m, 10 s episodes, random initial heading.

Group	Component	Dim.	$s^{\mathrm{proprio}}$
Me	Arm joint positions	7	✓
	Arm joint velocities ( $\times 0.1$ )	7	✓
	EE position (table frame)	3	✓
	EE linear velocity (table frame)	3	✓
	EE yaw $(\sin\psi,\cos\psi)$	2	✓
	EE yaw rate	1	✓

Appendix

Table 6: Franka Air Hockey observation space. Components marked with ✓ are included in

s^{\mathrm{proprio}}

; all components together form

s^{\mathrm{all}}

Term	Description	Expression	Weight
Puck progress	Forward puck displacement toward the opponent’s goal (backward motion ignored)	$\max(0,\,x^{p}_{t}-x^{p}_{t-1})$	10.0
Goal	Agent scores (once per goal)	$\mathbf{1}[\text{goal}]$	$+40$
Concede	Opponent scores (once per goal)	$\mathbf{1}[\text{concede}]$	$-40$
Match win	Two goals first, or leading at timeout / idle-puck termination	$\mathbf{1}[\text{match win}]$	$+80$
Match lose	Mirror of match win	$\mathbf{1}[\text{match lose}]$	$-80$
Draw	Tied at termination	$\mathbf{1}[\text{draw}]$	0

Appendix

Table 7: Reward terms for Franka Air Hockey. A match lasts 12 s and is won by the first player to score two goals; if the puck rests for 3 s inside (0.5 s outside) the workspace, the match ends with the current score.

x^{p}

is the puck position along the axis toward the opponent’s goal.

Group	Component	Dim.	$s^{\mathrm{proprio}}$
Me	Base yaw	1	✓
	Base height $z$	1	✓
	Base linear velocity (body frame)	3	✓
	Base angular velocity (body frame)	3	✓
	Projected gravity (body frame)	3	✓
	Glove positions (heading frame, $2\times 3$ )	6	✓

Appendix

Table 8: G1 Boxing observation space. Components marked with ✓ are included in

s^{\mathrm{proprio}}

; all components together form

s^{\mathrm{all}}

Term	Description	Expression	Weight
Punch	HP drained from the opponent minus HP lost	$(\Delta H^{\mathrm{opp}}_{t}-\Delta H^{\mathrm{me}}_{t})/H_{0}$	125
Facing	Torso and pelvis facing the opponent, in $[-2,2]$	$\mathbf{f}_{\mathrm{torso}}\!\cdot\!\hat{\mathbf{u}}+\mathbf{f}_{\mathrm{pelvis}}\!\cdot\!\hat{\mathbf{u}}$	0.03
Facing velocity	Penalises moving sideways relative to the body’s facing	$-\big(\|\mathbf{f}_{\mathrm{torso}}\!\cdot\!\mathbf{v}_{\perp}\|+\|\mathbf{f}_{\mathrm{pelvis}}\!\cdot\!\mathbf{v}_{\perp}\|\big)$	0.15
Approach	Decrease in distance to the opponent; zero when $d_{t}\leq 1$ m	$d_{t-1}-d_{t}$	3.0
Joint velocity	Squared joint velocity, clipped at $\pm 30$ rad/s	$-\sum_{j}\mathrm{clip}(\dot{q}_{j})^{2}$	$3\times 10^{-5}$
Joint limit	Excess beyond 80 % of the soft joint range, clipped at 0.3	$-\sum_{j}\mathrm{clip}\big(\max(\|\tilde{q}_{j}\|-0.8,\,0)\big)^{2}$	0.5

Appendix

Table 9: Reward terms for G1 Boxing, adapted from Won et al. Won et al. (2021) . Arena

10\times 10

m, 15 s episodes, initial HP

H_{0}=10^{5}

per player. Punch damage is

150\,\bar{v}_{\perp}^{2}

, where

\bar{v}_{\perp}

is the glove’s closing velocity normal to the struck face clipped at 12 m/s (head hits

\times 1.5

); it drains the victim’s HP. The outcome is decided by HP alone; a fall or arena exit voids the round.

\hat{\mathbf{u}}

is the unit vector toward the opponent,

\mathbf{f}_{\mathrm{torso}},\mathbf{f}_{\mathrm{pelvis}}

are the forward axes of the torso and pelvis,

\mathbf{v}_{\perp}

is the root XY velocity rotated by

90^{\circ}

(speed clipped at 1 m/s),

\tilde{q}

is the joint position normalised to

[-1,1]

within its soft limits, and

F_{ij}

are self-contact forces.

Hyper-parameter	Ant Sumo	Ant Fencing	Franka Hockey	G1 Boxing
PPO
Parallel environments	4096
Rollout length (steps / env)	32	32	24	32
Training iterations	70 000
Learning epochs / mini-batches	5 / 4
Learning rate	$1\times 10^{-4}$

Appendix

Table 10: Training hyper-parameters of the hierarchical self-play policies.

Explore similar work

May 11, 2026cs.AI

EmbodiSkill: Skill-Aware Reflection for Self-Evolving Embodied Agents

Embodied agents can benefit from skills that guide object search, action execution, and state changes across diverse environments. Since embodied environments vary across layouts, object states, and other execution factors, these skills must self-evolve from trajectories generated during task execution. However, existing skill self-evolution methods are mainly developed in digital environments and often convert trajectories into coarse skill updates. Directly applying this paradigm to embodied settings is problematic, because a failed task execution may reflect not only incorrect skill content, but also an execution lapse in which the agent fails to follow valid guidance. We propose EmbodiSkill, a training-free framework for embodied skill self-evolution through skill-aware reflection and targeted revision. EmbodiSkill interprets each trajectory with respect to the current skill, uses skill-changing evidence to update the skill body, and uses execution-lapse evidence to preserve and emphasize valid guidance. Experiments on ALFWorld and EmbodiedBench show that EmbodiSkill consistently improves embodied task success. On ALFWorld, EmbodiSkill enables a frozen Qwen3.5-27B executor to reach 93.28% task success, outperforming GPT-5.2 used as a direct agent without skills by 31.58%. These results show that skill-aware self-evolution helps embodied agents accumulate reusable procedural knowledge from their own trajectories.

Ruofei Ju, Xinrui Wang, Xin Ding +12

Nanjing University · Institute for AI Industry Research (AIR), Tsinghua University · Huazhong University of Science and Technology +2

Skill Evolution Embodied Agents Human-To-Robot Transfer Embodiment

Ask PDF

Jun 1, 2026cs.CL

MMG2Skill: Can Agents Distill In-the-Wild Guides into Self-Evolving Skills?

Abundant procedural knowledge on the Web holds great potential for helping agents solve long-horizon tasks. However, such knowledge is often multimodal, heterogeneous, noisy, and implicitly assumes human executors, making it difficult to use directly as the skills required by agents. To bridge the gap between human-oriented guides and agent-executable skills, we formalize this problem as guide-to-skill learning: converting in-the-wild guides into executable skills and continuously improving them from trajectories observable to the agent. To evaluate the capability of existing agents on this task, we introduce MMG2Skill-Bench, the first benchmark designed for this problem. We further propose MMG2Skill, a closed-loop framework that compiles guides into editable skills, conditions a fixed vision-language model (VLM) agent on these skills during execution, and revises the skills from trajectory-level root-cause feedback without using benchmark scores. Across GUI control, open-ended gameplay, and strategic card play with six VLM backbones, MMG2Skill consistently outperforms vanilla baseline agents in every model-domain setting, achieving macro-average gains of +12.8 to +25.3 percentage points across backbones. Ablation studies show that directly prompting agents with raw guides can degrade performance, while both structured skill construction and trajectory-driven revision are necessary for the observed improvements. On success-inferable tasks, analyzer-based early stopping further prevents late-stage performance regressions and saves 25%-53% of attempts when the success signal is properly calibrated.

Xinyu Che, Junqi Xiong, Yunfei Ge +10

Nanjing University · Kuaishou Technology

Skill Evolution Skills Trajectory-Level Credit In-The-Wild Videos

Ask PDF

Jun 17, 2026cs.RO

Playful Agentic Robot Learning

Current agentic robot systems can write executable Code-as-Policy programs, observe feedback, and revise behavior across multiple attempts, but they remain largely task-driven: reusable skills are acquired only after explicit instructions. We study Playful Agentic Robot Learning, where an embodied coding agent uses self-directed play as a continual skill-learning stage before downstream tasks arrive. We introduce RATs, Robotics Agent Teams designed for play-time skill acquisition. During play, RATs proposes novel yet learnable exploratory tasks, plans and executes robot-code policies, verifies intermediate progress, diagnoses failures, retries with dense, step-level feedback, and distills successful executions into a persistent code skill library. At test time, the agent reuses relevant skills from this frozen library to help solve new tasks. Experiments in LIBERO-PRO and MolmoSpaces show that play-learned skills improve held-out downstream tasks over no-play and random-play baselines, with 20.6 and 17.0 percentage-point gains over CaP-Agent0 on LIBERO-PRO and MolmoSpaces, respectively. Moreover, the learned skills can be plugged into other inference-time Code-as-Policy agents by simply retrieving them into the context, improving RoboSuite and real-world transfer by 8.9 and 8.8 points, respectively, without finetuning the underlying model.

Junyi Zhang, Jiaxin Ge, Hanjun Yoo +17

University of California, Berkeley · Impossible Research

Scalable Robot Learning Robot Policies Robot Systems

Ask PDF