Stack more levels: how to get general and human-like Mario playing
Organizations: New York University, Brooklyn, New York, USA
Abstract
Super Mario Bros. is widely used in game AI and procedural content generation (PCG), yet it remains unclear whether a model-free agent can reliably win unseen levels. To address this question, we train Proximal Policy Optimization (PPO) agents on increasingly large sets of PCG-generated Mario levels and evaluate their generalization on 1000 held-out test levels. We then shape competent PPO policies into three gameplay archetypes---runner, killer, and collector. Finally, we fine-tune these PPO policies using Diffusion-Reward Adversarial Imitation Learning (DRAIL) with human gameplay demonstrations. We find that increasing the number of distinct PCG-generated training levels generally improves unseen-level performance, and that fine-tuning PPO policies with DRAIL increases action-level human-likeness while preserving much of the persona induced by PPO pretraining.
Figures & tables
| Agent | Levels | Win | Compl. | Kill | Coin |
|---|---|---|---|---|---|
| Runner | 100 | 0.091 | 0.415 | 0.092 | 0.125 |
| 1k | 0.462 | 0.709 | 0.135 | 0.238 | |
| 10k | 0.630 | 0.806 | 0.142 | 0.248 | |
| 50k | 0.648 | 0.817 | 0.151 | 0.253 | |
| Killer | 100 | 0.222 | 0.561 | 0.384 | 0.158 |
| 1k | 0.540 | 0.776 | 0.638 | 0.221 |
| Agent | OR 10× Win | 10k 50k Win | 10k 50k Compl. |
|---|---|---|---|
| Runner | 3.70 [3.65, 3.74] | OR 1.12, | +0.011 [-0.014, 0.034] |
| Killer | 2.83 [2.76, 2.90] | OR 0.37, | -0.094 [-0.112, -0.075] |
| Collector | 4.34 [4.23, 4.46] | OR 2.30, | +0.041 [0.026, 0.057] |
| Comparison | Mean Diff. | 95% CI | Holm |
|---|---|---|---|
| K R steps | 116.737 | [96.796, 137.175] | |
| C R steps | 187.274 | [165.583, 210.795] | |
| K R kill | 0.598 | [0.578, 0.619] | |
| K C kill | 0.425 | [0.401, 0.448] | |
| C R coin | 0.593 | [0.575, 0.612] | |
| C K coin | 0.598 | [0.580, 0.616] |
| Method | Runner AAR | Killer AAR | Collector AAR | Avg. AAR | Avg. Action JS |
|---|---|---|---|---|---|
| PPO | 0.016 | 0.007 | 0.021 | 0.015 | 0.643 |
| Direct DRAIL | 0.355 (+0.339 [0.322, 0.356]) | 0.416 (+0.409 [0.394, 0.424]) | 0.420 (+0.399 [0.379, 0.417]) | 0.397 | 0.103 |
| PPO DRAIL | 0.387 (+0.371 [0.353, 0.388]) | 0.192 (+0.185 [0.172, 0.198]) | 0.393 (+0.372 [0.354, 0.388]) | 0.324 | 0.218 |
| Method | Runner | Killer | Collector | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Win | Compl. | Steps | Win | Compl. | Kill | Win | Compl. | Coin | |
| PPO | 0.995 | 0.998 | 435.0 | 0.985 | 0.993 | 1.023 | 0.990 | 0.996 | 0.927 |
| Direct DRAIL | 0.922 | 0.968 | 1042.1 | 0.927 | 0.973 | 0.853 | 0.904 | 0.967 | 0.565 |
| PPO DRAIL | 0.971 | 0.984 | 915.2 | 0.968 | 0.989 | 1.109 | 0.990 | 0.994 | 0.915 |
Game-Playing Agents
Game-playing agents use search, learning, and reasoning to choose actions in games with strategic, uncertain, or changing environments.
See all 215 papers