cs.AISep 28, 2026
SaveGlyphBench: A Playground for Language-Model Reinforcement Learning
Organizations: Mila - Quebec Artificial Intelligence Institute · Université de Montréal · Vmax
Abstract
We introduce GlyphBench, an environment suite for reinforcement learning (RL) post-training of language-model agents, with over 360 tasks spanning diverse games. GlyphBench renders spatial observations as two-dimensional Unicode grids and connects training, evaluation, and trajectory replay through a unified interface designed to support efficient and reproducible research. We use GlyphBench to study how observation interfaces, reasoning effort, and agent harnesses affect performance, and how RL configurations shape learning dynamics. Our results show that glyph observations outperform native text and pixels in our Craftax experiments, with further gains on several BALROG environments. RL on 100 GlyphBench tasks improves Qwen3.5-4B on held-out Reasoning Gym problems, reaching 63.48% accuracy and outperforming the base model, a math-trained baseline, and a code-trained baseline. These experiments provide empirical evidence that reasoning gains from gameplay can yield stronger transfer than math or code. Together, these results highlight GlyphBench's value as a testbed for systematic research on how language-model agents learn, interact, and generalize.
Figures & tables
Figure 1: GlyphBench connects nine game families through spatial text observations and named actions. A shared task interface supports policy evaluation, RL training, and trajectory replay. Six families form the standard track; three provide extended tasks.
Figure 2: The same Craftax state in pixels, Craftax’s original text renderer (native text), and GlyphBench’s glyph format. Native text is a comparison interface, not part of GlyphBench. We show excerpts here and both complete text observations in Figure 11 . Glyph colors aid visualization only.
Figure 3: Zero-shot performance on 303 tasks with 25 matched seeds each. Panel (a) shows task-mean returns, with interquartile ranges and median dots; (b–c) show episode length and token use over 7575 episodes per model. Tokens include prompts and completions under each model’s tokenizer.
Figure 4: Zero-shot Craftax performance across observation formats (a), harnesses (b), and reasoning efforts (c), as a percentage of maximum reward ( ). Each panel specifies the settings held fixed. Whiskers show 95% intervals around five-episode means; open markers show individual episodes in (a–b). The star marks one exploratory Astra episode with Guided Memory at maximum effort.
Figure 5: Policies learn on individual tasks across all six suites. Pale traces show training reward; dark traces show centered seven-point means. Dashed lines show mean initial-policy returns. Axes vary by task. Appendix F explains the selection and shows all 25 learning curves.
Figure 6: Learning in Snake under a fixed 6,144-rollout budget. The 45 conditions vary learning rate (rows), difficulty (columns), and group size (colors). Pale traces show batch returns; solid curves show trailing 512-rollout means; dots mark final observations. Return scales match within columns. We examine higher learning rates in Appendix J .
Figure 7: One policy learns across 100 tasks over 100 gradient steps. Panel (a) shows training returns with a trailing nine-step mean; (b) evaluates the same tasks on new environment seeds.
Figure 8: Reasoning Gym accuracy at a 32K output-token budget for the base model and policies trained on games, math, or code. Each policy answers the same 3,100 problems, with three attempts per problem. Panel (a) averages accuracy across 31 tasks; (b) separates the results by category. Whiskers show 95% problem-bootstrap intervals, resampling within tasks and keeping each problem’s three attempts together.
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
| Suite | Tasks | Track | Task scope |
| Classics | 50 | Standard | Puzzles and familiar games for planning, memory, and control. |
| MiniGrid | 71 | Standard | Navigation, keys, doors, obstacles, and memory. |
| MiniHack | 63 | Standard | Dungeon navigation, items, combat, and skill composition. |
| Craftax | 60 | Standard | Focused survival, crafting, combat, and progression tasks. |
| MiniAtari | 43 | Standard | Arcade control with compact dynamics and short horizons. |
| Procgen | 16 | Standard | Procedural platformers, shooters, and mazes. |
Table 1: The 362 GlyphBench tasks. Standard tasks bound episodes to 512 steps and returns to . Extended tasks support longer episodes and are excluded from the standard aggregate score.
Figure 9: The final decision of a successful MiniAtari BankHeist episode. Replay connects the agent’s observation and reasoning to its chosen action and the resulting environment feedback.
Figure 10: The nine Craftax floors in pixels and glyphs. Each pair shows the same local state, with HUDs and legends omitted. Blank glyph regions are unobserved cells; colors aid visualization only. The paired views show how the glyph interface represents the terrain and entities encountered during progression.
Figure 11: Complete environment observations for the initial Craftax state shown in Figure 2 . (a) GlyphBench’s glyph grid, full HUD, and symbol legend. (b) Craftax’s original text renderer: all 99 cells, inventory, and status information. The map listing continues down three columns. Line wrapping and column placement are adjusted for readability.
| Component | Basic | Guided Memory |
|---|---|---|
| Instructions | Task, core mechanics, observation conventions, and action format | Extended game guide, action format, memory protocol, and current-floor guidance |
| Current view | Glyph grid and standard wrapper HUD | Glyph grid and expanded inventory/status HUD |
| History | Growing conversation, with the last eight observation/action frames rendered in each turn | Last eight actions and rewards, plus persistent memory |
| External memory | No separate memory store | Six scratchpad sections and a floor-keyed landmark database |
| Exploration | Prior observations in the conversation | Summary of observed tiles, visited coordinates, frontiers, and remembered features |
| Memory calls | None | Every five steps and after relevant events; separate from action selection |
Table 2: Basic and Guided Memory in the Craftax harness comparison. Both use glyph observations, native game actions, and the selected model reasoning-effort setting. Neither requests an explicit output-token cap.
| Environment | Original BALROG language interface |
|---|---|
| BabyAI | Descriptions of visible objects and their positions relative to the agent, including door state and carried objects, together with the language task. |
| Baba Is AI | Active rules and descriptions of objects and movable rule-word blocks using horizontal and vertical offsets from the controlled object. |
| Crafter | Descriptions of nearby terrain and entities by distance and direction, the object directly ahead, survival statistics, and inventory. |
| MiniHack | Language-wrapper descriptions of surroundings, inventory, and player statistics, supplemented by a two-dimensional ASCII map. |
Table 3: Original BALROG observation formats for the four environments in the interface comparison. These are BALROG’s environment-specific representations, distinct from GlyphBench’s shared glyph format.
Figure 12: We compare the original BALROG interfaces with glyph observations on four environments. Bars show mean progression over paired episodes; differences are percentage points. Table 3 describes each original interface, including MiniHack’s existing ASCII map.
| Setting | Single-task runs | Snake | Multitask |
|---|---|---|---|
| Learning rate | Sweep | ||
| Action output tokens | 8,192 | 4,096 | 4,096 |
| Trainer sequence length | 32,768 | 32,768 | 65,536 |
| Trainer loss | default | ipo | ipo |
Table 4: Training settings. Output budgets apply per action-generation call; sequence length is the trainer’s token limit. Snake learning rates and group sizes are swept as described in Section 4 .
Figure 13: All 25 single-task learning curves, ordered by suite and task. Pale traces show training rewards and dark traces show centered seven-point means. Dashed lines show initial-policy rollout means where available. Endpoint ticks indicate each task’s training-step and reward range. All runs start from Qwen3.5-4B and use one training seed.
Figure 14: We evaluate the multitask policy on new environment seeds across all six training suites, using a common return scale. Each point averages task means within a suite. All suites improve over the displayed interval, while Procgen remains below zero.
Figure 15: The multitask policy learns jointly on all 100 tasks; here we show tasks 1–50, ordered by suite and task. Pale lines show task-mean batch returns; dark lines show trailing 15-gradient-step means. All panels share steps 0–250 and returns .
Figure 16: Training curves for the remaining 50 tasks of the same multitask policy. Axes and smoothing match Figure 15 ; together, the figures show each task once.
Figure 17: We evaluate transfer beyond gameplay by comparing performance before and after GlyphBench RL. (a) Reasoning Gym solve rates at three output budgets. (b) MathArena accuracy over all attempts. (c) SWE-bench pass rates. Bars start at zero; panel (c) spans 0–15%. Table 5 reports paired changes and uncertainty.
| Benchmark | Base | GlyphBench RL | (pp) | Paired 95% CI (pp) | |
|---|---|---|---|---|---|
| MathArena aggregate | 732 | 78.55 | 78.42 | ||
| AIME 2024 I | 60 | 90.00 | 93.33 | ||
| AIME 2024 II | 60 | 90.00 | 90.00 | ||
| AIME 2025 | 120 | 85.00 | 85.00 | ||
| AIME 2026 | 120 | 85.00 | 85.00 | ||
| HMMT Feb 2025 | 120 | 70.00 | 64.17 |
Table 5: External evaluation of the base and GlyphBench RL models. Rates are percentages; changes and paired 95% intervals are percentage points. counts attempts for mathematics and reasoning, and instances for coding. Resampling clusters attempts by problem.
Figure 18: Reasoning Gym performance before and after GlyphBench RL by category and output budget, on a common 0–100% scale. Every category improves at 8K and 16K; the smaller 2K budget yields a weaker and less consistent benefit. Table 6 reports the corresponding values.
Figure 19: Changes after GlyphBench RL on external benchmarks, in percentage points. The left panel shows aggregate changes with paired 95% bootstrap intervals; the right separates Reasoning Gym changes by category and output budget. Positive estimates whose intervals include zero do not establish an improvement.
| Category | 2K: base / ours ( ) | 8K: base / ours ( ) | 16K: base / ours ( ) |
|---|---|---|---|
| Algebra | 40.61 / 46.78 ( ) | 64.72 / 77.06 ( ) | 71.00 / 79.22 ( ) |
| ARC | 5.22 / 4.33 ( ) | 19.22 / 34.78 ( ) | 38.22 / 52.89 ( ) |
| Cognition | 24.29 / 25.62 ( ) | 36.05 / 38.00 ( ) | 37.86 / 40.57 ( ) |
| Games | 3.38 / 4.92 ( ) | 31.42 / 39.58 ( ) | 47.88 / 60.08 ( ) |
| Geometry | 19.33 / 19.00 ( ) | 48.50 / 51.50 ( ) | 50.67 / 53.33 ( ) |
| Graphs | 32.53 / 35.00 ( ) | 64.13 / 80.20 ( ) | 78.20 / 86.93 ( ) |
Table 6: Reasoning Gym solve rates (%): base / GlyphBench RL, with changes in percentage points. Categories contain 6, 3, 7, 8, 2, and 5 tasks in row order, each with 100 problems and three attempts per problem.
Figure 20: Math and code RL improve rewards in their respective training domains. The top panels show training reward (pale), trailing nine-step means (solid), and held-out domain reward (dashed). The bottom panels show average output tokens on held-out problems from each domain. These rewards measure learning within each domain; we evaluate reasoning transfer separately on Reasoning Gym.
| Reference policy | Accuracy difference | Paired 95% interval |
|---|---|---|
| Base | ||
| Math RL | ||
| Code RL |
Table 7: GlyphBench RL minus each comparison policy at the 32K output budget. Differences and paired 95% bootstrap intervals are percentage points.
Figure 21: We compare Reasoning Gym accuracy at additional output-token budgets. Each cap uses independently sampled responses to the same problems. Whiskers show 95% problem-bootstrap intervals; all three attempts for a problem remain in the same resampled cluster.
Figure 22: Reasoning Gym generation diagnostics. (a) Problems with at least one successful completion among three attempts. (b) Responses ending at the output-token limit. (c) Mean output tokens per completion. Truncated responses remain in the denominator and can still count as correct if the answer is extractable.
Figure 23: Peak versus final observed Snake return at , , and within 6,144 rollouts. Both coordinates use trailing 512-rollout means. Colors indicate group sizes, shapes indicate learning rates, and gray paths connect increasing group sizes within a rate. The diagonal marks full retention of the peak; vertical stems show the drop to final return. Gold stars mark each difficulty’s largest peak among plotted conditions. The sweep is exploratory.
Figure 24: Snake training over 25,600 rollouts at , comparing groups 32 and 512 across three difficulties. Pale traces show batch returns, solid curves show trailing 512-rollout means, and dots mark final observations. The dotted line marks 6,144 rollouts for reference. Group 32 achieves the higher final return on all three tasks.