cs.AISep 26, 2026
SaveWitness: Discovery, Deciphering, and Epiphany in Interactive Puzzle Environments
Organizations: Fleet AI · University of Nevada, Reno · Simon Fraser University · Independent
Abstract
Automated science needs agents that can work out the rules of an unfamiliar environment by interacting with it. Interactive rule-discovery puzzles offer a controlled setting for studying this ability: an agent infers hidden rules through experimentation and uses what it has inferred to reach a stated goal. We ask what limits current language models on these puzzles and whether reinforcement learning (RL) improves performance on rules held out from training. To study both, we introduce WITNESS, a 2D grid-based puzzle environment with ground-truth ASCII observations and controlled access to rules. An agentic pipeline generates games for WitnessGym, the RL training suite, and WitnessBench, comprising public validation and private test games. The validation set separately tests new compositions of trained rule primitives and primitives absent from training. Under a shared harness, the best of 18 frontier proprietary and open-weight models solves only 24% of private test level slots, with scores sensitive to the observation interface and agent configuration. Providing ground-truth rules raises Opus-5's validation RHAE-L5 (relative human action efficiency over the first five levels) from 59.9 to 97.8, whereas a 27B open-weight model gains only 2.1 points and remains limited even with the rules provided. RL on WitnessGym raises the 27B model's private test RHAE-L5 from 2.1 to 5.4 and yields a mean gain of 4.1 points on four external discovery benchmarks. Together, these results point to rule acquisition as a major difficulty for frontier models like Opus-5 while smaller models further struggle on rule-based execution, and indicate that RL on hidden-rule puzzles transfers to broader rules and real-world tasks beyond training. Benchmark is available at: https://witnessbench.ai
Figures & tables
Figure 1: Overview of W itness- I nspired T estbed of N ovel E nvironments with S ymbolic S tructure (WITNESS).
Figure 2: One level of a WITNESS game as the engine renders it and as the model observes it: the ASCII board with its legend, the current path, and the feedback shown after a rejected submission.
| Model | RHAE-L5 mean std | Beaten levels | Avg levels per game | OAE L5 / L20 | RHAE-uncap L5 / L20 | Deepest level | Beaten games | RHAE-L20 mean std |
| Fable-5 | 36.4 3.7 | 414/1700(24%) | 4.87/20 | 20.9 / 5.4 | 156.3 / 34.3 | 20/20 | 2/85 | 8.3 2.7 |
| Opus-5 | 30.6 2.0 | 405/1700(24%) | 4.76/20 | 21.2 / 6.6 | 210.9 / 38.6 | 20/20 | 2/85 | 9.4 2.2 |
| GPT-6-Astra-Pro | 29.8 2.0 | 399/1700(23%) | 4.69/20 | 19.5 / 9.1 | 224.1 / 59.5 | 20/20 | 10/85 | 12.8 1.1 |
| GPT-6-Astra | 27.6 0.9 | 389/1700(23%) | 4.58/20 | 17.6 / 7.7 | 182.7 / 50.0 | 20/20 | 10/85 | 11.3 0.9 |
| Kimi-K3 | 24.6 4.0 | 249/1700(15%) | 2.93/20 | 14.9 / 1.8 | 119.2 / 15.2 | 20/20 | 1/85 | 3.1 1.6 |
| GPT-5.6-Sol | 23.0 3.1 | 262/1700(15%) | 3.08/20 | 13.8 / 2.1 | 92.7 / 10.8 | 14/20 | 0/85 | 3.4 1.0 |
Table 1: Model Evals using our canonical harness (unseen games w/ novel rules, 20 lvls/game, 5 eval seeds). RHAE-L5 / RHAE-L20: RHAE computed over the first 5 levels (the benchmark number) and over all 20 levels. The first 5 levels are four rule-teaching levels with curriculum design plus a non-teaching testing level; the remaining levels are more complex levels of larger grid or harder. RHAE-uncap: the human-baseline score without the two caps. OAE (objective action efficiency): the same level-weighted efficiency score against the optimal action count. Deepest level: the most levels completed in any single rollout. Evaluated in max-effort where the endpoint exposed effort tiers.
| Harness | Opus-5 | Opus-4.8 | Kimi-K3 | Qwen3.8-27B |
| Our harness | 30.58 2.00 | 22.77 4.27 | 24.62 4.02 | 6.42 1.19 |
| Our harness, w/o perception | 30.29 4.92 | 10.95 2.80 | 8.71 2.91 | 1.03 0.30 |
| Minimal harness (ARC-AGI-3 alike) | 11.33 3.02 | 3.31 1.11 | 0.00 0.00 | 0.01 0.03 |
| [1pt/1.5pt] OSS harness: Prime-Agent | 68.11 6.33 ∗ | 59.88 0.75 ∗ | 47.99 3.96 | 26.60 6.39 |
| model-attended solving | 55.29 6.68 | 37.24 4.66 | 42.22 4.53 | 23.53 3.87 |
| verbal-only solving | 4.37 1.45 | 0.14 0.05 | 9.20 4.38 | 5.86 3.90 |
Table 2: Harness ablation on the 17 unseen games (5 seeds): how much of each model’s RHAE-L5 score depends on its harness. Our harness : our full harness. w/o perception : same harness without ground-truth ASCII board ( pixel input). Minimal harness : a minimalist loop, raw pixel input. Prime-Agent : A self-improving RLM Harness (95% on ARC-AGI-3 public games). Model-attended solving and Verbal-only solving : exclude cases – no model participation in level solving; winning moves computed by code. Details in Appendix A.5 . ∗ These models with Prime-Agent caused frequent refusals from Claude’s content filter; we report 2 seeds to show a spread and repeat no further.
| Policy | Extra prompt | RHAE-L5 | RHAE-L5 | Games cleared | Efficiency | Plan-less calls |
| Qwen3.8-27B | none | 8.0 5.0 | – | 5/40 | 0.39 | 1% |
| generic text | 9.1 2.5 | +1.1 | 6/40 | 0.44 | 0% | |
| GT rule card | 10.1 2.7 | +2.1 | 6/40 | 0.48 | 1% | |
| Opus 5 (frontier) | none | 59.9 11.4 | – | 30/40 | 0.74 | 3% |
| generic text | 63.6 3.7 | +3.7 | 34/40 | 0.72 | 4% | |
| GT rule card | 97.8 0.7 | +37.8 | 40/40 | 0.99 | 0% |
Table 3: Ground-truth rule cards as a prompt-time probe . A ground-truth rule card is a hand-written natural-language statement of a game’s true rules (win condition and mechanics, transcribed from the game engine), which we supply in the system prompt at evaluation time only, as an oracle probe that separates a policy’s failure to discover the rules from its failure to act on them. Eight held-out validation games, 5 evaluation seeds, 8192-token output cap, thinking off.
Figure 3: (a) RL training dynamics; and (b,c) generalization to test set (unseen rules) and external discovery tasks.
| Benchmark | Hidden rules | Observation | Training–evaluation split |
| ARC-AGI-3 ( ARC Prize Foundation, 2026a ) | ✓ | Pixel grid | Evaluation only |
| DiG-Bench ( Battleday et al., 2026 ) | ✓ | 1D string | Evaluation only |
| MazeBench ( Pappas & Pappas, 2026 ) | ✗ | 3D render (also ASCII/JSON) | Evaluation only |
| gg-bench ( Verma et al., 2025 ) | ✗ | Text | Evaluation only (LLMs) |
| lmgame-Bench ( Hu et al., 2026 ) | ✗ | Frames or text | By game |
| Enigmata ( Chen et al., 2026a ) | ✗ | Text | By instance |
Table 4: Comparison of related benchmarks. Hidden rules indicates interactive inference of rules not supplied to the agent. Split identifies the unit held out between training and evaluation in the reported experiments; evaluation only indicates that no training experiment is reported for evaluated models.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Cheatsheet for W itness- I nspired T estbed of N ovel E nvironments with S ymbolic S tructure (WITNESS).
| Policy | Extra prompt | RHAE-L5 | RHAE-L5 | RHAE-uncap | Games cleared | Efficiency | Plan-less calls |
| Base (no RL) | none | 8.0 5.0 | – | 9.9 6.7 | 5/40 | 0.39 | 1% |
| generic text | 9.1 2.5 | +1.1 | 11.4 5.6 | 6/40 | 0.44 | 0% | |
| GT rule card | 10.1 2.7 | +2.1 | 14.8 7.4 | 6/40 | 0.48 | 1% | |
| Broadcast RL, step 10 | none | 15.9 3.4 | – | 19.8 2.1 | 7/40 | 0.50 | 2% |
| generic text | 20.2 4.8 | +4.3 | 22.3 6.0 | 8/40 | 0.63 | 1% | |
| GT rule card | 47.7 5.0 | +31.8 | 82.6 14.8 | 31/40 | 0.69 | 57% |
Table 5: Ground-truth (GT) rule cards as extra prompts for test-time ablation. A ground-truth rule card is a hand-written natural-language statement of a game’s true rules (win condition and mechanics, transcribed from the game engine), which we supply in the system prompt at evaluation time only, as an oracle probe that separates a policy’s failure to discover the rules from its failure to act on them. Eight held-out validation games, 5 evaluation seeds, 8192-token output cap, thinking off. A rule card does not bring great improvements on the base model, while the same card improves the RL policies significantly. Frontier models such as Opus-5 are mainly bottle-necked by rule discovery. These hint-assisted scores are only for diagnostic purpose and excluded from all other results.
Figure 5: Why Opus-5 does not achieve perfect RHAE score (100) given GT rules: rare cases where human players complete a level more efficiently than Opus. In these levels, a human player selected a more efficient path, possibly unconsciously as it is a straight-forward path; the opus model selected non-optimal paths in 3/5 and 1/5 seeds, respectively. For most other cases, especially in earlier levels where a human player performed numerous steps to discover rules, Opus given GT rules executes much more efficiently, achieving RHAE-uncap scores that far exceed 100.
| Checkpoint | Layout | ft10 | ft14 | ft15 | ft16 |
| ( slots 4–5) | ( slots 1–2) | ( slots 0–4, 7) | ( slots 4–7) | ||
| Base (no RL) | trained | 1.19 0.32 / 0.29 / 0.6 | 2.25 0.37 / 0.46 / 3.0 | 2.81 0.32 / 0.45 / 3.4 | 2.06 0.46 / 0.59 / 4.5 |
| unseen | 0.75 0.37 / 0.38 / 1.1 | 1.31 0.38 / 0.41 / 1.9 | 2.12 0.38 / 0.43 / 2.7 | 3.69 0.38 / 0.40 / 4.5 | |
| RL, step 10 | trained | 6.44 0.80 / 0.67 / 40.0 | 3.12 0.40 / 0.50 / 5.8 | 2.50 0.48 / 0.59 / 4.6 | 4.31 0.30 / 0.48 / 8.2 |
| unseen | 5.94 0.63 / 0.64 / 25.2 | 2.88 0.48 / 0.41 / 4.9 | 2.62 0.38 / 0.44 / 3.9 | 4.62 0.34 / 0.62 / 13.7 | |
| RL, step 30 | trained | 3.75 0.81 / 0.52 / 12.9 | 3.44 0.27 / 0.51 / 7.0 | 1.88 0.32 / 0.58 / 2.7 | 2.44 0.46 / 0.58 / 5.3 |
Table 6: Transfer of RL gains to unseen layouts of the same rules. For each training game that has an alternate level set we evaluate checkpoints on the trained set and on a reserved set that was never used in training (same rules, different boards; the two layouts differ only in levels). Cells: levels completed / action efficiency on completed levels / official game score. The gains on trained layouts carry over to the unseen layouts; the gap between trained and unseen does not grow with training.
Figure 6: Comparing models of different sizes (9B vs 27B) with identical training settings. (a) 9B model does not experience useful learning signals while 27B learns to beat more levels. These findings corroborate the belief that complex behaviors emerge during model scaling; (b) the validation performance on composite of known primitive rules and unseen rules starts differently but converge. It suggests what model learns during RL training is generalized capability rather than memorizing specific patterns; (c) training on games of primitive rules generalized to test games of unseen rules.
Figure 7: Behavior comparison between 9B and 27B models. (a) In contrast to 9B model, 27B model learns to complete more levels with less LLM calls within a game rollout as the training progresses; (b) 27B model learns to reflect more within each LLM call to plan high-quality actions, while 9B model consistently outputs verbose tokens that do not improve level-solving; (c) 27B model learns to preserve smaller fraction of rules it proposed earlier, while 9B model keeps generating and preserving most low-level rule hypotheses that do not contribute to puzzle solving.
Figure 8: Comparison of RL algorithms: various credit assignment methods. Credit-C is shown with two curves, one with each seed, to exhibit its divergence during training, due to the noise in deterministically keyed rules.
| Model | Step | CipherBank custom ciphers (%) cipher decryption 786 items, 2 reps | PhysGym L4 success (%) law discovery 97 problems, 3 reps | WILT elim@5 (%) rule discovery 50 rules, 5 reps | Failing-to-Falsify incompat. fraction (%) Wason 2-4-6 80 instances, 5 reps | Mean over 4 benchmarks [95% CI] |
| Qwen3.8-27B (base) | – | 73.8 0.2 | 44.7 1.6 | 86.3 1.6 | 50.7 5.1 paired subset: 49.3–49.6 | – |
| + RL broadcast | 10 | 77.6 2.3 (+3.8) | 45.4 1.8 (+0.7) | 87.8 1.1 (+1.5) | 51.2 3.5 (+0.5) | +1.6 [ 0.3, +3.6] |
| 20 | 78.6 1.3 (+4.8) | 48.1 2.1 (+3.4) | 88.6 1.7 (+2.3) | 55.4 4.9 (+6.1) | +4.1 [+2.1, +6.2] | |
| + RL credit-C | 10 | 77.0 1.4 (+3.2) | 45.4 2.7 (+0.7) | 87.6 1.8 (+1.3) | 51.8 3.6 (+1.2) | +1.6 [ 0.6, +3.8] |
| 20 | 78.1 0.7 (+4.3) | 46.4 1.0 (+1.7) | 87.5 1.0 (+1.2) | 54.0 3.1 (+3.7) | +2.7 [+0.6, +4.9] |
Table 7: Training dynamics of OOD generalization: credit assignment (broadcast vs. credit-C) training step (10 vs. 20) on the four benchmarks. During RL training, both models have progressive OOD gains against the base model.
| Model | Requested identifier | Out cap / reasoning | Evaluated (2026) |
| Fable-5 | claude-fable-5 | 64k / adaptive-max | Jul 31 – Aug 5 |
| Opus-5 | anthropic/claude-opus-5 | 32k / 16k budget | Jul 29 – Aug 5 |
| GPT-6-Astra-Pro | openai/gpt-6-astra-pro | 64k / max | Sep 9 – Sep 17 |
| GPT-6-Astra | openai/gpt-6-astra | 64k / max | Sep 4 – Sep 12 |
| Kimi-K3 | moonshotai/kimi-k3 | 32k / max | Jul 30 – Aug 6 |
| GPT-5.6-Sol | openai/gpt-5.6-sol | 32k / max | Aug 12 – Aug 22 |
Table 8: Exact model identifiers and sampling settings behind Table 1 . Every model was run with the same canonical harness and protocol: 17 unseen games 5 evaluation seeds up to 20 levels; 300 actions per level (an unsolved level ends the rollout) and 1,000 LLM calls per rollout; temperature for all models (the client forces 1.0 whenever reasoning is enabled), no top_p , top_k or seed parameter, cached system prompt. Reasoning : “max” sends reasoning:{effort:"max"} ; “16k budget” sends reasoning:{max_tokens:16000} (endpoints without effort tiers); “adaptive-max” is the Anthropic API’s adaptive thinking with output_config.effort = max . Out cap is the per-call output-token limit.
More in this topic
AI Agent Benchmarks
AI agent benchmarks measure how reliably autonomous systems plan, use tools, interact, and complete tasks across controlled and real-world settings.
See all 1,494 papers