We present a method for synthesizing reactive character behaviors for continuous games as compact, human-readable programs. Game AI practice still relies heavily on manually authored behavior trees, state machines, and scripts, while academic reinforcement learning typically produces opaque neural controllers that are expensive to train and difficult to edit. Our approach bridges this gap by searching directly over a domain-specific language for continuous-space game policies. The language is designed around reactive geometric decisions and includes higher-order constructs such as direction maximization. These constructs help discretize a continuous behavior space into enumerable program structures. To make program search practical, we introduce a large set of synthesis antipatterns that remove redundant program forms while preserving behavioral coverage. We further combine bottom-up symbolic enumeration with top-down guidance from a coding agent. Our resulting method, agentic sketching, has the agent propose high-level policy structure and call an enumerator to complete local program slots. We evaluate the method on a benchmark of 14 continuous games, ranging from classic control tasks to multi-agent football. We find that pure enumeration is often more efficient than using a coding agent alone, while the combined method substantially outperforms both. Our results suggest that programmatic policy search can be a practical authoring tool for game AI: designers specify reward functions, and the system discovers editable behaviors that are effective, portable, and often surprising.
Figures & tables
Figure 1. (Top) The behavior of a mouse character in our ButtonCheese game, using a program synthesized by our agentic sketching method. The synthesizer discovers a surprising and effective policy in which the mouse goes to the button, presses it repeatedly to fill the map with cheese, and then rapidly collects the cheese. (Bottom) The last line is the automatically synthesized policy in our character behavior domain-specific language; the first three lines define game-provided sensor functions. A ButtonCheese game sequence shows the mouse repeatedly pressing a button to spawn many cheese objects and then collecting them, followed by the synthesized policy program.
Figure 2. An example program in our policy DSL for the Pathfinding game (right). The policy uses a conditional branch to decide whether the agent (the blue robot) should proceed directly toward the goal (yellow cheese) or use more complex logic to find a way around obstacles. The or function shown here is not part of the DSL; it is equivalent to nested if expressions and is used here for readability. In the false branch, the agent chooses the direction that maximizes the product of three quantities: the raycast length in that direction, the direction’s alignment with the goal, and its alignment with the vector (- (right (to goal)) (from goal)) . This policy was discovered by our agentic sketching algorithm. A Pathfinding game scene beside a synthesized policy program that conditionally moves toward the goal or follows raycast-based obstacle-avoidance logic.
Figure 3. Best scores normalized per game relative to Search. Averages are computed after per-game normalization. A grouped bar chart compares normalized scores for Coding agent, Search, Agentic sketching, and the best PPO minimum observed across all checkpoints for each benchmark game and their average.
Figure 4. Average normalized score of the three policy search methods over the full 20-minute search interval. Search initially outperforms agentic sketching, but agentic sketching eventually overtakes it; both outperform the coding agent. Line graph of average normalized score over time for Coding agent, Search, and Agentic sketching.
Method
Normalized score
Policy tests
Coding agent
0.76
5.39×102
Search
1.00
1.99×106
Agentic sketching
1.23
1.71×105
Table 1. Aggregate normalized score and average number of policy tests across benchmark games.
Method
ButtonCats
ButtonCheese
ButtonCheeseCats
Chase
Dodgeball
Football
Hide
Lawnmower
MouseCats
MouseCheeseCats
Pathfinding
ProtectTheChild
Seek
Vacuum
Search
21
12
7
6
12
-1
1747
57
1088
11
5
20810
4
39
Coding agent
17
9
6
2
3
-1
1675
52
758
6
6
16402
4
30
Agentic sketching
24
14
12
8
9
0
1858
60
1572
15
8
22684
4
50
Sk. w/o antipatterns
22
14
9
8
9
-2
1812
60
1266
11
6
20810
4
50
PPO
157
9
6
35
0
0
581
66
595
14
2
11098
1
33
PPO avg. score
172.0
13.0
8.0
62.0
1.9
0.4
1481.4
90.6
3031.4
16.2
7.6
15277.5
5.4
61.2
Table 2. Scores of the best policies found by each method. In our benchmark, a game’s aggregate score is the minimum score across N random seeds. For comparison, we also report the average PPO score across those seeds.
Method
ButtonCats
ButtonCheese
ButtonCheeseCats
Chase
Dodgeball
Football
Hide
Lawnmower
MouseCats
MouseCheeseCats
Pathfinding
ProtectTheChild
Seek
Vacuum
Search
20
20
18
21
22
10
19
25
23
20
20
23
19
30
Sketching, subexpr
17
18
14
18
20
9
15
20
17
17
20
36
16
22
Sketching, sketch
35
29
69
49
58
48
47
46
45
38
33
27
111
68
Coding agent
56
54
163
75
31
40
25
68
47
48
91
149
77
42
Table 3. Maximum expression sizes used by Search, Agentic Sketching, and the Coding Agent. For Agentic Sketching, we report both the maximum size of the searched subexpressions and the maximum sketch size.
Statistic
Button Cats
Button Cheese
Button Cheese Cats
Chase
Dodgeball
Football
Hide
Lawnmower
Mouse Cats
Mouse Cheese Cats
Pathfinding
Protect the Child
Seek
Vacuum
Total
Search attempts
16
13
13
21
20
5
11
32
12
17
20
5
11
17
213
Valid searches
14
13
13
20
18
5
10
31
12
16
19
5
10
17
203
Direct policy evals
7
3
5
5
7
7
1
1
1
1
7
0
5
2
52
Distinct sketches
9
10
13
18
14
4
5
21
10
11
9
3
8
15
150
Bound increases (imm.)
4 (3)
3 (0)
0 (0)
2 (0)
3 (2)
0 (0)
4 (4)
9 (2)
2 (0)
4 (2)
10 (6)
1 (1)
2 (1)
2 (0)
46 (21)
Table 4. Agentic sketching tool calls. Bound increases are revisits of the same sketch with a larger search bound (immediately consecutive revisits are shown in parentheses).
Figure 5. Search progress over 20 minutes for all benchmark games. Red shows pure bottom-up symbolic search, cyan shows pure coding-agent search, and violet shows agentic sketching. In most games, the coding agent is eventually dominated by search, which is itself eventually dominated by agentic sketching.
Figure 6. Fresh-seed PPO learning curves and bottom-up search progress for all benchmark games. Yellow and green show the average and minimum test scores, respectively, of the current PPO checkpoint; red shows the minimum score of the best policy found so far by bottom-up search. The horizontal axis shows simulator frames on a symmetric-log scale. Fourteen plots compare PPO average and minimum test scores with the minimum score found by bottom-up search as simulator frames increase.
Figure 7. Screenshots from all 14 games in our benchmark. See the code repository for full descriptions of each game, including formal specifications in our game description language. A grid of screenshots from the 14 benchmark games, showing their maps, characters, obstacles, goals, and interactive objects.
Recent LLM-based approaches to control either invoke a language model to select actions or synthesize world models that require planning at every decision, introducing latency that can limit real-time use. We introduce Code to Control, an approach that synthesizes Python controllers which execute directly as policies. Code to Control separates program structure from parameters. An LLM synthesizes the controller structure, while derivative-free search fits its parameters for continuous control using feedback from the environment. Once learned, the resulting controllers require neither LLM inference nor planning at decision time, enabling real-time gameplay and, under our timing protocol, faster action selection than a PPO policy. Across a suite of Atari games, Flappy Bird, and MuJoCo tasks, Code to Control outperforms planning-based program synthesis methods, remains competitive with deep reinforcement learning while using fewer environment interactions, transfers across substantial changes in environment dynamics, and scales to complex locomotion tasks.
Zergham Ahmed, Joshua B. Tenenbaum, Chris Bates +1
Harvard University · Massachusetts Institute of Technology · Florida Institute for Human and Machine Cognition
Recent game design agents have made substantial progress in generating playable games. However, program correctness does not ensure an enjoyable experience for players. We present Recursive Game Creator, an experience-oriented harness to advance agentic game development from rough game prototypes into entertaining games. Recursive Game Creator organizes recursive development around four components: Designer, Builder, Player, and Reviewer. The Designer translates user instructions and Reviewer's feedback into detailed plans. The Builder turns these plans into candidate games. The coding-native Player creates and executes reusable policies through programmatic interfaces to efficiently collect diverse gameplay trajectories, mitigating evaluation bias caused by slow GUI-based collection. The Reviewer uses carefully designed trajectory-based metrics to induce player preferences, integrating with visual evidence and explicit textual preferences to evaluate games against game-specific criteria. Finally, the Reviewer accepts the better version and provides improvement reviews for the next round, closing the recursive loop. Our method achieves state-of-the-art overall performance of 77.89 on GameCraft-Bench. On GameASG-Bench, it achieves a strict task success rate of 53.2%, a 34.1% improvement over the same-model baseline, and the highest mean runtime-check pass rate at 93.4% among compared methods. A user study shows longer playtime and higher ratings. Project page is at: https://imbaldy.github.io/recursive-game-creator/.
Jiajun Chen, Haoyu Wu, Mingda Jia +1
HKU MMLab · The University of Hong Kong · Shenzhen Loop Area Institute
This paper introduces a reinforcement learning framework that enables controllable and diverse player behaviors without relying on human gameplay data. Existing approaches often require large-scale player trajectories, train separate models for different player types, or provide no direct mapping between interpretable behavioral parameters and the learned policy, limiting their scalability and controllability. We define player behavior in an N-dimensional continuous space and uniformly sample target behavior vectors from a region that encompasses the subset representing real human styles. During training, each agent receives both its current and target behavior vectors as input, and the reward is based on the normalized reduction in distance between them. This allows the policy to learn how actions influence behavioral statistics, enabling smooth control over attributes such as aggressiveness, mobility, and cooperativeness. A single PPO-based multi-agent policy can reproduce new or unseen play styles without retraining. Experiments conducted in a custom multi-player Unity game show that the proposed framework produces significantly greater behavioral diversity than a win-only baseline and reliably matches specified behavior vectors across diverse targets. The method offers a scalable solution for automated playtesting, game balancing, human-like behavior simulation, and replacing disconnected players in online games.
Atahan Cilan, Atay Özgövde
Turkish Aerospace, Istanbul, Turkey · Department of Computer Engineering, Boğaziçi University, Istanbul, Turkey