The efficiency of game engines and policy optimization algorithms is crucial for training reinforcement learning (RL) agents in complex sequential decision-making tasks, such as Tetris. Existing Tetris implementations suffer from low simulation speeds, suboptimal state evaluation, and inefficient training paradigms, limiting their utility for large-scale RL research. To address these limitations, this paper proposes a high-performance Tetris AI framework based on bitboard optimization and improved RL algorithms. First, we redesign the Tetris game board and tetrominoes using bitboard representations, leveraging bitwise operations to accelerate core processes (e.g., collision detection, line clearing, and Dellacherie-Thiery Features extraction) and achieve a 53-fold speedup compared to OpenAI Gym-Tetris. Second, we introduce an afterstate-evaluating actor network that simplifies state value estimation by leveraging Tetris afterstate property, outperforming traditional action-value networks with fewer parameters. Third, we propose a buffer-optimized Proximal Policy Optimization (PPO) algorithm that balances sampling and update efficiency, achieving an average score of 3,829 on 10x10 grids within 3 minutes. Additionally, we develop a Python-Java interface compliant with the OpenAI Gym standard, enabling seamless integration with modern RL frameworks. Experimental results demonstrate that our framework enhances Tetris's utility as an RL benchmark by bridging low-level bitboard optimizations with high-level AI strategies, providing a sample-efficient and computationally lightweight solution for scalable sequential decision-making research.
Figures & tables
Method
Avg. Removed Lines
Training Samples
Feature
Generator
BCTS Thiery and Scherrer (2009a)
3,000
6.5×107
DT features
Random
CBMPI Gabillon et al. (2013) ; Scherrer et al. (2015)
4,300
8×106
DT features + RBF heights
Random
dSiLU-TD( λ ) Elfwing et al. (2018)
4,900
200,000
Bertsekas features + neural networks
Random
STEW Lichtenberg and Şimşek (2019)
4,800
238,000
DT features
7-Bag random
Buffered PPO (Ours)
3,850
61,440
DT features
Random
Table 1: Records in 10 × 10 Tetris AI.
Figure 1: Tetris Game Board with All Pieces
Figure 2: llustration of the Afterstate in Tetris Game
Feature
DT-10 weight
DT-20 weight
Landing height
-2.18
-2.68
Eroded piece cells
2.42
1.38
Row transitions
-2.17
-2.41
Column transitions
-3.31
-6.32
Holes
0.95
2.03
Board well
-2.22
-2.71
Table 2: DT-10 and DT-20 weights Gabillon et al. (2013) ; Scherrer et al. (2015) .
Figure 3: The O-Piece and Its Rotation(s)
Figure 4: The I-Piece and Its Rotations
Figure 5: The S-Piece and Its Rotations
Figure 6: The Z-Piece and Its Rotations
Figure 7: The L-Piece and Its Rotations
Figure 8: The J-Piece and Its Rotations
Figure 9: The T-Piece and Its Rotations
Figure 10: Collision Detection
Figure 11: The feedforward process of the Actor evaluating afterstates
Board Size
Weights
Gabillon et al. (2013) Gabillon et al. (2013)
Our bitboard impl
10 × 10(10,000 games)
DT-10
5,000(-)
5,152.02(5,137.51)
DT-20
4,300(-)
4,188.61(4,153.42)
10 × 20(10 games)
DT-10
29,000,000(-)
54,377,543(54,353,156)
DT-20
51,000,000(-)
75,093,204(65,000,224)
Table 3: Scores with different weights on different board sizes (Mean(SD))
Implementation
Time for 10,000 samples
OpenAI Gym Kauten (2019)
12.92 seconds
Bitboard Tetris (ours)
0.24 seconds
Bitboard Tetris via Python (ours)
0.59 seconds
Table 4: Time comparison of Tetris implementations for 10,000 samples
Figure 12: Comparison of Action-Value Evaluating Actor and Afterstate Evaluating Actor
Param
Value
Param
Value
Param
Value
γ
0.99
τ0
0.5
αθ
3e-4
λ
0.99
τk
0.00025
αω
3e-4
epoch
10
ϵ
0.1
episode
12,500
Table 5: Trajectory-Based PPO hyperparameters
Figure 13: Trajectory-Based PPO Algorithm
Parameter
Value
Parameter
Value
Parameter
Value
γ
0.99
ϵ
0.2
batchSize
2,048
λ
0.99
αθ
3e-4
miniBatchSize
256
epoch
10
αω
3e-4
totalSteps
61,440
Table 6: Buffer-based PPO hyperparameters
Figure 14: Buffer-Based PPO Algorithm
Algorithm
Sampling
Policy Updates
Total
Update Ratio
Trajectory PPO
10,536s
436s
10,972s
3.97%
Buffer PPO
112s
54s
166s
32.53%
Table 7: Time cost comparison of PPO variants
Algorithm
Total Steps
Avg. Converged Score
Trajectory PPO
69,046,726
3,840.30
Buffer PPO (optimized)
61,440
3,829.04
Table 8: Step cost comparison of PPO variants
Feature
Weight
Landing height
-0.51
Eroded piece cells
0.16
Row transitions
-0.40
Column transitions
-0.75
Holes
-0.18
Board well
-0.39
Table 9: Best weights obtained from PPO training
Algorithm
Score
Trajectory PPO
32,110,545.18(30,425,374.19)
Buffer PPO
13,936,917.72(13,198,381.24)
Table 10: Scores of two PPO algorithms on the 10 × 20 game board (Mean(SD))
Block Generation Rule
CBMPI
Trajectory PPO
Buffer PPO
Random (Baseline)
4,300
4,965.44 (4884.14)
3,590.90 (3545.05)
7-Bag
251,501.20
173,489.37 (177,212.68)
198,206.23 (198,903.07)
Adversarial Sequence
75
34(0)
53(0)
Table 11: Scores of agents under different block generation rules (Mean(SD))
Reinforcement learning (RL) research requires diverse, challenging environments that are both tractable and scalable. While modern video games may offer rich dynamics, they are computationally expensive and poorly suited for large-scale experimentation due to their CPU-bound execution. We introduce Octax, a high-performance suite of classic arcade game environments implemented in JAX, based on CHIP-8 emulation, a predecessor to Atari, which is widely adopted as a benchmark in RL research. Octax provides the JAX community with a long-awaited end-to-end GPU alternative to Atari games, offering image-based environments, spanning puzzle, action, and strategy genres, all executable at massive scale on modern GPUs. Our JAX-based implementation achieves orders-of-magnitude speedups over traditional CPU emulators. We demonstrate Octax's capabilities by training RL agents across multiple games, showing significant improvements in training speed and scalability compared to existing solutions. The environment's modular design enables researchers to easily extend the suite with new games or generate novel environments using large language models, making it an ideal platform for large-scale RL experimentation. Our open-source framework is available at https://github.com/riiswa/octax/.
Waris Radji, Thomas Michel, Hector Piteau
Univ. Lille, Inria, CNRS, Centrale Lille, UMR 9189-CRIStAL, France · Independent Researcher
An agent meets a model as served: through an endpoint with a price card, a shared cache and other tenants, or on whatever hardware a self-hosted model runs. Benchmarks rank the weights. We introduce a Tetris benchmark that measures what agents get from models as served: every move is scored against an oracle, and every agent receives the same pieces. In five pre-specified experiments with nine open-weight models on one serverless provider, plus an open decision model self-hosted on a CPU, we find that price and size do not predict decision quality; that resending history costs almost nothing when cached input is free, would cost eleven to twelve times more if it were not, and makes play worse; that deployments keep between 4 and more than 64 agent contexts warm, in line with their throughput rather than the model's KV-cache size; that an agent's own long requests slow its slowest short decisions more than tenfold, which a simple admission rule cuts by 59% without hurting play; and that the decision model plays mid-pack but takes 27 s per move on a CPU, about 650 times longer than reported on a GPU. Architecture predicts some of what an agent sees; the deployment sets the rest.
The Arcade Learning Environment (ALE) has been a cornerstone of Deep Reinforcement Learning research, providing a standardized benchmark for evaluating general agents. However, its lack of native support for object-extraction and environment modification restricts research into evaluating whether agents learn robust causal models or rely on brittle visual heuristics. Furthermore, the ALE's dependence on CPU-based emulation creates a significant bottleneck in modern GPU-accelerated training pipelines, limiting both experimental scale and iteration frequency. In this work, we introduce JAXtari, an open-source, high-performance reimplementation of Atari 2600 environments. By leveraging massive GPU parallelization, JAXtari allows PPO agents to finish training on 100M environment steps on a single A100-GPU in under an hour. Beyond speed, it provides object-centric observations and a flexible modification system, allowing researchers to, e.g. develop neuro-symbolic methods or discover novel shortcuts and subsequently train and adapt their agents rapidly, sometimes in only minutes.
Quentin Delfosse, Raban Emunds, Paul Seitz +5
Department of Computer Science, Technical University Darmstadt, Germany · Google Intrinsic AI Research · Hessian Center for Artificial Intelligence (hessian.AI) +2