Factoriax: A GPU-Accelerated Factorio-Style Simulator for Reinforcement Learning

Mickey Beurskens; Tristan Tomilin; Thiago D. Simão

cs.AIOct 4, 2026

Factoriax: A GPU-Accelerated Factorio-Style Simulator for Reinforcement Learning

Authors: Mickey Beurskens, Tristan Tomilin, Thiago D. Simão

Organizations: Eindhoven University of Technology

Abstract

We introduce Factoriax, a GPU-accelerated factory-building simulator written in JAX. In Factoriax, an agent must collect resources, build machines using those resources, and then automate the collection and crafting process by arranging machines on the map to build production pipelines. This paper discusses the structure of the Factoriax simulator and an initial benchmark called Easy Rocket in which an agent is tasked with building a resource-intensive machine called the Rocket in a limited number of game ticks to escape the planet. We also publish results from a number of PPO-based training runs on Easy Rocket. Factoriax is built to be fast. A 1-billion-step PPO training run, equivalent to 500,000 episodes, runs on Easy Rocket in about 8 minutes on a single NVIDIA A100. A standard laptop GPU can complete the same run in about 84 minutes. Our trained PPO agent learns to gather resources, craft machines from those resources, and place them on the map through a curriculum reward directly tied to a manually designed set of achievements. After training, the agent does not place machines in a functional spatial configuration, failing to fully complete the benchmark, and leaving the challenge open for future attempts.

Figures & tables

Figure 1: A typical early-game state in Factoriax. An agent (red dot) on a 2D grid mines raw materials using a miner (green), crafts machines, and arranges them so belts (yellow) carry the correct inputs to assemblers (purple). The assemblers create half-fabricates to speed up the production of more machines. A single coal miner on the right drops coal ore onto a belt, which is split between both lines through a belt splitter. Arms (brown) drop each assembler’s output onto pallets (grey) acting as high-capacity item buffers.

	GPU	Embodiment	Layout	Details	Reference
SafeOR-Gym	✗	✗	$\blacktriangle$	Operations research	( Ramanujam et al., 2026 )
Gym / Gymnasium	✗	✓	✗	Standard API, classic control	( Towers et al., 2025 )
NetHack	✗	✓	✗	Roguelike (procgen)	( Küttler et al., 2020 )
COOM	✗	✓	✗	Doom (continual RL)	( Tomilin et al., 2023 )
HASARD	✗	✓	✗	Doom (safe RL)	( Tomilin et al., 2025 )
MineRL	✗	✓	$\blacktriangle$	Minecraft building	( Guss et al., 2019 )

Table 1: Comparing Factoriax to other simulators and benchmarks. Comparisons assess whether a benchmark is GPU-accelerated , whether the agent in the benchmark is embodied and has to navigate through the environment, and whether the benchmark tests the agent’s capacity to create interconnected factory layout designs .

\blacktriangle

indicates partial fit.

Figure 2: A custom recipe book designed for the Easy Rocket task ( Section 4 ). Raw ores feed every intermediate. The rocket itself requires 100 hulls and 100 engine units (highlighted edges). This requires a working production line rather than hand assembly to finish within 2000 ticks. Appendix B shows the full standard recipe book.

Name	Role	Per-tick action	Key property
Miner	Producer	Pulls ore from the tile beneath into its output buffer	Needs an ore patch beneath; depletes the patch
Assembler	Producer	Pulls two inputs, advances a recipe timer, emits the output	2-input $\to$ 1-output; recipe set by inserted items
Conveyor belt	Logistics	Shifts contents one tile forward along its facing direction	Two parallel lanes (A and B)
Arm	Logistics	Picks one item from the tile it faces, deposits it on the opposite tile	One item transferred per tick
Splitter	Logistics	Alternates incoming belt items between two outputs	Balances two downstream consumers from one supply
Crossing	Logistics	Two belt lanes that cross perpendicularly without merging	No item transfer between the crossing lanes

Table 2: Subset of the placeable machines in Factoriax. Icons are the in-simulator sprites used by the renderer. Appendix B gives an overview of the machines’ crafting recipes.

Panel of Figure 3 — Figure 3: Scripted-baseline agent. (a) Cumulative factory at the end of each phase on seed 42. (b) First-unlock tick distribution across 16 random map seeds. The scripted baseline reaches all twenty-five achievements on every seed within the 2000-tick budget.

Figure 4: Mean episodic return on Easy Rocket over training, across ten seeds at learning rate $10^{-4}$ and entropy coefficient $0.01$ . The solid blue line shows the mean across seeds and the shaded area the standard deviation. The dashed reference at 25 is the scripted-baseline ceiling, or in other words, a fully successful run. Return saturates around 1.5 billion steps at a mean of $17.7$ and is flat thereafter.

Figure 5: Per-achievement unlock rate at 10 billion steps across ten PPO seeds.

Figure 6: Left: A plot of wall-clock time versus the batch size on the GPU. Right: Impact on wall-clock time for map size and entity limit, where blue indicates the server GPU and red the laptop GPU.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

	Name	Function
	Coal	Mined from coal patches. Burned in furnace recipes and used in several intermediates.
	Iron Ore	The base metal ore. Smelts to iron plate; direct input to several machine recipes.
	Copper Ore	The conductive ore. Smelts to copper plate; used in circuits and wires.
	Tin Ore	Smelts to tin plate; used in frames, wires, and pallets.
	Silicon	Refines to wafer; input to electronic intermediates.
	Limestone	Input to refractory brick and to several rocket-part recipes.

Appendix

Table 3: Raw resources in the standard recipe book, with the function of each.

Name	Recipe	Function
Miner	1 $\times$ Iron Plate, 1 $\times$ Wire	Extracts ore from the tile underneath into its output buffer.
Pallet	1 $\times$ Tin Plate, 1 $\times$ Wire	Passive storage tile that holds items in place.
Conveyor Belt	1 $\times$ Iron Plate, 1 $\times$ Copper Plate	Moves items one tile forward per tick along its facing direction.
Assembler	1 $\times$ Frame, 1 $\times$ Circuit	Combines two recipe inputs into an output; recipe selected by inserted items.
Arm	1 $\times$ Copper Plate, 1 $\times$ Wire	Picks an item from the tile it faces and deposits it onto the opposite tile.
Rocket	6 $\times$ Hull, 4 $\times$ Rocket Core	Capstone machine. Placing it on the map triggers liftoff.

Appendix

Table 4: Machines in the standard recipe book, with the recipe that produces each one and its function.

Name	Recipe	Function
Iron Plate	1 $\times$ Iron Ore, 1 $\times$ Coal	Smelted iron. Structural input to frames, miners, conveyors, and hulls.
Copper Plate	1 $\times$ Copper Ore, 1 $\times$ Coal	Smelted copper. Conductive input to circuits, wires, and conveyors.
Tin Plate	1 $\times$ Tin Ore, 1 $\times$ Coal	Smelted tin. Input to frames, wires, and pallets.
Wafer	1 $\times$ Silicon, 1 $\times$ Coal	Refined silicon disc. Input to circuits and advanced science packs.
Frame	1 $\times$ Iron Plate, 1 $\times$ Tin Plate	Structural plate-and-girder. Input to motors, assemblers, and hulls.
Circuit	1 $\times$ Copper Plate, 1 $\times$ Wafer	Logic substrate. Input to sensors, assemblers, and avionics.

Appendix

Table 5: Half-fabricates in the standard recipe book, with the recipe that produces each one and its function.

Figure 7: Standard production recipe book. Every recipe in the simulator’s BASE_RECIPES book, laid out by topological depth. The Easy Rocket scenario uses a simplified subset ( Figure 2 ).

#	Channel	x_ray	superficial	Encodes	Normalisation
1	block_type	✓	✓	Tile type (dirt, water, ore type, …)	$/\max(\texttt{BlockType})$
2	machine_type	✓	✓	Machine type placed on the tile (or NONE)	$/\max(\texttt{Machine})$
3	block_resources	✓	—	Ore count remaining under the tile	$/30000$
4	slot0_type	✓	—	Assembler input slot 0 item type	$/33$
5	slot0_count	✓	—	Assembler input slot 0 item count	$/1024$
6	slot1_type	✓	—	Assembler input slot 1 item type	$/33$

Appendix

Table 6: Spatial channels per observation profile. The x_ray profile uses all ten. The superficial profile uses three.

Group	Count	Encodes
Pose + time	4	Player $x$ (normalised by map_width ), $y$ (by map_height ), facing direction ( $/4$ ), current timestep (by max_timesteps )
Recipe affordability	NUM_RECIPES (26 in the default book)	One float per recipe in the active book; 1 if the player can hand-craft the recipe right now, 0 otherwise
Player inventory	NUM_ITEM_TYPES (33)	Item count per type, normalised by per-item PLAYER_MAX_STACK
Total (both profiles)	63

Appendix

Table 7: Common per-player scalars, shared by both profiles. Recipe-affordability and inventory sizes scale with the active recipe book and item set.

#	Scalar	Encodes
1	facing_machine_type	Machine type on the tile in front of the player
2	facing_buffer_type	Buffer item type on that tile
3	facing_buffer_count	Buffer item count on that tile
4	facing_asm_in0_type	Assembler input slot 0 item type
5	facing_asm_in0_count	Assembler input slot 0 item count
6	facing_asm_in1_type	Assembler input slot 1 item type

Appendix

Table 8: Facing-tile scalars appended by the x_ray profile only. Each scalar reads the entity on the tile directly in front of the player. All nine are zero if that tile is out of bounds.

Code	Action	Effect
0	NOOP	Skip the tick; player state is unchanged
1	UP	Move one tile up if passable, otherwise stay; sets facing UP
2	DOWN	Move one tile down if passable, otherwise stay; sets facing DOWN
3	LEFT	Move one tile left if passable, otherwise stay; sets facing LEFT
4	RIGHT	Move one tile right if passable, otherwise stay; sets facing RIGHT
5	FACE_UP	Snap facing direction to UP without moving

Appendix

Table 9: Movement family. Codes are the absolute Action integer (movement starts at 0). Cardinal moves implicitly set facing. FACE_* actions snap facing without moving.

Code	Action	Effect
9	MINE	Reduce the ore counter on the tile in front by one and add the ore to the player inventory
10	PICKUP	Pick up the machine on the tile in front into the player inventory
11	WITHDRAW	Drain the output slot of the machine in front into the player inventory
12	REPAIR	Restore the entity in front to full health (see note below)
13	ROTATE_LEFT	Rotate the machine on the tile in front to face LEFT
14	ROTATE_RIGHT	Rotate the machine on the tile in front to face RIGHT

Appendix

Table 10: Interaction family. Codes follow movement (starting at 9). Each action operates on the tile directly in front of the player.

Family	Base offset	Count	Items
PLACE_*	17	10	Machine (NONE excluded); see Appendix B , Table 2
CRAFT_*	27	26	HalfFabricate $\cup$ Machine (NONE excluded); see Appendix B , Tables 5 and 2
DEPOSIT_*	53	32	Resource $\cup$ HalfFabricate $\cup$ Machine (NONE excluded); see Appendix B

Appendix

Table 11: Parametric families. Each family covers a contiguous range of Action integers starting at the listed base offset. The item index inside the family follows the simulator’s enum order.

#	Engine name	Phase	Triggered when
1	mine_1_ore	Bootstrap	Player inventory holds any raw ore.
2	mine_3_ore	Bootstrap	Player inventory holds at least one of three distinct raw ore types.
3	prospector	Bootstrap	Player inventory holds at least one of every raw ore type.
4	place_miner	Bootstrap	At least one miner is placed on the map.
5	place_3_miners	Bootstrap	Three miners are placed on the map.
6	metal_in_motion	Bootstrap	Six miners are placed on the map.

Appendix

Table 12: Easy Rocket achievements in unlock order. Achievements 1-3 read the state of the agent inventory, and achievements 4-25 read the map state for placed machines or item buffer states.

Figure 8: Four reset states of Easy Rocket . The player spawns at the centre of a 16 $\times$ 16 grid scattered with patches of finite raw materials. The patches are placed differently for each random seed while preserving spawn-area clearance and a perimeter around the ore patches for belt routing.

Figure 9: Final scripted-baseline factory on seed 42 for Easy Rocket . Six independent miner setups, four more for the production lines. Three assemblers, two of which produce hulls and engines, and the final one combining the two to create the rocket according the the recipe graph in Figure 2 .

Figure 10: 10-billion-step run training runs. Each plot shows achievement unlock for that phase. Rates reported over training (ten seeds).

Explore similar work

Oct 2, 2025cs.LG

Octax: Accelerated CHIP-8 Arcade Environments for Reinforcement Learning in JAX

Reinforcement learning (RL) research requires diverse, challenging environments that are both tractable and scalable. While modern video games may offer rich dynamics, they are computationally expensive and poorly suited for large-scale experimentation due to their CPU-bound execution. We introduce Octax, a high-performance suite of classic arcade game environments implemented in JAX, based on CHIP-8 emulation, a predecessor to Atari, which is widely adopted as a benchmark in RL research. Octax provides the JAX community with a long-awaited end-to-end GPU alternative to Atari games, offering image-based environments, spanning puzzle, action, and strategy genres, all executable at massive scale on modern GPUs. Our JAX-based implementation achieves orders-of-magnitude speedups over traditional CPU emulators. We demonstrate Octax's capabilities by training RL agents across multiple games, showing significant improvements in training speed and scalability compared to existing solutions. The environment's modular design enables researchers to easily extend the suite with new games or generate novel environments using large language models, making it an ideal platform for large-scale RL experimentation. Our open-source framework is available at https://github.com/riiswa/octax/.

Waris Radji, Thomas Michel, Hector Piteau

Univ. Lille, Inria, CNRS, Centrale Lille, UMR 9189-CRIStAL, France · Independent Researcher

Multi-Turn Reinforcement Learning Action Space

Ask PDF

May 20, 2026cs.AI

Mahjax: A GPU-Accelerated Mahjong Simulator for Reinforcement Learning in JAX

Riichi Mahjong is a multi-player, imperfect-information game characterized by stochasticity and high-dimensional state spaces. These attributes present a unique combination of challenges that mirror complex real-world decision-making problems in reinforcement learning. While prior research has heavily relied on supervised learning from human play logs to pre-train the policy, algorithms capable of learning \textit{tabula rasa} (from scratch) offer greater potential for general applicability, as evidenced by the AlphaZero lineage. To facilitate such research, we introduce \textbf{Mahjax}, a fully vectorized Riichi Mahjong environment implemented in JAX to enable large-scale rollout parallelization on Graphics Processing Units (GPUs). We also provide a high-quality visualization tool to streamline debugging and interaction with trained agents. Experimental results demonstrate that Mahjax achieves throughputs of up to \textbf{2 million} and \textbf{1 million steps per second} on eight NVIDIA A100 GPUs under the no-red and red rules, respectively. Furthermore, we validate the environment's utility for reinforcement learning by showing that agents can be trained effectively to improve their rank against baseline policies.

Soichiro Nishimori, Shinri Okano, Keigo Habara +3

The University of Tokyo, Tokyo, Japan. · Nara Institute of Science and Technology, Nara, Japan. · Independent Researcher. +5

Visualization

Ask PDF

May 8, 2026cs.MA

SceneFactory: GPU-Accelerated Multi-Agent Driving Simulation with Physics-Based Vehicle Dynamics

Autonomous-driving simulators typically trade physical fidelity for scalable parallelism. Physics-based platforms such as CARLA and MetaDrive provide articulated vehicle dynamics and contact, but their non-vectorized interfaces make batched training difficult. GPU-batched systems such as Waymax and GPUDrive scale to hundreds of scenarios by replacing rigid-body physics with simplified kinematic models, omitting tire--road interaction, suspension, contact dynamics, and road-condition-dependent friction. We introduce SceneFactory, a GPU-vectorized platform for procedural scene construction, physics-based multi-agent simulation, and RL in autonomous-driving environments. Built on NVIDIA Isaac Sim + Isaac Lab, SceneFactory represents worlds and agents as batched tensors: control, observations, rewards, resets, and policy inference run as GPU tensor operations over the Isaac Lab tensor API. SceneFactory converts Waymo Open Motion Dataset road topologies into simulation-ready USD worlds, runs many worlds concurrently on one GPU, populates each with multiple articulated PhysX vehicles, and maps precipitation and road-surface type to PhysX material friction coefficients. With GPU vectorization, SceneFactory achieves up to 127

\times

higher throughput than a non-vectorized PhysX baseline on the same GPU and physics solver, reaching 19,250 controlled-agent simulation steps per second at 256 worlds

\times

16 agents. Cross-simulator transfer reveals an asymmetric dynamics gap: physics-grounded RL policies transfer to a simplified kinematic bicycle model with 99.5% success, whereas reverse transfer drops to 47.3%. Under wet-road friction, friction-aware policies reduce mean peak DRAC from 58.7 to 27.8,m/s

^2

without sacrificing goal reach. SceneFactory shows that scalable autonomous-driving training need not discard articulated rigid-body dynamics or physically grounded road-condition variation.

Yicheng Zhu, Yang Chen, Tao Li +1

Rochester Institute of Technology · City University of Hong Kong

Autonomous Driving Simulation Waymo Open Motion Dataset Vehicle Dynamics

Ask PDF