Progress in robot learning, particularly physical multi-agent reinforcement learning (MARL), is currently challenged by a limited availability of accessible, standardized benchmarks. While simulation-based MARL has produced remarkable emergent behaviors, from coordinated team play to complex tool use, translating these advances to physical systems remains difficult. A key barrier is infrastructural: physical platforms are often prohibitively expensive, require specialized facilities, or lack open mechanisms and designs for reproducibility. We introduce KlaskTron, an open-source, low-cost (<$2500 USD), desktop-scale robotic testbed for adversarial MARL based on the dynamic tabletop game KLASK. The platform features dual CoreXY gantries with high-torque brushless motors, enabling the high-speed, precise manipulation required for competitive play. Crucially, we provide a complete ecosystem: hardware designs (CAD, BOM), a GPU-native digital twin in NVIDIA Isaac Lab for high-fidelity modeling and simulation, and a validated sim-to-real baseline using a learned neural actuator model. We demonstrate that this baseline enables zero-shot policy transfer, with trained agents exhibiting emergent adversarial behaviors. To ensure full reproducibility of our results, all project assets are released publicly: hardware CAD/BOM and assembly documentation at [https://idscethzurich.github.io/klask_hardware], inference software stack at [https://github.com/IDSCETHZurich/klask_software] and the simulation and MARL setup at [https://github.com/IDSCETHZurich/klask_rl].
Figures & tables
Fig. 1: KlaskTron robot with the 2x 2D gantry for each player
Fig. 2: System architecture of the KlaskTron platform. The environment is interchangeable between the Isaac Lab simulation (for training) and the physical system (for deployment). State estimation in simulation is provided directly by the physics engine; on hardware, an Extended Kalman Filter processes camera observations. The control stack uses PPO during training and the frozen actor network during inference. Actions pass through the learned actuator model in simulation or directly to physical motors on hardware.
Fig. 3: The Isaac Lab simulation environment for KlaskTron. Each agent observes ball and striker positions/velocities and outputs continuous velocity commands.
Fig. 4: Adaptive process noise regions on the KLASK board (left) with corresponding covariance matrices (right). Higher values in Q matrices indicate reduced confidence in the process model, causing the EKF to rely more on measurements.
Fig. 5: Validation of the learned actuator model (10-step history, three hidden layers of 64 units). Solid lines show real hardware response; dashed lines show simulated response to identical command sequences. Shaded bands show the spread across 8 training seeds, indicating that prediction quality is robust to initialisation. The model achieves a global validation RMSE of 0.013m/s , with peak errors of 0.032m/s (x) and 0.049m/s (y) on these representative high-frequency trajectories.
Game
Winner
Duration (s)
Defensive
Offensive
Goal-Directed
Possession
#
Blocks
Strikes
Hits
(%)
1
Human
28
6
4
1
42%
2
Human
35
8
5
2
45%
3
Robot
42
12
9
4
51%
4
Human
21
4
3
1
38%
5
Human
30
7
5
2
44%
TABLE I: 10-Game Human-Robot Tournament Statistics. The agent demonstrates emerging defensive competence but significant offensive limitations.
Parameter
Value
Discount factor ( γ )
0.99
GAE parameter ( λ )
0.95
Learning rate
3×10−4
Clip parameter ( ϵ )
0.2
Mini-batch size
4096
Entropy coefficient
0.01
TABLE II: PPO hyperparameters.
Parameter
Range
Static friction
[0.2,0.5]
Dynamic friction
[0.3,0.45]
Restitution
[0.95,0.99]
Ball mass
[1.8,2.2] mg
TABLE III: Domain randomization ranges (disabled for best agent).
KlaskTron is an open-hardware physical platform for two-player adversarial reinforcement learning, with a learned actuator model that supports zero-shot sim-to-real transfer of trained policies. Building on this platform, we contribute KlaskTron-IL: a multi-tier imitation learning dataset (80k human demonstration timesteps collected directly on the physical robot, 320k symmetry-augmented timesteps, and synthetic expert trajectories up to 5M steps generated by the zero-shot-validated PPO policy), and diagnostic baselines that expose three characteristic failure modes of standard IL methods under contact-rich adversarial dynamics: BC's data scaling plateau, GAIL's interpretable but weak-gradient discriminator (Spearman ρ=−1.0 peg-ball proximity recovery, but a logit span under 0.1), and an asymmetric BC-prior x reward-density interaction. Because the policy interface used throughout this paper is the same one the actuator model bridges to hardware, a policy trained on this benchmark can run unchanged on the physical robot, as we demonstrate zero-shot for the strongest BC baseline, so long as its commands respect the actuator model's ±0.2 m/s identification range.
We built a robot called the Robotroller that actuates an Atari CX40+ controller and a device called the Atari Devbox that renders the game frame and the reward signal from the Arcade Learning Environment on a screen. The Robotroller and the Atari Devbox, together with an off-the-shelf camera and a desktop computer, constitute a system that can be used to study reinforcement learning algorithms in the physical world. We call the full system Physical Atari. In this paper, we detail the key decisions that make Physical Atari a robust and accessible platform. To make the system robust, we designed the Robotroller so that all movement is done through bearings, which reduces wear. Additionally, we wrote software that monitors the state of the servos at a high frequency and intervenes to limit stress. To make the system accessible, we used affordable off-the-shelf components and parts that can be manufactured using consumer 3D printers. Physical Atari can be built for under $1,000 and has been used for weeks of non-stop reinforcement learning experiments without any mechanical failures. We used it to validate that reinforcement learning algorithms can learn directly on robots and show that even small distribution shifts between learning and deployment can significantly degrade the performance of policies. Our results underscore the importance of on-device adaptation for strong performance on robots.
Khurram Javed, Joseph Modayil, Gloria Kennickell +2
Keen Technologies · Openmind Research Institute · University of Alberta, Canada
Reinforcement learning (RL) research has demonstrated success in both physical and simulated domains; however, the predominant methodology remains rooted in simulations. The predominance of simulations makes translating research to physical reality uncertain for both algorithms and researchers. We propose a physical platform that is designed to simplify the transition. In this paper, we present the Open Ant: a physical variant of the commonly used Gymnasium Ant environment, along with a simulation. We demonstrate that competent walking policies can be learned from scratch in approximately one hour directly from the physical robot's experience for two substantially different RL algorithms: SARSA(λ) and Soft Actor-Critic (SAC). Separately, we show policies that were learned in simulation transfer to reality. We also examine how well the platform supports a nimble experimental ecosystem. Specifically, we observe the speed with which new users from diverse backgrounds achieve their first success with the platform, and how easily the platform can be repaired and updated when hardware issues arise. Both the hardware design and software are available as open-source on GitHub for ease of customization. In summary, we advocate for the use of the Open Ant for RL researchers who frequently use simulated environments, so they can more easily include robot experiments in their evaluations.
Elena Sorina Lupu, Patrick Spieler, Khurram Javed +4
Openmind Research Institute, Canada · Keen Technologies · Department of Computing Science, University of Alberta, Canada