We introduce TAG (Testbed for Autonomous Games), a compact, low-cost robotic platform for studying learning and control in physically coupled dynamical systems. TAG combines an actively tilted and steered ball-in-maze board with a top-mounted rotary inverted pendulum. The pendulum's reaction torque and vibration disturb the board and marble, while the board's tilt, driven by the maze-solving policy, simultaneously disturbs the pendulum's balance. The main challenge in learning control policies for each subsystem is accounting for the impact of one subsystem's dynamics on the other's learning process. To address this challenge, we present complementary reinforcement learning paradigms, leveraging model-based and model-free approaches to learn the control policies for the maze and pendulum, respectively. Our key finding is that introducing the pendulum causes an immediate, sharp drop in the maze policy's success rate, but the policy adapts online and recovers to its pre-coupling performance within a small number of learning episodes. Experimental results show that both policies, trained independently, successfully control the maze board and the pendulum despite their physical coupling.
Figures & tables
Fig. 1: TAG: A T estbed for A utonomous G ames
Fig. 2: Rotary inverted pendulum.
Fig. 3: MuJuCo Simulator for Training Pendulum.
Fig. 4: Overview of the proposed method. The pendulum policy is trained in simulation with a model-free algorithm and deployed fixed on hardware; the maze policy is trained online, directly on the physical system, using DreamerV3. Camera images are processed locally into observations, sent to a remote server that runs DreamerV3 and updates the maze policy via a replay buffer, and returned as roll/pitch commands to the board. The two subsystems are physically coupled through the shared board, with the pendulum acting as a disturbance on the maze policy.
Parameter (unit)
Nominal
Range
Motor gain KM (N ⋅ m/V)
0.0127
[0.010,0.016]
Arm damping bϕ ( 10−4 N ⋅ m ⋅ s/rad)
9.4
[3.0,10.0]
Arm Coulomb friction (mN ⋅ m)
6.0
[4.0,8.0]
Pole damping bθ ( 10−5 N ⋅ m ⋅ s/rad)
3.4
[1.4,5.4]
Pole Coulomb friction (mN ⋅ m)
0.165
[0.065,0.265]
Pole inertia (scale)
1.00
[0.92,1.08]
TABLE I: Domain randomization for pendulum training
Fig. 5: Simulation results: success rates of the trained policy under 5 ms and 10 ms action delays across a range of board-disturbance conditions. Error bars indicate 95% Wilson confidence intervals.
Fig. 6: Real experimental results: success rates of the trained policy in real hardware across a range of board movements.
Fig. 7: Incremental training of the maze policy before and after pendulum activation. Blue and orange denote training with the inactive and active pendulum, respectively, and the shaded region marks the 69-episode ( ∼ 51 minutes) adaptation window under pendulum activation at episode 9,741.
State of the art quadrupedal locomotion approaches integrate Model Predictive Control (MPC) with Reinforcement Learning (RL), enabling complex motion capabilities with planning and terrain adaptive behaviors. However, they often face compounding errors over long horizons and have limited interpretability due to the absence of physical inductive biases. We address these issues by integrating Lagrangian Neural Networks (LNNs) into an RL MPC framework, enabling physically consistent dynamics learning. At deployment, our inverse dynamics infinite horizon MPC scheme avoids costly matrix inversions, improving computational efficiency by up to 4x with minimal loss of task performance. We validate our framework through multiple ablations of the proposed LNN and its variants. We show improved sample efficiency, reduced long-horizon error, and faster real time planning compared to unstructured neural dynamics. Lastly, we also test our framework on the Unitree Go1 robot to show real world viability.
the Robert Bosch Center for Cyber-Physical Systems, Indian Institute of Science, Bengaluru. · the Robert Bosch Center for Cyber-Physical Systems and the Department of Computer Science & Automation, Indian Institute of Science, Bengaluru.
Autonomous tugboating is central for automating maritime operations such as port logistics and vessel maneuvering, where multiple tugboats must cooperatively transport/manipulate a larger vessel. Collaborative pushing in this setting is challenging due to coupled hydrodynamics, low resistance, strong environmental disturbances, underactuated barge dynamics, and contact-rich interactions. Conventional control methods often rely on simplified models and fixed configurations, which limit their adaptability, while learning-based approaches are constrained by the lack of scalable and physically realistic training environments. We address these challenges by introducing a physics-based, GPU-accelerated simulation and learning framework for collaborative tugboat manipulation. Our simulator incorporates a customized buoyancy model, wave modeling, and hydrodynamic resistance, and supports large-scale multi-agent training under marine dynamics. In this simulator, we train a decentralized MAPPO (Multi-Agent PPO) policy augmented with a structured control prior (SCP) to improve training stability and maintain feasible pushing configurations. We evaluate our learned policy on straight-line transit, turning, and deceleration tasks, where we show that our decentralized framework yields more reliable and accurate maneuvering performance compared to a PID-based controller and a centralized PPO baseline. We further demonstrate zero-shot generalization to more challenging sea states and advanced maneuvers, as well as zero-shot scalability to larger teams of three and four tugboats despite training with only two agents.
Junkai Lu, Jiadong Zhao, Jiacheng Zhang +8
National University of Singapore Singapore, Singapore · ST Engineering Unmanned & Integrated Systems Pte. Ltd. Singapore, Singapore
We propose a Koopman model-based reinforcement learning (K-MBRL) algorithm for controlling nonlinear robotic systems with unknown dynamics. The K-MBRL learns a control policy by minimizing an approximate policy objective that combines the analytical immediate cost with an estimate of the future cost evaluated at the one-step predicted state. To this end, we develop a stochastic Koopman operator-based learning method to approximate the unknown nonlinear dynamics in a linear representation, while a critic function estimates the remaining future cost. This formulation offers two key advantages. First, the linear Koopman dynamics simplify the model-dependent policy gradient computation to a constant matrix product, enabling computationally efficient policy improvement. Second, one-step model propagation mitigates the accumulation of prediction errors associated with long-horizon rollouts while retaining long-term cost information through the critic. The learned policy enables real-time control through feedforward evaluation without online optimization. We establish policy learning convergence guarantees under standard stochastic approximation assumptions. Simulations on HalfCheetah and quadruped four-legged locomotion tasks demonstrate strong sample efficiency and improved performance compared with classical model-free and model-based reinforcement learning baselines. A pendulum swing-up task is used to assess policy gradient approximation accuracy and conduct ablation studies on the learning mechanism and key hyperparameters. The proposed K-MBRL is further validated on a physical quadruped robot performing a challenging sequence of maneuvers: rising to an upright posture supported by two rounded rear feet, maintaining balance during bipedal walking, and safely returning to the ground, as shown in Fig.1.
Wenjian Hao, Paulo C. Heredia, Shaoshuai Mou
School of Aeronautics and Astronautics, Purdue University, West Lafayette, IN 47907, USA