TwinLoop: Simulation-in-the-Loop Digital Twins for Online Multi-Agent Reinforcement Learning

Nan Zhang; Zishuo Wang; Shuyu Huang; Georgios Diamantopoulos; Nikos Tziritas; Panagiotis Oikonomou; Georgios Theodoropoulos

cs.LGApr 8, 2026

TwinLoop: Simulation-in-the-Loop Digital Twins for Online Multi-Agent Reinforcement Learning

Authors: Nan Zhang, Zishuo Wang, Shuyu Huang, Georgios Diamantopoulos, Nikos Tziritas, Panagiotis Oikonomou, Georgios Theodoropoulos

Organizations: Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen, China · Research Institute of Trustworthy Autonomous Systems, Southern University of Science and Technology, Shenzhen, China · Department of Informatics and Telecommunications, University of Thessaly, Lamia, Greece

Abstract

Decentralised online learning enables runtime adaptation in cyber-physical multi-agent systems, but when operating conditions change, learned policies often require substantial trial-and-error interaction before recovering performance. To address this, we propose TwinLoop, a simulation-in-the-loop digital twin framework for online multi-agent reinforcement learning. When a context shift occurs, the digital twin is triggered to reconstruct the current system state, initialise from the latest agent policies, and perform accelerated policy improvement with simulation what-if analysis before synchronising updated parameters back to the agents in the physical system. We evaluate TwinLoop in a vehicular edge computing task-offloading scenario with changing workload and infrastructure conditions. The results suggest that digital twins can improve post-shift adaptation efficiency and reduce reliance on costly online trial-and-error.

Figures & tables

Fig. 1: System model under consideration.

Fig. 2: Architecture of the DT-assisted VEC system.

Fig. 3: Workflow of the DT-assisted VEC system.

Fig. 4: 2km $\times$ 2km simulation scenario.

Digital Twins

Digital twins are virtual models synchronized with real systems to support monitoring, prediction, simulation, and decision-making.

See all 253 papers

Leveraging Human-In-The-Loop Demonstrations in Reinforcement Learning for Digital Twin-Driven Robot Flexibility
Oct 8, 2026Yuzhu Sun, Mien Van, Nguyen Minh Nhat +2Robotic RL Human-in-the-Loop AI
School of Electronics, Electrical Engineering and Computer Science, Queen’s University Belfast, Northern Ireland, UK
Adaptive digital twins for predictive decision-making: Online Bayesian learning of transition dynamics
Dec 15, 2025Eugenio Varetti, Matteo Torzoni, Marco Tezzele +1System Identification Digital Twins
MOX – Dipartimento di Matematica, Politecnico di Milano, Milan, 20133, Italy · Dipartimento di Ingegneria Civile e Ambientale, Politecnico di Milano, Milan, 20133, Italy · Mathematics Department, Emory University, Atlanta, 30322, GA, United States
Optimizing Reinforcement Learning Training over Digital Twin Enabled Multi-fidelity Networks
Mar 9, 2026Hanzhi Yu, Hasan Farooq, Julien Forgeat +4Hierarchical RL Digital Twins
Department of Electrical and Computer Engineering, University of Miami, Coral Gables, FL 33146 USA · Ericsson Research, Santa Clara, CA, 95054, USA · European AI Office, European Commission, Brussels, Belgium

Oct 8, 2026J/K move · Enter open · S save

Leveraging Human-In-The-Loop Demonstrations in Reinforcement Learning for Digital Twin-Driven Robot Flexibility

Yuzhu Sun, Mien Van, Nguyen Minh Nhat, Stephen McIlvanna+1

School of Electronics, Electrical Engineering and Computer Science, Queen’s University Belfast, Northern Ireland, UK

Robotic RL Human-in-the-Loop AI Continual Robot Learning Robot Imitation Learning Digital Twins

Open paper AskPDF arXiv

Growing automation makes collaborative robots work in more variable environments, increasing the need for adaptation. We propose a human-in-the-loop online training framework combining a digital twin (DT), reinforcement learning (RL), and human demonstrations. Unlike DTs used mainly to generate synthetic data before task execution, our DT is synchronized with the physical system in real time through camera feeds, allowing the virtual robot to update its observations and policy from real-world feedback. A dual actor framework integrates imitation learning (IL) without adding a direct imitation loss to the RL actor, so demonstrations can guide adaptation instead of manual reprogramming. The proposed framework is demonstrated on the Ufactory Xarm5 collaborative robot, where the robot's end-effector aims to reach the target position while avoiding obstacles. The experiments show that the framework can resume training after a change in the physical workspace and that, with a fixed set of non-optimal demonstrations, the dual actor framework achieves a much higher final success rate than two methods that add an imitation loss to the actor. The same pattern holds with real human demonstrations collected in virtual reality (VR): with demonstrations that never reach the goal, the dual actor framework reached 83-100% mean deterministic evaluation success, against 0-17% for the two imitation-loss methods.

Abstract

Digital Twins

Explore similar work

Leveraging Human-In-The-Loop Demonstrations in Reinforcement Learning for Digital Twin-Driven Robot Flexibility

Adaptive digital twins for predictive decision-making: Online Bayesian learning of transition dynamics

Optimizing Reinforcement Learning Training over Digital Twin Enabled Multi-fidelity Networks