TwinBridge: Digital Twin-Driven Policy and Reward Transfer for Real-World Reinforcement Learning in Laboratory Robotics
Authors: Shu'ang Sun, Yibo Qiu, Haoliang Ye, Zan Huang, Ao Tian, Kan Ding, Ronald Xu, Mingzhai Sun
Organizations: Suzhou Institute for Advanced Research, University of Science and Technology of China, Suzhou, Jiangsu, China · School of Biomedical Engineering, Division of Life Sciences and Medicine, University of Science and Technology of China, Hefei, Anhui, China
Online reinforcement learning offers an effective approach to precise laboratory manipulation, yet costly real-world demonstrations and limited access to dense process rewards hinder its practical application. We present TwinBridge, a digital twin-driven framework that supplies policy initialization and dense visual rewards for real-world online learning. We construct digital twins by combining 3D Gaussian splatting with physics simulation. Here, simulator states provide task events and geometric progress under physical prerequisites and ordering constraints, yielding rewards for policy pretraining with simulated demonstrations and reinforcement learning. We distill this supervision into a visual reward model. During real-world interaction, its predictions are combined with execution history to compute dense rewards for human-in-the-loop policy refinement. Across five real-world laboratory tasks, TwinBridge achieves a final mean autonomous success rate of 98% with at most 25 minutes of real-world interaction per task. Experiments show that dense rewards improve interaction efficiency over sparse rewards, and simulation-based policy initialization further accelerates learning.
Figures & tables
Fig. 1: Overview of TwinBridge. A digital twin supplies two resources for real-world online RL: a policy pretrained with simulated demonstrations and RL, and visual process rewards learned from physical supervision and a small set of real annotations. During real interaction, the frozen reward model guides policy refinement with human interventions. The bottom row shows the five laboratory tasks.
Fig. 2: Physics-grounded reward construction and visual transfer. (a) Ordered events and new valid progress records receive one-time credit in an execution-history ledger, defining the potential for potential-based reward shaping (PBRS). (b) Labeled simulated RGB sequences and a small set of real RGB sequences with event/stage annotations train a causal visual predictor. The trained predictor is frozen and transferred. (c) Real RGB predictions update the visual ledger under the same ordering and progress-record rules, and the same PBRS readout supplies online process rewards.
Fig. 3: Five laboratory tasks across four scenes. T1–T5 show tube pickup, lid removal, tube loading, tip attachment, and pipetting; T1 and T3 share a scene. Three representative stages run from top to bottom. Each stage pairs a real observation (left) with its digital-twin render (right).
Fig. 4: Simulation online learning across five laboratory tasks. Curves show ten-episode rolling success rates for TwinBridge and sparse-reward baseline, plotted against simulation steps. Both methods are displayed over the same interaction budget within each task.
Task
TwinBridge
HIL- ConRFT
HG- DAgger
HIL- SERL
w/ sim policy
w/o sim policy
Tube pickup
100
100
70
100
10
Lid removal
100
90
20
90
0
Tube loading
90
80
50
40
10
Tip attachment
100
100
60
100
0
Pipetting
100
90
20
50
0
TABLE I: Final Online Success Rates (%)
Fig. 5: Real-world online learning. Columns follow the task order in Fig. 3 . Ten-episode rolling averages show task success (top) and episode-level intervention fractions (bottom). Both TwinBridge variants use frozen visual process rewards; w/ and w/o sim policy denote the presence and absence of simulation policy initialization.
Figure 7
Task
Event F1 (%) ↑
Temporal ρ↑
Tube pickup
99.50
0.925
Lid removal
99.61
0.881
Tube loading
98.20
0.893
Tip attachment
99.51
0.882
Pipetting
99.69
0.862
Mean
99.30
0.889
TABLE II: Visual Reward Model Evaluation on Real Trajectories
Decentralised online learning enables runtime adaptation in cyber-physical multi-agent systems, but when operating conditions change, learned policies often require substantial trial-and-error interaction before recovering performance. To address this, we propose TwinLoop, a simulation-in-the-loop digital twin framework for online multi-agent reinforcement learning. When a context shift occurs, the digital twin is triggered to reconstruct the current system state, initialise from the latest agent policies, and perform accelerated policy improvement with simulation what-if analysis before synchronising updated parameters back to the agents in the physical system. We evaluate TwinLoop in a vehicular edge computing task-offloading scenario with changing workload and infrastructure conditions. The results suggest that digital twins can improve post-shift adaptation efficiency and reduce reliance on costly online trial-and-error.
Nan Zhang, Zishuo Wang, Shuyu Huang +4
Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen, China · Research Institute of Trustworthy Autonomous Systems, Southern University of Science and Technology, Shenzhen, China · Department of Informatics and Telecommunications, University of Thessaly, Lamia, Greece
Growing automation makes collaborative robots work in more variable environments, increasing the need for adaptation. We propose a human-in-the-loop online training framework combining a digital twin (DT), reinforcement learning (RL), and human demonstrations. Unlike DTs used mainly to generate synthetic data before task execution, our DT is synchronized with the physical system in real time through camera feeds, allowing the virtual robot to update its observations and policy from real-world feedback. A dual actor framework integrates imitation learning (IL) without adding a direct imitation loss to the RL actor, so demonstrations can guide adaptation instead of manual reprogramming. The proposed framework is demonstrated on the Ufactory Xarm5 collaborative robot, where the robot's end-effector aims to reach the target position while avoiding obstacles. The experiments show that the framework can resume training after a change in the physical workspace and that, with a fixed set of non-optimal demonstrations, the dual actor framework achieves a much higher final success rate than two methods that add an imitation loss to the actor. The same pattern holds with real human demonstrations collected in virtual reality (VR): with demonstrations that never reach the goal, the dual actor framework reached 83-100% mean deterministic evaluation success, against 0-17% for the two imitation-loss methods.
Yuzhu Sun, Mien Van, Nguyen Minh Nhat +2
School of Electronics, Electrical Engineering and Computer Science, Queen’s University Belfast, Northern Ireland, UK
Robots trained on real world data tend to be imprecise, slow, and brittle to perturbations. Improving these policies with reinforcement learning (RL) is an appealing alternative, but this process often requires expensive training in the real world. Performing policy improvement in simulation instead provides a far cheaper alternative, but unconstrained RL in simulation can exploit contact and dynamics mismatches, resulting in unsafe behaviors that do not transfer to hardware. Common forms of regularization can furthermore limit improvement by overconstraining to an imperfect behavior prior. In this work, we propose Support-Constrained Off-Domain REinforcement (SCORE), a real-to-sim-to-real framework that constrains RL in simulation to the support of a generative policy pretrained on real data. We instantiate this constraint through flow steering, restricting SCORE to actions the base policy can already produce, which ensures transferable behaviors while maximizing policy improvement. Improving a policy with SCORE requires minimal effort: it learns from sparse rewards, avoids distillation, and leaves the base policy untouched. Across eight real-world dexterous multi-fingered robotic manipulation tasks, SCORE improves average success rate from 37.8% to 89.9%, compared to 59.5% for the best baseline, and reaches success in 36.8% fewer steps than the base policy. Ultimately, through extensive experiments and ablations, we show that simulation can substantially improve real-world manipulation policies when policy optimization is appropriately constrained, introducing a new paradigm for real-to-sim-to-real policy improvement. Videos and code are available at https://weirdlabuw.github.io/score/.