Tackling Sim-to-Real Mismatch Through Sampling-Based Disturbance Observers: From Analytical Models to Learned World Models
Authors: Tianqi Zhu, Jun Yang, Jianliang Mao, Cong Li, Shihua Li
Organizations: School of Automation, Key Laboratory of Measurement and Control of CSE, Ministry of Education, Southeast University, Nanjing 210096, China · Robotics and Autonomous Systems Thrust, Systems Hub, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou 511453, China · Faculty of Artificial Intelligence, Shanghai University of Electric Power, Shanghai 200090, China
Robotic controllers increasingly rely on analytical models, simulators, cost-query interfaces, and learned world models. However, physical deployment can deviate from nominal assumptions, and additional disturbances may arise even when the model itself is accurate. In control systems, disturbance observers (DOB) are widely used to estimate such unmeasured effects from nominal models and measured feedback. Classical DOB formulations are generally built around explicit plant models. This paper develops the sampling-based disturbance observer (SDOB), extending the DOB principle to a broader range of models, including simulators and learned world models, through state-rollout or cost-query interfaces. SDOB separates two observable channels: state-effect disturbances, for which the feedback state differs from its prediction, and cost disturbances, for which the same query state receives different costs as the perceived environment changes. Diverse simulation and real-robot experiments across traditional and learned models demonstrate the effectiveness of SDOB in compensating for sim-to-real mismatch and improving control performance.
Figures & tables
Figure 1: Classical frequency-domain DOB control structure.
Figure 2: Overview of SDOB: model classes, disturbance channels, and key advantages.
Figure 3: Two disturbance channels used by SDOB. (a) Illustration under the state-effect disturbance dk−1x . (b) Illustration under the cost disturbance dk−1c .
Figure 4: Physical admissibility of sampled state-effect disturbances.
Figure 5: Cost disturbance estimation and planning compensation. The first two panels query costs at the same locations before and after an environment update. The third panel matches each current query to a position in the previous field with similar surrounding costs. Arrows connect the current queries, marked by white centers, to these matched earlier positions, marked by orange circles. The displacement between them estimates how the local cost pattern has moved. Red rings indicate that the current cost is higher than the matched previous cost, while blue rings indicate that it is lower. The matching minimizes the neighborhood loss in Eq. ( 27 ). The final panel uses the estimated motion to predict future costs and guide planning through Eqs. ( 29 ) and ( 30 ).
Figure 6: Schematic of the SDOB computation flow. Feedback updates the state-effect and cost disturbance estimates for subsequent rollout and cost compensation.
Task
SDOB channel
SDOB state samples mx
SDOB query states Nc
Controller rollout samples
Prediction horizon H
Environment step
Inverted pendulum
State effect
256
–
1024
15
0.050 s
Eight-obstacle navigation
State and cost
256
332
500
30
0.050 s
Latent Reacher
State effect
256
–
300
5
0.020 s
Rocket Collect
Cost
–
289
32
12
0.033 s
Landing
State and cost
256
121
32
24
0.033 s
Table 1: Core SDOB and controller parameters used in simulation.
Figure 7: Representative inverted-pendulum snapshots under sinusoidal torque bias. MPPI+SDOB effectively compensates for the torque disturbance, achieving disturbance-rejection performance comparable to the conventional MPPI+DOB.
Figure 9: Snapshots comparing navigation methods in an eight-obstacle environment with state-effect and cost disturbances. MPPI+SDOB successfully avoids collisions while achieving a shorter navigation path than the comparison methods.
Figure 11: Final Reacher observations from representative evaluations, where MPPI+SDOB achieves a smaller reaching error.
Table 2: Completion times in the representative real-car runs.
Figure 18: PointWorld box-lifting results. (a) Point-cloud motion predictions without and with SDOB. (b) Real-robot results with MPPI. (c) Real-robot results with MPPI+SDOB, showing a smaller box-lifting error.
Metric
MPPI
MPPI+SDOB
Final task RMSE (mm)
55.26±14.99
22.61±12.58
Final lift (mm)
287.39±24.91
306.60±12.73
Table 3: PointWorld box-lifting results over 20 paired trials.
Although conventional controllers and disturbance observers (DOBs) are the standard for precision tracking in manipulators, they suffer from parameter uncertainty, nonlinear friction, and compound disturbances. This study proposes a residual reinforcement learning DOB framework that pairs an analytical observer with an RL policy. The deterministic baseline operates within a reliable region, whereas the RL policy explicitly targets the residuals that the model cannot capture. To make this compensation disturbance-aware, an estimator network aligns the observation history with a privileged disturbance context, organizing the latent space by disturbance regime and enabling rapid adaptation across disturbance transitions. To guarantee stability, we derived and enforced a state-dependent action bound on the RL policy from an input-to-state stability (ISS) analysis such that the closed loop provably confines the tracking error to a certified envelope for arbitrary policy outputs. Experiments on a 6-DOF manipulator demonstrated consistent improvements in disturbance estimation and tracking, including a 27.8% tracking-error reduction on real hardware under zero-shot sim-to-real transfer and a 38.0% reduction under a base-vibration disturbance that was not observed during training.
Jihong Kim, Joonhyuk Kwon, Hwa Soo Kim +2
School of Mechanical Engineering, Hanyang University, Seoul 04763, Republic of Korea · Department of Mechanical System Design, Kyonggi University, Suwon 16227, South Korea · School of Electrical Engineering, Kookmin University, Seoul 02707, South Korea
Sim-to-real transfer has made substantial progress, but can still produce controllers that remain stable and functional on hardware while suffering from degraded tracking accuracy due to residual dynamics mismatch. Correcting these errors typically requires identifying the underlying system dynamics, adapting the control policy, or returning to simulation for additional training and finetuning, all of which can require substantial data and computation. We propose OSRAM (Online Sim-to-Real Adaptation via Closed-Loop System Modeling), a framework that instead adapts the reference commands provided to an existing controller. OSRAM treats the deployed robot and its policy as a unified closed-loop dynamical system and learns its task-level command-response behavior directly from tracking observations. A closed-loop dynamics model is meta-trained across randomized dynamics in simulation and rapidly finetuned after deployment using limited real-world interaction. The adapted model is then used to optimize future reference commands while leaving the underlying control policy unchanged. We evaluate OSRAM on bipedal velocity tracking and loco-manipulation in simulation and on hardware. Results show that closed-loop modeling improves prediction and tracking accuracy under unseen dynamics, while online reference adaptation reduces residual sim-to-real tracking errors across different control objectives and hardware configurations. These results demonstrate that adapting the behavior of the robot-policy closed loop provides a practical alternative to finetuning the policy or identifying the full physical dynamics for sim-to-real transfer. More information can be found at http://generalroboticslab.com/OSRAM.
We present OrbiSim, a novel robotic simulation paradigm that redefines world models as a fully differentiable physics engine for embodied intelligence. Unlike prior world models that focus on unconstrained imagination in latent or visual domains, OrbiSim establishes a unified, physically-grounded pathway that bridges structured scene assets, neural dynamics, and downstream reinforcement learning. By enabling end-to-end differentiability throughout the entire simulation loop -- spanning from explicit state transitions to visual observation generation -- OrbiSim supports tasks traditionally intractable for classical simulators, such as differentiable contact modeling, gradient-based policy optimization under sparse rewards, and intuitive physical inference. Empirical results demonstrate that OrbiSim significantly outperforms state-of-the-art world models in both predictive fidelity and control performance. Furthermore, its consistent responsiveness to asset configurations and physical parameters suggests its potential as a differentiable tool for enhancing robot simulation and policy training.
Jiajian Li, Jingyuan Huang, Junru Gong +3
MoE Key Lab of Artificial Intelligence, AI Institute, School of Computer Science Shanghai Jiao Tong University