Sim2Act: Robust Simulation-to-Decision Learning via Adversarial Calibration and Group-Relative Perturbation

Hongyu Cao; Jinghan Zhang; Kunpeng Liu; Dongjie Wang; Feng Xia; Haifeng Chen; Xiaohua Hu; Yanjie Fu

cs.LGMar 10, 2026

Sim2Act: Robust Simulation-to-Decision Learning via Adversarial Calibration and Group-Relative Perturbation

Authors: Hongyu Cao, Jinghan Zhang, Kunpeng Liu, Dongjie Wang, Feng Xia, Haifeng Chen, Xiaohua Hu, Yanjie Fu

Organizations: Arizona State University · Clemson University · University of Kansas · RMIT University ·

Abstract

Simulation-to-decision learning enables safe policy training in digital environments without risking real-world deployment, and has become essential in mission-critical domains such as supply chains and industrial systems. However, simulators learned from noisy or biased real-world data often exhibit prediction errors in decision-critical regions, leading to unstable action ranking and unreliable policies. Existing approaches either focus on improving average simulation fidelity or adopt conservative regularization, which may cause policy collapse by discarding high-risk high-reward actions. We propose Sim2Act, a robust simulation-to-decision framework that addresses both simulator and policy robustness. First, we introduce an adversarial calibration mechanism that re-weights simulation errors in decision-critical state-action pairs to align surrogate fidelity with downstream decision impact. Second, we develop a group-relative perturbation strategy that stabilizes policy learning under simulator uncertainty without enforcing overly pessimistic constraints. Extensive experiments on multiple supply chain benchmarks demonstrate improved simulation robustness and more stable decision performance under structured and unstructured perturbations.

Figures & tables

Panel of Figure 1 — Figure 1: Sim2Act Framework. (a) The overview illustrates the two-stage structure of our framework: simulator calibration and perturbed decision-making. (b) Module details: The simulator, implemented as an LSTM-based encoder-decoder model with linear feature extractor, predicts outcomes from state-action pairs and applies adversarial correction to reduce systematic prediction errors. During decision-making, latent states are perturbed using Gaussian noise derived from the simulator’s estimated covariance $\Sigma$ , producing a distribution of plausible states. The decision-maker $\mathcal{D}_{\theta}$ learns a robust policy by maximizing group-relative advantages across these perturbed states, enhancing generalization under latent uncertainty.

Dataset	DataCo (43, 165445)				GlobalStore (27, 51290)				OAS (22, 28136)
Sim Acc	Risk	Time	Status	Overall	Risk	Time	Status	Overall	Risk	Time	Status	Overall
Markov	0.4978	0.1487	0.5040	0.3835	0.4961	0.1355	0.4934	0.3750	0.5100	0.0011	0.5068	0.3393
Prediction	0.7019	0.3395	0.8161	0.6191	0.8440	0.6767	0.8430	0.7879	0.7157	0.3706	0.7510	0.6124
Generation	0.7024	0.3485	0.8156	0.6221	0.9366	0.8066	0.9355	0.8929	0.7149	0.3916	0.7503	0.6189
S2D	0.9508	0.8851	0.9695	0.9351	0.9743	0.9255	0.9756	0.9585	0.7215	0.3985	0.7574	0.6258
S2A(Ours)	0.9563	0.8875	0.9618	0.9352	0.9723	0.9744	0.9750	0.9650	0.7270	0.3937	0.7629	0.6279

Table 1: Simulation and Decision Performance Comparison (Refer to Appendix J for standard deviations).

Panel of Figure 2 — Figure 2: Decision-maker robustness under increasing perturbation across three datasets. Curves show the degradation of overall decision reward as perturbation strength increases, with slopes indicating sensitivity to uncertainty. Sim2Act maintains flatter degradation curves and smaller slopes than baselines, demonstrating stable performance under both latent-structured and unstructured perturbations (Goal 2).

Figure 3: CVaR@5 Robustness on DataCo, GlobalStore, and OAS datasets. Solid bars denote nominal performance (

p=0

), while hatched bars denote performance under perturbation (

p=0.5

Panel of Figure 4 — Figure 4: Simulator robustness under perturbation across three datasets. Radar plots summarize worst-case accuracy, variance, and drop rate, Sim2Act improves worst-case accuracy and reduces variability, demonstrating effective decision-critical calibration.

Panel of Figure 5 — Figure 5: Ablation study of Sim2Act components across three datasets. The first three panels report decision robustness under perturbation for different module combinations: None (S2D), +SimCal , +DecPert , and +Both . +SimCal improves robustness in decision-critical regions by reducing performance degradation, while +DecPert stabilizes decision behavior under uncertainty. The calibration heatmap (right) visualizes action-level reliability gains, showing that simulator calibration concentrates improvements on decision-critical actions.

Robust RL

Robust reinforcement learning develops agents that maintain performance and safety under uncertainty, disturbances, attacks, and distribution shifts.

See all 229 papers

Theoretical Foundations and Effective Algorithms for Policy-Aware Simulator Learning
May 27, 2026Christoph Dann, Yishay Mansour, Mehryar MohriSim-to-Real Transfer Active Learning
Google Research · Tel Aviv University and Google Research · Google Research and Courant Institute of Mathematical Sciences
Sim-to-Real of Humanoid Locomotion Policies via Joint Torque Space Perturbation Injection
Mar 23, 2026Junhyeok Rui Cha, Woohyun Cha, Jaeyong Shin +2Sim-to-Real Transfer Robotics Simulation
Department of Intelligence and Information, Graduate School of Convergence Science and Technology, Seoul National University, Republic of Korea · 1X Technologies. This work was conducted while the author was at Seoul National University · Department of Intelligence and Information, Graduate School of Convergence Science and Technology, ASRI, AIIS, Seoul National University, Republic of Korea, and Advanced Institute of Convergence Technology (AICT), Suwon, Republic of Korea. He is the corresponding author of this paper.
Abstract Sim2Real through Approximate Information States
Apr 16, 2026Yunfu Deng, Yuhao Li, Josiah P. HannaSim-to-Real Transfer State Abstraction
Department of Computer Sciences, University of Wisconsin–Madison, Madison, WI 53706 USA · Manning College of Information and Computer Sciences, University of Massachusetts Amherst, Amherst, MA 01003 USA

May 27, 2026J/K move · Enter open · S save

Theoretical Foundations and Effective Algorithms for Policy-Aware Simulator Learning

Christoph Dann, Yishay Mansour, Mehryar Mohri

Google Research · Tel Aviv University and Google Research · Google Research and Courant Institute of Mathematical Sciences

Sim-to-Real Transfer Active Learning Robust RL Model-Based RL

Open paper AskPDF arXiv

Model-based reinforcement learning (MBRL) agents typically learn world models by minimizing predictive loss. However, powerful RL optimizers inevitably exploit minor model inaccuracies, leading to simulator exploitation and a reality gap where policies succeed in simulation but fail in the real world. We propose that the objective for learning simulators should be strategic robustness rather than predictive accuracy, and formulate this as a zero-sum minimax game between a model player and an adversarial policy player. We provide a comprehensive theoretical analysis: (1) an online learning guarantee showing the game is learnable with sublinear regret bounds; (2) a tractable critic-based simplification bounding the global policy-value gap by the local critic's loss; and (3) an Error-MDP duality, proving that finding the worst-case policy is formally dual to a standard RL problem where the reward is the one-step critic error. This duality yields a provably convergent active data selection algorithm. Experiments on continuous control tasks demonstrate that our approach reduces prediction error in strategically important regions by

1.5

2.2\times

and enables policies trained purely in simulation to match near-optimal real-world performance.

Abstract

Robust RL

Explore similar work

Theoretical Foundations and Effective Algorithms for Policy-Aware Simulator Learning

Sim-to-Real of Humanoid Locomotion Policies via Joint Torque Space Perturbation Injection

Abstract Sim2Real through Approximate Information States