Getting Out and Getting Back: World and Behavior Grounding in Real2Sim2Real Co-Training

Samuel Liu; Youngsun Kim; Martin Matak; Gilwoo Lee

cs.ROSep 30, 2026

Getting Out and Getting Back: World and Behavior Grounding in Real2Sim2Real Co-Training

Authors: Samuel Liu, Youngsun Kim, Martin Matak, Gilwoo Lee

Organizations: University of Cambridge · Industrial Next

Abstract

Simulation can expand scarce real demonstrations for co-training, yet how world fidelity and similarity to human behavior affect policy performance remains unclear. We distinguish world grounding, which aligns simulation with the real system, and behavior grounding, which aligns simulated trajectories with human motion. We build a real2sim2real pipeline that varies these axes independently to generate data for co-training. On a dynamic dexterous pick-and-sort task, fully grounded co-training raises success from 52% to 86%; averaged across configurations, world grounding improves success by 18 percentage points and behavior grounding by 10. Deployed policies behave like a mixture of real-derived and simulation-derived policies, imitating real demonstrations in covered states and relying on simulated behavior elsewhere, which we examine through latent-space analysis. Together, these results suggest complementary roles: world grounding lets policies use simulated experience beyond real-data coverage, while behavior grounding matters mainly when world grounding is imperfect. Grounded simulation remains beneficial when co-training foundation models.

Figures & tables

Figure 1: Overview of world and behavior grounding in real2sim2real co-training. (Left) We vary simulated data along two axes, world grounding and behavior grounding, giving four simulated datasets. (Center) Each is combined with 100 real teleoperation demonstrations to co-train policies from scratch or post-train foundation models. (Right) Hypothesized mechanism (schematic, not real data): a deployed policy imitates the real demonstrations in states they cover, relies on simulated behavior elsewhere, and returns to real-like states once it can.

Panel of Figure 2 — Figure 2: Grounded co-training improves task success, for both (a) flow-matching policies and (b) foundation models post-trained with sim-real mixture data. All co-trained policies (Ungrounded, Behavior Grounded, World Grounded, Grounded) use 100 real demonstrations plus about 1,500 simulated trajectories. Error bars show 95% Wilson score intervals over 50 real-world trials.

Panel of Figure 3 — Figure 3: Visual grounding and rollout behavior in observation-embedding PCA projections. (a) CRAFT-style visual transfer places simulated observations closer to real data than native RTX rendering in this projection. (b) A selected rollout of a diagnostic policy trained with 10 real demonstrations and fully ungrounded simulation data moves toward the simulated data region while exhibiting a two-finger pinch, a characteristic of ungrounded simulated trajectories.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Factor	Grounded	Ungrounded
Wrist cameras	Fitted fisheye/mount; blur $\sigma=0.53$ px	Ideal $160^{\circ}$ fisheye; CAD mount; no blur
Head/base/belt poses	Registered using robot motion and AprilTags	Tape-measured head; nominal base orientation; supplied cell layout
Arm model/control	Factory kinematics; weighted IK, filtering, nullspace; 29 fitted parameters	Nominal kinematics; damped least-squares IK; calculated gains ( $\zeta=0.7$ ), no armature
Hand response	Fitted delay, deadband, and lag	Vendor defaults; direct commands

Appendix

Table 1: World configurations. All other listed controls are shared.

500M policy, trained from scratch
Policy	Nominal	Red light	Fast belt	Novel obj.	Combined	Shifted	Total
	/24	/8	/8	/9	/1	/26	/50
Real 100	16	2	4	4	0	10	26
Fully grounded	23	7	6	6	1	20	43
World grounded	21	3	8	6	0	17	38
Behavior grounded	20	3	6	4	1	14	34

Appendix

Table 2: Successes per condition. Shifted = all non-nominal trials.

Explore similar work

Abstract Sim2Real through Approximate Information States
Apr 16, 2026Yunfu Deng, Yuhao Li, Josiah P. HannaSim-To-Real Reinforcement Learning Sim-To-Real Gap
Department of Computer Sciences, University of Wisconsin–Madison, Madison, WI 53706 USA · Manning College of Information and Computer Sciences, University of Massachusetts Amherst, Amherst, MA 01003 USA
Efficient Sim-to-Real Transfer of World-Action Models from Synthetic Priors
Jun 30, 2026Zixing Wang, Kausik Sivakumar, Jinghuan Shang +5Sim-To-Real Reinforcement Learning Robotic Manipulation
1Purdue University · 2Robotics and AI Institute
SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation
Jun 26, 2026Nadun Ranawaka, Josiah Wong, Wei-Lin Pai +15Sim-To-Real Reinforcement Learning Simulation-Based Reinforcement Learning
NVIDIA · Stanford University · The University of Texas at Austin +2

Apr 16, 2026, cs.ROJ/K move · Enter open · S save

Abstract Sim2Real through Approximate Information States

Yunfu Deng, Yuhao Li, Josiah P. Hanna

Department of Computer Sciences, University of Wisconsin–Madison, Madison, WI 53706 USA · Manning College of Information and Computer Sciences, University of Massachusetts Amherst, Amherst, MA 01003 USA

Sim-To-Real Reinforcement Learning Sim-To-Real Gap Real-World Tasks

Open paper AskPDF arXiv

In recent years, reinforcement learning (RL) has shown remarkable success in robotics when a fast and accurate simulator is available for a given task. When using RL and simulation, more simulator realism is generally beneficial but becomes harder to obtain as robots are deployed in increasingly complex and widescale domains. In such settings, simulators will likely fail to model all relevant details of a given target task and this observation motivates the study of sim2real with simulators that leave out key task details. In this paper, we formalize and study the abstract sim2real problem: given an abstract simulator that models a target task at a coarse level of abstraction, how can we train a policy with RL in the abstract simulator and successfully transfer it to the real-world? Our first contribution is to formalize this problem using the language of state abstraction from the RL literature. This framing shows that an abstract simulator can be grounded to match the target task if the grounded abstract dynamics take the history of states into account. Based on the formalism, we then introduce a method that uses real-world task data to correct the dynamics of the abstract simulator. We then show that this method enables successful policy transfer both in sim2sim and sim2real evaluation.

Abstract

Explore similar work

Abstract Sim2Real through Approximate Information States

Efficient Sim-to-Real Transfer of World-Action Models from Synthetic Priors

SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation