AutoRSR: Automated Real-to-Sim-to-Real Data Generation for Scalable Robot Learning

Jiawei Zhang; Chang Liu; Litao Deng; Tao Zhang; Qin Jin

cs.ROOct 14, 2026

AutoRSR: Automated Real-to-Sim-to-Real Data Generation for Scalable Robot Learning

Authors: Jiawei Zhang, Chang Liu, Litao Deng, Tao Zhang, Qin Jin

Organizations: Renmin University of China

Abstract

Scaling embodied robot learning requires large and diverse manipulation datasets, but collecting real-world demonstrations remains expensive in human effort and robot time. Simulation provides a scalable alternative, yet converting a real workspace into a robot-learning-ready environment still requires substantial manual effort for object extraction, asset processing, coordinate calibration, and trajectory specification. We present AutoRSR, a highly automated real-to-sim-to real framework that converts scanned real-world workspaces into reusable embodied simulation assets and scalable manipulation data. Given multi-view observations, robot states, and open-vocabulary semantic queries, AutoRSR reconstructs the workspace, extracts individual object assets, and automatically registers the reconstructed scene to the robot base frame at metric scale. Starting from a small number of reference motions, AutoRSR then generates executable manipulation trajectories and scales them through trajectory interpolation and domain randomization. After workspace scanning and semantic queries, the remaining pipeline requires no manual object annotation, scene-to-robot calibration, or robot teleoperation. Across three real-world manipulation tasks, policies trained on 1K automatically generated episodes per task achieve 73.3--86.7% real-world success rates, comparable to pi 0.5 trained from 150 human teleoperated demonstrations, while reducing the human time required for data generation by an order of magnitude. These results demonstrate that real-world observations can be converted into metrically aligned, executable, and scalable simulation data with substantially reduced human effort.

Figures & tables

Fig. 1: Overview of AutoRSR . We first perform a hybrid scan of the workspace while recording robot states and wrist-camera poses, together with a reference manipulation motion. Given open-vocabulary semantic queries, AutoRSR reconstructs the scene and extracts individual object assets. The reconstructed workspace is then automatically registered to the robot base frame, enabling reference trajectories to be replayed and augmented in simulation. Finally, trajectory interpolation and domain randomization generate scalable manipulation data for robot policy learning.

Fig. 2: Object segmentation results on our Pour into the yellow bowl task and LERF [ 32 ] dataset.

Fig. 3: Example of demonstration alignment for the Pour into the yellow bowl task.

Fig. 4: Real-robot inference results for $\pi_{0.5}$ [ 39 ] fine-tuned on 1,000 AutoRSR-augmented episodes. The solid red box highlights the target manipulation region, and the dashed red box at the bottom‑left provides a magnified view of this region for better visualization

Fig. 5: Comparison of data collection time and physical policy success rates between teleoperation and AutoRSR.

Task	Teleop Trajectories			AutoRSR Trajectories
	50	100	150	100	300	1000
Pour into the yellow bowl	0	26.7	73.3	0	13.3	73.3
Put duck on the box	6.7	46.7	86.7	0	20	80
Close the laptop	20	60	86.7	6.7	46.7	86.7

TABLE I: Comparison of Physical Policy Success Rates Across Training Sources(%).

Real-only (150)	Sim-only (1k)	Co-Train (150+1k)
73.3	73.3	86.7

TABLE II: Training-data mixture comparison(%).

w/o Traj. Augmentation	w/ Traj. Augmentation
13.3	73.3

TABLE III: Motion trajectory augmentation ablation(%).

less Vis. Augmentation	more Vis. Augmentation
60	73.3

TABLE IV: Visual augmentation ablation(%).

task	Mean Error(R)	Mean Error(A)	RMSE(R)	RMSE(A)
grasp_duck	8.62	0.61	9.07	0.62
pour	4.58	0.91	5.02	0.91
computer	3.81	0.77	3.95	0.78

TABLE V: Alignment errors across different scenes (cm). (R) represents reconstruction coordinate frame and (A) represents our aligned coordinate frame.

Task	r2r2r [ 4 ]	GASE [ 7 ]	Ours
Scanning	H	H	H
Demonstration	H	-	H
Query	H	H	H
Reconstruction	A	A	A
Segmentation	H	H	A
Assert into Simulation	H	H	A

TABLE VI: Stage-by-stage comparison of human involvement across the entire pipeline. H indicates that human involvement is required, while A indicates that the stage is completed automatically by the system. Since GASE does not perform trajectory reproduction or augmentation, its corresponding entries in the demonstration and augmentation sections are left blank.

Task	Time to complete
Scanning	1 min
Demonstration	10 s
Mask processing	2 min
Reconstruction	10 min
Segmentation	$<$ 10 s
Meshification	2 min

TABLE VII: Upfront processing time until generation. We train the reconstruction model for 7,000 iterations and directly extract meshes using SuGaR [ 36 ] .

Task	Teleop	AutoRSR(Aug)
Pour into the yellow bowl	73.38 min	28.23 min
Put duck on the box	62.78 min	24.98 min
Close the laptop	52.11 min	27.40 min

TABLE VIII: Time required for collecting 150 demonstrations through teleoperation with one human operator and for generating 1,000 synthetic demonstrations using AutoRSR with single GPU.

Synthetic Data Generation for Robotics

Synthetic data generation for robotics creates simulated scenes, sensor readings, and action trajectories to train and evaluate robots.

See all 303 papers

Sep 21, 2026

ARSTAG: An Agentic Real2Sim2Real System for Task-Specific Robot Data Generation

Adapting visuomotor policies to new manipulation tasks often requires substantial manual engineering or teleoperated data collection. Simulation can provide task-specific data at scale, but constructing the scene, designing expert behavior, and configuring data generation still require significant per-task effort. We present ARSTAG, an agentic Real2Sim2Real system that turns a single RGB image and a natural-language instruction directly into robot policy-learning data. A hierarchy of language agents constructs a task-scoped simulation scene, generates robot-feasible demonstrations, and expands the training distribution through task-consistent randomization, while a coordinator agent manages cross-stage feedback and recovery. Across seven manipulation tasks spanning grasping, placement, and stacking, the ARSTAG-generated demonstrations enable sim-to-real transfer of three visuomotor policy architectures to a dual-arm robot, with pi0.5 achieving an average real-world success rate of 74.6%. Ablations show that task-consistent randomization substantially improves robustness, and policy performance increases with generated dataset size. Project webpage: https://boweili666.github.io/ARSTAG/.

Bowei Li, Yuner Zhang, Changliu Liu

Carnegie Mellon University, Pittsburgh, PA 15213, USA

Robotics Simulation Sim-to-Real Transfer Visuomotor Policy Learning Synthetic Data Generation for Robotics Robotic Manipulation Robot Manipulation Datasets

Ask PDF

Oct 7, 2026

Agentic RSR: Real-to-Sim-to-Real through Scene Reconstruction and Execution-Grounded Robot Policies

A simulation of a real robot workspace must preserve task-relevant interactions, while policies developed in it must operate on observations available to the real robot. Yet scene reconstruction and policy development are often treated separately. We present Agentic Real-to-Sim-to-Real (Agentic RSR), a framework that links scene reconstruction, policy development, and real-robot execution through the same manipulation task. Given a workspace video, a task description, and a known robot model, an agent recovers metric scale, iteratively refines the scene using visual feedback, and checks task-relevant interactions in MuJoCo. A coding agent then develops an executable policy, progressing from privileged object poses to visual observations and randomized simulation. The policy can interleave multiple observations and actions within one invocation, while the agent uses execution feedback to continue, retry, or revise its approach. A shared task-level interface carries the policy and accumulated experience to the real robot, where fresh observations and safety checks guide execution. Using Claude Opus 5.5 through DeepSeek Harness (DSH), all 18 scenes across two robots pass the joint visual check; mean four-view Depth MAE against reference depth estimates is 0.0808 m, mean Lab

ΔE_{76}

is 8.01, and mean grayscale SSIM is 0.7338. In real-robot experiments, the aggregate task success rate reaches 80% of the simulation task success rate, indicating substantial retention of simulated performance on hardware. Code and reconstructed scene data will be made publicly available.

Yihan Li, Yating Feng, Shengjiu Sun +7

Sun Yat-sen University · The Hong Kong University of Science and Technology (Guangzhou)

Sim-to-Real Transfer Robotic Manipulation Robotics Simulation Robot Policy Learning Robotic Perception

Ask PDF

Feb 12, 2026

Scene2Demo: Self-Evolving Embodied Data Generation via Object-Action Graph

We present Scene2Demo, an automated framework for offline embodied data generation. Given a single RGB scene image and a user query, Scene2Demo constructs an interactive simulated scene and generates executable task configurations, primitive action flows, multi-view execution videos, and offline robot-learning datasets. Its five role-specialized generation modules perform task understanding, symbolic decomposition, BDDL configuration generation, object-size adjustment, and action-flow planning. The resulting object-centric task specifications and action transitions are organized as path search over a scene-conditioned object-action graph. Failed or incomplete executions are refined by two feedback agents: a Robotic Safety Inspector that produces step-wise visual critiques and a Robotic Task Supervisor that revises action sequences or parameters using the rollout and previous evolution history. Across 102 automatically generated primitive scene-task pairs, Scene2Demo achieves a 71.6% execution success rate; on four representative long-horizon tasks, feedback-driven refinement improves both task success and subtask-level execution quality over primitive-only execution. Comparisons with RoboGen and GenSim2 show stronger execution performance under automated data-generation settings. Finally, behavior cloning policies achieve 96.0% and 92.0% success on two representative tasks, showing that the generated data supports downstream policy learning. Our project page is available at https://scene2demo-anon.github.io/.

Xiang Liu, Sen Cui, Guocai Yao +4

Robot Demonstration Generation Synthetic Data Generation for Robotics Robot Manipulation Datasets Robot Task Planning

Ask PDF

Abstract

Synthetic Data Generation for Robotics

Explore similar work

ARSTAG: An Agentic Real2Sim2Real System for Task-Specific Robot Data Generation

Agentic RSR: Real-to-Sim-to-Real through Scene Reconstruction and Execution-Grounded Robot Policies

Scene2Demo: Self-Evolving Embodied Data Generation via Object-Action Graph