Scene2Demo: Self-Evolving Embodied Data Generation via Object-Action Graph

Xiang Liu; Sen Cui; Guocai Yao; Zhong Cao; Jingheng Ma; Min Zhang; Changshui Zhang

cs.ROFeb 12, 2026

Scene2Demo: Self-Evolving Embodied Data Generation via Object-Action Graph

Authors: Xiang Liu, Sen Cui, Guocai Yao, Zhong Cao, Jingheng Ma, Min Zhang, Changshui Zhang

Abstract

We present Scene2Demo, an automated framework for offline embodied data generation. Given a single RGB scene image and a user query, Scene2Demo constructs an interactive simulated scene and generates executable task configurations, primitive action flows, multi-view execution videos, and offline robot-learning datasets. Its five role-specialized generation modules perform task understanding, symbolic decomposition, BDDL configuration generation, object-size adjustment, and action-flow planning. The resulting object-centric task specifications and action transitions are organized as path search over a scene-conditioned object-action graph. Failed or incomplete executions are refined by two feedback agents: a Robotic Safety Inspector that produces step-wise visual critiques and a Robotic Task Supervisor that revises action sequences or parameters using the rollout and previous evolution history. Across 102 automatically generated primitive scene-task pairs, Scene2Demo achieves a 71.6% execution success rate; on four representative long-horizon tasks, feedback-driven refinement improves both task success and subtask-level execution quality over primitive-only execution. Comparisons with RoboGen and GenSim2 show stronger execution performance under automated data-generation settings. Finally, behavior cloning policies achieve 96.0% and 92.0% success on two representative tasks, showing that the generated data supports downstream policy learning. Our project page is available at https://scene2demo-anon.github.io/.

Figures & tables

Figure 1: Pipeline overview. From one RGB scene image and a task query, Scene2Demo reconstructs an interactive scene, generates symbolic task configurations and action flows, and refines failed executions using visual-feedback agents. Successful rollouts provide multi-view videos and offline data for policy learning.

Figure 2: Structured task generation and feedback-driven refinement. Left: five role-specialized modules expand the query, decompose it into symbolic subtasks, instantiate BDDL-style conditions, adjust object scale, and plan initial action flows. Right: the Robotic Safety Inspector provides step-wise visual critiques, and the Robotic Task Supervisor revises failed flows using the rollout and evolution history.

Figure 3: Object-action task graph. Nodes represent object-conditioned primitive actions at local time steps. Intra-task edges form action flows, while transfer edges connect adjacent subtasks; task generation searches a scene-conditioned path from the initial state to the user goal.

Category	Count	Success	SR (%)
$\mathcal{T}_{5}:$ Open/Close	36	24	66.7
$\mathcal{T}_{6}:$ Pick/Place	66	49	74.2
Total	102	73	71.6

Table 1: Primitive-task execution. Success rates on 102 scene-task pairs from 34 generated RGB scene images with three asset variations.

\mathcal{T}_{5}

denotes Open/Close and

\mathcal{T}_{6}

denotes Pick/Place.

Method	$\mathcal{T}_{1}$	$\mathcal{T}_{2}$	$\mathcal{T}_{3}$	$\mathcal{T}_{4}$	$\mathcal{T}_{5}$	$\mathcal{T}_{6}$
M1: Ours (Primitive)	0 / 50	0 / 50	40 / 70	0 / 50	66.7 / 66.7	74.2 / 74.2
M2: Ours (Prim+Evo)	40 / 84	30 / 79	70 / 93	70 / 85	–	–
M3: GenSim2 (Primitive)	0 / 73	5 / 78	55 / 91	0 / 68	100 / 100	38.8 / 82.1
M4: RoboGen (Primitive)	0 / 50	0 / 46.5	0 / 50	0 / 50	0 / 50.2	0 / 42.8
M5: RoboGen (Prim+RL)	fail	fail	fail	fail	success	success

Table 2: Main execution comparison. Each cell reports SR / Subtask-Level Score (%). M1/M2 are Scene2Demo with primitive-only planning and with refinement; M3/M4 are GenSim2 and RoboGen; M5 is RoboGen with task-specific RL. Per-model refinement results are given in Table 3 .

Figure 4: Feedback-driven refinement on $\mathcal{T}_{1}$ (Glass $\to$ Refrigerator). The four subtasks are opening the refrigerator, picking up the glass, placing it inside, and closing the refrigerator.

Tasks	VLM	Demos	Subtask 1		Subtask 2			Subtask 3			Subtask 4			Complete Task
Tasks	VLM	Demos	SR	Iter	SR	ESR	Iter	SR	ESR	Iter	SR	ESR	Iter	SR	Iter
$\mathcal{T}_{1}$	Gemini-3-Flash	10	100	0	100	–	0	30	30	1.7	100	100	0.7	30	2.3
	GPT-5.2	10	100	0	100	–	0	30	30	2	66.7	66.7	1.0	20	3.5
	Qwen3-VL	20	100	0	100	–	0	45	45	2.7	88.9	66.7	0.5	40	3.1
$\mathcal{T}_{2}$	Qwen3-VL	10	100	0	100	100	0.3	40	40	1.8	75	66.7	0.7	30	2.3
$\mathcal{T}_{3}$	Gemini-3-Flash	10	100	0	70	62.5	1.6	–	–	–	–	–	–	70	1.6

Table 3: Refinement performance on long-horizon tasks. We report execution success (SR), failed first attempts rescued by refinement (ESR), and mean iterations for successful executions (Iter).

\mathcal{T}_{3}

\mathcal{T}_{4}

have two subtasks (–).

Panel of Figure 5 — Figure 5: Evaluation and ablations. (a) Execution over 102 primitive scene-task pairs. (b) Repeated decomposition of one query. (c) Wall-clock time (mean and observed range). (d–e) Camera-view and context-window ablations on $\mathcal{T}_{1}$ .

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: Visual Comparison of Camera Perspectives. The figure is divided into two task scenarios. Left (Columns 1-3): The task “Put the cup into the refrigerator”. The Global View reveals the arm reaching inside the fridge, though the specific arm posture is occluded. The Head View clearly shows the robot’s joint state and the open fridge door. The Wrist View captures the close-up of the cup within the gripper. Right (Columns 4-6): The task “Place the apple into the bowl”. The Global View shows the overall layout, but depth ambiguity makes it hard to judge if the apple is perfectly over the bowl. The Head View compensates for this by showing the alignment relative to the base. However, since the gripper is in mid-air, the Wrist View captures mostly empty space, providing little actionable information.

Panel of Figure 7 — Figure 7: Visualization of 20 Representative Scene-Task Pairs. Top Row: RGB scene images generated by Qwen3-Max or Seedream-4.5. Bottom Row: Reconstructed interactive simulation environments. Each column corresponds to a specific atomic task proposed by the VLM, demonstrating our framework’s ability to generate diverse household scenarios.

Method	$\mathcal{T}_{1}$	$\mathcal{T}_{2}$	$\mathcal{T}_{3}$	$\mathcal{T}_{4}$	$\mathcal{T}_{5}$	$\mathcal{T}_{6}$
Ours (primitive)	10 $(10{\times}1)$	10 $(10{\times}1)$	10 $(10{\times}1)$	10 $(10{\times}1)$	36 $(1{\times}36)$	66 $(1{\times}66)$
Ours (prim+Evo)	20 $(20{\times}1)$	10 $(10{\times}1)$	10 $(10{\times}1)$	10 $(10{\times}1)$	–	–
GenSim2 (primitive)	20 $(20{\times}1)$	20 $(20{\times}1)$	20 $(20{\times}1)$	20 $(20{\times}1)$	80 $(20{\times}4)$	80 $(20{\times}4)$
RoboGen (primitive)	10 $(10{\times}1)$	10 $(10{\times}1)$	10 $(10{\times}1)$	10 $(10{\times}1)$	40 $(10{\times}4)$	40 $(10{\times}4)$
RoboGen (prim+RL)	1	1	1	1	1	1

Appendix

Table 4: Exact number of evaluated demos and episodes for each method and task. Each cell reports all demos as demos/episode

\times

episodes .

Panel of Figure 8 — Figure 8: Visual Overview of Complex Long-Horizon Tasks. Row 1: Pairs of generated RGB scene images and reconstructed simulation environments. Row 2: Close-up views of the main target object and support object matched from the simulation environments. Row 3: The user-provided keywords that seed the autonomous task generation process.

Tasks	Subtask Index	Initial State	Target State
$\mathcal{T}_{1}$ : Glass $\to$ Fridge	1	(not (open refrigerator_0)	(open refrigerator_0)
	2	(ontop glass_0 table_0)	(not (ontop glass_0 table_0))
	3	(inside glass_0 gripper)	(inside glass_0 refrigerator_0)
	4	(open refrigerator_0)	(not (open refrigerator_0))
$\mathcal{T}_{2}$ : Apple $\to$ Fridge	1	(not (open refrigerator_0)	(open refrigerator_0)
	2	(ontop apple_0 table_0)	(not (ontop apple_0 table_0))

Appendix

Table 5: Initial and goal states for the sub-tasks of our complex long-horizon tasks

Tasks	Subtask	Initial Action Flow (Pre-evolution)
$\mathcal{T}_{1}$	1	APPROACH $\to$ CONVERGE $\to$ GRASP $\to$ ARTICULATE_OPEN $\to$ UNGRASP
	2	NAVIGATE_TO_TARGET $\to$ APPROACH $\to$ CONVERGE $\to$ GRASP $\to$ LIFT_EEF_UP(0.2)
	3	NAVIGATE_TO_SUPPORT $\to$ MOVE_BASE_FORWARD(0.4) $\to$ MOVE_EEF_FORWARD(0.1) $\to$ UNGRASP
	4	NAVIGATE_TO_TARGET $\to$ APPROACH $\to$ CONVERGE $\to$ GRASP $\to$ ARTICULATE_CLOSE(0.0, 0.5) $\to$ UNGRASP
$\mathcal{T}_{2}$	1	APPROACH $\to$ CONVERGE $\to$ GRASP $\to$ ARTICULATE_OPEN(0.0, 0.7) $\to$ UNGRASP $\to$ RETREAT
	2	NAVIGATE_TO_TARGET $\to$ RETREAT $\to$ APPROACH $\to$ CONVERGE $\to$ GRASP $\to$ LIFT_EEF_UP(0.2)

Appendix

Table 6: Initial action flows for the sub-tasks of our representative complex long-horizon tasks

Subtask	Iter	Action Flow
1	0	APPROACH $\to$ CONVERGE $\to$ GRASP $\to$ ARTICULATE_OPEN $\to$ UNGRASP
2	0	NAVIGATE_TO_TARGET $\to$ APPROACH $\to$ CONVERGE $\to$ GRASP $\to$ LIFT_EEF_UP (0.2)
3	0	NAVIGATE_TO_SUPPORT $\to$ MOVE_BASE_FORWARD (0.4) $\to$ MOVE_EEF_FORWARD (0.1) $\to$ UNGRASP
	1	NAVIGATE_TO_SUPPORT $\to$ MOVE_BASE_FORWARD (0.3) $\to$ MOVE_EEF_FORWARD (0.45) $\to$ UNGRASP
	2	NAVIGATE_TO_SUPPORT $\to$ MOVE_BASE_FORWARD (0.3) $\to$ MOVE_EEF_FORWARD (0.45) $\to$ MOVE_EEF_FORWARD (0.2)
	2	$\to$ LIFT_EEF_DOWN (0.3) $\to$ UNGRASP

Appendix

Table 7: Evolutionary Trajectory for Task

\mathcal{T}_{1}

: “Put the glass into the refrigerator”. Subtasks 1 and 2 succeeded initially. Subtask 3 required 2 iterations to correct placement height and depth. Subtask 4 required 1 iteration to fully close the door.

Subtask	Iter	Action Flow
1	0	APPROACH $\to$ CONVERGE $\to$ GRASP $\to$ ARTICULATE_OPEN (0.0, 0.7) $\to$ UNGRASP $\to$ RETREAT
2	0	NAVIGATE_TO_TARGET $\to$ RETREAT $\to$ APPROACH $\to$ CONVERGE $\to$ GRASP $\to$ LIFT_EEF_UP (0.2)
3	0	NAVIGATE_TO_SUPPORT $\to$ MOVE_BASE_FORWARD (0.4) $\to$ MOVE_EEF_FORWARD (0.1) $\to$ UNGRASP
	1	NAVIGATE_TO_SUPPORT $\to$ MOVE_BASE_BACKWARD (0.3) $\to$ MOVE_EEF_FORWARD (0.4) $\to$ MOVE_EEF_FORWARD (0.2) $\to$ UNGRASP
	2	NAVIGATE_TO_SUPPORT $\to$ MOVE_BASE_BACKWARD (0.3) $\to$ MOVE_EEF_FORWARD (0.4) $\to$ TURN_BASE_LEFT (45)
	2	$\to$ MOVE_EEF_FORWARD (0.2) $\to$ UNGRASP

Appendix

Table 8: Evolutionary Trajectory for Task

\mathcal{T}_{2}

: “Put the apple into the refrigerator”. Subtask 3 underwent 3 iterations, introducing base rotation and finer arm control to navigate the apple onto the fridge shelf.

Subtask	Iter	Action Flow
1	0	APPROACH $\to$ CONVERGE $\to$ GRASP $\to$ LIFT_EEF_UP (0.2)
2	0	NAVIGATE_TO_SUPPORT $\to$ MOVE_BASE_FORWARD (0.4) $\to$ MOVE_EEF_FORWARD (0.1) $\to$ LIFT_EEF_DOWN (0.3) $\to$ UNGRASP
	1	NAVIGATE_TO_SUPPORT $\to$ MOVE_BASE_BACKWARD (0.15) $\to$ MOVE_EEF_FORWARD (0.4) $\to$ TURN_BASE_LEFT (15)
	1	$\to$ MOVE_EEF_FORWARD (0.1) $\to$ LIFT_EEF_DOWN (0.4) $\to$ UNGRASP
	2	NAVIGATE_TO_SUPPORT $\to$ MOVE_BASE_BACKWARD (0.3) $\to$ MOVE_EEF_FORWARD (0.2) $\to$ TURN_BASE_RIGHT (20)
	2	$\to$ MOVE_EEF_FORWARD (0.25) $\to$ LIFT_EEF_DOWN (0.4) $\to$ UNGRASP

Appendix

Table 9: Evolutionary Trajectory for Task

\mathcal{T}_{3}

: “Put the apple into the bowl”. The placement subtask (Subtask 2) required 2 iterations to prevent collision with the bowl’s rim.

Subtask	Iter	Action Flow
1	0	APPROACH $\to$ CONVERGE $\to$ GRASP $\to$ LIFT_EEF_UP (0.2)
2	0	NAVIGATE_TO_SUPPORT $\to$ MOVE_BASE_FORWARD (0.4) $\to$ MOVE_EEF_FORWARD (0.1) $\to$ LIFT_EEF_DOWN (0.3) $\to$ UNGRASP
	1	NAVIGATE_TO_SUPPORT $\to$ TURN_BASE_RIGHT (60) $\to$ MOVE_EEF_FORWARD (0.4) $\to$ MOVE_EEF_FORWARD (0.1)
	1	$\to$ LIFT_EEF_DOWN (0.3) $\to$ UNGRASP
	2	NAVIGATE_TO_SUPPORT $\to$ TURN_BASE_RIGHT (60) $\to$ MOVE_EEF_FORWARD (0.4) $\to$ MOVE_EEF_FORWARD (0.2)
	2	$\to$ LIFT_EEF_DOWN (0.3) $\to$ UNGRASP

Appendix

Table 10: Evolutionary Trajectory for Task

\mathcal{T}_{4}

: “Put the cup on the table”. Subtask 2 required 2 iterations, heavily relying on base rotation to align with the table surface.

Panel of Figure 9 — Figure 9: Failure Analysis for $\mathcal{T}_{1}$ Subtask 3. Despite 4 rounds of evolution (Iter 5 also failed), the robot consistently fails to insert the glass. The root cause, visible in the Head View, is that the lower link of the robotic arm collides with the left wall of the refrigerator, preventing the end-effector from advancing. This kinematic constraint is difficult for the VLM to diagnose from 2D images, as it requires a subtle backward adjustment of the mobile base to clear the singularity, which is a correction that iterative parameter tuning failed to find.

Panel of Figure 10 — Figure 10: Failure Analysis for $\mathcal{T}_{3}$ Subtask 2. The robot struggles to align the apple directly above the bowl. Over 4 iterations, the VLM attempts to correct the drift, but lacks the precision to calculate the exact offset. The open-loop nature means the VLM oscillates between over-correcting (moving too far left) and under-correcting, failing to find the ”sweet spot” within the limited iteration budget.

Figure 11: Policy architecture for downstream behavior cloning. The policy encodes randomized point clouds with a PointNet encoder, fuses the point-cloud feature with proprioceptive states, and predicts 10-DoF robot actions using a 4-layer causal Transformer and an action MLP.

Synthetic Data Generation for Robotics

Synthetic data generation for robotics creates simulated scenes, sensor readings, and action trajectories to train and evaluate robots.

See all 288 papers

Oct 1, 2026

Awomo-SimDataEngine: Agentic Simulation-ReadyWorld Generation

Generating useful robot-training data requires more than visually plausiblescenes: objects must support interaction, placements must remain physicallyvalid, and tasks must admit repeatable execution. We presentAwomo-SimDataEngine, an agentic system that connects asset and scenegeneration to robot demonstration synthesis. Shared asset services providerigid and articulated objects, including structure-grounded part and jointgeneration with ISArt. Scene generation supports two complementary routes:Unravel reconstructs editable scenes from images, while SimForge buildssingle-room and multi-room environments from text. A graph-native harnesscoordinates construction, validation, andbounded repair, routing failures to the responsible module while retainingunaffected scene state. PolicyForge binds validated worlds to tasks and robotembodiments to produce replayable demonstrations. Evaluations cover assetgeometry, scene quality, and downstream policy learning. On MuJoCo-basedLIBERO-Plus, co-training with Isaac Sim demonstrations improves the overallsuccess rate of a World-Action Model (WAM) from

77.17\%

89.43\%

. Goal and spatialsuccess improve by

31.66

and

6.25

percentage points, respectively.These results support the utility of the generated data for cross-simulatorpolicy training, with more limited gains on long-horizon tasks.

Awomo-PhysicalRSI Team, Danjiao Ma, Enhui Ma +21

Awomo-PhysicalRSI Team

Synthetic Data Generation for Robotics Robotics Simulation Functional 3D Scene Generation Indoor Scene Generation

Ask PDF

Oct 6, 2026

EmbodiedSmith: Scaling Embodied Data through Recursive Self-Improvement Flywheel in Simulation

Scaling robotic foundation models requires diverse training data and reliable evaluation environments. Simulation offers a scalable solution, yet existing generation pipelines remain constrained by predefined assets and skills, a disconnect between scene generation and task generation, and limited support for complex embodiments and physics. We introduce EmbodiedSmith, a framework for scalable embodied data generation through recursive self-improvement (RSI). EmbodiedSmith unifies asset, scene, and task generation in a pipeline that supports autonomous creation and language-driven customization. Its core is an agentic refinement loop: scene generation anticipates downstream task requirements, while task generation guides targeted scene edits, allowing scenes and tasks to iteratively improve one another. This joint refinement improves task generation success, including for long-horizon tasks. The framework further supports mobile manipulators, humanoids, and dexterous hands, as well as interactions involving deformable objects and fluids, broadening the range of behaviors and physical phenomena represented in generated data. Together, these capabilities provide a flexible simulation engine for both robot pretraining and evaluation. Extensive experiments validate the quality, diversity, and generation efficiency of the resulting data, while downstream policy experiments demonstrate that increased data diversity improves generalization.

Yikai Qin, Yifei Deng, Mingjian Liang +13

The Hong Kong University of Science and Technology (Guangzhou) · Xiaomi Robotics · Southeastern University +7

Robot Foundation Models Robotics Simulation Synthetic Data Generation for Robotics Functional 3D Scene Generation

Ask PDF

Sep 21, 2026

ARSTAG: An Agentic Real2Sim2Real System for Task-Specific Robot Data Generation

Adapting visuomotor policies to new manipulation tasks often requires substantial manual engineering or teleoperated data collection. Simulation can provide task-specific data at scale, but constructing the scene, designing expert behavior, and configuring data generation still require significant per-task effort. We present ARSTAG, an agentic Real2Sim2Real system that turns a single RGB image and a natural-language instruction directly into robot policy-learning data. A hierarchy of language agents constructs a task-scoped simulation scene, generates robot-feasible demonstrations, and expands the training distribution through task-consistent randomization, while a coordinator agent manages cross-stage feedback and recovery. Across seven manipulation tasks spanning grasping, placement, and stacking, the ARSTAG-generated demonstrations enable sim-to-real transfer of three visuomotor policy architectures to a dual-arm robot, with pi0.5 achieving an average real-world success rate of 74.6%. Ablations show that task-consistent randomization substantially improves robustness, and policy performance increases with generated dataset size. Project webpage: https://boweili666.github.io/ARSTAG/.

Bowei Li, Yuner Zhang, Changliu Liu

Carnegie Mellon University, Pittsburgh, PA 15213, USA

Robotics Simulation Sim-to-Real Transfer Visuomotor Policy Learning Synthetic Data Generation for Robotics Robotic Manipulation Robot Manipulation Datasets

Ask PDF

Abstract

Synthetic Data Generation for Robotics

Explore similar work

Awomo-SimDataEngine: Agentic Simulation-ReadyWorld Generation

EmbodiedSmith: Scaling Embodied Data through Recursive Self-Improvement Flywheel in Simulation

ARSTAG: An Agentic Real2Sim2Real System for Task-Specific Robot Data Generation