Scene2Demo: Self-Evolving Embodied Data Generation via Object-Action Graph

Xiang Liu; Sen Cui; Guocai Yao; Zhong Cao; Jingheng Ma; Min Zhang; Changshui Zhang

cs.ROFeb 12, 2026

Scene2Demo: Self-Evolving Embodied Data Generation via Object-Action Graph

Authors: Xiang Liu, Sen Cui, Guocai Yao, Zhong Cao, Jingheng Ma, Min Zhang, Changshui Zhang

Abstract

We present Scene2Demo, an automated framework for offline embodied data generation. Given a single RGB scene image and a user query, Scene2Demo constructs an interactive simulated scene and generates executable task configurations, primitive action flows, multi-view execution videos, and offline robot-learning datasets. Its five role-specialized generation modules perform task understanding, symbolic decomposition, BDDL configuration generation, object-size adjustment, and action-flow planning. The resulting object-centric task specifications and action transitions are organized as path search over a scene-conditioned object-action graph. Failed or incomplete executions are refined by two feedback agents: a Robotic Safety Inspector that produces step-wise visual critiques and a Robotic Task Supervisor that revises action sequences or parameters using the rollout and previous evolution history. Across 102 automatically generated primitive scene-task pairs, Scene2Demo achieves a 71.6% execution success rate; on four representative long-horizon tasks, feedback-driven refinement improves both task success and subtask-level execution quality over primitive-only execution. Comparisons with RoboGen and GenSim2 show stronger execution performance under automated data-generation settings. Finally, behavior cloning policies achieve 96.0% and 92.0% success on two representative tasks, showing that the generated data supports downstream policy learning. Our project page is available at https://scene2demo-anon.github.io/.

Figures & tables

Figure 1: Pipeline overview. From one RGB scene image and a task query, Scene2Demo reconstructs an interactive scene, generates symbolic task configurations and action flows, and refines failed executions using visual-feedback agents. Successful rollouts provide multi-view videos and offline data for policy learning.

Figure 2: Structured task generation and feedback-driven refinement. Left: five role-specialized modules expand the query, decompose it into symbolic subtasks, instantiate BDDL-style conditions, adjust object scale, and plan initial action flows. Right: the Robotic Safety Inspector provides step-wise visual critiques, and the Robotic Task Supervisor revises failed flows using the rollout and evolution history.

Figure 3: Object-action task graph. Nodes represent object-conditioned primitive actions at local time steps. Intra-task edges form action flows, while transfer edges connect adjacent subtasks; task generation searches a scene-conditioned path from the initial state to the user goal.

Category	Count	Success	SR (%)
$\mathcal{T}_{5}:$ Open/Close	36	24	66.7
$\mathcal{T}_{6}:$ Pick/Place	66	49	74.2
Total	102	73	71.6

Table 1: Primitive-task execution. Success rates on 102 scene-task pairs from 34 generated RGB scene images with three asset variations.

\mathcal{T}_{5}

denotes Open/Close and

\mathcal{T}_{6}

denotes Pick/Place.

Method	$\mathcal{T}_{1}$	$\mathcal{T}_{2}$	$\mathcal{T}_{3}$	$\mathcal{T}_{4}$	$\mathcal{T}_{5}$	$\mathcal{T}_{6}$
M1: Ours (Primitive)	0 / 50	0 / 50	40 / 70	0 / 50	66.7 / 66.7	74.2 / 74.2
M2: Ours (Prim+Evo)	40 / 84	30 / 79	70 / 93	70 / 85	–	–
M3: GenSim2 (Primitive)	0 / 73	5 / 78	55 / 91	0 / 68	100 / 100	38.8 / 82.1
M4: RoboGen (Primitive)	0 / 50	0 / 46.5	0 / 50	0 / 50	0 / 50.2	0 / 42.8
M5: RoboGen (Prim+RL)	fail	fail	fail	fail	success	success

Table 2: Main execution comparison. Each cell reports SR / Subtask-Level Score (%). M1/M2 are Scene2Demo with primitive-only planning and with refinement; M3/M4 are GenSim2 and RoboGen; M5 is RoboGen with task-specific RL. Per-model refinement results are given in Table 3 .

Figure 4: Feedback-driven refinement on $\mathcal{T}_{1}$ (Glass $\to$ Refrigerator). The four subtasks are opening the refrigerator, picking up the glass, placing it inside, and closing the refrigerator.

Tasks	VLM	Demos	Subtask 1		Subtask 2			Subtask 3			Subtask 4			Complete Task
Tasks	VLM	Demos	SR	Iter	SR	ESR	Iter	SR	ESR	Iter	SR	ESR	Iter	SR	Iter
$\mathcal{T}_{1}$	Gemini-3-Flash	10	100	0	100	–	0	30	30	1.7	100	100	0.7	30	2.3
	GPT-5.2	10	100	0	100	–	0	30	30	2	66.7	66.7	1.0	20	3.5
	Qwen3-VL	20	100	0	100	–	0	45	45	2.7	88.9	66.7	0.5	40	3.1
$\mathcal{T}_{2}$	Qwen3-VL	10	100	0	100	100	0.3	40	40	1.8	75	66.7	0.7	30	2.3
$\mathcal{T}_{3}$	Gemini-3-Flash	10	100	0	70	62.5	1.6	–	–	–	–	–	–	70	1.6

Table 3: Refinement performance on long-horizon tasks. We report execution success (SR), failed first attempts rescued by refinement (ESR), and mean iterations for successful executions (Iter).

\mathcal{T}_{3}

\mathcal{T}_{4}

have two subtasks (–).

Panel of Figure 5 — Figure 5: Evaluation and ablations. (a) Execution over 102 primitive scene-task pairs. (b) Repeated decomposition of one query. (c) Wall-clock time (mean and observed range). (d–e) Camera-view and context-window ablations on $\mathcal{T}_{1}$ .

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: Visual Comparison of Camera Perspectives. The figure is divided into two task scenarios. Left (Columns 1-3): The task “Put the cup into the refrigerator”. The Global View reveals the arm reaching inside the fridge, though the specific arm posture is occluded. The Head View clearly shows the robot’s joint state and the open fridge door. The Wrist View captures the close-up of the cup within the gripper. Right (Columns 4-6): The task “Place the apple into the bowl”. The Global View shows the overall layout, but depth ambiguity makes it hard to judge if the apple is perfectly over the bowl. The Head View compensates for this by showing the alignment relative to the base. However, since the gripper is in mid-air, the Wrist View captures mostly empty space, providing little actionable information.

Panel of Figure 7 — Figure 7: Visualization of 20 Representative Scene-Task Pairs. Top Row: RGB scene images generated by Qwen3-Max or Seedream-4.5. Bottom Row: Reconstructed interactive simulation environments. Each column corresponds to a specific atomic task proposed by the VLM, demonstrating our framework’s ability to generate diverse household scenarios.

Method	$\mathcal{T}_{1}$	$\mathcal{T}_{2}$	$\mathcal{T}_{3}$	$\mathcal{T}_{4}$	$\mathcal{T}_{5}$	$\mathcal{T}_{6}$
Ours (primitive)	10 $(10{\times}1)$	10 $(10{\times}1)$	10 $(10{\times}1)$	10 $(10{\times}1)$	36 $(1{\times}36)$	66 $(1{\times}66)$
Ours (prim+Evo)	20 $(20{\times}1)$	10 $(10{\times}1)$	10 $(10{\times}1)$	10 $(10{\times}1)$	–	–
GenSim2 (primitive)	20 $(20{\times}1)$	20 $(20{\times}1)$	20 $(20{\times}1)$	20 $(20{\times}1)$	80 $(20{\times}4)$	80 $(20{\times}4)$
RoboGen (primitive)	10 $(10{\times}1)$	10 $(10{\times}1)$	10 $(10{\times}1)$	10 $(10{\times}1)$	40 $(10{\times}4)$	40 $(10{\times}4)$
RoboGen (prim+RL)	1	1	1	1	1	1

Appendix

Table 4: Exact number of evaluated demos and episodes for each method and task. Each cell reports all demos as demos/episode

\times

episodes .

Panel of Figure 8 — Figure 8: Visual Overview of Complex Long-Horizon Tasks. Row 1: Pairs of generated RGB scene images and reconstructed simulation environments. Row 2: Close-up views of the main target object and support object matched from the simulation environments. Row 3: The user-provided keywords that seed the autonomous task generation process.

Tasks	Subtask Index	Initial State	Target State
$\mathcal{T}_{1}$ : Glass $\to$ Fridge	1	(not (open refrigerator_0)	(open refrigerator_0)
	2	(ontop glass_0 table_0)	(not (ontop glass_0 table_0))
	3	(inside glass_0 gripper)	(inside glass_0 refrigerator_0)
	4	(open refrigerator_0)	(not (open refrigerator_0))
$\mathcal{T}_{2}$ : Apple $\to$ Fridge	1	(not (open refrigerator_0)	(open refrigerator_0)
	2	(ontop apple_0 table_0)	(not (ontop apple_0 table_0))

Appendix

Table 5: Initial and goal states for the sub-tasks of our complex long-horizon tasks

Tasks	Subtask	Initial Action Flow (Pre-evolution)
$\mathcal{T}_{1}$	1	APPROACH $\to$ CONVERGE $\to$ GRASP $\to$ ARTICULATE_OPEN $\to$ UNGRASP
	2	NAVIGATE_TO_TARGET $\to$ APPROACH $\to$ CONVERGE $\to$ GRASP $\to$ LIFT_EEF_UP(0.2)
	3	NAVIGATE_TO_SUPPORT $\to$ MOVE_BASE_FORWARD(0.4) $\to$ MOVE_EEF_FORWARD(0.1) $\to$ UNGRASP
	4	NAVIGATE_TO_TARGET $\to$ APPROACH $\to$ CONVERGE $\to$ GRASP $\to$ ARTICULATE_CLOSE(0.0, 0.5) $\to$ UNGRASP
$\mathcal{T}_{2}$	1	APPROACH $\to$ CONVERGE $\to$ GRASP $\to$ ARTICULATE_OPEN(0.0, 0.7) $\to$ UNGRASP $\to$ RETREAT
	2	NAVIGATE_TO_TARGET $\to$ RETREAT $\to$ APPROACH $\to$ CONVERGE $\to$ GRASP $\to$ LIFT_EEF_UP(0.2)

Appendix

Table 6: Initial action flows for the sub-tasks of our representative complex long-horizon tasks

Subtask	Iter	Action Flow
1	0	APPROACH $\to$ CONVERGE $\to$ GRASP $\to$ ARTICULATE_OPEN $\to$ UNGRASP
2	0	NAVIGATE_TO_TARGET $\to$ APPROACH $\to$ CONVERGE $\to$ GRASP $\to$ LIFT_EEF_UP (0.2)
3	0	NAVIGATE_TO_SUPPORT $\to$ MOVE_BASE_FORWARD (0.4) $\to$ MOVE_EEF_FORWARD (0.1) $\to$ UNGRASP
	1	NAVIGATE_TO_SUPPORT $\to$ MOVE_BASE_FORWARD (0.3) $\to$ MOVE_EEF_FORWARD (0.45) $\to$ UNGRASP
	2	NAVIGATE_TO_SUPPORT $\to$ MOVE_BASE_FORWARD (0.3) $\to$ MOVE_EEF_FORWARD (0.45) $\to$ MOVE_EEF_FORWARD (0.2)
	2	$\to$ LIFT_EEF_DOWN (0.3) $\to$ UNGRASP

Appendix

Table 7: Evolutionary Trajectory for Task

\mathcal{T}_{1}

: “Put the glass into the refrigerator”. Subtasks 1 and 2 succeeded initially. Subtask 3 required 2 iterations to correct placement height and depth. Subtask 4 required 1 iteration to fully close the door.

Subtask	Iter	Action Flow
1	0	APPROACH $\to$ CONVERGE $\to$ GRASP $\to$ ARTICULATE_OPEN (0.0, 0.7) $\to$ UNGRASP $\to$ RETREAT
2	0	NAVIGATE_TO_TARGET $\to$ RETREAT $\to$ APPROACH $\to$ CONVERGE $\to$ GRASP $\to$ LIFT_EEF_UP (0.2)
3	0	NAVIGATE_TO_SUPPORT $\to$ MOVE_BASE_FORWARD (0.4) $\to$ MOVE_EEF_FORWARD (0.1) $\to$ UNGRASP
	1	NAVIGATE_TO_SUPPORT $\to$ MOVE_BASE_BACKWARD (0.3) $\to$ MOVE_EEF_FORWARD (0.4) $\to$ MOVE_EEF_FORWARD (0.2) $\to$ UNGRASP
	2	NAVIGATE_TO_SUPPORT $\to$ MOVE_BASE_BACKWARD (0.3) $\to$ MOVE_EEF_FORWARD (0.4) $\to$ TURN_BASE_LEFT (45)
	2	$\to$ MOVE_EEF_FORWARD (0.2) $\to$ UNGRASP

Appendix

Table 8: Evolutionary Trajectory for Task

\mathcal{T}_{2}

: “Put the apple into the refrigerator”. Subtask 3 underwent 3 iterations, introducing base rotation and finer arm control to navigate the apple onto the fridge shelf.

Subtask	Iter	Action Flow
1	0	APPROACH $\to$ CONVERGE $\to$ GRASP $\to$ LIFT_EEF_UP (0.2)
2	0	NAVIGATE_TO_SUPPORT $\to$ MOVE_BASE_FORWARD (0.4) $\to$ MOVE_EEF_FORWARD (0.1) $\to$ LIFT_EEF_DOWN (0.3) $\to$ UNGRASP
	1	NAVIGATE_TO_SUPPORT $\to$ MOVE_BASE_BACKWARD (0.15) $\to$ MOVE_EEF_FORWARD (0.4) $\to$ TURN_BASE_LEFT (15)
	1	$\to$ MOVE_EEF_FORWARD (0.1) $\to$ LIFT_EEF_DOWN (0.4) $\to$ UNGRASP
	2	NAVIGATE_TO_SUPPORT $\to$ MOVE_BASE_BACKWARD (0.3) $\to$ MOVE_EEF_FORWARD (0.2) $\to$ TURN_BASE_RIGHT (20)
	2	$\to$ MOVE_EEF_FORWARD (0.25) $\to$ LIFT_EEF_DOWN (0.4) $\to$ UNGRASP

Appendix

Table 9: Evolutionary Trajectory for Task

\mathcal{T}_{3}

: “Put the apple into the bowl”. The placement subtask (Subtask 2) required 2 iterations to prevent collision with the bowl’s rim.

Subtask	Iter	Action Flow
1	0	APPROACH $\to$ CONVERGE $\to$ GRASP $\to$ LIFT_EEF_UP (0.2)
2	0	NAVIGATE_TO_SUPPORT $\to$ MOVE_BASE_FORWARD (0.4) $\to$ MOVE_EEF_FORWARD (0.1) $\to$ LIFT_EEF_DOWN (0.3) $\to$ UNGRASP
	1	NAVIGATE_TO_SUPPORT $\to$ TURN_BASE_RIGHT (60) $\to$ MOVE_EEF_FORWARD (0.4) $\to$ MOVE_EEF_FORWARD (0.1)
	1	$\to$ LIFT_EEF_DOWN (0.3) $\to$ UNGRASP
	2	NAVIGATE_TO_SUPPORT $\to$ TURN_BASE_RIGHT (60) $\to$ MOVE_EEF_FORWARD (0.4) $\to$ MOVE_EEF_FORWARD (0.2)
	2	$\to$ LIFT_EEF_DOWN (0.3) $\to$ UNGRASP

Appendix

Table 10: Evolutionary Trajectory for Task

\mathcal{T}_{4}

: “Put the cup on the table”. Subtask 2 required 2 iterations, heavily relying on base rotation to align with the table surface.

Panel of Figure 9 — Figure 9: Failure Analysis for $\mathcal{T}_{1}$ Subtask 3. Despite 4 rounds of evolution (Iter 5 also failed), the robot consistently fails to insert the glass. The root cause, visible in the Head View, is that the lower link of the robotic arm collides with the left wall of the refrigerator, preventing the end-effector from advancing. This kinematic constraint is difficult for the VLM to diagnose from 2D images, as it requires a subtle backward adjustment of the mobile base to clear the singularity, which is a correction that iterative parameter tuning failed to find.

Panel of Figure 10 — Figure 10: Failure Analysis for $\mathcal{T}_{3}$ Subtask 2. The robot struggles to align the apple directly above the bowl. Over 4 iterations, the VLM attempts to correct the drift, but lacks the precision to calculate the exact offset. The open-loop nature means the VLM oscillates between over-correcting (moving too far left) and under-correcting, failing to find the ”sweet spot” within the limited iteration budget.

Figure 11: Policy architecture for downstream behavior cloning. The policy encodes randomized point clouds with a PointNet encoder, fuses the point-cloud feature with proprioceptive states, and predicts 10-DoF robot actions using a 4-layer causal Transformer and an action MLP.

Synthetic Data Generation for Robotics

Synthetic data generation for robotics creates simulated scenes, sensor readings, and action trajectories to train and evaluate robots.

See all 288 papers

Awomo-SimDataEngine: Agentic Simulation-ReadyWorld Generation
Oct 1, 2026Awomo-PhysicalRSI Team, Danjiao Ma, Enhui Ma +21Synthetic Data Generation for Robotics Robotics Simulation
Awomo-PhysicalRSI Team
EmbodiedSmith: Scaling Embodied Data through Recursive Self-Improvement Flywheel in Simulation
Oct 6, 2026Yikai Qin, Yifei Deng, Mingjian Liang +13Robot Foundation Models Robotics Simulation
The Hong Kong University of Science and Technology (Guangzhou) · Xiaomi Robotics · Southeastern University +7
ARSTAG: An Agentic Real2Sim2Real System for Task-Specific Robot Data Generation
Sep 21, 2026Bowei Li, Yuner Zhang, Changliu LiuRobotics Simulation Sim-to-Real Transfer
Carnegie Mellon University, Pittsburgh, PA 15213, USA

Oct 1, 2026J/K move · Enter open · S save

Awomo-SimDataEngine: Agentic Simulation-ReadyWorld Generation

Awomo-PhysicalRSI Team, Danjiao Ma, Enhui Ma, Haohan Liu+20

Awomo-PhysicalRSI Team

Synthetic Data Generation for Robotics Robotics Simulation Functional 3D Scene Generation Indoor Scene Generation

Open paper AskPDF arXiv

Generating useful robot-training data requires more than visually plausiblescenes: objects must support interaction, placements must remain physicallyvalid, and tasks must admit repeatable execution. We presentAwomo-SimDataEngine, an agentic system that connects asset and scenegeneration to robot demonstration synthesis. Shared asset services providerigid and articulated objects, including structure-grounded part and jointgeneration with ISArt. Scene generation supports two complementary routes:Unravel reconstructs editable scenes from images, while SimForge buildssingle-room and multi-room environments from text. A graph-native harnesscoordinates construction, validation, andbounded repair, routing failures to the responsible module while retainingunaffected scene state. PolicyForge binds validated worlds to tasks and robotembodiments to produce replayable demonstrations. Evaluations cover assetgeometry, scene quality, and downstream policy learning. On MuJoCo-basedLIBERO-Plus, co-training with Isaac Sim demonstrations improves the overallsuccess rate of a World-Action Model (WAM) from

77.17\%

89.43\%

. Goal and spatialsuccess improve by

31.66

and

6.25

percentage points, respectively.These results support the utility of the generated data for cross-simulatorpolicy training, with more limited gains on long-horizon tasks.

Abstract

Synthetic Data Generation for Robotics

Explore similar work

Awomo-SimDataEngine: Agentic Simulation-ReadyWorld Generation

EmbodiedSmith: Scaling Embodied Data through Recursive Self-Improvement Flywheel in Simulation

ARSTAG: An Agentic Real2Sim2Real System for Task-Specific Robot Data Generation