Generative simulation can automate robot environment construction, but physically valid environments are not necessarily useful for training. The challenge is to identify environments that address a policy's failures and supervision that helps it overcome them. We introduce CoDA, a framework in which a simulation data engine and an action policy co-evolve through a shared heterogeneous memory of environment construction, policy failures, and learning progress. The memory compiles validated diagnoses from a vision-language critic into generation programs that recreate difficulties and training programs that help the policy overcome them. We further propose Text2potential to convert these diagnoses into staged potentials for action-chunk value learning, providing dense supervision for policy updates while preserving the task's optimal policy under standard shaping assumptions. Measured learning progress guides further generation and program refinement. Across four manipulation task families, CoDA improves out-of-distribution success from 5.3% after supervised fine-tuning to 27.6%, exceeding the strongest closed-loop baseline by 9.5 percentage points with the same generation budget. On LIBERO-Pro, 15 rounds of generation and training raise success from 25% to 84% under position perturbations and from 1% to 76% under instruction perturbations.
Figures & tables
Figure 1: The 0.64314 0.68627 0.84706C0.34902 0.7451 0.88235o __color_backend_reset: __color_backend_reset: 0.44706 0.78039 0.59608D0.5451 0.81176 0.3098A __color_backend_reset: __color_backend_reset: framework. The data engine G generates executable scenes with independently editable task, spatial, and object-attribute factors (τ,σ,α) and a sparse success verifier ν . Construction traces and rollout-grounded failure diagnoses enter a shared memory, which organizes them into reusable difficulty modes. Validated modes are compiled into generation programs that reproduce the diagnosed conditions and training programs that specify subgoals, potentials, and learning tools. Text2potential supplies dense feedback through chunk-level value targets that guide policy updates, and measured learning progress guides the next generation round.
OOD by family
Average
Data generation
Pick-place
Articulated
Put-inside
Stack/insert
In
OOD
pretrained, zero-shot
3.1 ± 0.6
1.6 ± 0.5
2.7 ± 0.6
1.8 ± 0.6
3.4 ± 0.2
2.3 ± 0.3
+ seed demonstrations (SFT)
6.4 ± 0.6
4.7 ± 0.4
5.9 ± 0.5
4.2 ± 0.4
11.7 ± 0.4
5.3 ± 0.3
domain randomization
8.8 ± 0.4
8.3 ± 0.6
7.6 ± 0.4
6.9 ± 0.5
17.2 ± 0.3
7.9 ± 0.4
uniform variants [ 29 ]
10.6 ± 0.4
6.9 ± 0.6
9.7 ± 0.4
6.8 ± 0.3
21.6 ± 0.3
8.5 ± 0.2
scalar-feedback co-evolution [ 10 ]
20.9 ± 0.7
15.3 ± 0.7
19.6 ± 1.2
16.6 ± 0.7
33.8 ± 0.4
18.1 ± 0.5
Table 1: Policy generalization across four manipulation task families. OOD success rates (%) under different generation and training strategies. In denotes success on each generator’s own distribution. Values are mean ± standard error over three seeds. Best and second best highlighted.
Figure 2: Co-evolution over rounds. (a) Average OOD success as number of accepted environments per task family increases. Dashed line marks the comparison reported in Table 1 . Curves show mean over three seeds, with standard error shaded for CoDA. (b) Number of difficulty modes newly selected for training, retained for further training, and removed from the memory in each round. Labels above bars separately report the number of generation programs reused from other task families. (c) OOD success gain (percentage points) per round by CoDA and generation guided by task success rates.
Object
Goal
Spatial
Overall
Method
StdLBR
Pos
Task
Pos
Task
Pos
Task
Pos
Task
OpenVLA [ 12 ]
0.765
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
π0 [ 1 ]
0.942
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
π0.5 [ 8 ]
0.971
0.17
0.01
0.38
0.00
0.20
0.01
0.25
0.01
Human-written programs
—
0.14
0.00
0.00
0.00
0.23
0.05
0.12
0.02
ASPIRE (code-as-policy) [ 18 ]
—
0.98
0.95
0.81
0.45
0.51
0.60
0.77
0.67
Table 2: LIBERO-Pro generalization to position and instruction perturbations. Success rates under position (Pos) and instruction (Task) perturbations for the Object, Goal, and Spatial suites. StdLBR reports success on standard LIBERO. Best and second best highlighted.
Critic
OOD success (%) ↑
Refuted (%) ↓
Human agr. (%) ↑
Qwen3-VL-235B (open weights)
24.9 ± 0.8
27.5
73.8
InternVL3-78B (open weights)
25.7 ± 0.7
24.1
76.3
Claude Opus 5
26.1 ± 0.6
20.3
81.5
GPT-5.5 (default)
27.6 ± 0.5
18.2
82.4
Table 3: Critic ablations. OOD success at a fixed training budget, diagnoses refuted by control evaluations, and agreement with two human annotators. Best and second best results are highlighted.
Training signal
OOD success
Steps to 25% OOD (k)
Agreement
sparse verifier only ( Φ≡0 )
22.3 ± 0.8
31
—
potential from task description [ 42 ]
24.6 ± 0.7
29
95.3
free-form dense reward, not a potential
23.5 ± 1.0
33
72.4
frozen actor: Qϕ -weighted selection only [ 7 ]
21.4 ± 0.9
—
96.8
Text2potential without refinement
26.1 ± 0.6
27
95.9
Text2potential (ours)
27.6 ± 0.5
21
97.2
Table 4: Text2potential ablations. We report OOD success (%) at a fixed training budget, policy updates needed to reach 25% OOD success (in thousands), and agreement (%) with the sparse-only critic’s action-chunk selections after convergence. Best and second best highlighted.
Figure 3: Successful rollouts. Left: Pre-defined LIBERO-Pro tasks. Right: Evolved scenes, one row per variant. Columns are the initial state, grasp, transport, place, and success.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Memory configuration
OOD success
Rounds to target
No memory
19.2 ± 0.7
14
Ω only (engine side)
22.8 ± 0.7
9
Φ only (policy side)
22.1 ± 0.8
11
Single memory: raw trajectory retrieval
20.1 ± 0.9
13
Single memory: distilled insights
21.6 ± 0.7
12
Single memory: skill library [ 34 ]
21.0 ± 0.8
12
Appendix
Table 5: Effect of memory design on policy learning. OOD success (%), mean ± standard error over three seeds at the same fixed generation budget, and # rounds required to reach 25% OOD success. Ω and Φ denote scene generation and policy training programs, respectively. The full method retains both program types. Best and second highlighted.
Learned on ↓ / lifted to →
Pick-place
Articulated
Put-inside
Stack/insert
Pick-place
72 / 100
38 / 46
61 / 79
55 / 68
Articulated
41 / 52
69 / 100
44 / 55
37 / 48
Put-inside
63 / 81
40 / 50
70 / 100
52 / 64
Stack/insert
57 / 70
36 / 45
54 / 66
66 / 100
Re-learned from scratch on target
31 / 38
28 / 35
30 / 37
27 / 33
Appendix
Table 6: Transfer of generation programs across task families. Rows indicate the family in which programs were learned, and columns indicate the family in which they are evaluated. Each entry reports two percentages: variants within the success band / learning progress relative to programs learned within the target family. Diagonal entries provide the within-family reference, with learning progress normalized to 100%. The final row reports programs learned from scratch in the target family using the same LLM token budget as transfer.
Loop run around ↓ / policy trained →
π0.5
OpenVLA-OFT
π0.5
27.6
21.1
OpenVLA-OFT
25.8
22.7
no memory
5.3
3.8
Appendix
Table 7: Transfer of shared memory across policy backbones. OOD success (%), with rows identifying the backbone used to collect memory and columns identifying the backbone trained and evaluated. Off-diagonal entries use frozen memory from the other backbone, with no additional environment generation. Diagonal entries report the full CoDA framework run with the corresponding backbone. The final row gives each backbone’s SFT baseline without memory. Best and second best results in each column are highlighted.
Table 8: Generation quality and validation of failure diagnoses. Verification (%) measures the percentage of scene proposals passing physical validity checks. Coverage is the entropy over task, spatial, and object-attribute factors (τ,σ,α) . Targeting measures the percentage of generated variants within the success band. Refuted reports the percentage of failure diagnoses rejected after comparing policy performance on targeted and control scene variants. Best and second best highlighted.
Table 9: Computational cost and effect of round scheduling. LLM token usage, simulation time, and GPU time are reported per 1,000 accepted environments. Token counts include scene generation, failure diagnosis, and potential compilation. The fixed schedule starts a new generation round every Kπ policy-update steps, while the adaptive schedule uses the remaining difficulty modes and recent learning progress. Best and second best highlighted.
Figure 4: Learning dynamics and memory accumulation. (a) In-distribution success on pick-and-place tasks during policy training, using the same generated data with potential shaping or sparse rewards alone. Shaping accelerates learning, with both conditions reaching similar final success rates. (b) Agreement between the action chunks selected by the two value critics, reaching approximately 97% at the end of training. (c) OOD success at successive memory snapshots, indexed by the number of validated difficulty modes. Using both program stores yields higher final success than retaining generation or training programs alone.
Figure 5: Scene variations generated by CoDA. Four examples show changes in object layout, size, and identity for tool sorting, serving-board arrangement, craft-station organization, and noodle preparation tasks. Each panel contains a seed scene, three spatial perturbations of increasing magnitude, a target-size change, and a target-object replacement. All displayed variants pass validity checks.
Figure 6: Successful policy rollouts on benchmark and generated environments. Left: five LIBERO-Pro tasks involving object placement into a basket or cabinet. Right: a fruit-preparation seed scene and five generated variants, covering three spatial perturbations, target resizing, and target replacement. Each row shows one rollout, with columns depicting the initial scene, grasp, transport, placement, and successful task completion.
Sim-and-real co-training is an effective way to scale robot learning by augmenting limited real-world demonstrations with simulation data. However, existing approaches typically treat simulators as fixed data sources, relying on uniform generation, offline retrieval, or manual parameter alignment by domain experts. We formulate Active Generative Simulation: optimizing a simulator's data-generation distribution to maximize downstream real-world performance. To solve this problem, we propose MetaGenSim, a meta-learning framework that trains an auxiliary rater network via meta-gradients to automatically discover high-utility data-generation distributions within a user-specified simulation parameter space, without manual parameter alignment. Across simulation and real-world manipulation tasks, MetaGenSim outperforms uniform generation and retrieval baselines, achieves a 150% relative improvement in real-world full-task success over uniform generation, and matches expert manual alignment. We further show that distributions curated with a lightweight proxy policy transfer to VLA fine-tuning, and that MetaGenSim can exploit partial alignment through atomic skill stitching and object-receptacle composition when exact sim-and-real task alignment is unavailable. Project website: https://meta-gensim.github.io
We present Scene2Demo, an automated framework for offline embodied data generation. Given a single RGB scene image and a user query, Scene2Demo constructs an interactive simulated scene and generates executable task configurations, primitive action flows, multi-view execution videos, and offline robot-learning datasets. Its five role-specialized generation modules perform task understanding, symbolic decomposition, BDDL configuration generation, object-size adjustment, and action-flow planning. The resulting object-centric task specifications and action transitions are organized as path search over a scene-conditioned object-action graph. Failed or incomplete executions are refined by two feedback agents: a Robotic Safety Inspector that produces step-wise visual critiques and a Robotic Task Supervisor that revises action sequences or parameters using the rollout and previous evolution history. Across 102 automatically generated primitive scene-task pairs, Scene2Demo achieves a 71.6% execution success rate; on four representative long-horizon tasks, feedback-driven refinement improves both task success and subtask-level execution quality over primitive-only execution. Comparisons with RoboGen and GenSim2 show stronger execution performance under automated data-generation settings. Finally, behavior cloning policies achieve 96.0% and 92.0% success on two representative tasks, showing that the generated data supports downstream policy learning. Our project page is available at https://scene2demo-anon.github.io/.
Generating useful robot-training data requires more than visually plausiblescenes: objects must support interaction, placements must remain physicallyvalid, and tasks must admit repeatable execution. We presentAwomo-SimDataEngine, an agentic system that connects asset and scenegeneration to robot demonstration synthesis. Shared asset services providerigid and articulated objects, including structure-grounded part and jointgeneration with ISArt. Scene generation supports two complementary routes:Unravel reconstructs editable scenes from images, while SimForge buildssingle-room and multi-room environments from text. A graph-native harnesscoordinates construction, validation, andbounded repair, routing failures to the responsible module while retainingunaffected scene state. PolicyForge binds validated worlds to tasks and robotembodiments to produce replayable demonstrations. Evaluations cover assetgeometry, scene quality, and downstream policy learning. On MuJoCo-basedLIBERO-Plus, co-training with Isaac Sim demonstrations improves the overallsuccess rate of a World-Action Model (WAM) from 77.17% to 89.43%. Goal and spatialsuccess improve by 31.66 and 6.25 percentage points, respectively.These results support the utility of the generated data for cross-simulatorpolicy training, with more limited gains on long-horizon tasks.