RoboGene: Boosting VLA Pre-training via Diversity-Driven Agentic Framework for Real-World Task Generation
Authors: Yixue Zhang, Kun Wu, Zhi Gao, Zhen Zhao, Pei Ren, Zhiyuan Xu, Fei Liao, Xinhua Wang, +7 more
Organizations: Beijing Innovation Center of Humanoid Robotics · The School of Advanced Manufacturing and Robotics, Peking University · Beijing Institute of Technology · The School of Mechanical Engineering and Automation, Beihang University · State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University
The pursuit of general-purpose robotic manipulation is hindered by the scarcity of diverse, real-world interaction data. Unlike data collection from web in vision or language, robotic data collection is an active process incurring prohibitive physical costs. Consequently, automated task curation to maximize data value remains a critical yet under-explored challenge. Existing manual methods are unscalable and biased toward common tasks, while off-the-shelf foundation models often hallucinate physically infeasible instructions. To address this, we introduce RoboGene, an agentic framework designed to automate the generation of diverse, physically plausible manipulation tasks across single-arm, dual-arm, and mobile robots. RoboGene integrates three core components: diversity-driven sampling for broad task coverage, self-reflection mechanisms to enforce physical constraints, and human-in-the-loop refinement for continuous improvement. We conduct extensive quantitative analysis and large-scale real-world experiments, collecting datasets of 18k trajectories and introducing novel metrics to assess task quality, feasibility, and diversity. Results demonstrate that RoboGene significantly outperforms state-of-the-art foundation models (e.g., GPT-4o, Gemini 2.5 Pro). Furthermore, real-world experiments show that VLA models pre-trained with RoboGene achieve higher success rates and superior generalization, underscoring the importance of high-quality task generation. Our project is available at https://robogene-boost-vla.github.io.
Figures & tables
Figure 2: Overview of RoboGene. The framework comprises three key components: (1) a diversity-driven sampling mechanism based on a Least Frequently Used (LFU) strategy; (2) a self-reflection mechanism that enhances task generation quality; and (3) a long-term memory module that consolidates Human-in-the-Loop (HITL) feedback.
Method
Task ( ↑ )
Type ( ↑ )
Logical ( ↑ )
Obj. ( ↑ )
Skill ( ↑ )
Phys. ( ↑ )
Clarity
Consist.
Validity
Cov.
Cov.
Feas.
Rule-based
0.5533
0.3900
0.4044
0.2145
0.4840
0.4811
Human
0.9644
0.8411
0.9489
0.3580
0.1769
0.9478
GPT-4o
0.7922
0.8555
0.7622
0.3105
0.2542
0.7467
Gemini 2.5 Pro
0.8791
0.8671
0.6691
0.2172
0.2458
0.6790
RoboGene
0.9910
0.9876
0.9899
0.6323
0.9152
0.9899
Table 1: Individual task evaluation results on the 900 tasks generated by each method. Higher scores indicate better task quality.
Figure 3: Scenario diversity results across 8 pre-defined scenario categories. RoboGene demonstrates a highly balanced distribution.
Figure 4: Skill diversity results given a total inventory of 118 skills. RoboGene significantly outperforms baselines in skill coverage and demonstrates a highly balanced distribution.
Figure 5: (a) Usage frequency of the top-15 objects across the 900 tasks generated by each method. (b) Total object count for tasks generated by each method, given a total inventory of 1,137 objects.
Metric / Method
RoboGene
GPT-4o
Gemini 2.5 Pro
Rule-based
Human
BLEU-1 ( ↓ )
28.93
36.61
38.97
80.32
72.80
BLEU-2 ( ↓ )
6.62
9.84
9.76
67.82
65.62
BLEU-3 ( ↓ )
3.15
4.82
5.73
62.11
60.19
BLEU-4 ( ↓ )
1.75
2.71
3.23
56.45
55.55
ROUGE-L ( ↓ )
0.1918
0.2512
0.2767
0.5786
0.4886
Cosine Similarity ( ↓ )
29.07
34.64
34.63
69.84
68.88
Table 2: Results on task semantic diversity. Lower scores indicate lower similarity between task and greater dataset diversity.
Figure 6: Real-world experimental setup. We employ three robots: the Single-arm UR-5e (SUR), the Dual-arm Franka (DFR), and the AgileX Cobot Magic V2.0 (AGX).
Figure 7: Experimental setup for generalization evaluation on the DFR-HangCups task, including novel objects, background variations, static distractors, illumination changes, and language instruction changes.
Method
SUR-Sort
SUR-Pour
SUR-Replenish
Button
Water
Shelves
ACT
90%
55%
60%
DFR-Hand
DFR-Open
DFR-Rinse
Tape
Box
Lettuce
ACT
90%
85%
60%
AGX-Organize
AGX-Tranfer
AGX-Water
Table 3: Validation of data collection across three robots. We trained on three tasks and reported the success rates of various models.
Figure 8: Success rates on 5 tasks after fine-tuning. Models were first pre-trained using data collected from the 300 tasks generated by each method. The number appended to each method name indicates the number of fine-tuning epochs.
Method
Novel
Background
Static
Illumination
Instruction
Avg.
Objects
Variations
Distractors
Changes
Changes
Human
10%
20%
20%
15%
10%
15%
GPT-4o
10%
5%
10%
5%
10%
8%
Gemini 2.5 Pro
10%
15%
10%
10%
15%
12%
RoboGene
30%
25%
30%
25%
35%
29%
Table 4: Generalization results on the DFR-HangCups task.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Exp.
Reflection
Skill
Object
Memory
Unique
Unique
Physical
Mechanism
Sampling
Sampling
Module
Skills
Objects
Feasibility
1
30
353
0.75
2
✓
58
512
0.88
3
✓
✓
101
524
0.91
4
✓
✓
✓
96
685
0.85
RoboGene
✓
✓
✓
✓
108
719
0.99
Appendix
Table 5: Ablation study of RoboGene. We evaluate the impact of the Reflection Mechanism, Skill/Object Sampling, and Memory Module on task diversity (Unique Skills/Objects) and Physical Feasibility.
Method
DFR-Grill
DFR-Hang
DFR-Build
DFR-Lubricate
DFR-Weigh
Average
Skewers
Cups
Blocks
Gears
Beaker
Human
75
40
5
45
25
38
GPT-4o
55
40
15
50
20
36
Gemini 2.5 Pro
80
30
15
35
15
35
RoboGene
80
50
30
55
25
48
Appendix
Table 6: Comparative results of downstream fine-tuning performance on unseen dual-arm tasks. We report the success rates (%) of the π0 model pre-trained on different datasets and fine-tuned with 50 demonstrations per task.
Synthetic data generated by video generative models has shown promise for robot learning as a scalable pipeline, but it often suffers from inconsistent action quality due to imperfectly generated videos. Recently, vision-language models (VLMs) have been leveraged to validate video quality, but they have limitations in distinguishing physically accurate videos and, even then, cannot directly evaluate the generated actions themselves. To tackle this issue, we introduce RoboCurate, a novel synthetic robot data generation framework that evaluates and filters the quality of annotated actions by comparing them with simulation replay. Specifically, RoboCurate replays the predicted actions in a simulator and assesses action quality by measuring the consistency of motion between the simulator rollout and the generated video. In addition, we unlock observation diversity beyond the available dataset via image-to-image editing and apply action-preserving video-to-video transfer to further augment appearance. We observe RoboCurate's generated data yield substantial relative improvements in success rates compared to using real data only, achieving +70.1% on GR-1 Tabletop (300 demos), +16.1% on DexMimicGen in the pre-training setup, and +179.9% in the challenging real-world ALLEX humanoid dexterous manipulation setting.
Achieving generalizable robotic manipulation remains a central challenge in embodied intelligence. Despite rapid advances in model architectures and learning algorithms, progress is often limited by the scarcity and narrow diversity of real-world data. The RoboSynChallenge competition introduces a unified benchmark to evaluate and advance the generalizability of manipulation policies across a spectrum of tasks, environments, and difficulty levels. To alleviate the shortage of realistic data, the challenge integrates large-scale synthetic data generation with standardized real-world robotic evaluation. Participants are encouraged to leverage synthesized state-action trials to improve general-purpose policy learning, while final assessments are conducted exclusively on unseen real-world manipulation environments. Baseline implementations, including Transformer-, Diffusion-, Vision-Language-Action, and World-Action-Model-based policies, are provided to ensure reproducibility and comparability. By coupling scalable simulation-based training with rigorous real-world validation, RoboSynChallenge aims to foster the development of broadly capable, data-efficient, and adaptable manipulation systems, thereby paving the way toward truly general robotic intelligence.
Runyi Zhao, Ruixin Wu, Chengkun Li +15
Shenzhen Loop Area Institute (SLAI) · The Chinese University of Hong Kong, Shenzhen · DexForce +8
Training generalist robots demands large-scale, diverse manipulation data, yet real-world collection is prohibitively expensive, and existing simulators are often constrained by fixed asset libraries and manual heuristics. To bridge this gap, we present V-Dreamer, a fully automated framework that generates open-vocabulary, simulation-ready manipulation environments and executable expert trajectories directly from natural language instructions. V-Dreamer employs a novel generative pipeline that constructs physically grounded 3D scenes using large language models and 3D generative models, validated by geometric constraints to ensure stable, collision-free layouts. Crucially, for behavior synthesis, we leverage video generation models as rich motion priors. These visual predictions are then mapped into executable robot trajectories via a robust Sim-to-Gen visual-kinematic alignment module utilizing CoTracker3 and VGGT. This pipeline supports high visual diversity and physical fidelity without manual intervention. To evaluate the generated data, we train imitation learning policies on synthesized trajectories encompassing diverse object and environment variations. Extensive evaluations on tabletop manipulation tasks using the Piper robotic arm demonstrate that our policies robustly generalize to unseen objects in simulation and achieve effective sim-to-real transfer, successfully manipulating novel real-world objects.