RoboGene: Boosting VLA Pre-training via Diversity-Driven Agentic Framework for Real-World Task Generation
Authors: Yixue Zhang, Kun Wu, Zhi Gao, Zhen Zhao, Pei Ren, Zhiyuan Xu, Fei Liao, Xinhua Wang, +7 more
Organizations: Beijing Innovation Center of Humanoid Robotics · The School of Advanced Manufacturing and Robotics, Peking University · Beijing Institute of Technology · The School of Mechanical Engineering and Automation, Beihang University · State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University
The pursuit of general-purpose robotic manipulation is hindered by the scarcity of diverse, real-world interaction data. Unlike data collection from web in vision or language, robotic data collection is an active process incurring prohibitive physical costs. Consequently, automated task curation to maximize data value remains a critical yet under-explored challenge. Existing manual methods are unscalable and biased toward common tasks, while off-the-shelf foundation models often hallucinate physically infeasible instructions. To address this, we introduce RoboGene, an agentic framework designed to automate the generation of diverse, physically plausible manipulation tasks across single-arm, dual-arm, and mobile robots. RoboGene integrates three core components: diversity-driven sampling for broad task coverage, self-reflection mechanisms to enforce physical constraints, and human-in-the-loop refinement for continuous improvement. We conduct extensive quantitative analysis and large-scale real-world experiments, collecting datasets of 18k trajectories and introducing novel metrics to assess task quality, feasibility, and diversity. Results demonstrate that RoboGene significantly outperforms state-of-the-art foundation models (e.g., GPT-4o, Gemini 2.5 Pro). Furthermore, real-world experiments show that VLA models pre-trained with RoboGene achieve higher success rates and superior generalization, underscoring the importance of high-quality task generation. Our project is available at https://robogene-boost-vla.github.io.
Figures & tables
Figure 2: Overview of RoboGene. The framework comprises three key components: (1) a diversity-driven sampling mechanism based on a Least Frequently Used (LFU) strategy; (2) a self-reflection mechanism that enhances task generation quality; and (3) a long-term memory module that consolidates Human-in-the-Loop (HITL) feedback.
Method
Task ( ↑ )
Type ( ↑ )
Logical ( ↑ )
Obj. ( ↑ )
Skill ( ↑ )
Phys. ( ↑ )
Clarity
Consist.
Validity
Cov.
Cov.
Feas.
Rule-based
0.5533
0.3900
0.4044
0.2145
0.4840
0.4811
Human
0.9644
0.8411
0.9489
0.3580
0.1769
0.9478
GPT-4o
0.7922
0.8555
0.7622
0.3105
0.2542
0.7467
Gemini 2.5 Pro
0.8791
0.8671
0.6691
0.2172
0.2458
0.6790
RoboGene
0.9910
0.9876
0.9899
0.6323
0.9152
0.9899
Table 1: Individual task evaluation results on the 900 tasks generated by each method. Higher scores indicate better task quality.
Figure 3: Scenario diversity results across 8 pre-defined scenario categories. RoboGene demonstrates a highly balanced distribution.
Figure 4: Skill diversity results given a total inventory of 118 skills. RoboGene significantly outperforms baselines in skill coverage and demonstrates a highly balanced distribution.
Figure 5: (a) Usage frequency of the top-15 objects across the 900 tasks generated by each method. (b) Total object count for tasks generated by each method, given a total inventory of 1,137 objects.
Metric / Method
RoboGene
GPT-4o
Gemini 2.5 Pro
Rule-based
Human
BLEU-1 ( ↓ )
28.93
36.61
38.97
80.32
72.80
BLEU-2 ( ↓ )
6.62
9.84
9.76
67.82
65.62
BLEU-3 ( ↓ )
3.15
4.82
5.73
62.11
60.19
BLEU-4 ( ↓ )
1.75
2.71
3.23
56.45
55.55
ROUGE-L ( ↓ )
0.1918
0.2512
0.2767
0.5786
0.4886
Cosine Similarity ( ↓ )
29.07
34.64
34.63
69.84
68.88
Table 2: Results on task semantic diversity. Lower scores indicate lower similarity between task and greater dataset diversity.
Figure 6: Real-world experimental setup. We employ three robots: the Single-arm UR-5e (SUR), the Dual-arm Franka (DFR), and the AgileX Cobot Magic V2.0 (AGX).
Figure 7: Experimental setup for generalization evaluation on the DFR-HangCups task, including novel objects, background variations, static distractors, illumination changes, and language instruction changes.
Method
SUR-Sort
SUR-Pour
SUR-Replenish
Button
Water
Shelves
ACT
90%
55%
60%
DFR-Hand
DFR-Open
DFR-Rinse
Tape
Box
Lettuce
ACT
90%
85%
60%
AGX-Organize
AGX-Tranfer
AGX-Water
Table 3: Validation of data collection across three robots. We trained on three tasks and reported the success rates of various models.
Figure 8: Success rates on 5 tasks after fine-tuning. Models were first pre-trained using data collected from the 300 tasks generated by each method. The number appended to each method name indicates the number of fine-tuning epochs.
Method
Novel
Background
Static
Illumination
Instruction
Avg.
Objects
Variations
Distractors
Changes
Changes
Human
10%
20%
20%
15%
10%
15%
GPT-4o
10%
5%
10%
5%
10%
8%
Gemini 2.5 Pro
10%
15%
10%
10%
15%
12%
RoboGene
30%
25%
30%
25%
35%
29%
Table 4: Generalization results on the DFR-HangCups task.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Exp.
Reflection
Skill
Object
Memory
Unique
Unique
Physical
Mechanism
Sampling
Sampling
Module
Skills
Objects
Feasibility
1
30
353
0.75
2
✓
58
512
0.88
3
✓
✓
101
524
0.91
4
✓
✓
✓
96
685
0.85
RoboGene
✓
✓
✓
✓
108
719
0.99
Appendix
Table 5: Ablation study of RoboGene. We evaluate the impact of the Reflection Mechanism, Skill/Object Sampling, and Memory Module on task diversity (Unique Skills/Objects) and Physical Feasibility.
Method
DFR-Grill
DFR-Hang
DFR-Build
DFR-Lubricate
DFR-Weigh
Average
Skewers
Cups
Blocks
Gears
Beaker
Human
75
40
5
45
25
38
GPT-4o
55
40
15
50
20
36
Gemini 2.5 Pro
80
30
15
35
15
35
RoboGene
80
50
30
55
25
48
Appendix
Table 6: Comparative results of downstream fine-tuning performance on unseen dual-arm tasks. We report the success rates (%) of the π0 model pre-trained on different datasets and fine-tuned with 50 demonstrations per task.