RoboCurate: Harnessing Diversity with Action-Verified Neural Trajectory for Robot Learning

Seungku Kim; Suhyeok Jang; Byungjun Yoon; Dongyoung Kim; John Won; Jinwoo Shin

cs.ROFeb 21, 2026

RoboCurate: Harnessing Diversity with Action-Verified Neural Trajectory for Robot Learning

Authors: Seungku Kim, Suhyeok Jang, Byungjun Yoon, Dongyoung Kim, John Won, Jinwoo Shin

Organizations: KAIST · RLWRLD

Abstract

Synthetic data generated by video generative models has shown promise for robot learning as a scalable pipeline, but it often suffers from inconsistent action quality due to imperfectly generated videos. Recently, vision-language models (VLMs) have been leveraged to validate video quality, but they have limitations in distinguishing physically accurate videos and, even then, cannot directly evaluate the generated actions themselves. To tackle this issue, we introduce RoboCurate, a novel synthetic robot data generation framework that evaluates and filters the quality of annotated actions by comparing them with simulation replay. Specifically, RoboCurate replays the predicted actions in a simulator and assesses action quality by measuring the consistency of motion between the simulator rollout and the generated video. In addition, we unlock observation diversity beyond the available dataset via image-to-image editing and apply action-preserving video-to-video transfer to further augment appearance. We observe RoboCurate's generated data yield substantial relative improvements in success rates compared to using real data only, achieving +70.1% on GR-1 Tabletop (300 demos), +16.1% on DexMimicGen in the pre-training setup, and +179.9% in the challenging real-world ALLEX humanoid dexterous manipulation setting.

Figures & tables

Figure 1: Overview of RoboCurate . (1) We generate diverse neural trajectory by applying image-to-image (I2I) model for scene diversity and video-to-video (V2V) model for appearance diversity, respectively. (2) We then filter neural trajectory using simulator-replay consistency, retaining only those for which a classifier predicts the motion in the generated video matches the simulator rollout.

Panel of Figure 2 — Figure 2: Examples of neural trajectory. (Top): original videos, (Bottom): visually augmented neural trajectory. The two bottom-left frames indicate a video whose initial frame is edited by I2I model, while the two bottom-right frames indicate a video processed with V2V transfer.

Figure 3: Examples of negative pairs for attentive probe training. We construct negative pairs from real-world dataset by inducing temporal shifts or sampling video from different episodes.

Figure 4: An overview of experimental design for RoboCurate. We conduct two-phase experiments: (1) pre-training on real data and neural trajectory followed by fine-tuning on simulation data, and (2) co-finetuning on real data and neural trajectory.

Method	Synth.	Filtering	300 Demos			1000 Demos
Method	Synth.	Filtering	Rearrangement	Articulated	Avg.	Rearrangement	Articulated	Avg.
Real	✗	✗	16.1	13.3	15.4	29.7	32.3	30.3
w/ DreamGen	✓	✗	21.1	14.7	19.5	31.7	33.7	32.2
w/ RoboCurate (Ours)	✓	✗	23.2	21.0	22.7	33.2	39.3	34.8
w/ RoboCurate (Ours)	✓	✓	25.4	28.7	26.2	38.2	37.0	37.9

Table 1: Performance comparison on GR-1 Tabletop ( NVIDIA et al., 2025 ) . We report the average success rate (%) over 50 trials across 24 tasks (18 rearrangement and 6 articulated), trained with varying number of demonstrations. DreamGen ( Jang et al., 2025 ) denotes neural trajectory generated by I2V model and labeled with IDM-predicted actions, without our visual diversity pipeline. For a fair comparison, all methods that leverage synthetic data use the same pre-filtering dataset size of 10K neural trajectory.

Method	Synth.	Filtering	GR-1 Humanoid	Bimanual Panda Arms (Dexterous Hands)	Avg.
Real	✗	✗	56.9	32.2	44.6
w/ DreamGen	✓	✗	57.1	35.6	46.4
w/ RoboCurate (Ours)	✓	✗	59.3	39.3	49.3
w/ RoboCurate (Ours)	✓	✓	62.7	40.9	51.8

Table 2: Performance comparison on DexMimicGen ( Jiang et al., 2025 ) . We report the average success rate (%) over 50 trials across 6 tasks (3 GR-1 Humanoid and 3 Bimanual Panda Arms (Dexterous Hands)), trained with 100 demonstrations per task. Results for tables are averaged over 3 random seeds. DreamGen ( Jang et al., 2025 ) denotes neural trajectory generated by I2V model and labeled with IDM-predicted actions, without our visual diversity pipeline. For a fair comparison, all methods that leverage synthetic data use the same pre-filtering dataset size of 10K neural trajectory.

Panel of Figure 5 — Figure 5: Visualization of benchmarks. We visualize our benchmark settings (from left to right): (1) GR-1 Tabletop ( NVIDIA et al., 2025 ) , (2) DexMimicGen ( Jiang et al., 2025 ) with bimanual Panda arms with dexterous hands, (3) DexMimicGen with GR-1 humanoid, and (4) a real-robot benchmark on dexterous-hand humanoid robot ALLEX.

Method	Synth.	Filtering	In-distribution	Novel Object	Novel Behavior	Avg.
Method	Synth.	Filtering	Pick and Place Can	Pick and Place Cup	Pour Can	Avg.
Real	✗	✗	25.0	16.7	0.0	13.9
w/ DreamGen	✓	✗	37.5	33.3	12.5	27.8
w/ RoboCurate* (Ours)	✓	✓	47.9	43.8	25.0	38.9

Table 3: Performance comparison on ALLEX humanoid. We report the average success rate (%) over 24 trials across 3 tasks (1 seen task and 2 unseen tasks). We collect 48 demonstrations only for the seen task. We generate synthetic data for both seen and unseen tasks (48 episodes for the seen task and 50 episodes for each unseen task) and co-finetune with real data. DreamGen ( Jang et al., 2025 ) denotes neural trajectory generated by I2V model and labeled with IDM-predicted actions, without our visual diversity pipeline. * denotes that our filtering strategy is applied during the generation stage via Best-of-N sampling, and visual augmentation is not used in this case.

Method	Filtering	Rearrangement	Articulated	Avg.
Real + Neural	✗	30.7	36.3	32.1
w/ DreamGenBench	✓	33.7	40.7	35.4
w/ VideoCon-Physics	✓	33.9	39.0	35.2
w/ RoboCurate (Ours )	✓	37.7	40.3	38.3

Table 4: Comparison with other filtering strategies on GR-1 Tabletop ( NVIDIA et al., 2025 ) . We report the average success rate (%) over 50 trials across 24 tasks (18 rearrangement and 6 articulated), trained with 1,000 demonstrations per task. DreamGenBench ( Jang et al., 2025 ) and VideoCon-Physics ( Bansal et al., 2024 ) are physical plausibility benchmarks for generated videos. For a fair comparison, all methods use the same 10K pre-filtering neural trajectory.

Diversity		300 Demos
Visual	Task	Rearrangement	Articulated	Avg.
✗	25%	12.7	12.0	12.5
✗	50%	18.2	14.7	17.3
✗	100%	18.9	22.0	19.7
✓	100%	22.2	26.7	23.3

Table 5: Diversity analysis of RoboCurate on GR-1 Tabletop ( NVIDIA et al., 2025 ) . We report the average success rate (%) over 50 trials across 24 tasks (18 rearrangement and 6 articulated), trained with 300 demonstrations per task. For a fair comparison, we fix the dataset size to 10K neural trajectory across all diversity setups.

Method	Filtering	Attentive Probe	Human Label	Rearrangement	Articulated	Avg.
Real + Neural	✗	-	-	23.2	21.0	22.7
w/ V-JEPA2	✓	✗	-	26.1	17.0	23.8
w/ V-JEPA2	✓	✓	✓	24.8	19.7	23.5
w/ RoboCurate (Ours )	✓	✓	✗	25.4	28.7	26.2

Table 6: Component-wise ablation of RoboCurate’s filtering strategy on GR-1 Tabletop ( NVIDIA et al., 2025 ) . We report the average success rate (%) over 50 trials across 24 tasks (18 rearrangement and 6 articulated), trained with 300 demonstrations per task. With attentive probe, we discard video pairs classified as motion-inconsistent; without the attentive probe, we discard pairs whose video embedding cosine similarity falls below a fixed threshold. With human labels, we train the attentive probe on 2K manually binary-labeled video pairs; without human labels, we train it using our proposed positive and negative pairs constructed from the real-world dataset (see Figure 3 ). For a fair comparison, all methods use the same 10K pre-filtering neural trajectory and use V-JEPA2 ( Assran et al., 2025 ) as the pre-trained video encoder for filtering.

Figure 6: Aggregation of RoboCurate performance. Comparison of VLA performance across different pre-training or co-finetuning settings on GR-1 Tabletop ( NVIDIA et al., 2025 ) , DexMimicGen ( Jiang et al., 2025 ) , and real-world benchmark. RoboCurate shows strong performance across all settings.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Cross-attn heads ( $n_{\text{heads}}$ )	8
Cross-attn layers ( $L$ )	1
Clip length ( $H$ )	16
Temporal stride (frames)	4
Input resolution	256 $\times$ 256
Optimizer	AdamW

Appendix

Table 7: Attentive probe training hyperparameters.

Explore similar work

RoboRender: Robot-Oriented Video Generation for Visual Sim-to-Real Transfer
Oct 7, 2026Huang Huang, Wensi Ai, Ziyu Chen +6Sim-to-Real Transfer Synthetic Data Generation for Robotics
Stanford University · Nvidia · Microsoft Research
RoboGene: Boosting VLA Pre-training via Diversity-Driven Agentic Framework for Real-World Task Generation
Feb 18, 2026Yixue Zhang, Kun Wu, Zhi Gao +12Synthetic Data Generation for Robotics Robotic Data Collection
Beijing Innovation Center of Humanoid Robotics · The School of Advanced Manufacturing and Robotics, Peking University · Beijing Institute of Technology +2
Dream.exe: Can Video Generation Models Dream Executable Robot Manipulation?
Jun 3, 2026Rui Zhao, Kaiming Yang, Jifeng Zhu +6Physical Consistency in Video Generation Robotic Manipulation
Show Lab, National University of Singapore · University of Oxford · Tencent.

Oct 7, 2026J/K move · Enter open · S save

RoboRender: Robot-Oriented Video Generation for Visual Sim-to-Real Transfer

Huang Huang, Wensi Ai, Ziyu Chen, Youhui Wang+5

Stanford University · Nvidia · Microsoft Research

Sim-to-Real Transfer Synthetic Data Generation for Robotics Controllable Video Generation Zero-Shot Robotic Manipulation

Open paper AskPDF arXiv

Simulation enables large-scale, low-cost robot data generation, but policies trained in simulation often fail to transfer to the real world due to the sim-to-real visual discrepancies. Existing approaches often rely on intermediate representations, which can discard rich semantic information or require additional perception modules at deployment. We address this visual sim-to-real gap with RoboRender, a framework that converts simulated trajectories into photorealistic RGB videos for policy learning. RoboRender trains a robot-oriented video generation model conditioned on simulated depth videos, language instructions, and robot RGB mask videos, preserving simulator geometry, robot motion, and action labels while synthesizing realistic textures, backgrounds, and distractors. The generated RGB videos are paired with simulator-provided states and actions to train policies for zero-shot real-world deployment. On robot video test sets, our video model outperforms depth-conditioned video generation baselines in generation quality. In real-world experiments across pick-and-place, articulated-object manipulation, and mobile manipulation tasks, policies trained on RoboRender-generated data achieve a 71% average success rate, outperforming raw simulation renderings and conventional visual domain randomization by approximately 7.1x and 3.6x, respectively. We further show that policy performance improves with more generated videos per simulation trajectory, increasing opening-task success by 65 percentage points. These results demonstrate that generative video rendering mitigates the visual sim-to-real gap for zero-shot policy transfer. Project website: https://robo-render.github.io/.

Abstract

Explore similar work

RoboRender: Robot-Oriented Video Generation for Visual Sim-to-Real Transfer

RoboGene: Boosting VLA Pre-training via Diversity-Driven Agentic Framework for Real-World Task Generation

Dream.exe: Can Video Generation Models Dream Executable Robot Manipulation?