cs.ROApr 4, 2026
SaveCRAFT: Video Diffusion for Bimanual Robot Data Generation
Organizations: Thomas Lord Department of Computer Science at the University of Southern California, USA
Abstract
Bimanual robot learning from demonstrations is fundamentally limited by the cost and narrow visual diversity of real-world data, which constrains policy robustness across viewpoints, object configurations, and embodiments. We present Canny-guided Robot Data Generation using Video Diffusion Transformers (CRAFT), a video diffusion-based framework for scalable bimanual demonstration generation that synthesizes temporally coherent manipulation videos while producing action labels. By conditioning video diffusion on edge-based structural cues extracted from simulator-generated trajectories, CRAFT produces physically plausible trajectory variations and supports a unified augmentation pipeline spanning object pose changes, camera viewpoints, lighting and background variations, cross-embodiment transfer, and multi-view synthesis. We leverage a pre-trained video diffusion model to convert simulated videos, along with action labels from the simulation trajectories, into action-consistent demonstrations. Starting from only a few real-world demonstrations, CRAFT generates a large, visually diverse set of photorealistic training data, bypassing the need to replay demonstrations on the real robot (Sim2Real). Across simulated and real-world bimanual tasks, CRAFT improves success rates over existing augmentation strategies and straightforward data scaling, demonstrating that diffusion-based video generation can substantially expand demonstration diversity and improve generalization for dual-arm manipulation tasks. Our project website is available at: https://craftaug.github.io/
Figures & tables
Fig. 2: Method Overview. (1) Trajectory Expansion: Real-world teleoperation data is first collected, and a digital twin pipeline transfers the objects and robot into simulation (Real2Sim). This simulation environment is then used for large-scale data generation. (2) Video Generation: The simulation trajectories are rendered into source videos and passed through a Canny-Edge Converter to extract structural edge representations, which are then combined with a real-world reference image and language instructions to condition a video diffusion model that synthesizes photorealistic video outputs. (3) Augmented Dataset Construction: The resulting generated videos support a wide range of visual variations, including object pose, lighting conditions, object color, background, cross-embodiment transfer, camera viewpoint, and combined wrist and third-person camera perspectives. (4) Generated Dataset: The synthesized videos are paired with action labels from the simulation trajectories, producing action-consistent demonstrations for downstream policy training.
Fig. 3: Reference Image and Canny-edge Visualization. Examples of reference images used to condition the video diffusion model for different augmentation techniques: (1) A standard reference image of the scene capturing gripper-object contact, used to condition the video diffusion model. (2) An example Canny-edge frame extracted from the simulation source video , used as structural control input. (3) A lighting-modified reference image generated using Veo3 [ 45 ] under green ambient illumination. (4) An empty table reference image with no objects, used for object color generation. (5) A tiled reference image combining a third-person view (top left), left wrist (top right), and right wrist (bottom left), with the fourth tile left blank, supporting up to four simultaneous camera viewpoints. Reference images include top and bottom padding (not shown).
Fig. 4: Simulation Environment. The SB (see Section V ) task adapted from RoboTwin [ 14 ] on the bimanual UR5 with WSG grippers, showing the initial state (left) and success state (right).
| Simulation (%) | Real-World ( / 20 ) | |||||
|---|---|---|---|---|---|---|
| Method | LP | PC | SB | LR | PC | SB |
| Collected Target | 55.0% | 69.0% | 59.0% | 5 / 20 | 2 / 20 | 3 / 20 |
| Shadow [ 10 ] | 2.0% | 2.3% | 6.0% | 2 / 20 | 1 / 20 | 1 / 20 |
| CRAFT (Target) | 11.3% | 6.0% | 21.6% | 4 / 20 | 1 / 20 | 2 / 20 |
| CRAFT (Ours) | 82.6% | 89.3% | 86.0% | 17 / 20 | 15 / 20 | 16 / 20 |
TABLE I: Cross-Embodiment Results. Simulation success rates (%) and real-world successes out of 20 trials for cross-embodiment transfer. Simulation evaluates transfer from a bimanual UR5 to a bimanual Franka Panda on LP , PC , and SB (see Section V-A ). Real-world evaluates transfer from a bimanual xArm7 to a bimanual Franka Panda on LR , PC , and SB (see Section VI-B ). CRAFT (Target) denotes source-to-target transfer demos, with the same number of demos as Target. CRAFT (Ours) uses 1000 generated demos without collecting any target robot demos, in contrast to Collected Target which requires 100 demos on the target robot. All methods are trained and evaluated using ACT.
| Method | Stack Bowls (SB) |
|---|---|
| Collected Demos (Upper Bound) | 59.0% |
| CRAFT w/o Canny | 10.3% |
| CRAFT w/ Canny | 21.6% |
TABLE II: Ablation Study. Success rates out of 150 demonstrations on Stack Bowls comparing video generation with and without Canny-edge control input. Collected Demos (Upper Bound) serves as the upper bound. No augmentation is performed in this ablation, as the objective is to isolate and evaluate the effect of synthesized image quality on ACT performance.
| Lighting | Background | Camera View | Object Color | Wrist + 3rd Person | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Method | LR | PC | SB | LR | PC | SB | LR | PC | SB | LR | PC | SB | LR | PC | SB |
| ACT w/o Aug | 3 / 20 | 1 / 20 | 0 / 20 | 4 / 20 | 0 / 20 | 0 / 20 | 6 / 20 | 3 / 20 | 2 / 20 | 2 / 20 | 0 / 20 | 1 / 20 | 15 / 20 | 11 / 20 | 13 / 20 |
| CRAFT Pose-Only | 5 / 20 | 3 / 20 | 2 / 20 | 7 / 20 | 2 / 20 | 3 / 20 | 13 / 20 | 5 / 20 | 7 / 20 | 5 / 20 | 2 / 20 | 3 / 20 | 13 / 20 | 8 / 20 | 10 / 20 |
| ACT w/ Baseline Aug | 13 / 20 | 9 / 20 | 8 / 20 | 4 / 20 | 5 / 20 | 6 / 20 | 14 / 20 | 8 / 20 | 6 / 20 | 15 / 20 | 9 / 20 | 11 / 20 | N/A † | N/A † | N/A † |
| CRAFT (Ours) | 17 / 20 | 14 / 20 | 12 / 20 | 18 / 20 | 15 / 20 | 10 / 20 | 19 / 20 | 18 / 20 | 18 / 20 | 18 / 20 | 18 / 20 | 17 / 20 | 20 / 20 | 19 / 20 | 20 / 20 |
TABLE III: Real-World Augmentation Results. Success rates out of 20 for LR , PC , and SB (see Section VI-B ) across five augmentation techniques. For each augmentation technique, all methods are evaluated under test conditions that vary only along that specific dimension (e.g., unseen lighting conditions for Lighting, unseen backgrounds for Background), while all other visual factors remain fixed. All CRAFT (Ours) columns use 1000 generated demonstrations combined with the real-world collected demonstrations (100 for LR , 200 for PC , and 150 for SB ), where the number of collected demonstrations per task was determined based on the minimum required to achieve reasonable ACT policy performance. Baseline methods use the same number of collected demonstrations augmented with their respective augmentation technique. The “ACT w/ Baseline Aug” row refers to a different baseline method for each augmentation type, chosen as the most relevant prior work for a specific augmentation: Lighting (Color Jitter), Background (RoboEngine [ 7 ] ), Camera View (VISTA [ 5 ] ), and Object Color (SAM3 [ 46 ] ). All methods are trained and evaluated using an ACT policy on the bimanual Franka.
Fig. 5: Real-world rollout trained on ACT of the PC (see Section VI-B ), with the bottom-right corner of each image showing the progression of the task execution. Images are labeled in order of task progression.
Fig. 6: Simulation Environments. Bimanual manipulation tasks adapted from RoboTwin [ 14 ] , shown for the bimanual UR5 with WSG grippers (left) and the bimanual Franka Panda (right). Tasks from top to bottom: Lift Pot (LP) , Place Cans in Plasticbox (PC) , and Stack Bowls (SB) .
Fig. 7: Real-World Environments. Bimanual manipulation tasks adapted from RoboTwin [ 14 ] , shown for the bimanual Franka Panda. Tasks from top to bottom: Lift Roller (LR) , Place Cans in Plasticbox (PC) , and Stack Bowls (SB) . The left images show the initial state and the right images show the final states.
Fig. 8: Real-World Setup. Bimanual cross-embodiment setup showing the Franka Panda (left) and xArm7 (right). Pink dotted circles indicate the Intel RealSense D435i camera placements.
Fig. 9: Video Generation Quality Comparison. Generated video frames without (left) and with (right) a reference image capturing gripper-object contact. As seen in the blue bordered images, including gripper-object contact in the reference image leads to higher fidelity contact synthesis in the generated video.
Fig. 10: Lighting Environments. Test lighting conditions used for evaluation: blue (top left), green (top right), red (bottom left), and yellow (bottom right) ambient illumination.
Fig. 11: Background Environments. Test background conditions used for evaluation: flower fabric (top left), rose fabric (top right), and no curtain (bottom left).
Fig. 12: Object Color Conditions. Object color variations used for evaluation across each task. From top to bottom: Lift Roller (LR) , Place Cans in Plasticbox (PC) , and Stack Bowls (SB) .
Fig. 13: Canny-Edge Filtering Comparison. Unfiltered (left) vs. filtered (right) Canny-edge representations, showing how edge thickening and connectivity post-processing produce cleaner structural control signals for the video diffusion model.
More in this topic
Synthetic Data Generation for Robotics
Synthetic data generation for robotics creates simulated scenes, sensor readings, and action trajectories to train and evaluate robots.
See all 303 papers