The generalization of vision-language-action (VLA) models heavily relies on diverse training data. However, acquiring large-scale robot manipulation data across varied object appearances is costly and labor-intensive. To address this, we introduce Embodied Manipulation Media Adaptation (EMMA), a framework for augmenting VLA policies that combines a generative data engine with an effective training pipeline. We introduce DreamTransfer, a diffusion Transformer-based architecture for generating multi-view consistent and geometrically plausible embodied manipulation videos. It supports text-driven video editing on foreground, background and lighting while preserving 3D structure and geometric fidelity. We also utilize a hybrid training set of real and generated data and propose AdaMix to enhance the training process. AdaMix is a training strategy that adaptively weights samples according to policy performance to emphasize challenging samples. Evaluations show DreamTransfer outperforms previous video generation models in multi-view consistency, geometric fidelity, and text-conditioning precision. In real-world robotic tasks with zero-shot visual settings, our framework achieves a relative performance increase of over 92% compared to training with real data alone, and improves by an additional 17% with AdaMix.
Figures & tables
Figure 1: DreamTransfer demonstrates strong controllability in embodied manipulation video generation. It excels in text-guided appearance editing while preserving 3D structure and geometric plausibility, and supports both real-to-real and sim-to-real transfer. The first column: real-world and simulation data. Next two columns: diverse data generated by DreamTransfer from the first column.
Figure 2: Overview of the EMMA framework. First, DreamTransfer generates multi-view consistent videos via text-guided visual editing of the foreground, background, and lighting conditions, conditioned on depth and text prompts. The generated videos are then evaluated by a video quality filter. Low-quality videos are initially assigned zero sampling weight to stabilize early-stage training. The AdaMix module further adaptively reweights training samples based on trajectory quality metrics, up-weighting challenging samples to improve policy robustness and generalization.
Figure 3: Overview of the DreamTransfer framework. Multi-view depth maps are concatenated along the width dimension.
Task
Model
Mat.Pix.(↑)
RMSE(↓)
Abs.Rel.(↓)
Sq.Rel.(↓)
CLIPSim.(↑)
Fold Clothes
RoboTransfer [ 50 ]
1736
3.97
0.50
2.13
24.63
Cosmos-Transfer1 [ 49 ]
2210
2.78
0.36
1.14
25.02
DreamTransfer
2604
2.50
0.32
0.97
24.54
Clean Desk
RoboTransfer [ 50 ]
3213
2.50
0.32
0.85
23.87
Cosmos-Transfer1 [ 49 ]
2484
1.55
0.19
0.36
25.02
DreamTransfer
4311
1.45
0.18
0.33
25.75
Table 1: Comparison of video generation quality on robot manipulation tasks. Best results are in bold . The result demonstrates that DreamTransfer achieves the best average performance across all evaluation metrics.
Figure 4: Impact of data mix ratios on real-world robotic tasks performance.
VLA
Method
Fold Clothes
Clean Desk
Throw Bottle
Average
Score
SR
Score
SR
Score
SR
Score
SR
π0 [ 1 ]
No.Aug.
3
10%
4.1
67.5%
2.1
12.5%
3.1
30%
Cosmos-Transfer1 † [ 49 ]
3.3
40%
4
70%
3.2
40%
3.5
50%
DreamTransfer †
4.4
65%
4.4
77.5%
3.2
52.5%
4
65%
DreamTransfer +FixMix
4.4
70%
4.5
77.5%
3.6
57.5%
4.2
68.3%
DreamTransfer + AdaMix
4.6
77.5%
4.7
90%
4.4
70%
4.6
79.2%
Table 2: Comprehensive real-world robotic task performance comparison across video generation models, and training strategies. No.Aug. denotes training on real data only. † denotes that the generated data in the training set are used without applying the video quality filter. Both FixMix and AdaMix employ a video quality filter before training. FixMix uses uniform sampling with constant weights during training. Score is the behavior score; SR is success rate. Results are averaged over 10 trials across 4 distinct visual variations of the foreground object. Best results are in bold .
Method
Fold Clothes
Clean Desk
Throw Bottle
Average
Time
Smth.
JOL.
Time
Smth.
JOL.
Time
Smth.
JOL.
Time
Smth.
JOL.
FixMix
40.4
2.2
56.1
12.1
2.6
60.0
46.0
1.2
14.6
32.8
2.0
43.6
AdaMix
39.8
2.2
53.4
11.3
2.4
64.9
38.9
1.2
10.4
30.0
1.9
42.9
Table 3: Comparison of execution time (Time, seconds), trajectory smoothness (Smth., angular acceleration in rad /s2 ) and joint overlimit (JOL., frames) between FixMix and AdaMix training strategies on real-world robotic tasks. Detailed calculations for these metrics are provided in Appendix F.3 . Lower values are better for all metrics. Best results are in bold .
Figure 5: Task success rate and behavior score under different training data and mixing strategies in simulation experiments.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Qualitative comparison of embodied manipulation video transfer methods.
Table 10
Figure 7: Ablation studies on the sampling hyperparameter λ and the effect of real data volume.
Figure 8: Example from our multi-view embodied manipulation dataset built upon Agibot World dataset. Each sample consists of 121 aligned RGB and depth frames, along with a caption describing object and scene appearance.
Configurations
Stage 1
Stage 2
Input Resolution (W × H)
576×128
1920×480
Batch Size
32
4
Training Steps
3500
4500
Optimizer
AdamW
AdamW
Learning Rate
1×10−5
1×10−5
Trainable Parameters
Main Branch + ControlNet
Main Branch + ControlNet
Appendix
Table 6: Training configurations for the two-stage training of DreamTransfer .
Model
Training Time
GPUs
Cosmos-Transfer1 [ 49 ]
336 hours
1024 × NVIDIA H100 GPUs
RoboTransfer [ 50 ]
96 hours
8 × NVIDIA H20 GPUs
DreamTransfer
131 hours
4 × NVIDIA H20 GPUs
Appendix
Table 7: Training time and computational resources for each model.
Metric
Mat.Pix.
RMSE
Abs.Rel.
Sq.Rel.
CLIPSim.
Threshold
>1000
<3
<0.4
<1
>20
Appendix
Table 8: Quality thresholds for filtering generated videos in AdaMix . A video is retained only if all its metric values meet the corresponding criteria.
Figure 9: Samples for training the VLA policy. The left column shows real data with fixed environments and appearances (simulation data collected from simulation environments for the Stack Bowls Two ), while the right column shows generated data with diverse object appearances.
Configuration
Fold Clothes
Clean Desk
Throw Bottle
Stack Bowls Two
# Real Data
50
20
20
–
# Simulated Data
–
–
–
50
# Generated Data
50
20
20
50
# Filtered Generated Data
46
19
19
47
Batch Size
64
64
64
64
Total Training Steps ( π0 )
20000
5000
10000
1000
Appendix
Table 9: The number of real/simulation and generated data for each task used in VLA policy training, as well as the detailed training configurations on each task. All pre-trained policies (e.g., π0 and π0.5 ) are fine-tuned on a single task based on their open-source pre-trained weights.
Figure 10: Real-world robot execution of π0 policies trained with AdaMix on three manipulation tasks.
Figure 11: Simulated robot execution in RoboTwin 2.0 of π0 policies trained with AdaMix on the Stack Bowls Two task.
Figure 12: Qualitative comparison between DreamTransfer and general video generation models (CogVideoX [ 45 ] and Wan2.5 [ 42 ] ) in embodied scenarios.
Figure 13: Diverse data generated by DreamTransfer from the simulation data (first column of the last row in Figure 9 ) for the Stack Bowls Two task. DreamTransfer exhibits strong controllability in embodied manipulation video generation.