cs.ROOct 25, 2026
SaveX-World: Transferring Video Priors to Variable-View Robot Control
Organizations: School of Data Science, Fudan University · Shanghai Innovation Institute · Logos Intelligence Lab
Abstract
Video diffusion models provide a source of interaction representations for robot control, but transferring these representations requires reconciling variable camera inputs with heterogeneous action interfaces and closed-loop execution. We present X-World, a framework that transfers a video prior through a fixed-length token sequence to a shared action expert. A residual cross-view attention branch extends a pretrained video backbone to one, two, or three views while retaining its original self-attention path. A layer-aligned Q-Former aggregates backbone features, and dataset-specific modules map between native state and action interfaces and the shared action expert. Training separates video adaptation from action-only pretraining by freezing the video backbone in the latter stage, followed by joint target-domain post-training. The policy extracts video features with a single backbone pass and uses asynchronous action chunking for execution. On LIBERO, X-World reaches 98.5% mean success; its LIBERO-trained checkpoint obtains 76.5% on LIBERO-Plus without further adaptation. The seven-shift breakdown reveals gains in robot, language, lighting, and layout perturbations, alongside weaknesses under camera, background, and noise shifts relative to OpenVLA-OFT. On six dual-arm real-world tasks, it achieves 81.7% mean success over 20 trials per task. Removing video adaptation or cross-view fusion produces the largest observed success drops among the tested removals.
Figures & tables
Figure 1: Overview of X-World . The video backbone is first adapted on robot and human videos with one to three camera views (left). A layer-aligned Q-Former then compresses its features into fixed-length tokens that condition the action DiT, while dataset-specific projections map native states and actions to and from the shared expert (right). Action pretraining learns this interface with the video backbone frozen; target-domain post-training updates all modules using video and action supervision.
Figure 2: Cross-view adaptation of the video transformer. (a) LTX-Video ( HaCohen et al., 2024 ) applies self-attention to a single view. (b) Genie-Envisioner ( Liao et al., 2025 ) fuses concatenated views through cross-view attention. (c) X-World retains pretrained per-view self-attention and learns cross-view fusion through a separate residual branch. Zero initialization preserves the pretrained computation at the start of adaptation while allowing cross-camera interactions to be learned. (d) Training-time masks for one, two, and three available views. Shaded entries allow attention; replicated slots are isolated from available views, supporting joint training across camera counts.
| Stage | Data | Loss | Updated modules |
|---|---|---|---|
| 1: Video adaptation | Robot/human videos | Video backbone and view modules | |
| 2: Action pretraining | Robot trajectories | Q-Former, action expert, native projections | |
| 3: Post-training | Target trajectories | All modules |
Table 1: Three-stage training of X-World . Stage 1 adapts the video backbone and view modules through video prediction on robot and human data. Stage 2 freezes the backbone and trains the Q-Former, action expert, and native interfaces on action-labeled trajectories. Stage 3 jointly optimizes video and action objectives on LIBERO or real-world demonstrations, updating all modules.
| Method | Spatial | Object | Goal | Long | Average |
|---|---|---|---|---|---|
| Diffusion Policy ( Chi et al., 2025 ) | 78.5 | 87.5 | 73.5 | 64.8 | 76.1 |
| OpenVLA ( Kim et al., 2024 ) | 84.7 | 88.4 | 79.2 | 53.7 | 76.5 |
| SpatialVLA ( Qu et al., 2025 ) | 88.2 | 89.9 | 78.6 | 55.5 | 78.1 |
| CoT-VLA ( Zhao et al., 2025a ) | 87.5 | 91.6 | 87.6 | 69.0 | 83.9 |
| -Fast ( Pertsch et al., 2025 ) | 96.4 | 96.8 | 88.6 | 60.2 | 85.5 |
| GR00T-N1 ( Bjorck et al., 2025 ) | 94.4 | 97.6 | 93.0 | 90.6 | 93.9 |
Table 2: LIBERO success rates (%) on the Spatial, Object, Goal, and Long suites. Average is the mean of the four suite scores. Best results are bold with pink backgrounds; second-best results are underlined with lavender backgrounds.
| Method | Cam. | Robot | Lang. | Light | Bg. | Noise | Layout | Total |
|---|---|---|---|---|---|---|---|---|
| OpenVLA ( Kim et al., 2024 ) | 0.8 | 3.5 | 23.0 | 8.1 | 34.8 | 15.2 | 28.5 | 15.6 |
| ( Black et al., 2024 ) | 13.8 | 6.0 | 58.8 | 85.0 | 81.4 | 79.0 | 68.9 | 53.6 |
| OpenVLA-OFT ( Kim et al., 2025 ) | 56.4 | 31.9 | 79.5 | 88.7 | 93.3 | 75.8 | 74.2 | 69.6 |
| OpenVLA-OFT w ( Kim et al., 2025 ) | 10.4 | 38.7 | 70.5 | 76.8 | 93.6 | 49.9 | 69.9 | 55.8 |
| NORA ( Hung et al., 2025 ) | 2.2 | 37.0 | 65.1 | 45.7 | 58.6 | 12.8 | 62.1 | 39.0 |
| WorldVLA ( Cen et al., 2025 ) | 0.1 | 27.9 | 41.6 | 43.7 | 17.1 | 10.9 | 38.0 | 25.0 |
Table 3: Zero-shot success rates (%) across seven LIBERO-Plus distribution shifts ( Fei et al., 2025 ) . Cam., Lang., and Bg. denote camera, language, and background changes, respectively. The w and m variants use no wrist input and mixed-data supervised fine-tuning, respectively. OpenVLA-OFT+ is excluded because it is trained on LIBERO-Plus. Best results are bold with pink backgrounds; second-best results are underlined with lavender backgrounds.
Figure 3: Dual-arm evaluation platform. Two UR5e arms use Robotiq 2F85 grippers and wrist-mounted RealSense D405 cameras; a shared D455 camera supplies the third-person view.
Figure 4: Real-world manipulation examples. Each row corresponds to a separate rollout of the language instruction above it, with three third-person snapshots arranged in execution order from left to right.
Figure 5: Real-world task success. Bars show success fractions over 20 trials per policy and task on six dual-arm UR5e tasks under a shared evaluation protocol and identical initial placements. X-World achieves the highest observed success on every task.
| Setting | Spatial | Object | Goal | Long | Average |
|---|---|---|---|---|---|
| w/o Stage 1 | 95.4 | 97.6 | 92.4 | 72.4 | 89.5 |
| w/o Stage 2 | 99.2 | 98.8 | 97.2 | 94.2 | 97.4 |
| w/o Stage 1 and Stage 2 | 97.4 | 96.2 | 93.4 | 81.2 | 92.1 |
| Shared projections | 98.6 | 98.4 | 98.2 | 94.4 | 97.4 |
| w/o Q-Former | 99.8 | 98.6 | 98.6 | 94.6 | 97.9 |
| w/o video loss | 99.8 | 100.0 | 97.2 | 96.2 | 98.3 |
Table 4: LIBERO success rates (%) for ablations of training stages, model components, and video loss. Stage 1 is video adaptation and Stage 2 is action pretraining; Average is the mean across the four suites. Best results are bold with pink backgrounds; second-best results are underlined with lavender backgrounds.
| Model timing | Deployment timing | ||
|---|---|---|---|
| T5 encoder (first call) | 13 | Cached model response | 46 |
| VAE | 7 | Network transfer | 32 |
| Video diffusion | 12 | Data processing | 7 |
| Q-Former | 5 | Control-side overhead | 6 |
| Action expert | 8 | End-to-end refresh | 91 |
Table 5: Single-H100 inference latency (ms). Component timings exclude 14 ms of model-side overhead included in the 46 ms cached response; T5 encoding adds 13 ms on the first call. The 91 ms deployed refresh additionally includes network transfer, data processing, and control-side overhead.
Figure 6: Action-prefix and view-availability analyses. The left panel compares inference without real-time chunking (w/o RTC) with prefix lengths . The right panel uses the same checkpoint to compare one, two, and three camera views and a shuffled three-view order. Bars report success fractions.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset | Views | Summary |
|---|---|---|
| RoboTwin 2.0 ( Chen et al., 2025 ) | 3 | Domain-randomized dual-arm benchmark designed for robust bimanual manipulation and transfer. |
| CALVIN ( Mees et al., 2022 ) | 2 | Long-horizon language benchmark built around static and gripper observations for sequential manipulation. |
| AgiBotWorld ( Bu et al., 2025a ) | 3 | Quality-controlled manipulation data collected in realistic settings with broad skill coverage. |
| Behavior-1K ( Li et al., 2024 ) | 3 | Household simulation benchmark emphasizing realistic environments and long-horizon daily activities. |
| RoboCOIN ( Wu et al., 2025 ) | 3 | Bimanual multi-embodiment data with hierarchical annotations for coordinated manipulation skills. |
| Galaxea ( Jiang et al., 2025 ) | 3 | Open-world robot data from everyday scenes with fine-grained subtask language annotations. |
Table 6: Datasets for Stage 1 video adaptation. Robot and human videos supply future-frame supervision, while Views gives the number of available camera streams per training example.
| Dataset | Views | Summary |
|---|---|---|
| RoboTwin 2.0 ( Chen et al., 2025 ) | 3 | Domain-randomized dual-arm trajectories for robust bimanual manipulation and transfer. |
| CALVIN ( Mees et al., 2022 ) | 2 | Long-horizon language-conditioned trajectories with static and gripper observations. |
| AgiBotWorld ( Bu et al., 2025a ) | 3 | Large-scale manipulation trajectories collected in realistic settings with diverse skills. |
| Behavior-1K ( Li et al., 2024 ) | 3 | Household embodied benchmark providing long-horizon trajectories in realistic simulation. |
| RoboCOIN ( Wu et al., 2025 ) | 3 | Bimanual multi-embodiment trajectories with hierarchical annotations for coordinated skills. |
| Galaxea ( Jiang et al., 2025 ) | 3 | Open-world manipulation trajectories with fine-grained subtask language supervision. |
Table 7: Datasets for Stage 2 action pretraining. Robot trajectories provide action supervision, while Views gives the available camera count per example; dataset-specific projections accommodate native state and action formats.
Figure 7: Future-video predictions after Stage 1 adaptation. The three examples use one, two, and three input camera views. Within each example, the first column contains observed frames and the remaining columns show predictions at frames 8, 16, and 24 of the 24-frame horizon; multiple rows correspond to synchronized views of the same sequence.
Figure 8: Future-video predictions after real-world post-training. Table clearing (top) and block stacking (bottom) each show three synchronized camera views. Observed frames occupy the first column, followed by predictions at frames 8, 16, and 24, so each row traces the predicted evolution of one camera view.
More in this topic
Robotic Control
Robotic control studies how machines use sensing and computation to produce reliable movement and interaction in changing environments.
See all 587 papers