Organizations: PrimeBot Research · Southeast University · Shanghai Jiao Tong University · The Chinese University of Hong Kong · University of Chinese Academy of Sciences · Peking University · Shanghai Innovation Institute · Beihang University
A robot world model that serves as a simulator must do more than generate plausible videos: it must faithfully follow commanded actions, maintain consistency across camera views, and sustain coherent long-horizon rollouts. We present PrimeSim, an action-conditioned, multiview world model simulator that addresses these challenges through progressive midtraining and posttraining. Starting from a pretrained video generator, we midtrain on 30,110 hours of robot demonstrations spanning 22 platforms. To improve action following, we introduce a unified cross-embodiment action representation and frame-aligned dual-path conditioning to strengthen action-frame correspondence. To maintain multiview consistency, we develop view-aware interleaved attention with explicit camera identities, enabling cross-view information exchange while preserving view-specific representations. We further introduce a fast-video, slow-memory posttraining framework that transforms the bidirectional video generator into an efficient autoregressive simulator. By combining fast causal generation, self-rollout training, and hierarchical temporal memory, PrimeSim mitigates error accumulation from self-generated histories and supports coherent long-horizon simulation. PrimeSim runs at 10.1 FPS, remains stable over long-horizon rollouts on rigid and deformable manipulation tasks, and transfers to held-out platforms within 100 to 200 gradient steps where the pretrained backbone does not. Against five open-source action-conditioned world models it leads four of the five WorldSimProbe suites and posts the best overall EWMBench score, while fine-grained local calibration still trails the field; its rollouts are faithful enough to support GRPO-based policy optimization. We release the model weights, inference code, and training code.
Figures & tables
Figure 1: PrimeSim. (a) The midtraining corpus: roughly 30.1K hours of curated robot data spanning humanoid, dual-arm, single-arm and handheld UMI [ 14 , 3 ] platforms, mixing real and simulated footage. (b) The model is conditioned on a language prompt, a unified action representation over head, torso, waist, end-effector, gripper and chassis, and three camera views, and predicts future frames; post-training makes generation autoregressive with a selected KV-cache memory at four denoising steps. (c) Capabilities: fidelity against prior systems, action following measured against simulation ground truth, few-shot adaptation to an unseen embodiment, and use as a simulator: success rate in PrimeSim predicts success rate in the LIBERO [ 51 ] physics simulator with r=0.99 , so policies can be trained in the model and the result carried back to the robot.
Figure 2: Overview of the data pipeline. Raw demonstrations from teleoperation, UMI, and simulation are converted through four stages: video quality filtering, action quality filtering, action preprocessing, and data alignment. Each stage lists the operations applied to the incoming data. The pipeline produces a unified corpus of 30,110 hours across 20 dataset families and 22 robot platforms (Table 12 ).
Figure 3: Overview of the PrimeSim training corpus: 30,110 hours, 2.38B frames and 1.52M episodes from 1,773 repositories over 22 robot platforms, at an 84:16 real-to-simulated ratio. Left: the task-category and embodiment distributions. Right: example frames from each of the five embodiment classes.
Figure 4: The PrimeSim model architecture. Left: the whole system, including the input data modalities, the VAE, the text encoder and the PrimeSim decoder. Right: the action conditioning mechanisms, a framewise attention mask and a frame-action adaptor.
Figure 5: General action representation in PrimeSim, covering single-arm, bimanual, UMI, and wheeled humanoid embodiments.
Figure 6: Overview of the post-training pipeline, from the midtrained base model through the four adaptation stages to the deployed simulator.
Model
T1 ↑
T2 ↑
T3 ↑
T4 ↑
T5 ↑
Mean ↑
LingBot-VA
77.0
60.2
51.0
57.8
22.4
53.7
DreamDojo
69.5
41.5
50.3
65.3
23.5
50.0
Cosmos-3-Nano
67.3
55.9
47.4
61.1
25.0
51.3
IRASim
54.0
54.7
47.9
68.2
22.4
49.4
Ctrl-World
62.7
61.7
50.4
77.5
25.0
55.5
PrimeSim
70.2
67.9
72.3
85.7
37.1
66.6
Table 1: WorldSimProbe results on LIBERO [ 51 ] , against five open-source action-conditioned world models. T1–T3 measure action realization (local action calibration, global trajectory coverage, action-source behavior preservation); T4–T5 measure interaction response (interaction grounding, interaction dynamics). Bold marks the best value per suite; higher is better. PrimeSim leads four of the five, all but local action calibration.
Method
SceneC ↑
HSD ↑
Dyn ↑
nDTW ↑
Div ↑
BLEU ↑
CLIP ↑
Logics ↑
Overall ↑
Action-conditioned
GE-Sim (LTX) [ 50 ]
0.900
0.940
0.780
0.980
0.011
0.330
0.908
0.970
5.82
GE-Sim (COSMOS2) [ 50 ]
0.910
0.870
0.850
0.970
0.010
0.310
0.902
0.970
5.79
Thinker-WM (action-cond.) [ 24 ] ‡
–
0.737
0.364
0.874
–
–
–
–
5.03
PrimeSim (ours)
0.887
1.000
0.666
1.000
0.010
0.570
0.954
1.000
6.09
Instruction- or text-conditioned (different protocol)
Table 2: EWMBench scores. Higher is better; Overall is the sum of the eight dimensions. The first block is action-conditioned like ours; the second block is instruction- or text-conditioned and therefore not iso-protocol with ours (e.g. Thinker-WM generates at 5 FPS upsampled to 30 FPS for scoring). GE-Sim rows are computed from the published Genie Envisioner simulation evaluation [ 50 ] , aligned to the standard EWMBench column names. Thinker-WM, Qwen-RobotWorld, LVP, Sora2 and Kling26 rows are as reported on the same 21-episode split in [ 24 ] , compiled there from [ 95 ] . † EnerVerse-FT is from the EWMBench paper’s own evaluation [ 93 ] . ‡ Thinker-WM (action-cond.) conditions on the ground-truth action sequence; only its motion dimensions and Overall are reported [ 24 ] , averaged over the evaluated checkpoints.
Model
Views
Res. / view
Future T
Steps
TempFPS ↑
ViewFPS ↑
Ours (PrimeSim Stage-3)
3
288×384
32 (33)
4
10.12
30.35
GE-Sim 2.0 [ 63 ]
3
384×512
25
4
2.24
6.73
BWM [ 67 ] (fair-aligned)
1
384×512
25
50
4.04
4.04
Ctrl-World [ 29 ]
3
192×320
5
50
0.87
2.60
BWM [ 67 ] (official)
1
672×896
48
50
0.75
0.75
CausalWM [ 83 ] †
1
480×640
25
4+4+4
0.66
0.66
Table 3: Inference throughput on a single NVIDIA A100-80GB (bfloat16), measured as steady-state wall-clock throughput per autoregressive chunk under each method’s released inference recipe. TempFPS counts rollout time steps per second; ViewFPS counts each camera view as one frame and coincides with TempFPS for single-view models. Rows are not matched in resolution, horizon or denoising steps. PrimeSim is measured at 288×384 per view under the RoboTwin 2.0 Stage-3 contract (33-frame windows, continuous KV, 32 usable frames per window), a different geometry from the 192×256 setting of the quality evaluations above. † CausalWM is text- and first-frame-conditioned, not action-conditioned.
Platform
Morphology
Midtrained
Pretrained
xArm 7
7-DoF single arm
200
2000 †
ViperX 300
bimanual
100
2000 †
UMI gripper
handheld
200
2000 †
Table 4: Few-shot adaptation to held-out embodiments. Both initialisations are finetuned under an identical recipe with single-trajectory batches at a batch size of 1 and a learning rate of 3×10−5 , and neither receives an embodiment identifier or any embodiment-specific parameters. Entries are gradient steps to an action-following rollout, meaning one in which the arm tracks the commanded trajectory and the scene responds to the motion actually realized. † No action-following rollout was reached within this budget.
Figure 7: Few-shot adaptation to unseen embodiments. Each pair shows the midtrained model above the pretrained backbone for one platform held out from training: a single 7-DoF arm ( xArm 7 ) drawn from Open X-Embodiment [ 17 ] , a bimanual platform ( ViperX 300 , as used by ALOHA [ 98 ] ) and the original Stanford UMI gripper [ 14 ] . Similar embodiments do occur in the corpus, and they are what make transfer possible at all, since a wholly alien platform would leave nothing to build on. They are still new embodiments, however, and adaptation between embodiments does not happen for free: each brings its own kinematics, camera mounting and data distribution, which have to be acquired rather than inherited. All initialisations are finetuned under an identical recipe with single-trajectory batches at a batch size of 1, and none receives an embodiment identifier or any embodiment-specific parameters. The midtrained model reaches an action-following rollout within 100 to 200 gradient steps on all three platforms. The pretrained backbone, which must initialise the action-conditioning and multiview modules from scratch, does not reach the same regime even at ten to twenty times that budget.
Figure 8: Zero-shot and few-shot generation on an unseen setting and camera pose. Both subfigures show the AgileX Piper bimanual arm on XVLA [ 100 ] , held out of pre-training entirely: the embodiment is in the corpus, but the scene, the garments and the camera placements are not. Rows are cameras (head, left wrist, right wrist) and columns are seven frames sampled uniformly from a single rollout. (a) Without adaptation the model controls the arms and renders the cloth, but the generated cloth state drifts from the commanded action once the motion becomes contact-rich. (b) After 200 gradient steps the arm trajectories and the cloth deformation track the commanded actions much more closely and the three views stay mutually consistent, so most of the cost of the transfer is recovered.
Data
Regime
PSNR ↑
SSIM ↑
LPIPS ↓
XR2
in-distribution
19.79±1.64
0.731±0.053
0.192±0.052
XVLA
zero-shot
15.82±2.24
0.762±0.049
0.297±0.072
Δ
−3.97
+0.031
+0.105
Table 5: Zero-shot transfer to a held-out dataset. Both rows share the checkpoint, sampler (50 steps, seed 42) and metrics at 192×256 over three jointly generated views; only the data differs. XR2 [ 82 ] is held-out episodes of repositories seen in pre-training; XVLA is excluded and differs in scene, garments and camera placement. Means ± std over eight validation windows. SSIM is not comparable across the two datasets, so only PSNR and LPIPS should be read as a degradation.
Figure 9: (a) Each point is one OpenVLA-OFT policy, evaluated on the same 300 LIBERO-Spatial initial states in the world model (x) and in the LIBERO simulator (y). The dotted line marks perfect agreement; points above it mean the world model under-estimates the policy. (b) Success rate of the SFT policy before and after 200 steps of GRPO post-training inside PrimeSim, measured in the simulator (solid) and in PrimeSim (hatched).
Action Representation
PSNR ↑
SSIM ↑
LPIPS ↓
Absolute action
22.23
0.79
0.14
Delta action
21.34
0.81
0.18
Relative action chunk
24.56
0.84
0.13
Table 9: Ablation study on action representation. Best result per metric in bold . ↑ : higher is better; ↓ : lower is better.
Figure 10: Counterfactual action following on LIBERO. Each row shows seven frames sampled uniformly from a single PrimeSim rollout, generated conditional on an action sequence rather than a task prompt. Our evaluation is built on counterfactual commands, meaning action trajectories that do not correspond to the demonstrated behaviour. Faithful generation therefore requires the model to respond to the commanded motion rather than replay the recorded one. Static scene elements remain stable across the rollout while the commanded motion is carried out, indicating that the generated future is driven by the action rather than by visual plausibility alone.
Action Conditioning Mechanism
PSNR ↑
Framewise
Adapter
Baseline
Ours
✓
×
23.23
24.10
×
✓
20.10
20.24
✓
✓
23.31
24.56
Table 10: Ablation study on action conditioning. The Baseline column uses vanilla cross-attention for framewise conditioning and Adaptive LayerNorm (AdaLN) for the adapter; Ours uses modified cross-attention and token fusion and shifting, respectively. The final row combines both mechanisms. Higher PSNR is better.
Figure 11: Joint multiview generation. Rows are cameras, from top to bottom head, left wrist and right wrist; columns are seven frames sampled uniformly across a single 33-frame PrimeSim rollout. The three views are generated together under a single action sequence rather than independently, and all respond to the same commanded motion. The wrist views change most over the interval, consistent with their proximity to the manipulation, while the head view tracks the arms throughout.
Multiview Mechanism
PSNR ↑
SSIM ↑
LPIPS ↓
Frame concatenation only
16.30
0.62
0.58
+ View token conditioning
23.78
0.83
0.18
+ Intra-view attention
24.56
0.84
0.15
Table 11: Ablation study on multiview consistency. Components are added incrementally from naive frame concatenation. Best result in bold .
Figure 12: Memory under occlusion. Five frames from a single RoboTwin 2.0 rollout, head camera. The arm sweeps across the workspace and its hand passes in front of the shoe in the lower left (fourth frame), so the shoe leaves the camera’s view entirely. When the hand clears, the shoe is present again at the same position, scale and orientation rather than being re-invented or dropped. Each window is conditioned on a selected KV-cache memory rather than on the previous frame alone, so an object occluded for several seconds is carried forward instead of being lost when it re-enters view.
Figure 13: Cloth manipulation rollouts, seven frames sampled uniformly from each. All three are AgiBot G2 humanoid episodes shown from the head camera: a white cloth on a bamboo table beside a metal bowl (top two rows), and a blue cloth on a bedding surface with a folded garment to its right (bottom row). The garment deforms continuously across each rollout and stays distinguishable from the surface beneath it rather than dissolving into the background, which is the failure mode deformable objects expose over long horizons.
Figure 14: Generated rollouts on two EWMBench tasks, the qualitative companion of Table 2 (Sec. 4.1.2 ). Each row shows seven frames sampled uniformly from a single rollout, generated open-loop at 10 FPS from the first frame, the task instruction and the ground-truth action trajectory, and shown at the 480×640 frame size used for scoring. Top: the arm picks a bottle from the conveyor and packs it into a cardboard box already holding other items. Bottom: the arm places a slice of bread onto the plate beside the toaster.
Hours
Frames
Dataset family
Real
Sim
Total
Real
Sim
Total
In-House Data
9,540
New A
9,540
375M
New A
375M
InternData-A1 [ 71 ]
4,000
170
4,170
410M
1M
411M
AgiBot World [ 2 ]
2,300
New A
2,300
243M
New A
243M
Hy-Embodied (UMI) [ 94 ]
2,100
New A
2,100
226M
New A
226M
BEHAVIOR-1K 2026 [ 44 ]
New A
2,000
2,000
New A
211M
211M
Table 12: Training data mixture, measured from dataset metadata rather than from upstream release figures. Hours are derived as frames/fps/3600 and so are consistent with the frame counts by construction. The mixture spans 20 dataset families, 1,773 LeRobot repositories, 1.52M episodes and 22 distinct robot platforms; it is 84% real by trajectory time. Real and Sim split each family by data source, and two families contribute both.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Component
Value
Backbone init
Wan2.2-TI2V-5B [ 75 ]
Parameters (total)
5.05 B
Hidden dim
3072
Transformer blocks
30
Attention heads
24
Patch size (spatial)
2 × 2
Appendix
Table 13: Model architecture.
Hyperparameter
Value
Optimizer
AdamW
Learning rate
1×10−4
LR schedule
cosine decay
Warmup steps
10,000
Min LR ratio
0.05 ( 5×10−6 )
β1,β2
0.9, 0.95
Appendix
Table 14: Midtraining hyperparameters. The architecture is initialized from the Wan2.2-TI2V-5B checkpoint, whose own training is not reported here.
Hyperparameter
Value
Initialization
Midtrained PrimeSim
Training objective
Flow matching with teacher forcing
Trainable parameters
LoRA adapters
Optimizer
AdamW
Learning rate
1×10−4
LR schedule
Cosine decay
Appendix
Table 15: Stage 1 autoregressive modelling hyperparameters. The model is initialized from the midtrained PrimeSim checkpoint and adapted to chunk-causal generation using teacher forcing.
Hyperparameter
Value
Initialization
Stage 1 causal model
Training objective
Causal consistency distillation
Trainable parameters
Student LoRA adapters
Optimizer
AdamW
Learning rate
1×10−4
Precision
bfloat16
Appendix
Table 16: Stage 2 causal consistency distillation hyperparameters. Stage 2 initializes both teacher and student from the Stage 1 causal model and distills the 50-step teacher into a four-step causal student using one online teacher ODE step per iteration.
Hyperparameter
Value
Initialization
Stage 2 causal student
Training objective
Asymmetric DMD
Trainable parameters
Student LoRA adapters and fake-score network
Optimizer
AdamW
Learning rate
1×10−4
Precision
bfloat16
Appendix
Table 17: Stage 3 asymmetric distribution matching distillation hyperparameters. Stage 3 initializes the generator from the four-step causal student produced by Stage 2 and refines it using a frozen bidirectional midtrained model as the real-score teacher and a trainable fake-score network that tracks the student’s generated distribution.
Hyperparameter
Value
RGB frames per sample
33
Latent frames per window, T
9
Sampling steps
4
Global sink capacity
3
Slow-memory capacity
9
Local-memory capacity
9
Appendix
Table 18: Stage 4 episode-level rollout and hierarchical temporal memory hyperparameters. Cache capacities are measured in temporal latent slots.
Item
Value
GPU
NVIDIA A100-SXM4-80GB
Precision
bfloat16
Batch size
1 (single episode / prompt stream)
Timing API
time.perf_counter() with torch.cuda.synchronize()
Reported aggregate
steady-state mean, warm-up discarded
Appendix
Table 19: Timing environment. All reported numbers use a single GPU at batch size one.
Model
Warm-up
Timed
Aggregation
PrimeSim Stage-3
1 window
3 repeats
mean
GE-Sim 2.0
3 chunks
12 chunks
mean of last 5 (steady state)
BWM (fair-aligned)
1 chunk
3 chunks
mean
BWM (official)
1 chunk
2 chunks
mean
Ctrl-World
bench default
5 chunks
mean
CausalWM
weights preloaded
2 chunks
mean
Appendix
Table 20: Warm-up schedule, timed repetitions and aggregation per model.
Model
History
Future T
Views V
Denoise steps
PrimeSim Stage-3
continuous KV
32 (33-frame window)
3
4 (1000, 750, 500, 250)
GE-Sim 2.0
4 (memory band)
25
3
4 (distilled)
BWM (fair-aligned)
4
25
1
50
BWM (official)
9
48
1
50
Ctrl-World
6
5
3 (stacked)
50
CausalWM
first frame + text
25
1
4+4+4 per stage
Appendix
Table 21: Chunk semantics per model: conditioning history, future frames per chunk T , number of camera views V and denoising steps.
Model
#tokens
PrimeSim Stage-3
46,656
GE-Sim 2.0
73,728
BWM (fair-aligned)
24,576
BWM (official)
141,120
Ctrl-World
14,400
CausalWM
∼ 19,200
Appendix
Table 22: Approximate latent tokens per forward (Eq. 14 ). GE-Sim 2.0 counts its effective temporal extent including memory; Ctrl-World uses its stacked 576×320 canvas; CausalWM’s temporal stride may differ, so its value is an estimate.
In this technical report, we propose Pelican-Sim 1.0, a general world model simulator for embodied intelligence that predicts future observations from visual context and robot actions to support downstream learning and decision making. The model incorporates four key design features: (1) Unified action representation: a 28-dimensional action value space covering most mainstream embodiments, keeping one model valid across heterogeneous devices. (2) Action-visual injection: URDF- and camera-rendered action videos bridge actions and pixels, giving markedly better controllability across embodiments, scenes, and tasks (PSNR +0.904 over alternative fusion baselines). (3) Sparse mixture-of-experts (MoE): sparse MoE layers add capacity for heterogeneous dynamics and absorb the action modality while reducing inter-modality conflict (FVD -6.530 vs. the dense backbone). (4) Efficient rollout generation: causal adaptation and few-step distillation yield a four-step autoregressive simulator, achieving a 5.67-fold speedup over the 35-step model. Benefiting from these designs, we train on approximately one million real-world and simulated trajectories and obtain large gains in action controllability and video quality: PSNR improves over the strongest evaluated baselines by 4.636 on AgiBotWorld Beta, 2.080 on RoboMIND, and 10.343 on RoboTwin, with the adapted EWMBench DYN score up 0.426 on RoboTwin. Relying on this, four downstream applications on RoboTwin succeed: 500 generated trajectories added to 50 demonstrations per task raise policy success from 70% to 93%; policy evaluation reaches a Pearson correlation of 0.994 across five checkpoints; and relative success gains reach 47.7% for action selection and 20.3% for policy improvement. Qualitative generalization across trajectory, scene, object, embodiment, and viewpoint shifts highlights its potential as a general-purpose world model simulator.
Shilong Zou, Shilin Zhang, Yingji Zhang +8
Beijing Innovation Center of Humanoid Robotics (X-Humanoid) WFM System Group
Real-world robot learning is constrained by the cost of collecting experience and evaluating candidate behaviors. Video generation models offer a scalable foundation for visual simulators that predict action outcomes before physical execution. Yet they often favor visual plausibility over accurate action following and coherent robot--object dynamics, while action-conditioned simulators depend on scarce, embodiment-specific data that are difficult to share across incompatible control spaces. We introduce WorldLine, an action-driven visual simulator that decouples transferable dynamics learning from heterogeneous action grounding. WorldLine learns manipulation dynamics from more than 10,000 hours of action-free robot videos and grounds them using over 2,000 hours of action trajectories across more than ten embodiments. An image-space action representation provides a shared control interface across embodiments, while multi-view and failure-enriched training with relational regularization improves interaction-sensitive prediction. Robot-focused few-step distillation enables efficient causal rollout while preserving action-critical motion. Across held-out and out-of-domain settings, WorldLine maintains strong visual quality and robot-motion agreement; on failed trajectories, it improves robot-mask IoU by 0.1626 over the strongest baseline. It predicts trajectory success with 74% mean accuracy across RoboTwin and AgiBot, one percentage point above the strongest baseline. Without RoboTwin training or adaptation, its rollouts improve task success by up to 21.4 percentage points over direct policy execution. Together, these capabilities make WorldLine a scalable and efficient visual simulator for policy evaluation and embodied planning. More results are available at project page.
Shenghe Zheng, Wenbo Li, Jiyao Zhang +4
The Hong Kong University of Science and Technology · Joy Future Academy · Peking University +1
Action-conditioned video prediction models (often referred to as world models) have shown strong potential for robotics applications, but existing approaches are often slow and struggle to capture physically consistent interactions over long horizons, limiting their usefulness for scalable robot policy training and evaluation. We present Interactive World Simulator, a framework for building interactive world models from a moderate-sized robot interaction dataset. Our approach leverages consistency models for both image decoding and latent-space dynamics prediction, enabling fast and stable simulation of physical interactions. In our experiments, the learned world models produce interaction-consistent pixel-level predictions and support stable long-horizon interactions for more than 10 minutes at 15 FPS on a single RTX 4090 GPU. Our framework enables scalable demonstration collection solely within the world models to train state-of-the-art imitation policies. Through extensive real-world evaluation across diverse tasks involving rigid objects, deformable objects, object piles, and their interactions, we find that policies trained on world-model-generated data perform comparably to those trained on the same amount of real-world data. Additionally, we evaluate policies both within the world models and in the real world across diverse tasks, and observe a strong correlation between simulated and real-world performance. Together, these results establish the Interactive World Simulator as a stable and physically consistent surrogate for scalable robotic data generation and faithful, reproducible policy evaluation.
Yixuan Wang, Rhythm Syed, Fangyu Wu +7
Columbia University · Toyota Research Institute · Amazon +1