GE-Act 2.0: Pretraining and Scaling a World-Action Model for Robotic Manipulation

Renhang Liu; Wenzhi Zhao; Zhuo Yang; Liliang Chen; Pengfei Zhou; Shengcong Chen; Guanghui Ren; Youlun Peng

cs.ROSep 4, 2026

GE-Act 2.0: Pretraining and Scaling a World-Action Model for Robotic Manipulation

Authors: Renhang Liu, Wenzhi Zhao, Zhuo Yang, Liliang Chen, Pengfei Zhou, Shengcong Chen, Guanghui Ren, Youlun Peng, +37 more

Abstract

World-action models (WAM) predict future states to guide robot actions, enabling learning from both action-free video and action-labeled interaction. Most inherit pretrained video generators, leaving WAM pretraining and scaling underexplored. We introduce Genie Envisioner Act 2.0 (GE-Act 2.0), a world-action model whose trainable generative and action components are all initialized from scratch on manipulation data. It combines a control-oriented autoencoder (CoAE), a single-step visual planner (SVP), and an inverse dynamics model (IDM). CoAE retains action- and instruction-relevant information under aggressive compression, while SVP produces a complete future state in one differentiable pass, so visual planning and inverse dynamics can be pretrained separately on complementary data. The components are then jointly trained with knowledge-aligned selective optimization (KASO), which reduces mismatched supervision by selecting only predicted futures judged behaviorally compatible with the recorded action. We evaluate pretrained checkpoints directly, without per-task fine-tuning, on 100 tasks across 20 manipulation skill groups with held-out scenes, backgrounds, lighting, and object instances. Scaling co-training data from 300 to 30,000 hours raises success from 17.1% to 44.1% on G1-OP and from 13.4% to 31.1% on G2-90D; despite comprising less than 2% of the co-training data, G2-90D improves by 17.7 points, suggesting cross-embodiment transfer. Gains span 19/20 and 18/20 skill groups, and skill-specific coverage strongly correlates with zero-shot out-of-distribution (OOD) success (Pearson r=0.80; Spearman rho=0.85). Under the same protocol, the model grounds object, color, shape, and position references in at least 90% of trials and follows explicit instructions even when they conflict with an already-committed behavior or a conventional scene association.

Figures & tables

Figure 1: GE-Act 2.0 system architecture. The SVP grounds the instruction in the current observation and fully denoises multi-view future visual latents in a single flow-generator pass; the IDM combines current and predicted latents with proprioception to produce dense and sparse action sequences.

Figure 2: CoAE architecture and training objectives. At the

256{\times}384

frame resolution used by the SVP, the encoder maps one input frame to a

4{\times}6

grid of 512-channel latents, giving 24 tokens per frame. A reconstruction decoder provides pixel-level supervision, while three alignment heads match language-aligned, spatiotemporal, and dense visual features from frozen visual teachers.

Representation	Spatial downsampling	Channels	Pixel decoder	Action MAE $\downarrow$	Caption-match error (%) $\downarrow$
CoAE	$64{\times}64$	512	✓	0.01673	2.05
DC-AE	$64{\times}64$	128	✓	0.02573	2.71
DINOv3	$16{\times}16$	1024	✗	0.01273	2.75
V-JEPA 2.1	$16{\times}16$	1024	✗	0.01487	4.15
SigLIP	$32{\times}32$	2048	✗	0.03138	3.28

Table 1: Controlled probes of five frozen visual representations. Spatial downsampling is reported as the height

\times

width factor relative to the input; for example,

64{\times}64

means that one latent location spans a

64{\times}64

-pixel input region. Channels gives the latent feature dimension. Lower is better for both probe metrics.

Figure 4: Comparison of Decoupled Pretraining, End-to-End Co-training, E2E + Pretraining Losses, and KASO. Decoupled Pretraining is the pretraining stage itself: the SVP and the IDM are trained on their own objectives and never exchange generated futures. The four strategies differ in whether the IDM receives generated future visual states, whether the pretraining losses for the SVP and IDM remain active, and whether generated candidates are filtered for action compatibility.

Figure 5: Video samples projected onto the $z_{1}$ – $z_{2}$ plane (top, with ground-truth contours overlaid) and induced action densities (bottom) learned by each strategy on the toy system. The displayed run uses seed 0 and single-sample evaluation for every learned strategy; seed-to-seed variation is discussed in Appendix B.7 . Ground truth contains four video modes and two action modes; in the displayed run, Decoupled Pretraining preserves the four video modes but yields a diffuse action density; End-to-End Co-training collapses both the video and action distributions; with the deterministic IDM, E2E + Pretraining Losses preserves the video modes but collapses the action density to the mean of the two action modes; and KASO recovers the four video modes and the bimodal action structure.

Figure 6: The KASO pipeline. Candidate visual predictions are sampled from the SVP and scored without gradients; selected noises are then replayed exactly so that the joint action loss updates both the IDM and the SVP.

(a) Follow score: four-object scene (10 trials per target)
Method		Sponge	Red apple	Mouse	Remote	Macro avg.
E2E		90	100	90	70	87.5
E2E+PT		100	100	90	90	95
KASO		100	100	90	90	95
(b) Pick success
	Single-object scene (25 trials)	Four-object scene (10 trials per target)

Table 2: Controlled real-robot ablation of retaining pretraining losses and action-compatible selection. Follow score requires contacting the instructed target and no distractor; pick success additionally requires completing the pickup. Four-object macro averages assign equal weight to the four targets. All variants initialize from the same full-scale pretrained components (SVP, 39,000 hours; IDM, 32,000 hours) and use the same 300-hour connection/alignment mixture, including 30 hours from G2-90D, without evaluation-specific SFT. Entries are percentages.

Figure 7: Stage-specific training-data mixtures. The panels show the 39,000-hour SVP pretraining mixture, the 32,000-hour IDM pretraining mixture, and the 30,000-hour co-training mixture. Slice angle denotes the share of data hours; colors and source order are held fixed across panels. Violet and coral highlights emphasize the action-free egocentric/human video used for SVP pretraining and the rollout/failure trajectories used for IDM pretraining, respectively. The keys report source hours and percentages.

Figure 8: G1-OP zero-shot OOD scaling across 100 atomic tasks and four data volumes. Top: the 100 tasks as one mosaic per training-data scale, with tile intensity denoting per-task SR. Middle: representative zero-shot OOD rollouts of the 30,000-hour model. Bottom: per-task SR at every scale, grouped by skill; colors follow the skill-group legend.

Figure 9: Skill-group zero-shot OOD scaling trajectories for G1-OP (more than 50% of the co-training mixture) and G2-90D (less than 2%). Each panel reports one group’s mean SR across the 300-, 1,200-, 5,000-, and 30,000-hour models.

Figure 10: Representative successful zero-shot OOD rollouts on G2-90D across skill groups, each shown as three frames spanning the start, mid-execution, and outcome of the trial. Skill labels are colored by skill group, matching the palette of Figure 8 .

Figure 11: Skill-group training coverage and G1-OP zero-shot OOD success for the 30,000-hour model. Each point is one of the 19 groups retained after omitting the mechanically trivial Close group; the horizontal axis reports the corresponding training hours across all embodiments on a logarithmic scale, and the vertical axis reports SR on a logit scale. Coverage and success are strongly correlated (Pearson

r=0.80

; Spearman

\rho=0.85

), with a fitted gain of 1.94 logit units per decade of data.

Figure 12: Controlled scenes for fine-grained instruction following, ordered as introduced in the text: object identity, color, size, shape, and position & order. Position and order share one setup, evaluated in a horizontal layout (left/right and ordinal references) and a vertical layout (near/far). Top row: Pick scenes; bottom row: Place scenes.

Figure 13: Fine-grained instruction following and language distribution across six grounding dimensions. Left: occurrence in 95,010 human-written RefCOCOg expressions (Natural) and the GE-Act 2.0 training corpus, shown on a logarithmic scale. Middle and right: Follow Score and full-task SR for Pick and Place, respectively. Dimensions are ordered by their occurrence in the GE training corpus; descriptor categories are counted independently and may overlap.

Figure 14: Instruction-conflict stress tests. Each panel shows the active instruction, frames from one rollout, and the outcome; red rules mark where the instruction changed. (a) Target switch: the commanded cup changes from green to blue, then to orange, while the gripper is already at the previous target; the robot redirects each time and grasps the final target. (b) Arm switch: the command moves from the right arm on the green cup to the left arm on the orange cup, and the left arm completes it. (c) Semantic conflict: the robot places a cup into a shoebox, following the explicit object–destination relation over the conventional scene association. Illustrative only; not part of the quantitative evaluation.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Easy	Background	Light	Clutter	Height	Hard
StarVLA	58.10	27.10	50.90	24.20	48.40	10.60
GR00T-N1.7	43.60	40.40	41.90	27.10	39.00	20.70
$\pi_{0.5}$	73.10	67.00	69.20	57.90	67.60	47.90
GE-Act 2.0	76.71	70.28	65.92	66.52	70.39	60.52

Appendix

Table 3: RoboTwin Clean-to-Random success rate (%,

\uparrow

). Models are adapted only on the Easy/Clean setting; the remaining columns evaluate OOD environment variations. The best result in each column is bolded.

Task	$\pi_{0}$	GR00T-N1.7	LingBot-VLA 2.0	$\pi_{0.5}$	ACoT-VLA	GE-Act 2.0
pick_billiards_color	0.50	0.73	0.88	0.83	0.91	0.95
pick_block_color	0.49	0.80	0.80	0.87	0.96	0.92
pick_block_number	0.40	0.61	0.72	0.82	0.76	0.71
pick_block_shape	0.20	0.44	0.43	0.55	0.52	0.52
pick_block_size	0.34	0.62	0.68	0.79	0.83	0.76
pick_common_sense	0.19	0.68	0.63	0.59	0.66	0.74

Appendix

Table 4: Results on the OOD GenieSim-Instruction benchmark (normalized score,

\uparrow

). Evaluation configurations vary background textures, object positions, and other scene factors relative to the training demonstrations. The best result in each row is bolded.

Model	Camera	Robot	Language	Light	Background	Noise	Layout	Overall
StarVLA	52.5	49.8	88.5	95.7	95.7	73.0	76.9	74.1
$\pi_{0.5}$	78.4	73.6	80.8	96.2	94.1	89.0	84.5	84.4
GE-Act 2.0	94.1	50.7	81.4	94.0	60.5	95.5	83.1	80.4

Appendix

Table 5: LIBERO-Plus success rate (%,

\uparrow

) after adaptation on standard LIBERO. Each column evaluates a held-out perturbation axis. Overall is the success rate over all 10,030 task instances, so axes are weighted by their instance counts, following the benchmark’s protocol; all models are scored the same way. The best result in each column is bolded.

Component	Configuration
CoAE	Framewise deep-compression autoencoder (DC-AE layout) with $64\times$ spatial downsampling and 512 latent channels; a $256\times 384$ frame gives a $4\times 6$ grid of 24 tokens. Latents are standardized per channel with corpus statistics.
SVP flow generator	2.51 B parameters. A 36-block diffusion transformer of width 2048 (32 heads): 20 shared blocks followed by 8 blocks for the mean-velocity head $u_{\theta}$ and 8 parallel blocks for the instantaneous-velocity head $v_{\theta}$ . Every block cross-attends to the fused VLM states; every third block attends across views. Conditioning enters through adaptive layer normalization as the sum of sinusoidal embeddings of $t$ and of $r$ and a learned view-identity embedding. Per view the input is one conditioning frame plus six predicted frames, i.e., $7\times 24=168$ tokens.
VLM (frozen)	Qwen3.5-2B. The prompt holds the current head-view frame (96 vision tokens) and the instruction in a fixed window of 192 tokens; the text-span states of all 25 layers are fused by the layer gate of Eq. ( 5 ).
IDM	0.56 B parameters. 28 transformer blocks of width 1152 (16 heads) with self-attention over action tokens and cross-attention to the visual latent tokens of the current and predicted frames. The flow time and a learned embodiment embedding enter through adaptive layer normalization; actions are 32-dimensional joint-space vectors, zero-padded across embodiments, and the proprioceptive state enters through a small MLP.

Appendix

Table 6: Trained components. Parameter counts are read from the final co-trained checkpoint; the frozen Qwen3.5-2B VLM (2.72 B parameters) is excluded.

Skill group	Atomic task instances
Pick (13)	Red apple; toothpaste; red cube; water bottle (two scene variants); marker pen from water cup; toothbrush; water cup; controller; towel; stapler; whisk; towel from rod.
Place (6)	Apple into container; blue cube into plate; orange into small bowl; marker pen into pen holder; red cube into paper cup; marker pen into open drawer.
Wipe (3)	Plate with rag; water stain with sponge; table with eraser.
Stack (7)	Red cube on yellow block; paper cup on paper cup; bowl on bowl; coaster on coaster; cola can on cola can; cutlery-box lid; trash-can lid.
Pass (5)	Remote control, left-to-right; orange ball, right-to-left; sponge eraser, left-to-right; Rubik’s cube, right-to-left; cola can, left-to-right.
Fold (4)	Towel, one arm; towel, both arms; rag twice, both arms; unfold towel, both arms.

Appendix

Table 7: Atomic-task taxonomy for the 100-task real-robot zero-shot OOD suite. Parenthetical counts sum to 100. Unless an arm or both arms are explicitly specified, the task is executed with the right arm.

Task	G1-OP SR (%)				G2-90D SR (%)
	0.3k h	1.2k h	5k h	30k h	0.3k h	1.2k h	5k h	30k h
Pick (13 tasks)
pick up red apple	90	100	100	100	60	100	100	100
pick up toothpaste	0	20	40	80	30	40	60	60
pick up red cube	80	80	70	90	70	70	50	50
pick up water bottle	40	50	70	90	60	100	90	100

Appendix

Table 8: Task-level zero-shot OOD success rate (SR, %) across data scales. Every G1-OP and G2-90D result is shown separately; each SR is computed over 10 trials, and all 100 tasks are included.

Dimension	Instruction	Follow	SR
Pick (33 instructions)
Object	pick up the white badminton ball with the right arm.	100	80
Object	pick up the yellow knife with the left arm.	100	80
Object	pick up the blue comb with the right arm.	100	80
Object	pick up the black mouse with the left arm.	100	100
Object	pick up the black tv controller with the right arm.	100	100

Appendix

Table 9: Instruction-level results of the fine-grained grounding study (G1-OP, five trials per instruction). Follow and SR are percentages of the five trials.

Dimension	Matching rule	RefCOCOg count	RefCOCOg (%)	GE corpus (%)
Object	Residual: no color, size, shape, position, or order descriptor	39,474	41.5	58.3
Color	Explicit color words; orange as a fruit name excluded	42,161	44.4	29.6
Size	large , small , big , tiny , medium and comparative/superlative forms	5,299	5.6	0.96
Shape	round , square , rectangular , cylindrical , triangular	574	0.6	0.67
Position	left , right , near , far ; body-side phrases such as right hand excluded unless another spatial relation is present	11,927	12.6	13.8
Order	middle , center , ordinals, leftmost , rightmost ; generic between excluded	2,440	2.6	0.13

Appendix

Table 10: Lexical matching rules and descriptor frequencies used in Figure 13 . RefCOCOg counts are out of 95,010 expressions.

Explore similar work

What Limits Cross-Task Generalization in World Action Models?
Oct 12, 2026Chen Zhao, Zipeng Ye, Yihan Lin +2World Action Models World Models for Robotics
Renmin University of China, Beijing 100872, China · Key Laboratory of Data Engineering and KnowledgeEngineering, MOE, China · Meituan, Beijing 100102, China
$τ_0$ -WM: A Unified Video-Action World Model for Robotic Manipulation
May 31, 2026Pengfei Zhou, Shengcong Chen, Di Chen +17Action-Conditioned Video Generation Long-Horizon Robotic Manipulation
1Shanghai Innovation Institute · 2AGIBOT Finch
Riemann-1.0: An Embodied World Action Model for Physical AI
Aug 27, 2026Haofeng Sun, Jiangbo Pei, Fei Kang +13World Models for Robotics Long-Horizon Robotic Manipulation
Riemann Dynamics · Research Intern at Riemann Dynamics

Oct 12, 2026J/K move · Enter open · S save

What Limits Cross-Task Generalization in World Action Models?

Chen Zhao, Zipeng Ye, Yihan Lin, Yuchen Xie+1

Renmin University of China, Beijing 100872, China · Key Laboratory of Data Engineering and KnowledgeEngineering, MOE, China · Meituan, Beijing 100102, China

World Action Models World Models for Robotics Action-Conditioned World Models

Open paper AskPDF arXiv

Cross-task generalization remains difficult for world action models (WAMs), even when their predicted futures depict plausible manipulation. This gap can be attributed directly: we make this gap directly testable by decoupling a WAM into a world model and an inverse-dynamics model (IDM), then holding the IDM fixed while replacing the source of future images. We introduce Align Once, Evaluate Everywhere (AoE), a camera-space inverse-dynamics model pretrained on large-scale robot data, which provides this recovery interface for different future sources across tasks and environments. Through it, we find that actions recovered from recorded futures of unseen tasks execute almost as well as replayed demonstrations, whereas futures predicted by world models collapse across benchmarks. The failures are not generic: they concentrate on the gripper and interaction geometry while generation capacity is spent on task-irrelevant background. This diagnosis suggests that supervision should target exactly these regions. We therefore introduce the vision-to-action (VITA) loss, which passes generated futures through the frozen AoE and penalizes discrepancies in the recovered actions, redirecting generation toward action-relevant detail. Across two world models and three manipulation benchmarks, this simple change raises the execution success of generated futures from near zero to 47.0%. The same loss also transfers to WAMs and yields consistent gains in native control.

Abstract

Explore similar work

What Limits Cross-Task Generalization in World Action Models?

τ0τ_0τ0​-WM: A Unified Video-Action World Model for Robotic Manipulation

Riemann-1.0: An Embodied World Action Model for Physical AI

$τ_0$ -WM: A Unified Video-Action World Model for Robotic Manipulation