cs.ROOct 2, 2026
SaveXGenAct: Geometry-Enhanced World Action Models through Cross-Task Generation
Organizations: University of Wisconsin–Madison · University of Maryland, College Park
Abstract
World action models (WAMs) have advanced robot control by predicting how observations and actions evolve over time. Despite this progress, RGB and action based future prediction does not explicitly address the spatial understanding needed for robot manipulation. Existing efforts often add a limited set of spatial prediction tasks through specialized heads or branches, leaving both the range of spatial supervision and the model architecture fragmented. We introduce XGenAct, a world action model that represents RGB observations, robot actions, metric depth, surface normals, and functional role segmentation as RGB videos through deterministic codecs. By sampling perception and action tasks during training, XGenAct uses one video diffusion transformer and one objective to learn temporal prediction across these spaces without modality specific learned heads. On held out RLBench tasks, structured perception training improves average closed loop success over RGB only training, and XGenAct achieves 52% success in the five task external comparison, versus 26% for the strongest evaluated baselines. It also predicts future depth and segmentation more accurately than the evaluated pipelines that generate RGB first and then apply a frozen perception expert.
Figures & tables
Figure 1: One checkpoint, multiple future spaces. Given initial observation and action anchors and the instruction “slide the bottom drawer open,” XGenAct generates future RGB, action images, metric depth, functional-role segmentation, and surface normals at matched temporal indices. The column is given; the rest are generated. The visual rows are obtained through separate prompts to the same model rather than one five-stream decode; one camera view is shown.
Figure 2: Training and inference, shared architecture, and conditioning plans.
| Method | Close microwave | Toilet seat down | Reach target | Close drawer | Close laptop | Avg. |
|---|---|---|---|---|---|---|
| PAD-Depth ( Guo et al., 2024 ) | 5 | 5 | 10 | 40 | 0 | 12 |
| ( Physical Intelligence et al., 2025 ) | 30 | 5 | 35 | 35 | 25 | 26 |
| MolmoAct ( Lee et al., 2025 ) | 0 | 40 | 10 | 0 | 5 | 11 |
| UWM ( Zhu et al., 2025 ) | 0 | 10 | 5 | 50 | 5 | 14 |
| LaWAM ( Chen et al., 2026a ) | 15 | 70 | 10 | 25 | 5 | 25 |
| Evo-1 ( Lin et al., 2026 ) | 10 | 10 | 0 | 10 | 5 | 7 |
Table 1: Closed-loop success rate (%) on five unseen RLBench tasks , 20 trials each. All methods except MolmoAct are fine-tuned on our 16-task RLBench split; MolmoAct is evaluated zero-shot with its output adapted to our controller, following Zhen et al. (2026) . Training and evaluation details are in Appendix H . Best result per column in bold.
Figure 3: Paired closed-loop rollouts against external policies. Final frame of one episode per task and method; all methods start from the same scene (trials are paired by scene seed). The trial shown is the first one XGenAct solves, and the baselines are shown on that trial whatever their outcome. Success rates over 20 trials are in Table 1 ; per-task key frames and failure cases are in Appendix J .
| Model | RLBench | ||||
|---|---|---|---|---|---|
| Close microwave | Toilet seat down | Close drawer | Meat on grill | RL avg. | |
| RGB | 5 | 20 | 95 | 10 | 32.50 |
| RGB+D | 45 | 20 | 85 | 60 | 52.50 |
| RGB+S | 95 | 5 | 60 | 70 | 57.50 |
| RGB+N | 75 | 40 | 70 | 45 | 57.50 |
| RGB+D+S | 70 | 25 | 60 | 60 | 53.75 |
Table 2: Closed-loop success (%) across training-time modality menus. Bold and underlined values are the best and second-best results per column.
Figure 4: Direct generation versus generated RGB followed by a frozen expert. Each column is one training menu; the dots below mark the perception streams it adds to RGB. Teal bars: the model generates the perception space directly. Gray bars: the same model generates future RGB, which Depth Anything V2, Lotus-G or CLIPSeg converts. Numbers above a pair give direct minus expert (green: direct is better). Scores average four held-out RLBench tasks, measured against the simulator state reached by each route’s own actions; a task is left out of a panel if its executed poses deviate from the commanded ones by 10 mm or more (Appendix G ; per-task values in Table 7 ).
Figure 5: Generated metric depth lifted to 3-D on held-out tasks. The last generated depth frame (view 1, open loop) and the demonstration’s depth at the same frame are back-projected with the camera intrinsics and rendered from the same novel viewpoint and axis limits; colour encodes height. Pixels on depth discontinuities are dropped in both clouds. Further qualitative results are in Appendix J .
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
| Source | Training tasks |
|---|---|
| RLBench | open_drawer , slide_block_to_target , sweep_to_dustpan , meat_off_grill , turn_tap , put_item_in_drawer , close_jar , reach_and_drag , stack_blocks , light_bulb_in , put_money_in_safe , stack_wine , put_groceries_in_cupboard , place_shape_in_shape_sorter , push_buttons , insert_onto_square_peg |
| ManiSkill3 | peg_insertion_side , pick_cube , plug_charger , pull_cube_tool , push_cube , stack_cube , stack_pyramid |
Table 3: Training tasks of the two sources. The external comparison trains on the RLBench tasks only; the menu ablation and the perception study train on both sources at a ratio. Names follow the identifiers used by the data loader.
| Stream | Encoding | Codec round-trip | VAE round-trip |
|---|---|---|---|
| Depth | Barron RGB-cube path, equation 11 | AbsRel | AbsRel 0.362% ; off-path |
| Normal | , equation 12 | mean cosine | mean cosine 0.9906 ( ) |
| Segmentation | Role LUT with a 9-color palette | IoU (spec.) | target IoU median 0.965 |
| Action | Three Gaussian channels, equation 9 | mm 3D | Not measured |
| RGB control (same VAE, same clips) | N/A | 29.6–31.5 dB PSNR | |
Table 4: Admission-gate results for each output stream. All streams are evaluated after a codec round-trip and, where applicable, a round-trip through the frozen VAE before training. Metrics are computed against the decoded ground-truth stream. The RGB control uses the same VAE and clips and provides an approximate upper bound on reconstruction quality.
| IIII | 1I1I | FIFI | |
|---|---|---|---|
| Conditional |
Table 5: The shared conditioning distribution used by every trained model, including RGB , conditional on the action stream being retained (RLBench samples).
Figure 6: Perception evaluation by simulator replay. (a) One replanning step of the depth route (close microwave, view 1). From the current depth and pose, XGenAct generates a chunk of depth frames and action images . The decoded poses are executed, and after each step the simulator renders the depth of the state that the model’s own actions reached (middle row, raw simulator output in the same colour code); is scored against rather than against a demonstration. Scores use only the dynamic region , outlined in white: pixels whose simulator depth changed by more than 1 cm since the chunk began, which includes both where the arm is and where it was. The anchor and the ramp steps are never scored. (b) The three routes compared in Table 7 and Figure 4 . Each is a separate rollout scored against its own replay: direct generation of , the same checkpoint’s generated RGB passed through a frozen expert, and generated RGB itself.
| Method | Training | Cam. | Chunk | Exec. | Action target |
| 20,500 steps, batch 32 | 50 | 10 | SE(3) to current pose | ||
| LaWAM | 20,500 steps, batch 32 | 8 | 8 | absolute pose | |
| Evo-1 | 20,500 steps, batch 32 | ✓ | 50 | 25 | per-step delta, accumulated |
| VLANeXt | 20,500 steps, batch 32 | ✓ | 8 | 8 | absolute pose |
| MolmoAct | zero-shot | ✓ | 16 | 16 | triangulated 2D trace |
| PAD-Depth | 2,560 steps, batch 256 | ✓ | 3 poses | 16 | absolute pose |
Table 6: Training and execution settings of the methods in Table 1 . Cam. : camera parameters are an input. Chunk : actions predicted per query; Exec. : actions executed before replanning.
Figure 7: Close microwave , trial 0 (the first trial XGenAct solves). XGenAct closes the door; PAD-Depth, UWM and MolmoAct time out, and VLA-JEPA stops on repeated inverse-kinematics failures.
Figure 8: Failure cases of XGenAct : the first trial of each task that XGenAct does not solve. Both failures are timeouts; on the same scenes every baseline also fails, except MolmoAct in (b).
Figure 9: Paired closed-loop executions on RLBench. Each row uses the same task instance and scene seed for RGB-only and XGenAct . The basketball row is shown from a recording-only right-shoulder camera; both policies retain the original front and overhead inputs used throughout the closed-loop evaluation.
Figure 10: Imagined versus executed future, meat on grill (view 1, first replanning step). Top: RGB frames XGenAct generated for step . Bottom: the simulator after executing the decoded actions up to (corner: PSNR to the imagined frame). Across the 40-step chunk, the generated and executed sequences show the same grasp, transport, and placement in nearly the same arm poses (whole-frame PSNR – dB).
Figure 11: Imagination versus execution, close microwave (trial 0, both views). For each view: the generated RGB frame, the simulator after executing the decoded actions up to the same step, and the generated action image that produced them.
Figure 12: Open-loop generation across output spaces. For each episode, the upper row is generated (frame 40 of 40, view 1) and the lower row is the demonstration. Each output space is a separate prompt of the same checkpoint, so the arm pose can differ slightly between columns.
Figure 13: End-effector paths decoded from generated action images (red) versus the demonstration (green, dashed), drawn on the frame-0 observation of both views; the top two rows are training tasks and the bottom two held-out tasks. Each point is the centroid of the position blob of one frame; open circle: start, square: end, ring: first frame with the gripper closed. The corner value is the mean 2-D distance between the two paths in pixels ( images); it is not the 3-D pose error of the tables.