Robot actions are inherently embodiment-specific and only weakly aligned with image-space visual changes, limiting their effectiveness as conditioning signals for robot world models. In contrast, visual tracks provide an embodiment-agnostic representation of how task-relevant points move through a scene, offering dense image-space guidance for accurate and spatially precise future video prediction. Building on this observation, we propose TrAct, a world-model-based robot decision-making framework that uses visual tracks as an intermediate interface between control and prediction. TrAct consists of three components: a Vision-Language-Action-and-Track model (VLAT) that jointly predicts candidate actions and corresponding visual tracks from the current observation and language instruction; a track-conditioned world model (TWM) that predicts future visual outcomes conditioned on the proposed tracks; and a vision-language reward model (VLAC) that scores the predicted outcomes. At inference time, VLAT generates candidate action-track pairs, TWM rolls out their visual consequences, and VLAC selects the track whose predicted outcome best satisfies the instruction; the action paired with the selected track is then executed by the robot. Experiments on the proposed LIBERO-INTEGRAL benchmark and real-world Franka manipulation show that TrAct improves success rates from 27% to 55% in simulation and from 49% to 76% on real-world tasks compared with the strong VLA baseline π0.5. Furthermore, TWM consistently improves video prediction quality over the action-conditioned world model (AWM). These results demonstrate that visual tracks provide an effective shared interface between robot control and visual prediction, enabling more accurate world modeling and stronger robot generalization.
Figures & tables
Figure 1: Overview of TrAct. TrAct uses VLAT (left, purple), pretrained on large-scale cross-embodiment data, to propose candidate action–track pairs. TWM (middle, gray) rolls out the visual outcome for each candidate conditioned on the tracks, and VLAC (middle, green) selects the highest-reward rollout; the robot executes the paired action. Visual tracks thus serve as an intermediate interface between future prediction and robot control. We evaluate TrAct on diverse simulation and physical robot tasks (right).
Figure 2: Overview of the TrAct framework. Given an observation ot and language instruction l , the VLAT model jointly predicts K candidate actions and corresponding visual tracks. Each set of predicted tracks conditions a world model to generate future video frames. A VLM-based reward model scores the predicted outcomes and selects the action whose rollout best fits the instruction.
Metric
Real / Agent
Real / Wrist
Sim / Agent
Sim / Wrist
Act.
Track
Act.
Track
Act.
Track
Act.
Track
PSNR ↑
20.70
21.27
16.89
20.62
15.12
24.51
16.25
22.71
SSIM ↑
0.767
0.802
0.621
0.738
0.482
0.843
0.668
0.822
LPIPS ↓
0.126
0.111
0.372
0.225
0.438
0.106
0.452
0.182
FID ↓
105
95
176
130
248
156
222
92
FVD ↓
148
105
246
140
129
38
207
162
Table 1: Video generation quality of action-conditioned and track-conditioned world models. Results are reported for real-world and simulation scenes under agent-view and wrist-view cameras. Act. denotes action-conditioned world model, and Track denotes track-conditioned world model. ↑ indicates higher is better and ↓ indicates lower is better. Best results for each scene are in bold.
Figure 3: Qualitative video comparison. TWM produces more spatially accurate and motion-coherent frames than AWM across both agent-view and wrist-view cameras.
Method
LIBERO-10
Goal
Object
Spatial
Avg.
π0.5
94.0
96.0
98.0
99.0
96.8
VLAT
94.0
98.0
100.0
100.0
98.0
VLAT+AWM
94.0
98.0
100.0
100.0
98.0
TrAct
94.0
100.0
99.0
100.0
98.3
Table 2: Success rates (%) on four standard LIBERO benchmark suites. Each suite contains 100 evaluation episodes. The best result in each column is shown in bold.
Task
Bg.
π0.5
π0.5 +VLAC
VLAT
VLAT+AWM
TrAct
Put the green block in the bowl
R
0.5
0.7
0.6
0.7
0.7
D
0.4
0.5
0.4
0.5
0.7
Put the half pink noodle in the bowl
R
0.6
0.9
0.8
1.0
1.0
D
0.6
0.8
0.6
0.8
0.8
Put the hotdog bun in the bowl
R
0.6
0.8
0.6
0.8
0.8
D
0.4
0.6
0.5
0.6
0.6
Table 4: Success rates on five unseen physical robot manipulation tasks, each evaluated under a regular training background ( R ) and a visually different non-training background ( D ). We compare five models. The best result for each row is shown in bold.
Figure 4: Rollout Demonstrations of two tasks: put the half-length pink noodle in the bowl (left) and close the right cabinet door (right). Each step is overlaid with track predictions.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Component
VLAT
World Model
VLAC
Total
Latency (ms)
52
1810
569
2431
Appendix
Table 5: Inference latency breakdown per planning step, measured with K=16 candidates, rolled out in parallel, on NVIDIA L40S GPUs. Each planning step yields a 16-step action chunk.
Figure 5: Additional Qualitative video comparison. TWM continues to produce more spatially accurate and motion-coherent frames than AWM across both agent-view and wrist-view cameras.
Figure 6: Overview of the physical experiments. We use a base and wrist-view camera and finetune the model on four training tasks. We then test five out-of-distribution tasks, as well as an unseen background for each task.
Task
Var.
π0.5
VLAT
VLAT+AWM
TrAct
Pick the alphabet soup and place it in the basket
Swap
0.4
0.5
0.7
0.7
Put both the cream cheese box and the butter in the basket
Swap
0.4
0.5
0.6
0.6
Put both moka pots on the stove
Object
0.5
0.5
0.6
0.7
Put the yellow and white mug in the microwave and close it
Object
0.2
0.3
0.4
0.5
Pick the akita black bowl next to the ramekin and place it on the plate
Task
0.1
0.3
0.5
0.5
Put the plate on the stove
Task
0.2
0.4
0.5
0.5
Appendix
Table 6: Results on the LIBERO-INTEGRAL benchmark. The benchmark includes robustness variations from LIBERO-PRO and LIBERO-Plus, and the last ten tasks evaluate cross-embodiment generalization by replacing the Franka Panda with a UR5 robot. Scores denote task success rates. The best result for each row is shown in bold.
Table 12
Task
π0.5
TrAct
TrAct+
Put both the alphabet soup and the cream cheese box in the basket (UR5)
0.1
0.3
0.3
Pick the ketchup and place it in the basket (UR5)
0.0
0.1
0.2
Put the bowl on the stove (UR5)
0.2
0.5
0.6
Pick the orange juice and place it in the basket (UR5)
0.0
0.5
0.5
Pick the tomato sauce and place it in the basket (UR5)
0.0
0.2
0.3
Average
0.06
0.32
0.38
Appendix
Table 9: Additional results on the five most difficult UR5 cross-embodiment tasks from LIBERO-INTEGRAL. TrAct+ denotes TrAct pretrained with the larger DROID+BridgeData V2+EgoDex mixture. Scores denote task success rates. The best result in each row is shown in bold.
Figure 7: Rollout examples Rollout with four key frames showing 16-horizon track predictions under the training background (left) and unseen background (right). Task list from top row to bottom row: put the half-length pink noodle in the bowl (1), put the hotdog bun in the bowl (2), put the green brick in the bowl (3), pour the angled scooper’s pasta into the pan (4), and close the partially open right cabinet door (5).
Figure 8: Rollout examples Rollout with four key frames. Task list from top row to bottom row, left to right column: pick the alphabet soup and place it in the basket (1), put both the cream cheese box and the butter in the basket (2), put both moka pots on the stove (3), put the yellow and white mug in the microwave and close it (4), pick the akita black bowl next to the ramekin and place it on the plate (5), put the plate on the stove (6), put the white mug on the left plate and put the yellow and white mug on the right plate (7), put the wine bottle on the top of the drawer (8), put both the cream cheese box and the butter in the basket (9), and pick up the book and place it in the back compartment of the caddy (10).
Figure 9: UR5 Rollout examples Rollout with four key frames. Task list from top row to bottom row, left to right column: put both the alphabet soup and the cream cheese box in the basket (1), put both the alphabet soup and the tomato sauce in the basket (2), put the bowl on the stove (3), pick the orange juice and place it in the basket (4), pick the akita black bowl between the plate and the ramekin and place it on the plate (5), pick the akita black bowl next to the plate and place it on the plate (6), pick the ketchup and place it in the basket (7), pick the tomato sauce and place it in the basket (8), pick the akita black bowl on the cookies box and place it on the plate (9), and put the bowl on the plate (10).
Robot manipulation uses temporal context to select actions and visual foresight to assess their consequences, yet dense representations of past and future observations incur substantial processing costs. We introduce PACT-WAM, a world-action model that jointly generates a 16-step action trajectory and its temporally corresponding visual forecast through conditional flow sampling. Hierarchical history encoding assigns coarse spatial representations to earlier observations and finer representations to recent ones, retaining 16 observations with 256 tokens per view, 75% fewer than dense encoding of the same frames. A shared flow module jointly updates continuous action and visual states through two modality-specific heads under transition-wise causal attention, and a TiTok-VAE decoder reconstructs multi-view future images from the visual latents. Decoded forecasts also support Proposal Review (PR), a vision-language model component for execution-prefix selection and proposal rejection. Without PR, PACT-WAM achieves average success rates of 98.6%, 92.3%, and 78.0% on LIBERO, RoboTwin 2.0, and real-world Piper tasks, respectively. PR provides a test-time enhancement, raising these rates to 99.5%, 93.4%, and 86.7%. Ablations show that hierarchical history allocation and joint action-visual generation improve control success, while analyses of visual capacity and forecast-guided execution characterize the trade-offs between success and proposal-generation cost.
Yushan Liu, Jingjing Fan, Shoujie Li +3
Tsinghua University · Nanyang Technological University · Xspark AI
Vision-language-action models (VLAs) are promising general-purpose robot policies, but adapting them to new tasks typically requires costly task-specific teleoperation data. As an alternative, we study one-shot demo-conditioned VLAs, where a robot policy is conditioned on a single demonstration video of an unseen task. We find that existing end-to-end approaches often struggle when successful execution requires precisely localizing small target regions. To address this limitation, we propose SeeTraceAct, a demo-conditioned VLA framework that encourages precise spatial grounding through visibility-aware prediction of future end-effector traces. To enable reproducible evaluation with cross-embodiment demonstrations, we introduce and release RoboCasa-DC, a demo-conditioned extension of RoboCasa that pairs its robot trajectories with newly collected humanoid demonstrations. Experiments on RoboCasa-DC and a real-world benchmark, where a Franka Panda arm is conditioned on human demonstrations, show that SeeTraceAct outperforms baselines, achieving the best success rate across all four RoboCasa-DC settings and improving real-world average success by 9.5 percentage points.
Jaehyeon Son, Junhyun Kim, Kyle Kam +7
Georgia Institute of Technology · Allen Institute for AI · Johns Hopkins University +1
World-Action Models (WAMs) build robot control on video-generation backbones, which jointly predict dense future visual trajectories and robot actions. We argue that video generation is an unnecessary intermediate objective for world-action modeling. For robotic manipulation, the goal of a world model is not to reproduce how the world looks at every intermediate moment, but to predict the state that the world will reach after an action is executed. The intermediate frames only describe the visual transition between physical states, which consumes substantial model capacity and computation, but do not directly specify the physical outcome that the robot action is intended to produce. In this paper, we propose DELE-w0.5, which infers robot actions from predicted future states without relying on video generation. Concretely, DELE-w0.5 infers the action sequence from its corresponding compact future latent state. The future latent state captures the action-relevant physical outcome of robot interaction and serves as an explicit bridge between world modeling and action generation. The core design principle of DELE-w0.5 is to model how the physical world changes under robot actions, rather than how its visual appearance evolves frame by frame. This formulation removes the high-dimensional visual redundancy introduced by dense video representations, and it therefore enables cheaper training and low-latency inference. Across 640 real-robot trials on four long-horizon manipulation tasks, our DELE-w0.5 achieves the best performance among all compared policies, attaining 62.5% overall full-task success and 81.3% macro ordered-stage progress. It outperforms the strongest baseline by 32.5 percentage points in full-task success and 20.1 percentage points in macro progress.