Robot actions are inherently embodiment-specific and only weakly aligned with image-space visual changes, limiting their effectiveness as conditioning signals for robot world models. In contrast, visual tracks provide an embodiment-agnostic representation of how task-relevant points move through a scene, offering dense image-space guidance for accurate and spatially precise future video prediction. Building on this observation, we propose TrAct, a world-model-based robot decision-making framework that uses visual tracks as an intermediate interface between control and prediction. TrAct consists of three components: a Vision-Language-Action-and-Track model (VLAT) that jointly predicts candidate actions and corresponding visual tracks from the current observation and language instruction; a track-conditioned world model (TWM) that predicts future visual outcomes conditioned on the proposed tracks; and a vision-language reward model (VLAC) that scores the predicted outcomes. At inference time, VLAT generates candidate action-track pairs, TWM rolls out their visual consequences, and VLAC selects the track whose predicted outcome best satisfies the instruction; the action paired with the selected track is then executed by the robot. Experiments on the proposed LIBERO-INTEGRAL benchmark and real-world Franka manipulation show that TrAct improves success rates from 27% to 55% in simulation and from 49% to 76% on real-world tasks compared with the strong VLA baseline π0.5. Furthermore, TWM consistently improves video prediction quality over the action-conditioned world model (AWM). These results demonstrate that visual tracks provide an effective shared interface between robot control and visual prediction, enabling more accurate world modeling and stronger robot generalization.
Figures & tables
Figure 1: Overview of TrAct. TrAct uses VLAT (left, purple), pretrained on large-scale cross-embodiment data, to propose candidate action–track pairs. TWM (middle, gray) rolls out the visual outcome for each candidate conditioned on the tracks, and VLAC (middle, green) selects the highest-reward rollout; the robot executes the paired action. Visual tracks thus serve as an intermediate interface between future prediction and robot control. We evaluate TrAct on diverse simulation and physical robot tasks (right).
Figure 2: Overview of the TrAct framework. Given an observation ot and language instruction l , the VLAT model jointly predicts K candidate actions and corresponding visual tracks. Each set of predicted tracks conditions a world model to generate future video frames. A VLM-based reward model scores the predicted outcomes and selects the action whose rollout best fits the instruction.
Metric
Real / Agent
Real / Wrist
Sim / Agent
Sim / Wrist
Act.
Track
Act.
Track
Act.
Track
Act.
Track
PSNR ↑
20.70
21.27
16.89
20.62
15.12
24.51
16.25
22.71
SSIM ↑
0.767
0.802
0.621
0.738
0.482
0.843
0.668
0.822
LPIPS ↓
0.126
0.111
0.372
0.225
0.438
0.106
0.452
0.182
FID ↓
105
95
176
130
248
156
222
92
FVD ↓
148
105
246
140
129
38
207
162
Table 1: Video generation quality of action-conditioned and track-conditioned world models. Results are reported for real-world and simulation scenes under agent-view and wrist-view cameras. Act. denotes action-conditioned world model, and Track denotes track-conditioned world model. ↑ indicates higher is better and ↓ indicates lower is better. Best results for each scene are in bold.
Figure 3: Qualitative video comparison. TWM produces more spatially accurate and motion-coherent frames than AWM across both agent-view and wrist-view cameras.
Method
LIBERO-10
Goal
Object
Spatial
Avg.
π0.5
94.0
96.0
98.0
99.0
96.8
VLAT
94.0
98.0
100.0
100.0
98.0
VLAT+AWM
94.0
98.0
100.0
100.0
98.0
TrAct
94.0
100.0
99.0
100.0
98.3
Table 2: Success rates (%) on four standard LIBERO benchmark suites. Each suite contains 100 evaluation episodes. The best result in each column is shown in bold.
Task
Bg.
π0.5
π0.5 +VLAC
VLAT
VLAT+AWM
TrAct
Put the green block in the bowl
R
0.5
0.7
0.6
0.7
0.7
D
0.4
0.5
0.4
0.5
0.7
Put the half pink noodle in the bowl
R
0.6
0.9
0.8
1.0
1.0
D
0.6
0.8
0.6
0.8
0.8
Put the hotdog bun in the bowl
R
0.6
0.8
0.6
0.8
0.8
D
0.4
0.6
0.5
0.6
0.6
Table 4: Success rates on five unseen physical robot manipulation tasks, each evaluated under a regular training background ( R ) and a visually different non-training background ( D ). We compare five models. The best result for each row is shown in bold.
Figure 4: Rollout Demonstrations of two tasks: put the half-length pink noodle in the bowl (left) and close the right cabinet door (right). Each step is overlaid with track predictions.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Component
VLAT
World Model
VLAC
Total
Latency (ms)
52
1810
569
2431
Appendix
Table 5: Inference latency breakdown per planning step, measured with K=16 candidates, rolled out in parallel, on NVIDIA L40S GPUs. Each planning step yields a 16-step action chunk.
Figure 5: Additional Qualitative video comparison. TWM continues to produce more spatially accurate and motion-coherent frames than AWM across both agent-view and wrist-view cameras.
Figure 6: Overview of the physical experiments. We use a base and wrist-view camera and finetune the model on four training tasks. We then test five out-of-distribution tasks, as well as an unseen background for each task.
Task
Var.
π0.5
VLAT
VLAT+AWM
TrAct
Pick the alphabet soup and place it in the basket
Swap
0.4
0.5
0.7
0.7
Put both the cream cheese box and the butter in the basket
Swap
0.4
0.5
0.6
0.6
Put both moka pots on the stove
Object
0.5
0.5
0.6
0.7
Put the yellow and white mug in the microwave and close it
Object
0.2
0.3
0.4
0.5
Pick the akita black bowl next to the ramekin and place it on the plate
Task
0.1
0.3
0.5
0.5
Put the plate on the stove
Task
0.2
0.4
0.5
0.5
Appendix
Table 6: Results on the LIBERO-INTEGRAL benchmark. The benchmark includes robustness variations from LIBERO-PRO and LIBERO-Plus, and the last ten tasks evaluate cross-embodiment generalization by replacing the Franka Panda with a UR5 robot. Scores denote task success rates. The best result for each row is shown in bold.
Table 12
Task
π0.5
TrAct
TrAct+
Put both the alphabet soup and the cream cheese box in the basket (UR5)
0.1
0.3
0.3
Pick the ketchup and place it in the basket (UR5)
0.0
0.1
0.2
Put the bowl on the stove (UR5)
0.2
0.5
0.6
Pick the orange juice and place it in the basket (UR5)
0.0
0.5
0.5
Pick the tomato sauce and place it in the basket (UR5)
0.0
0.2
0.3
Average
0.06
0.32
0.38
Appendix
Table 9: Additional results on the five most difficult UR5 cross-embodiment tasks from LIBERO-INTEGRAL. TrAct+ denotes TrAct pretrained with the larger DROID+BridgeData V2+EgoDex mixture. Scores denote task success rates. The best result in each row is shown in bold.
Figure 7: Rollout examples Rollout with four key frames showing 16-horizon track predictions under the training background (left) and unseen background (right). Task list from top row to bottom row: put the half-length pink noodle in the bowl (1), put the hotdog bun in the bowl (2), put the green brick in the bowl (3), pour the angled scooper’s pasta into the pan (4), and close the partially open right cabinet door (5).
Figure 8: Rollout examples Rollout with four key frames. Task list from top row to bottom row, left to right column: pick the alphabet soup and place it in the basket (1), put both the cream cheese box and the butter in the basket (2), put both moka pots on the stove (3), put the yellow and white mug in the microwave and close it (4), pick the akita black bowl next to the ramekin and place it on the plate (5), put the plate on the stove (6), put the white mug on the left plate and put the yellow and white mug on the right plate (7), put the wine bottle on the top of the drawer (8), put both the cream cheese box and the butter in the basket (9), and pick up the book and place it in the back compartment of the caddy (10).
Figure 9: UR5 Rollout examples Rollout with four key frames. Task list from top row to bottom row, left to right column: put both the alphabet soup and the cream cheese box in the basket (1), put both the alphabet soup and the tomato sauce in the basket (2), put the bowl on the stove (3), pick the orange juice and place it in the basket (4), pick the akita black bowl between the plate and the ramekin and place it on the plate (5), pick the akita black bowl next to the plate and place it on the plate (6), pick the ketchup and place it in the basket (7), pick the tomato sauce and place it in the basket (8), pick the akita black bowl on the cookies box and place it on the plate (9), and put the bowl on the plate (10).