Pri4R: Learning World Dynamics for Vision-Language-Action Models with Privileged 4D Representation

Jisoo Kim; Jungbin Cho; Sanghyeok Chu; Ananya Bal; Jinhyung Kim; Gunhee Lee; Sihaeng Lee; Seung Hwan Kim

cs.CVMar 2, 2026

Pri4R: Learning World Dynamics for Vision-Language-Action Models with Privileged 4D Representation

Authors: Jisoo Kim, Jungbin Cho, Sanghyeok Chu, Ananya Bal, Jinhyung Kim, Gunhee Lee, Sihaeng Lee, Seung Hwan Kim, +4 more

Organizations: KAIST AI · LG AI Research · Yonsei University · Carnegie Mellon University · Seoul National University

Abstract

Humans learn not only how their bodies move, but also how the surrounding world responds to their actions. In contrast, while recent Vision-Language-Action (VLA) models exhibit impressive semantic understanding, they often fail to capture the spatiotemporal dynamics governing physical interaction. In this paper, we introduce Pri4R, a simple yet effective approach that endows VLA models with an implicit understanding of world dynamics by leveraging privileged 4D information during training. Specifically, Pri4R augments VLAs with a lightweight point track head that predicts 3D point tracks. By injecting VLA features into this head to jointly predict future 3D trajectories, the model learns to incorporate evolving scene geometry within its shared representation space, enabling more physically aware context for precise control. Due to its architectural simplicity, Pri4R is compatible with dominant VLA design patterns with minimal changes. During inference, we run the model using the original VLA architecture unchanged; Pri4R adds no extra inputs, outputs, or computational overhead. Across simulation and real-world evaluations, Pri4R significantly improves performance on challenging manipulation tasks, including a +10% gain on LIBERO-Long and a +40% gain on RoboCasa. We further show that 3D point track prediction is an effective supervision target for learning action-world dynamics, and validate our design choices through extensive ablations. Project page: https://jiiiisoo.github.io/Pri4R/

Figures & tables

Fig. 2 : Overview of Pri4R. We augment two common VLA architectures with an auxiliary point track head that predicts per-step 3D point displacements $\widehat{\Delta P}_{t:t+H}$ from backbone embeddings $\mathbf{z}_{t}$ and the current point set $P_{t}$ . (a) For backbone-centric VLAs (e.g., OpenVLA-OFT [ 29 ] ), we set $\mathbf{z}_{t}$ to the final layer action-query token embeddings. (b) For expert-style VLAs (e.g., $\pi$ [ 7 , 24 ] ), we condition an embedding module on the backbone’s final layer hidden states to produce $\mathbf{z}_{t}$ . (c) The point track head encodes $P_{t}$ with a PointMLP, then fuses the resulting point features with $\mathbf{z}_{t}$ via a FusionMLP to predict future point tracks. Privileged 3D point track supervision during training forces the VLA to model how scene geometry evolves, yielding more reliable interaction and higher task success, while leaving the test-time interface and compute completely unchanged.

	Average	LIBERO - Spatial	LIBERO - Object	LIBERO - Goal	LIBERO - Long
	SR ( $\uparrow$ )	SR ( $\uparrow$ )	SR ( $\uparrow$ )	SR ( $\uparrow$ )	SR ( $\uparrow$ )
Diffusion Policy [ 14 ] *	72.4	78.3	92.5	68.3	50.5
Octo [ 50 ] *	75.1	78.9	85.7	84.6	51.1
DiT Policy [ 20 ] *	82.4	84.2	96.3	85.4	63.8
OpenVLA [ 29 ] *	76.5	84.7	88.4	79.2	53.7
$\pi_{0}$ [ 7 ]	87.4 $\pm$ 0.2	87.8 $\pm$ 0.2	84.9 $\pm$ 0.4	91.2 $\pm$ 0.9	85.7 $\pm$ 0.7

TABLE I : Results on LIBERO. Success rates (SRs) are reported across four task suites (500 trials per suite). Entries marked with * are taken from the OpenVLA-OFT paper [ 28 ] . Best performances for each task are marked in bold . Pri4R improves average success rates of every state-of-the-art VLAs (

\pi

series and OpenVLA-OFT) and almost every task in LIBERO.

	Average	Pick and Place	Open/Close doors	Open/Close drawers	Twisting knobs	Turning levers	Pressing buttons	Insertion
	SR ( $\uparrow$ )	SR ( $\uparrow$ )	SR ( $\uparrow$ )	SR ( $\uparrow$ )	SR ( $\uparrow$ )	SR ( $\uparrow$ )	SR ( $\uparrow$ )	SR ( $\uparrow$ )
$\pi_{0}~$ [ 7 ]	38.8	24.0	45.0	78.0	29.0	57.3	57.3	0.0
$\pi_{0}~$ [ 7 ] + Pri4R	42.2 ( +3.4 )	24.0 (+0.0)	49.0 ( +4.0 )	84.0 ( +6.0 )	30.0 ( +1.0 )	71.3 ( +14.0 )	59.3 ( +2.0 )	2.0 ( +2.0 )
$\pi_{0.5}~$ [ 24 ]	52.9	54.3	51.0	75.0	28.0	79.3	60.0	4.0
$\pi_{0.5}$ [ 24 ] + Pri4R	57.0 ( +4.1 )	52.0 ( -2.3 )	68.5 ( +17.5 )	89.0 ( +14.0 )	33.0 ( +5.0 )	86.7 ( +7.4 )	54.7 ( -5.3 )	5.0 ( +1.0 )
OpenVLA-OFT [ 28 ]	33.1	21.8	45.7	59.0	8.0	36.0	56.0	27.0

TABLE II : Results on Robocasa. Success rate (SR) for each method across 24 task suites, grouped following [ 40 ] , and averaged over 50 trials. Pri4R improves average success rates of every state-of-the-art VLAs and almost every task in Robocasa.

Fig. 3 : Training dynamics. Pri4R learns slowly at the early stage due to the 3D point track objective, but improves performance rapidly, reaching the baseline peak $2.7\times$ faster.

What supervision signal?
Method	SR $\uparrow$	$\Delta$
OpenVLA-OFT [ 28 ]	33.1	–
+ Goal point set	33.8	+0.7
+ 2D point track	37.0	+3.9
+ Depth	42.3	+8.3
+ Ours (3D point track)	46.3	+13.2

TABLE III : Analysis of supervision. Average success rates on RoboCasa. 3D point track is the most effective supervision, while using both robot and scene points is crucial.

	Need $P_{t}$ at test?	Avg.	PnP	Doors	Drawers	Knobs	Lever	Button	Insertion
		SR / $\Delta$	SR / $\Delta$	SR / $\Delta$	SR / $\Delta$	SR / $\Delta$	SR / $\Delta$	SR / $\Delta$	SR / $\Delta$
OpenVLA-OFT [ 28 ]	No	33.1 / +0.0	21.8 / +0.0	45.7 / +0.0	59.0 / +0.0	8.0 / +0.0	36.0 / +0.0	56.0 / +0.0	27.0 / +0.0
+ $P_{t}$ input	Yes	33.3 / +0.2	14.0 / -7.8	48.0 / +2.3	59.0 / +0.0	10.0 / +2.0	49.3 / +13.3	59.3 / +3.3	26.0 / -1.0
+ $P_{t}$ input + $P_{t:t+H}$ track	Yes	34.5 / +1.4	14.3 / -7.5	49.0 / +3.3	59.0 / +0.0	10.0 / +2.0	54.0 / +18.0	60.7 / +4.7	29.0 / +2.0
+ $P_{t:t+H}$ track (Ours)	No	46.3 / +13.2	23.0 / +1.2	61.7 / +16.0	80.0 / +21.0	25.0 / +17.0	66.7 / +30.7	79.3 / +23.3	34.0 / +7.0

TABLE IV : Ablation on

P_{t}

as input.

\Delta

denotes absolute improvement over the OpenVLA-OFT baseline [ 28 ] . “

P_{t}

input” indicates that the point set

P_{t}

is appended to the backbone as additional tokens alongside image and language inputs. “

P_{t:t+H}

track” refers to our proposed displacement supervision over the action horizon.

Method	SR $\uparrow$	$\Delta$
$\pi_{0.5}$ [ 24 ]	52.9	–
Design choice of embedding module
+ Point expert	53.4	+0.5
+ Backbone query token (attend action)	54.4	+1.5
+ Backbone query token	54.8	+1.9
+ Ours	57.0	+4.1

TABLE V : Ablations on embedding module with

\pi_{0.5}

. Average success rate (SR) on RoboCasa. Each design choice of our embedding module is important.

Fig. 4 : Predicted point tracks in simulation and the real world. Future trajectories are visualized in a rainbow color map. As highlighted in the red boxes, Pri4R accurately predicts point tracks for both scene elements and the robot.

Point Encoder	Fusion Module	SR $\uparrow$	$\Delta$
OpenVLA-OFT (no point-track head)		89.2	–
PointNet [ 12 ]	Ours	80.8	$-8.4$
PtTrans. [ 60 ]	Ours	92.4	$+3.2$
Ours	Transformer [ 53 ]	92.2	$+3.0$
Ours	Ours	94.4	$\mathbf{+5.2}$

TABLE VI : Point-track head ablations. SR of OpenVLA-OFT on LIBERO Long. Our point-track head has two slots: Point MLP and Fusion MLP . We plug in different architectures for each slot and report SR;

\Delta

is w.r.t. the no-head baseline.

Panel of Figure 10 — Fig. 5 : Real-world setup. Blue boxes indicate the target. In Pick the farthest object, the red box marks a closer distractor object; in Pick up the doll and place in the white bin, the red box marks a distractor.

Panel of Figure 11 — Fig. 6 : Qualitative comparison across tasks. Baseline (left) failures vs. Our method (right) successes.

Model	Height	Spatial		Depth		Tracking
Model	Unseen	Seen	Unseen	Seen	Unseen	Seen	Unseen	OOD
OpenVLA-OFT	83.3	100.0	60.0	45.9	50.0/ 0.0	75.0	41.7	50.0
+ Pri4R	96.7	100.0	80.0	79.2	50.0 / 37.5	100.0	66.7	66.7
$\pi_{0.5}$	60.0	90.0	30.0	66.7	50.0/ 0.0	50.0	0.0	33.8
+ Pri4R	66.7	100.0	60.0	95.8	100.0 /0.0	66.7	0.0	50

TABLE VII : Quantitative results on the real-world setup.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Shared settings
	OpenVLA-OFT	OpenVLA-OFT + Pri4R
# GPUs	4 $\times$ NVIDIA H200
learning rate (LR)	$5\mathrm{e}{-4}$
total batch size	32 (8 per GPU)
input images	3 camera images: third-person, wrist, top-down
input image size	224 $\times$ 224

Appendix

TABLE S1 : Hyperparameter details for real-world training. Unless otherwise specified, OpenVLA-OFT and OpenVLA-OFT+Pri4R share the same settings.

Shared settings
	$\pi_{0.5}$	$\pi_{0.5}$ + Pri4R
# GPUs	4 $\times$ NVIDIA H200
optimizer	AdamW
input images	3 camera images: third-person, wrist, top-down
input image size	224 $\times$ 224
learning rate scheduler	Cosine decay

Appendix

TABLE S2 : Hyperparameter details for training

\pi

series on real-world tasks. Unless otherwise specified,

\pi

and

\pi

+Pri4R share the same settings.

Dataset	Method	Input images	# GPUs	Batch size	# steps
LIBERO	OpenVLA-OFT	third-view + wrist-view	8	total 64	90K
LIBERO	$\pi_{0}$ / $\pi_{0.5}$	third-view + wrist-view	4	total 128	30K
RoboCasa	OpenVLA-OFT	left-view + right-view + wrist-view	8	total 64	120K
RoboCasa	$\pi_{0}$ / $\pi_{0.5}$	left-view + right-view + wrist-view	4	total 128	30K

Appendix

TABLE S3 : Simulation training details. We report settings that differ from real-world training.

Fig. S2 : Depth reconstruction: (Top) ground-truth depth maps from the simulator, and (Bottom) depth maps reconstructed by our variational autoencoder (VAE).

Method	Total	PnP	Doors	Drawers	Knobs	Lever	Press	Insert
OpenVLA-OFT	0.331	0.218	0.457	0.590	0.080	0.360	0.560	0.270
OpenVLA-OFT + 2D tracks	0.370	0.210	0.520	0.620	0.140	0.473	0.600	0.290
OpenVLA-OFT + Goal point set	0.338	0.165	0.533	0.810	0.080	0.413	0.587	0.220
OpenVLA-OFT + Environment points	0.352	0.190	0.487	0.470	0.170	0.540	0.493	0.280
OpenVLA-OFT + Robot-only points	0.438	0.223	0.607	0.790	0.280	0.660	0.673	0.270
OpenVLA-OFT + Ours (Pri4R)	0.463	0.230	0.617	0.800	0.250	0.667	0.793	0.340

Appendix

TABLE S4 : Robocasa success rates by task. We report the average success rate ( Total ) and per-task success rates on RoboCasa for all experiments.

Method	SR $\uparrow$	$\Delta$
OpenVLA-OFT [ 28 ]	33.1	–
+ 3D point sequence w/o input pointcloud	28.7	-4.4
+ Ours (3D point track)	46.3	+13.2

Appendix

TABLE S5 : Analysis of Point Set

P_{t}

input. We present the average success rates on RoboCasa.

	Average	Spatial	Object	Goal	Long
OpenVLA-OFT	92.7	92.8	98.4	96.4	83.0
+SpatialForcing	94.2	96.8	99.0	96.2	84.8
+ Pri4R	95.0	93.2	96.6	96.8	93.2

Appendix

TABLE S6 : Comparison with SpatialForcing. We compare Pri4R with a recent SOTA method, SpatialForcing.

weight	Avg	PnP	Doors	Drawers	Turn	Twist	Press	Insert
0.1	54.7	54.3	62.5	90.0	28.0	73.3	55.3	3.0
1.0	57.0	52.0	68.5	89.0	33.0	86.7	54.7	5.0
10.0	50.7	47.0	54.5	77.0	35.0	82.0	47.3	5.0

Appendix

TABLE S7 : Ablations on tracking loss weight for

\pi_{0.5}

. We present the average success rate (SR) across 24 task suites on RoboCasa, averaged over 50 trials per suite.

number	Avg	PnP	Doors	Drawers	Turn	Twist	Press	Insert
256	48.8	49.3	55.5	77.0	28.0	73.3	41.3	1.0
512	53.9	48.8	60.0	87.0	27.0	81.3	59.3	7.0
1024	57.0	52.0	68.5	89.0	33.0	86.7	54.7	5.0

Appendix

TABLE S8 : Ablations on number of points for 3D track prediction for

\pi_{0.5}

. We present the average success rate (SR) across 24 task suites on RoboCasa, averaged over 50 trials per suite.

Fig. S3 : Visualization of point track data. Purple lines denote 3D point tracks. Points are sub-sampled for clarity.

Fig. S4 : Pri4R on real-world tasks. Additional qualitative results are provided in the supplementary videos.

Explore similar work

Aug 31, 2026

Temporal Forcing: 4D Representation Alignment for Vision-Language-Action Models

Long-horizon robotic manipulation requires vision-language-action (VLA) models to track scene states and their evolution beyond the current observation. However, simply conditioning policies on observation history does not guarantee that the history is effectively utilized: action supervision constrains what the policy should do, but only indirectly constrains what its history representations should retain. To resolve this, we present Temporal Forcing, a 4D representation alignment framework that explicitly supervises latent temporal states and their transitions. Specifically, we first introduce a history pathway that compresses past observations into compact latent tokens. We then align these tokens and current-frame features with geometric targets from a pretrained 4D foundation model, providing direct supervision at both the state and transition levels. The 4D foundation model and alignment heads are used only for training-time supervision. Temporal Forcing improves average success from 96.6% to 98.8% on LIBERO, with the largest gain on LIBERO-Long (93.8% to 97.2%), and from 53.5% to 62.8% across twelve RoboTwin 2.0 tasks. Furthermore, Temporal Forcing increases full-task success from 20.0% to 43.3% on a physical multi-stage hidden-placement task. Controlled experiments show that 4D representation alignment is crucial for making observation history beneficial to the model. Code will be publicly available.

Xingyu Ding, Yuzhong Zhao, Chunhai Zhao +3

Nanjing University, Nanjing, China · Institute of Automation, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China

Representation Learning Robotic Manipulation Vision-Language-Action Models Latent Space Alignment

Ask PDF

Aug 27, 2026

PHR-VLA: Planning Horizon Reasoning for Vision-Language-Action Models

Vision-language-action models (VLAs) have shown strong promise for general-purpose robotic manipulation by mapping language instructions and vision observations directly to actions. However, most VLAs primarily condition action prediction on current observations and lack an explicit mechanism for reasoning over future task dynamics, which is particularly important for fine-grained, contact-rich manipulation. We present PHR-VLA, a framework that enables planning-horizon reasoning in VLAs through privileged latent representations of future dynamics. PHR-VLA introduces a lightweight auxiliary future head that, during training, aligns the VLA's internal representations with latent dynamics extracted from future observations. Evaluation results demonstrate that local, contact-centric, patch-level latent dynamics supervision from the wrist camera improves success rate on LIBERO from 84.1% to 88.4% and on real-world disassembly tasks from 63.3% to 82.5%. Patch-level supervision from a third-person camera also improves performance on Meta-World from 56.70% to 57.8%. These results demonstrate that privileged latent dynamics alignment provides an effective training signal for improving anticipatory reasoning in VLA policies. Project website: https://davoodsz.github.io/PHR-VLA.github.io/

Davood Soleymanzadeh, Kaidi Zhang, Zhiyuan Zhang +4

J. Mike Walker ’66 Department of Mechanical Engineering, Texas A&M University, College Station, TX 77843, USA · Department of Industrial Engineering, Purdue University, West Lafayette, IN 47907, USA · Zachry Department of Civil and Environmental Engineering, Texas A&M University, College Station, TX 77843 USA

Vision-Language-Action Models Reasoning for VLA Models Robotic Manipulation Language-Conditioned Robot Manipulation

Ask PDF

Jan 29, 2026

DynamicVLA: A Vision-Language-Action Model for Dynamic Object Manipulation

Manipulating dynamic objects remains an open challenge for Vision-Language-Action (VLA) models. Although recent VLAs generalize well in static manipulation, dynamic scenes introduce a latency-induced perception-execution mismatch: object states continue to evolve during inference, making actions predicted from past observations stale at execution time. We present DynamicVLA, a latency-aware VLA model for dynamic object manipulation. It combines a compact 0.4B architecture and convolutional vision encoder for efficient multimodal inference with a continuous inference schedule that overlaps reasoning and execution for non-blocking control. Latent-aware Action Streaming then discards latency-invalid action prefixes and executes only the temporally valid suffix of each predicted chunk, preserving action-time alignment under dynamic object motion. To fill the missing foundation of dynamic manipulation data, we introduce the Dynamic Object Manipulation (DOM) benchmark, built with an automated collection pipeline that gathers 200K synthetic episodes across 2.8K scenes and 206 objects, and enables fast collection of 2K real-world episodes without teleoperation. Extensive evaluations in simulation and on real robots show that DynamicVLA improves dynamic manipulation success under changing object motion, perception-heavy instructions, and unseen motion patterns.

Haozhe Xie, Beichen Wen, Jiarui Zheng +4

Inference-Time Optimization Efficient VLM Inference Efficient VLA Model Inference Vision-Language-Action Models Robot Manipulation Benchmarks

Ask PDF