PDRVLA: A Vision-Language-Action Model with Physical Dynamics Representations

Hanqing Wang; Zhaotian Cai; Ruichuan An; Bohan Zeng; Tao Zhang; Yilin Chen; Xu Jiang; Kaiquan Chen

cs.ROOct 27, 2026

PDRVLA: A Vision-Language-Action Model with Physical Dynamics Representations

Authors: Hanqing Wang, Zhaotian Cai, Ruichuan An, Bohan Zeng, Tao Zhang, Yilin Chen, Xu Jiang, Kaiquan Chen,

Organizations: Peking University

Abstract

Vision--Language--Action (VLA) models generate robot actions from visual observations and language instructions, but their understanding of physical dynamics remains limited. Video world models can capture scene dynamics, yet generating future videos introduces high latency into the control loop. We introduce PDRVLA, whose lightweight Physical Dynamics Module extracts physical tokens from vision--language features to condition action generation without online video generation. The module learns through direct prediction of future RGB, depth, and optical-flow features, with auxiliary supervision through a frozen pretrained video world model. Training relies entirely on publicly available data, expanded through human-to-robot synthesis, subtask annotation, and embodied chain-of-thought (ECoT) annotation. Across four simulation benchmarks, PDRVLA achieves success rates of 99.3% on LIBERO, 89.9% on LIBERO-Plus, 93.6% on RoboTwin~2.0, and 81.5% on SimplerEnv. Ablations show improved future prediction with world-model supervision and evaluate how subtask conditioning, future object-pose supervision, and ECoT training affect action success.

Figures & tables

Figure 1: PDRVLA overview. (a) Simplified architecture. (b) Success rates on four simulation benchmarks.

Category	Statistics
VLA Data
Real-Robot	1.26M trajectories / 14,893 hours
UMI	1.22M trajectories / 10,259 hours
Simulation	1.87M trajectories / 8,868 hours
VL Data
General VL	8.05M VL examples

Table 1: Core training corpus by category.

Augmentation	Statistics
Egocentric Conversion	0.90M robot trajectories / 2,646 hours
Subtask Annotation	1.79B annotated frames / 1.1M VL examples
ECoT	2.56M annotated keyframes

Table 2: Statistics of synthesized and annotated training data.

Figure 2: Six types of subtask VL examples, differing in query timing, answer content, and subtask-history context. No observations after the query frame are provided.

Source Group	Labels	Examples	Source Group	Labels	Examples
EgoDex	VLM	125,000	RoboCasa-pretrain	VLM	40,000
InternVLA-sim	VLM	103,415	UniFoLM	VLM	35,789
ABC-130K	VLM	93,621	Table30	VLM	34,884
RoboCOIN	Native	91,642	RW-RL	VLM	26,397
OpenNeoData	VLM	83,917	Humanoid Everyday	VLM	24,866
GALAXEA	Native	75,022	community-v3	VLM	20,838

Table 3: Subtask VL examples by source group. Native denotes source-provided subtask labels.

Dataset	Data Type	Annotated Keyframes
AgiBot World ( Bu et al., 2025 )	Real	530,301
ABC-130K ( Allshire et al., 2026 )	Real	530,289
RoboCOIN ( Wu et al., 2025b )	Real	691,908
RoboMIND ( Wu et al., 2025a )	Real	153,508
HiFi-UMI-2K ( Simple AI et al., 2026 )	UMI	575,197
FastUMI-100K ( Liu et al., 2025 )	UMI	83,648

Table 4: ECoT keyframes annotated by our pipeline.

Figure 3: PDRVLA architecture. (a) Physical tokens complement final-layer VLM features as inputs to the action expert. Starting from Gaussian noise, Euler updates integrate predicted velocities into an action chunk. (b) Learnable queries attend to shared VLM features at every physical-module block before projection to physical tokens. Camera views and token sequences are schematic.

Figure 4: Training stages. (a) Physical pretraining combines direct future-feature prediction with supervision through a frozen video model. (b) Joint policy training removes the video model, retains direct prediction, and adds action and VL objectives. Grid stacks represent feature tensors; sg marks the detached intermediate-feature path. Camera views and video frames are schematic.

Method	LIBERO	LIBERO-Plus	RoboTwin 2.0	SimplerEnv
$\pi_{0}$ Black et al. (2024)	94.4	53.6	62.2	27.1
$\pi_{0.5}$ Physical Intelligence (2025)	96.9	84.4	79.8	57.1
OpenVLA-OFT Kim et al. (2025)	97.1	69.6	–	–
GR00T-N1.7 Bjorck et al. (2025) ; NVIDIA (2026)	97.0	–	–	59.3
Cosmos-Policy Kim et al. (2026)	98.5	82.2	–	–
LingBot-VLA 1.0 Wu et al. (2026a)	–	–	87.6	–

Table 5: Overview of simulation results (success rate, %). – denotes unavailable results.

Method	Spatial	Object	Goal	Long	Avg.
$\pi_{0}$ Black et al. (2024)	98.0	96.8	94.4	88.4	94.4
$\pi_{0.5}$ Physical Intelligence (2025)	98.8	98.2	98.0	92.4	96.9
OpenVLA-OFT Kim et al. (2025)	97.6	98.4	97.9	94.5	97.1
GR00T-N1.7 Bjorck et al. (2025)	97.7	98.5	97.5	94.4	97.0
Cosmos-Policy Kim et al. (2026)	98.1	100.0	98.2	97.6	98.5
LingBot-VA Li et al. (2026)	98.5	99.6	97.2	98.5	98.5

Table 6: LIBERO success rates (%).

Method	Sensor noise	Camera viewpoints	Robot initial states	Light conditions	Language instructions	Background textures	Object layout	Overall
$\pi_{0}$ Black et al. (2024)	79.0	13.8	6.0	85.0	58.8	81.4	68.9	53.6
$\pi_{0.5}$ Physical Intelligence (2025)	89.0	78.4	73.6	96.2	80.8	94.1	84.5	84.4
OpenVLA-OFT Kim et al. (2025)	75.8	56.4	31.9	88.7	79.5	93.3	74.2	69.6
Cosmos-Policy Kim et al. (2026)	92.7	75.8	63.3	96.5	81.7	88.9	82.2	82.2
Qwen-RobotManip Yuan et al. (2026)	97.7	87.2	75.5	96.6	85.6	97.7	87.3	89.0
InternVLA-A1.5 InternRobotics (2026)	95.6	83.1	55.1	96.4	86.9	98.2	85.2	84.8

Table 7: LIBERO-Plus success rates by perturbation category (%).

Method	Clean	Rand.	Avg.
$\pi_{0}$ Black et al. (2024)	65.9	58.4	62.2
$\pi_{0.5}$ Physical Intelligence (2025)	82.7	76.8	79.8
LingBot-VLA 1.0 Wu et al. (2026a)	88.6	86.7	87.6
LingBot-VLA 2.0 Wu et al. (2026b)	93.5	92.8	93.2
LingBot-VA Li et al. (2026)	92.9	91.6	92.3
Qwen-VLA-Instruct Wang et al. (2026a)	86.1	87.2	86.7

Table 8: RoboTwin 2.0 success rates (%) in clean and randomized settings and their average.

Method	Spoon $\to$ Towel	Carrot $\to$ Plate	Stack Blocks	Eggplant $\to$ Basket	Avg.
$\pi_{0}$ Black et al. (2024)	29.1	0.0	16.6	62.5	27.1
$\pi_{0.5}$ Physical Intelligence (2025)	49.3	64.7	44.7	69.7	57.1
GR00T-N1.7 NVIDIA (2026)	78.0	58.0	48.0	53.0	59.3
Xiaomi-Robotics-0 Cai et al. (2026b)	95.8	62.5	75.0	83.3	79.2
Qwen-VLA-Instruct Wang et al. (2026a)	–	–	–	–	73.7
InternVLA-A1.5 InternRobotics (2026)	92.4	85.4	69.5	75.7	80.8

Table 9: SimplerEnv (WidowX / Bridge, visual matching) success rates (%).

Figure 5: Normalized training MSE with direct prediction alone or additional world-model supervision. Each curve shows sample-weighted means over the preceding 1,000 updates; insets show updates 1,000–6,000 in greater detail.

Target	Direct only	Direct + WM	Reduction (%)	95% interval (%)
RGB	0.2558	0.2454	4.06	[3.04, 4.99]
Depth	0.3154	0.3087	2.12	[0.93, 3.31]
Flow	1.3606	1.3522	0.62	[ $-0.07$ , 1.28]
Mean	0.6439	0.6354	1.32	[0.71, 1.89]

Table 10: Final normalized prediction MSE on held-out RoboCOIN clips. Mean denotes the equally weighted average across the three modalities. Relative reductions are measured against Direct only; intervals are 95% paired episode-bootstrap intervals.

Variant	Sensor noise	Camera viewpoints	Robot initial states	Light conditions	Language instructions	Background textures	Object layout	Overall
SFT baseline	95.57	88.18	75.16	99.12	90.70	99.44	86.49	89.93
+ Subtask conditioning	96.50	89.18	76.32	98.51	89.92	99.07	86.75	90.23

Table 11: Self-generated subtask conditioning on LIBERO-Plus by perturbation category (success rate, %).

		Subtask refresh interval (chunks)
Task	Baseline	1	8	32
CuttingToolSelection	46	60	50	33
RinseSinkBasin	39	49	44	37
PrepareCoffee	7	13	7	7
PanTransfer	22	25	21	15
LoadDishwasher	48	55	58	33

Table 12: RoboCasa365 success rates (%) with different subtask refresh intervals.

	LIBERO		LIBERO-Plus
Setting	Long	Average	Long	Overall
No ECoT training	92.8	97.1	65.18	79.59
ECoT-trained, no generation	96.0	98.2	69.87	79.25
ECoT-trained, with generation	95.6	98.3	68.72	79.12

Table 13: Effects of ECoT training and inference-time generation on success rates (%).

Variant	Sensor noise	Camera viewpoints	Robot initial states	Light conditions	Language instructions	Background textures	Object layout	Overall
SFT baseline	95.57	88.18	75.16	99.12	90.70	99.44	86.49	89.93
+ Object-pose supervision	96.56	89.56	74.52	98.25	90.50	98.88	87.15	90.12

Table 14: Auxiliary object-pose supervision on LIBERO-Plus by perturbation category (success rate, %).

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Design choice	Mean Prediction Skill $\uparrow$
VLM input layer
Block 8	0.3617
Block 16	0.3413
Block 24	0.3314
Final block	0.2432
Cross-attention placement within the physical module

Appendix

Table 15: Physical-module design comparisons using frozen-representation probes.

Source task suite	SFT baseline	+ Subtask conditioning	+ Object-pose supervision
Spatial	96.50	96.59	95.71
Object	92.85	93.21	93.05
Goal	83.94	84.06	85.18
Long	86.90	87.53	86.94
Overall	89.93	90.23	90.12

Appendix

Table 16: Subtask conditioning and object-pose supervision on LIBERO-Plus by source task suite (success rate, %).

Suite	Median $z$ (m)	$z$ p01–p99 (m)	Above upper bound (%)
Spatial	1.041	0.915–1.285	100.0
Object	0.225	0.010–0.336	0.0
Goal	1.076	0.911–1.290	100.0
Long	0.712	0.447–1.267	49.1

Appendix

Table 17: Sampled absolute EEF-height statistics in LIBERO. The pretraining upper bound is 0.772 m.

Vision-Language-Action Models

Vision-language-action models turn visual observations and natural-language instructions into actions for robots and other embodied agents.

See all 1,194 papers

VLA- $Δ$ : Learning Physical State Transition for Vision-Language-Action Models
Oct 15, 2026Qi Lu, Jiarun Li, Ce Zhang +5VLA Model Training Vision-Language-Action Models
Zhejiang University (zju.edu.cn) · Central South University (csu.edu.cn)
ViDAL: A Visual Dynamics-Grounded Action Latent Space for Vision-Language-Action Models
Oct 6, 2026Yuan Xu, Yixiang Chen, Qisen Ma +10Latent Action Learning Visuomotor Policy Learning
CASIA · UCAS · Alibaba Group +1
Pri4R: Learning World Dynamics for Vision-Language-Action Models with Privileged 4D Representation
Mar 2, 2026Jisoo Kim, Jungbin Cho, Sanghyeok Chu +9World Models for VLA Models Vision-Language-Action Models
KAIST AI · LG AI Research · Yonsei University +2

Oct 15, 2026J/K move · Enter open · S save

VLA- $Δ$ : Learning Physical State Transition for Vision-Language-Action Models

Qi Lu, Jiarun Li, Ce Zhang, Miao Pan+4

Zhejiang University (zju.edu.cn) · Central South University (csu.edu.cn)

VLA Model Training Vision-Language-Action Models VLMs for Robotics Robot Dynamics Learning

Open paper AskPDF arXiv

Vision-language-action (VLA) models transfer semantic knowledge acquired through vision-language pretraining to robot control, achieving strong performance across various benchmarks. However, most VLAs directly map observations to demonstrated actions without explicit supervision of intermediate physical state changes, making them prone to overfitting observation-action correspondences. Some methods introduce intermediate reasoning or future-state prediction in language, visual, or latent spaces to guide action generation, but do not explicitly supervise the physical state transition itself. We therefore propose VLA-

Δ

, a general framework that uses physical changes between current and future states as explicit learning targets. To support this objective, we construct a large-scale pretraining dataset and derive physical state-transition annotations covering task-relevant semantics and visual motion changes, providing more direct supervision for continuous action generation. During inference, the model generates actions directly from current observations without additional inference overhead. VLA-

Δ

achieves performance competitive with the state of the art on LIBERO and RoboTwin, and outperforms

π_{0.5}

in real-robot experiments. Our code is available at https://github.com/ZJU-OmniAI/VLA-Delta

Abstract

Vision-Language-Action Models

Explore similar work

VLA-ΔΔΔ: Learning Physical State Transition for Vision-Language-Action Models

ViDAL: A Visual Dynamics-Grounded Action Latent Space for Vision-Language-Action Models

Pri4R: Learning World Dynamics for Vision-Language-Action Models with Privileged 4D Representation

VLA- $Δ$ : Learning Physical State Transition for Vision-Language-Action Models