VLA-$Δ$: Learning Physical State Transition for Vision-Language-Action Models

Qi Lu; Jiarun Li; Ce Zhang; Miao Pan; Yi Pan; Xu Wang; Wenqi Zhang; Xuhong Zhang

cs.CVOct 15, 2026

VLA- $Δ$ : Learning Physical State Transition for Vision-Language-Action Models

Authors: Qi Lu, Jiarun Li, Ce Zhang, Miao Pan, Yi Pan, Xu Wang, Wenqi Zhang, Xuhong Zhang

Organizations: Zhejiang University (zju.edu.cn) · Central South University (csu.edu.cn)

Abstract

Vision-language-action (VLA) models transfer semantic knowledge acquired through vision-language pretraining to robot control, achieving strong performance across various benchmarks. However, most VLAs directly map observations to demonstrated actions without explicit supervision of intermediate physical state changes, making them prone to overfitting observation-action correspondences. Some methods introduce intermediate reasoning or future-state prediction in language, visual, or latent spaces to guide action generation, but do not explicitly supervise the physical state transition itself. We therefore propose VLA- $Δ$ , a general framework that uses physical changes between current and future states as explicit learning targets. To support this objective, we construct a large-scale pretraining dataset and derive physical state-transition annotations covering task-relevant semantics and visual motion changes, providing more direct supervision for continuous action generation. During inference, the model generates actions directly from current observations without additional inference overhead. VLA- $Δ$ achieves performance competitive with the state of the art on LIBERO and RoboTwin, and outperforms $π_{0.5}$ in real-robot experiments. Our code is available at https://github.com/ZJU-OmniAI/VLA-Delta

Figures & tables

Figure 1: Comparison of transition modeling paradigms in vision–language–action policies. (a) Direct VLA maps observations and instructions directly to actions. (b) Reasoning VLA describes task plans and subgoals in language space. (c) WAM predicts future visual states, while (d) Latent-WAM predicts future latent representations. (e) VLA- $\Delta$ models physical state changes through multilevel supervision.

Figure 2: Overall architecture of VLA- $\Delta$ . Given multi-view observations, a task instruction, and the robot state, the model jointly learns a physical state transition representation from Structured Transition Slots, Optical Flow, and Future-State Difference Text supervision, and uses these representations to condition continuous action generation. The right panel illustrates the attention-visibility constraints among the different information streams.

Figure 3: Physical state-transition annotations and the automated data annotation pipeline.

Methods	VLM Size	Spatial	Object	Goal	Long	Average
TraceVLA	7B	84.6	85.2	75.1	54.1	74.8
openVLA	7B	84.7	88.4	79.2	53.7	76.5
SpatialVLA	4B	88.2	89.9	78.6	55.5	78.1
WorldVLA	7B	87.6	96.2	83.4	60.0	81.8
CoT-VLA	7B	87.5	91.6	87.6	69.0	83.9
ThinkAct	7B	88.3	91.4	87.1	70.9	84.4

Table 1: Performance comparison on LIBERO (Success Rates,%).

Method	AdjustBottle		BeatBlockHammer		BlocksRankingSize		ClickAlarmclock		HangingMug		…	AVG
Method	Clean	Random	Clean	Random	Clean	Random	Clean	Random	Clean	Random	…	Clean	Random
X-VLA	100	99	92	88	67	74	99	99	23	27	…	72.88	72.84
$\pi_{0}$	99	95	79	84	14	5	77	68	14	11	…	65.92	58.4
$\pi_{0.5}$	100	99	96	93	49	26	98	89	18	17	…	82.74	76.76
Motus	89	93	95	88	75	63	100	100	38	38	…	88.52	87.02
ZR-0	100	99	85	88	70	81	96	82	35	33	…	88.70	87.98

Table 2: Performance comparison on RoboTwin 2.0 (Success Rates, %) under Clean and Random settings. Full results are provided in Appendix C.2 .

Figure 5: Real-robot evaluation on the AgileX PiPER 6-DoF platform. (a) Dual-arm robot platform. (b) Real-robot experimental results (Success Rates, %). (c) Overview of the five tasks.

Variant	Query	Pretraining	Difference Text	Slots	Optical Flow	LIBERO Avg. (%)
Dense VLM Features	No	No	No	No	No	95.70
Dense VLM Features	No	Yes	No	No	No	95.85
Difference Queries Only	Yes	No	No	No	No	96.25
Difference Queries Only	Yes	Yes	No	No	No	96.45
Query + Difference Text	Yes	Yes	Yes	No	No	97.20
Query + Structured Slots	Yes	Yes	No	Yes	No	96.90

Table 3: Ablation results on 40 LIBERO tasks.

Figure 6: t-SNE visualization of Slot Query hidden states.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Frames	Hours	No. of Tasks	Introduction
MolmoAct (Household + Tabletop)	1,104,942	30.693	96	Everyday household and tabletop manipulation with a Franka robot, covering organizing, cleaning, opening and closing, grasping, and placement tasks; includes a fixed external view, a second external view, and a wrist-camera view.
DROID	27,630,375	511.674	49,630	Open-ended everyday manipulation with Franka robots across different data collectors, buildings, and real-world scenes; includes two external views and a wrist-camera view.
RH20T	3,898,056	108.279	141	Multi-robot real-world manipulation. The Stage05 release subset covers Flexiv, Franka, KUKA, and UR5 across seven configurations, 19 users, and 21 scene IDs; includes two external views, a wrist-camera view, and force/torque fields.

Appendix

Table 4: Overview of the datasets used to construct the pretraining corpus.

Panel of Figure 8 — Figure 8: Physical state-transition annotation examples: a current–future observation pair with Future-State Difference Text and Structured Transition Slots (top), and slot annotations for loading a plate into a metal rack (middle) and hanging a mug (bottom).

Evaluation target	Reference criteria	Valid count	Percentage
Text: Task Temporal	Human-verified task phases, task progress, and subgoal changes	290	96.67%
Text: Spatial Motion	Human-verified object and gripper positions, relative relations, and their changes	289	96.33%
Text: Contact Interaction	Gripper opening and closing, contact establishment, maintenance, and release confirmed in video	293	97.67%
Text: Object Constraints	Manipulation constraints specified by task instructions or supported by visual evidence	298	99.33%
Slot 1: Task progress	Human reference values in $[0,1]$ based on a consistent definition of progress	292	97.33%
Slot 2: Task phase	Human labels based on predefined phase categories	293	97.67%

Appendix

Table 5: Counts and proportions from the audit of physical state-transition annotations.

Task	$\bm{\pi_{0}}$		$\bm{\pi_{0.5}}$		X-VLA		Motus		ZR-0		VLA- $\Delta$
Task	Cle	Ran.	Cle	Ran.	Cle	Ran.	Cle	Ran.	Cle	Ran.	Cle	Ran.
AdjustBottle	99	95	100	99	100	99	89	93	100	99	100	98
BeatBlockHammer	79	84	96	93	92	88	95	88	85	88	90	86
BlocksRankingRGB	80	63	92	85	83	83	99	97	92	91	95	94
BlocksRankingSize	14	5	49	26	67	74	75	63	70	81	80	74
ClickAlarmclock	77	68	98	89	99	99	100	100	96	82	92	94

Appendix

Table 6: Per-task success rates (%) on RoboTwin 2.0 under the Clean and Random settings.

Capability Dimension	Task Instruction
Obstacle Avoidance and Physical Constraint Compliance	Place the carrots on a rack, making sure they do not touch the cups.
Long-Horizon Execution	Put all the items on the tabletop onto the tray.
Fine-Grained Manipulation	Put the spoon into the cup.
Spatial Reasoning	Place the item closest to the bowl onto the plate.
Generalization under Changed Tablecloth	Place the item closest to the bowl onto the plate.

Appendix

Table 7: Capability dimensions and task instructions for real-robot evaluation.

Supervision Target	Query Group	Output Content	Output Shape	Output Constraints and Supervision
$S_{1}$ : Task progress	G12	Task progress at the current and next keyframes	$2$	Sigmoid; Smooth L1 and a progress monotonicity penalty
$S_{2}$ : Task phase	G12	Seven manipulation phases at two time steps	$2\times 7$	Classification logits; class-weighted cross-entropy with label smoothing
$S_{3}$ : Target position	G3	Target-object boxes at two time steps	$2\times 4$	Valid box parameterization; Smooth L1 and GIoU
$S_{4}$ : Gripper position	G4	Gripper boxes at two time steps	$2\times 4$	Valid box parameterization; Smooth L1 and GIoU
$S_{5}$ : Motion and relative displacement	G5	Target, gripper, and relative 3D displacements	$3\times 3$	Standardized regression and displacement consistency in meters
$S_{6}$ : Interactable position	G6	Two affordance contact points in the current frame	$2\times 2$	Sigmoid; pointwise Smooth L1

Appendix

Table 8: Structured Transition Slot outputs, Query groups, and supervision. Output shapes exclude the batch dimension.

Target	Classes in ID Order and Meanings
$S_{2}$ : Task phase	0: align (alignment); 1: approach (approaching); 2: contact (contact); 3: manipulate (object manipulation); 4: release (release); 5: retract (retraction); 6: transport (transport)
$S_{7}$ : Gripper state transitions	0: closed_to_open (closed to open); 1: maintain_closed (remaining closed); 2: maintain_open (remaining open); 3: open_to_closed (open to closed)
$S_{8}$ : Contact transitions	0: break_contact (breaking contact); 1: establish_contact (establishing contact); 2: maintain_contact (maintaining contact); 3: remain_separated (remaining separated)

Appendix

Table 9: Fixed class vocabularies for task phase, gripper state transitions, and contact transitions. Class IDs start at zero.

VLA Model Training

See all 39 papers

Vision-Language-Delta-Action: Learning Compact Deltas as Actionable Transition Interfaces for Robotic Manipulation
Oct 19, 2026Lingsen Zhang, Wei Li, Jie He +5Robotic Manipulation Vision-Language-Action Models
Harbin Institute of Technology, Shenzhen · Baidu, Inc. · Shenzhen Loop Area Institute
In-Context VLA: Endowing Vision-Language-Action Models with Language via In-Context Post-Training and Agentic Tool Use
Aug 6, 2026Jiarui Yang, Wen Huang, Jiale Zhang +2Reasoning for VLA Models ICL for Robotics
Nankai University · Tsinghua University · Swiss Federal Institute of Technology in Lausanne (EPFL)
QuoVLA: Quotient Space for Vision-Language-Action Models
May 24, 2026Xuan Wang, Yinan Wu, Haoran Duan +1Quantization Representation Learning
Department of Automation Tsinghua University

Oct 19, 2026J/K move · Enter open · S save

Vision-Language-Delta-Action: Learning Compact Deltas as Actionable Transition Interfaces for Robotic Manipulation

Lingsen Zhang, Wei Li, Jie He, Henghua Zhang+4

Harbin Institute of Technology, Shenzhen · Baidu, Inc. · Shenzhen Loop Area Institute

Robotic Manipulation Vision-Language-Action Models World Models for VLA Models Latent Dynamics Modeling

Open paper AskPDF arXiv

Robotic manipulation is not merely observation-to-action generation, but the realization of task-specified state changes in the physical world. Existing vision-language-action (VLA) methods directly learn observation-to-action mappings, imposing a substantial learning burden due to the gap between semantic understanding and action execution. Although world-action models (WAMs) improve control through future prediction, modeling complete future states wastes capacity on static appearance and action-irrelevant changes. We propose Vision-Language-Delta-Action (VLDA), a paradigm that reformulates full future prediction as compact actionable transition modeling. Its core is a compact Delta variable that captures latent world-state changes, serving as an interface between vision-language understanding and action generation. VLDA learns Delta through a unified objective composed of the Local Transition Filter (LT-Filter) and Global Transition Anchor (GT-Anchor). LT-Filter captures task-relevant manipulation-centric changes while suppressing static reconstruction biases. GT-Anchor provides coarse future-level regularization to preserve the overall direction of state changes. Delta-Driven Action Refinement (DDAR) further grounds Delta in the action manifold and injects it as a structured seed to guide action denoising. We implement VLDA within LoongForge, an open-source training framework for large-scale foundation and embodied models, using its modular and distributed infrastructure for unified two-stage optimization. Experiments show that VLDA improves long-horizon consistency, robustness, and generalization, validating actionable Delta as an alternative to direct action prediction and full future-state modeling.

Abstract

VLA Model Training

Explore similar work

Vision-Language-Delta-Action: Learning Compact Deltas as Actionable Transition Interfaces for Robotic Manipulation

In-Context VLA: Endowing Vision-Language-Action Models with Language via In-Context Post-Training and Agentic Tool Use

QuoVLA: Quotient Space for Vision-Language-Action Models