ΔVLA: Prior-Guided Vision-Language-Action Models via World Knowledge Variation
Authors: Yijie Zhu, Jie He, Rui Shao, Kaishen Yuan, Tao Tan, Xiaochen Yuan, Zitong Yu
Organizations: Harbin Institute of Technology, Shenzhen, Shenzhen 518055, China · Great Bay University, Dongguan 523000, China · Information Hub, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou 511400, China · Macao Polytechnic University, Macao 999078, China
Recent vision-language-action (VLA) models have significantly advanced robotic manipulation by unifying perception, reasoning, and control. To achieve such integration, recent studies adopt a predictive paradigm that models future visual states or world knowledge to guide action generation. However, these models emphasize forecasting outcomes rather than reasoning about the underlying process of change, which is essential for determining how to act. To address this, we propose ΔVLA, a prior-guided framework that models world-knowledge variations relative to an explicit current-world knowledge prior for action generation, rather than regressing absolute future world states. Specifically, 1) to construct the current world knowledge prior, we propose the Prior-Guided WorldKnowledge Extractor (PWKE). It extracts manipulable regions, spatial relations, and semantic cues from the visual input, guided by auxiliary heads and prior pseudo labels, thus reducing redundancy. 2) Building upon this, to represent how world knowledge evolves under actions, we introduce the Latent World Variation Quantization (LWVQ). It learns a discrete latent space via a VQ-VAE objective to encode world knowledge variations, shifting prediction from full modalities to compact latent. 3)Moreover, to mitigate interference during variation modeling, we design the Conditional Variation Attention (CV-Atten), whichpromotes disentangled learning and preserves the independence of knowledge representations. Extensive experiments on both simulated benchmarks and real-world robotic tasks demonstrate ΔVLA achieves state-of-the-art performance while improving efficiency. Code and real-world execution videos are available at https://github.com/JiuTian-VL/DeltaVLA.
Figures & tables
Fig. 1 : Comparison with Previous Paradigms. (a) Image-based paradigms predict subgoal images for action generation. (b) World knowledge-based paradigms replace pixel-level prediction with future world states. (c) Δ VLA constructs an explicit current-world knowledge prior via PWKE and models discrete world-knowledge variations via LWVQ , enabling prior-grounded reasoning about how the world should change under actions. (d) Comparison of performance and efficiency with previous methods.
Fig. 2 : Overview of Δ VLA Framework. It first extracts current world knowledge via the Prior-Guided World Knowledge Extractor , which encodes semantic, regional, and depth cues from SigLIP and DINOv2 guided by auxiliary heads and prior pseudo labels to form a unified world prior. These representations are concatenated with world variation tokens and modeled through the Conditional Variation Attention for structured perception–variation reasoning. Finally, the Latent World Variation Quantization module learns a discrete codebook representing world knowledge variations, providing causal guidance for variation token learning and consistent action generation.
Fig. 3 : Illustration of PWKE. World tokens have different meanings across encoders: they correspond to semantic tokens in SigLIP and depth tokens in DINOv2. The dashed box indicates auxiliary heads used only for training supervision.
Fig. 4 : Illustration of CV-Atten. In red dashed boxes, each variation token attends to its corresponding world prior, ensuring type-specific modeling and minimizing cross-modality leakage. For action generation, we employ parallel decoding.
Method
Spatial
Object
Goal
Long
Average
SR ↑
RK ↓
SR ↑
RK ↓
SR ↑
RK ↓
SR ↑
RK ↓
SR ↑
RK ↓
OpenVLA [CoRL’24] [ 52 ]
84.7
10
88.4
10
79.2
10
53.7
12
76.5
10
MDT [RSS’24] [ 69 ]
78.5
11
87.5
11
73.5
11
64.8
11
76.1
11
π0 [RSS’25] [ 3 ]
96.8
5
98.8
2
95.8
3
85.2
9
94.2
7
OpenVLA-OFT [RSS’25] [ 70 ]
97.6
3
98.4
3
97.9
1
94.5
2
97.1
2
UniVLA [RSS’25] [ 71 ]
96.5
6
96.8
6
95.6
4
92.0
3
95.2
4
TABLE I : Simulation Experimental Results on the LIBERO Benchmark. Comparison of task success rates (SR) and their ranks (RK). “†” indicates our reproduced results. Bold indicates the highest score.
Method
Click A.
Open L.
Place C. P.
Dump B. B.
Press S.
Click B.
Stack B. T.
Place B. S.
Average
RDT [ 75 ]
61.0
59.0
78.0
64.0
41.0
80.0
76.0
5.0
58.0
ACT [ 76 ]
32.0
56.0
72.0
68.0
31.0
58.0
82.0
7.0
50.8
Diffusion Policy [ 77 ]
61.0
49.0
41.0
49.0
6.0
54.0
61.0
11.0
41.5
3D Diffusion Policy [ 55 ]
77.0
82.0
86.0
85.0
69.0
90.0
83.0
19.0
73.9
π0 [ 3 ]
63.0
85.0
88.0
83.0
62.0
44.0
91.0
23.0
67.4
OpenVLA-OFT† [ 70 ]
82.0
73.0
89.0
86.0
68.0
84.0
83.0
13.0
72.3
TABLE II : Simulation Experimental Results on the RoboTwin 2.0 Benchmark. Comparison of success rates across eight tasks, along with the overall average performance. “†” indicates our reproduced results. Bold indicates the highest score.
Method
Drawer Manipulation
Align Shoes
T-shirt Folding
Plate Lemon Apple
Average
Pull
+Place
+Push
Retrieve
+Retrieve
+Align
Step 1
+Step 2
+Step 3
Open
+Retrieve
+Cover
Galaxea R1 Lite platform
OpenVLA† [ 52 ]
15/25
13/25
10/25
13/25
12/25
10/25
10/25
8/25
6/25
14/25
13/25
10/25
36%
OpenVLA-OFT† [ 70 ]
18/25
17/25
16/25
16/25
16/25
15/25
12/25
10/25
8/25
13/25
12/25
12/25
51%
UniVLA† [ 71 ]
16/25
14/25
13/25
15/25
14/25
14/25
15/25
14/25
12/25
11/25
10/25
9/25
48%
Seer† [ 18 ]
14/25
14/25
12/25
16/25
13/25
10/25
11/25
11/25
11/25
15/25
13/25
12/25
45%
TABLE III : Real-World Results. “†” indicates results reproduced using the same settings as Δ VLA. The table shows success rates for the progressive stage across four long-horizon tasks, with the average success rate representing only the completion of the entire task.
Method
Latency ↓
Throughput ↑
Cost ↓
LIBERO SR ↑
OpenVLA† [ 52 ]
0.254 s
3.9 Hz
11.7 h
76.5%
OpenVLA-OFT† [ 70 ]
0.132 s
60.6 Hz
12.5 h
97.1%
PD-VLA† [ 72 ]
0.143 s
55.9 Hz
11.7 h
94.7%
Seer† [ 18 ]
0.125 s
8.0 Hz
14.3 h
-
DreamVLA† [ 25 ]
0.130 s
7.7 Hz
14.5 h
92.6%
Δ VLA
0.105 s
76.2 Hz
4.9 h
97.8%
TABLE IV : Efficiency Comparison Results. Latency denotes inference delay; Throughput denotes the number of predicted actions per second; Cost is training time per 10k steps. “†” denotes results reproduced under identical settings as Δ VLA. All results are reported on A800 GPUs for a fair comparison.
PWKE
LWVQ
CV-Atten
Spatial
Object
Goal
Long
95.2
96.4
95.4
91.8
✓
96.0
97.4
96.6
92.6
✓
96.4
97.0
96.2
93.0
✓
✓
98.0
98.6
96.6
94.8
✓
✓
✓
98.6
99.4
97.4
95.6
TABLE V : Ablation Study on Model Components. LWVQ denotes the use of latent world variation prediction.
Region
Semantic
Depth
Spatial
Object
Goal
Long
w/ FiLM
w/o FiLM
✓
95.6
97.2
96.4
93.4
✓
✓
96.8
98.0
96.8
93.8
✓
✓
97.0
98.2
96.6
94.4
✓
✓
✓
98.6
99.4
97.4
95.6
✓
✓
✓
97.4
98.2
96.2
94.0
TABLE VI : Ablation Study on PWKE. w/ FiLM indicates the use of textual guidance in extracting manipulable regions.
Method
Spatial
Object
Goal
Long
The Number of Region (R) and World (W) Tokens
(R:128, W:4)
98.4
98.6
96.6
95.2
(R:128, W:9)
98.4
98.8
97.0
94.8
(R:64, W:4)
98.0
99.0
97.2
94.8
(R:64, W:9)
98.6
99.4
97.4
95.6
(R:64, W:16)
98.2
98.8
96.8
95.0
TABLE VII : Ablation Study on the Region and World Tokens in PWKE. We analyze how varying the number of tokens and their attention modeling influences performance.
Method
Spatial
Object
Goal
Long
w/o noise
98.6
99.4
97.4
95.6
w/ 10%
98.4
99.4
97.2
95.4
w/ 20%
98.0
99.0
96.8
95.4
w/ 30%
98.0
98.6
96.6
95.0
w/ 50%
97.0
97.4
96.0
93.8
TABLE VIII : Robustness Evaluation with Noisy Prior Labels. We evaluate the effect of injecting different levels of noise into the prior labels. The w/o noise row provides the clean-label.
Method
Spatial
Object
Goal
Long
Representation of Future States
Full Future Modalities
95.6
96.8
95.6
92.0
Continuous Variation
96.8
97.4
96.4
93.2
Latent Variation
98.6
99.4
97.4
95.6
Shared or Separated Variation Tokens
Shared
96.8
98.6
96.0
94.4
TABLE IX : Ablation Study on the LWVQ Module. We investigate the representation of future states and the design of latent variation, comparing shared versus separated settings.
Method
Spatial
Object
Goal
Long
4
98.6
99.4
97.4
95.6
9
98.4
99.2
97.0
95.4
16
98.0
98.8
96.8
95.0
TABLE X : Ablation Study on the Number of Each Type of Variation Tokens. We select a moderate number of variation tokens to achieve better performance.
Fig. 5 : Visualization of Manipulation Region Extraction and World Variation Modeling. On the left, the manipulable regions extracted by the PWKE module are shown. On the right, the future states, reconstructed from the current state and latent variation through the LWVQ module, closely align with the real future states and accurately highlight dynamic changes, marked by red circles.
Fig. 6 : Visualization of Real-World Execution Comparison. Given the long-horizon instruction, Δ VLA successfully completes all stages of the manipulation task, while DreamVLA fails at multiple critical steps.
Fig. 7 : Visualization of Executions on Two Additional Real-World Tasks. Δ VLA successfully completes all sub-stages in each task, demonstrating reliable long-horizon planning, robust manipulation skills, and accurate reasoning.
Fig. 8 : Diagnostic Analysis of CV-Atten. During geometry-critical grasping, CV-Atten suppresses cross-modal interference by keeping attention on contact-relevant geometry. Without it, attention leaks to surrounding semantic cues, causing grasp-point drift and failure.
Vision-Language-Action (VLA) models have recently emerged as a promising paradigm for robotic manipulation, in which reliable action prediction critically depends on accurately interpreting and integrating visual observations conditioned on language instructions. Although recent works have sought to enhance the visual capabilities of VLA models, most approaches treat the LLM backbone as a black box, providing limited insight into how visual information is grounded into action generation. Therefore, we perform a systematic analysis of multiple VLA models across different action-generation paradigms and observe that sensitivity to visual tokens progressively decreases in deeper layers during action generation. Motivated by this observation, we propose DeepVision-VLA, built on a Vision-Language Mixture-of-Transformers (VL-MoT) framework. This framework enables shared attention between the vision foundation model and the VLA backbone, injecting multi-level visual features from the vision expert into deeper layers of the VLA backbone to enhance visual representations for precise and complex manipulation. In addition, we introduce Action-Guided Visual Pruning (AGVP), which leverages shallow-layer attention to prune irrelevant visual tokens while preserving task-relevant ones, reinforcing critical visual cues for manipulation with minimal computational overhead. DeepVision-VLA outperforms prior state-of-the-art methods by 9.0% and 7.5% on simulated and real-world tasks, respectively, providing new insights for the design of visually enhanced VLA models.
Yulin Luo, Hao Chen, Zhuangzhe Wu +10
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · The Chinese University of Hong Kong · Simplexity Robotics
Vision-Language-Action (VLA) models have shown strong potential for robotic manipulation, but many existing methods still rely on direct mappings from language and visual observations to dense actions. This formulation can weaken the semantic reasoning capability inherited from pre-trained Vision-Language Models (VLMs), which are mainly optimized for visual-linguistic understanding rather than low-level control, and becomes fragile under spatial variations, including changes in object positions, scene layouts, robot embodiments, and camera viewpoints. To address these limitations, we propose H-VLA, a hierarchical VLA framework that decouples high-level key-action reasoning from low-level motion generation. H-VLA combines a Key-Action Model for predicting a key-action as the next manipulation subgoal, a Motion Planning Model for generating dense future actions conditioned on the predicted key-action, and a Unified Camera-Centric Action Space for consistent representation across datasets, embodiments, and viewpoints. We further adopt a two-stage training strategy that emphasizes key-action reasoning during pre-training and dense motion generation during fine-tuning. Experiments show that H-VLA achieves strong performance on SimplerEnv, reaching 91% on Google Robot visual matching, 84% on Google Robot variant aggregation, and 81% on WidowX visual matching. On Agilex real-robot tasks, H-VLA improves over the strongest baseline by 10, 47, and 16 percentage points under in-distribution, out-of-distribution position, and out-of-distribution scene/object settings, respectively.
Xiongfeng Peng, Lu Xu, Yandong Wang +10
Advanced Research Lab, Samsung R&D Institute China-Beijing (SRCB), China · Samsung AI Center, DS Division, South Korea
Vision-language-action (VLA) models built on pretrained vision-language models have demonstrated strong performance across diverse robotic manipulation tasks. However, VLA models that directly map current 2D observations to actions often lack sufficient spatial and temporal understanding, limiting their performance in precise and long-horizon manipulation. Recent methods enhance VLA models through geometric supervision and future-state prediction across the entire scene. However, these methods can suffer from redundant scene information, distracting the model from learning the geometry and dynamics relevant to the current interaction. To address this issue, we propose FOCAL-VLA, a framework that combines subtask-guided geometry distillation with implicit world modeling to learn representations of current spatial structure and future interaction dynamics. To focus geometric learning on the current subtask, we transfer geometric knowledge from VGGT to the VLA model by aligning geometry latents with features from subtask-relevant image regions. To capture the future 3D evolution of the current interaction, we incorporate implicit world modeling using Track4World features from current and future demonstration frames. The two complementary representations jointly guide action generation without running VGGT or Track4World at inference time. Experiments show that FOCAL-VLA outperforms baselines on both simulation benchmarks and real-world manipulation tasks. Project website: https://zhiyuan-gao.github.io/FOCAL-VLA/.
Zhiyuan Gao, Di Wen, Yanxiang Zhan +4
University of Bremen, Germany · Karlsruhe Institute of Technology (KIT), Germany · Robotics Institute Germany (RIG)