DIAL: Decoupling Intent and Action via Latent World Modeling for End-to-End VLA

Yi Chen; Yuying Ge; Hui Zhou; Mingyu Ding; Yixiao Ge; Xihui Liu

cs.ROMar 31, 2026

DIAL: Decoupling Intent and Action via Latent World Modeling for End-to-End VLA

Authors: Yi Chen, Yuying Ge, Hui Zhou, Mingyu Ding, Yixiao Ge, Xihui Liu

Organizations: The University of Hong Kong · XPENG Robotics · University of North Carolina at Chapel Hill

Abstract

The development of Vision-Language-Action (VLA) models has been significantly accelerated by pre-trained Vision-Language Models (VLMs). However, most existing end-to-end VLAs treat the VLM primarily as a multimodal encoder, directly mapping vision-language features to low-level actions. This paradigm underutilizes the VLM's potential in high-level decision making and introduces training instability, frequently degrading its rich semantic representations. To address these limitations, we introduce DIAL, a framework bridging high-level decision making and low-level motor execution through a differentiable latent intent bottleneck. Specifically, a VLM-based System-2 performs latent world modeling by synthesizing latent visual foresight within the VLM's native feature space; this foresight explicitly encodes intent and serves as the structural bottleneck. A lightweight System-1 policy then decodes this predicted intent together with the current observation into precise robot actions via latent inverse dynamics. To ensure optimization stability, we employ a two-stage training paradigm: a decoupled warmup phase where System-2 learns to predict latent futures while System-1 learns motor control under ground-truth future guidance within a unified feature space, followed by seamless end-to-end joint optimization. This enables action-aware gradients to refine the VLM backbone in a controlled manner, preserving pre-trained knowledge. Extensive experiments on the RoboCasa GR1 Tabletop benchmark show that DIAL establishes a new state-of-the-art, achieving superior performance with 10x fewer demonstrations than prior methods. Furthermore, by leveraging heterogeneous human demonstrations, DIAL learns physically grounded manipulation priors and exhibits robust zero-shot generalization to unseen objects and novel configurations during real-world deployment on a humanoid robot.

Figures & tables

Figure 1: DIAL bridges high-level decision making and low-level motor control through a differentiable latent intent bottleneck . (Left) System-2 (VLM) performs latent world modeling (LWM) to synthesize visual foresight within its native ViT feature space. This foresight serves as a structural bottleneck to convey the VLM’s intent, which System-1 (Policy) then decodes into actions via latent inverse dynamics. A decoupled-to-unified training paradigm ensures stability, leveraging initial alignment in a consistent latent space to facilitate subsequent end-to-end refinement via action-aware gradients. (Right) Powered by this structural grounding, DIAL scales across heterogeneous human and robot data to achieve SOTA performance with high data efficiency. In real-world deployments, it demonstrates robust zero-shot generalization across diverse configurations, mastering tasks ranging from semantic object manipulation to complex multi-stage coordination.

Figure 2: Comparison of VLA Architectures. (Left) Hierarchical Models decouple reasoning and execution via text or pixels, resulting in non-differentiable gaps and significant deployment latency. (Middle) End-to-End VLAs map multimodal features directly to actions. Even when auxiliary tasks are used, they are typically treated as optional context, which cannot strictly guarantee that actions are grounded in the VLM’s intent. (Right) DIAL (Ours) introduces a differentiable latent bottleneck. By requiring System-1 to bridge the gap between current visual features and System-2’s predicted latent foresight, DIAL ensures that execution is inherently anchored to the VLM’s predictive intent.

Figure 3: The Dual-System Architecture of DIAL. Built upon a pre-trained VLM, System-2 (top) synthesizes a latent foresight ( $x_{t}$ ) from language ( $l_{t}$ ), current visual observation ( $o_{t}$ ), and learnable queries via its LLM backbone and an MLP head. System-1 (bottom) employs self-attention to fuse current and foresight visual features, serving as the cross-attention condition for a DiT-based action decoder. This decoder directly takes the projected proprioceptive state ( $q_{t}$ ) and noisy action tokens to generate action chunks. To ensure feature consistency, both systems share the VLM’s frozen pre-trained ViT. As indicated by the switches, the training transitions from a decoupled warmup (conditioned on ground-truth features of $o_{t+H}$ ) to end-to-end optimization (conditioned on $x_{t}$ ). Throughout both stages, an MSE loss is applied to align the latent foresight with ground-truth features.

Figure 4: Examples from the 24 RoboCasa GR1 Tabletop Tasks, including object rearrangement (e.g., Croissant to Box) and interaction with articulated fixtures (e.g., Bottle to Cabinet).

Figure 5: Real-world Tasks and Data Sources. We evaluate our framework across manipulation tasks categorized by data composition and complexity: (Top) Cross-embodiment learning , featuring Pick & Place and Pouring , jointly trained on EgoDex human and robot trajectories. (Bottom) Multi-stage coordination , including Handover and Sweeping . While top tasks leverage heterogeneous data, bottom tasks use robot-native sequences to capture intricate synchronization and tool-use.

Figure 6: Real-world Generalization Scenarios. Comparison of in-distribution tasks and five OOD categories: combinatorial generalization (multiple seen objects), distractor robustness (unseen background items), instance-level transfer (novel object types), fixture-level transfer (novel shelf types), and surface-level transfer (unseen tablecloths).

Figure 7: Results on RoboCasa GR1 Tabletop Simulation with full training data.

Figure 8: Results on RoboCasa GR1 Tabletop Simulation under the few-shot setting.

Figure 9: Impact of incorporating EgoDex basic_pick_place human demonstrations on few-shot performance in RoboCasa GR1 simulation tasks.

Figure 10: In-distribution experiment results on the real-world humanoid robot for cross-embodiment learning tasks, showing the impact of joint training with human demonstrations.

Figure 11: Real-world OOD results for cross-embodiment learning tasks across three generalization challenges: combinatorial generalization, distractor robustness, and instance-level transfer.

Figure 12: Real-world experiment results on multi-stage coordination tasks, evaluating both in-distribution performance and zero-shot generalization across novel fixture and surface configurations.

Figure 13: Visualization of latent representations for current observations, ground-truth futures, and predicted foresight, with colors encoding the first three PCA components mapped to RGB. The last column shows the per-patch cosine distance between predicted foresight and current observation features, highlighting regions where the model anticipates future change.

Explore similar work

Oct 6, 2026

ViDAL: A Visual Dynamics-Grounded Action Latent Space for Vision-Language-Action Models

Vision-Language-Action (VLA) models have become a central paradigm for robot policy learning, which predict actions in three forms: raw action chunks, discrete action tokens, or continuous action latents. However, existing action representations primarily model action trajectories, with limited consideration of the visual dynamics induced by these actions. We introduce ViDAL, a Visual Dynamics-grounded Action Latent Space that anchors continuous action latents in the future visual dynamics of the scene. Specifically, ViDAL learns action latent space by training an Action Variational Autoencoder (Action VAE) to reconstruct action chunks while aligning its latent with future scene dynamics. When integrated into downstream robot policies, the proposed Action VAE serves as a plug-in action interface compatible with multiple VLA architectures and enables optional future-video prediction as an additional capability. Empirically, ViDAL outperforms competitive baselines on LIBERO with 98.1% average success, improves a multi-task

π_{0.5}

policy on RoboTwin 2.0 from 54.3% to 65.5% (Clean) and from 33.2% to 43.1% (Random) success rates over 50 dual-arm tasks, and yields 20.0% and 23.4% absolute success-rate gains on real-world single-arm Franka and dual-arm ARX robot platforms.

Yuan Xu, Yixiang Chen, Qisen Ma +10

CASIA · UCAS · Alibaba Group +1

Latent Action Learning Visuomotor Policy Learning Robotic Manipulation Vision-Language-Action Models

Ask PDF

Jun 5, 2026

LARA: Latent Action Representation Alignment for Vision-Language-Action Models

Visual-language action (VLA) models enable robots to predict actions directly from observations and language instructions, but their performance depends on large-scale, high-quality data and is limited by the scarcity of real-world robot action datasets. To facilitate VLA model learning with abundant unlabeled human videos, Latent Action Models (LAM) learn latent action representations from visual dynamics to provide additional supervision for VLA learning. However, LAM and VLA are typically trained separately, leaving LAM ungrounded during VLA training and VLA models constrained by frozen LAM representations. To address these issues, we propose Latent Action Representation Alignment (LARA), a plug-and-play framework that jointly optimizes LAM and VLA via representation alignment. This enables reciprocal benefits where LAMs learn with action trajectories to avoid spurious visual changes, while VLAs are regularized by forward dynamics learned within LAMs to reduce hallucinations of functionally ineffective trajectories. We demonstrate LARA versatility and effectiveness for pre-training, post-training enhancement of pre-trained VLA models, and LAM refinement, achieving an average of ~10%, ~5%, and ~15% improvement over 3 simulation and 1 meticulously designed real-world robotic manipulation benchmarks.

Mengya Liu, Baoxiong Jia, Jiangyong Huang +2

State Key Laboratory of General Artificial Intelligence, BIGAI · Peking University · Delta Intelligence

Latent Action Learning Robotic Manipulation Vision-Language-Action Models

Ask PDF

Sep 19, 2026

H-VLA: Hierarchical Vision-Language-Action Model with Key-Action Reasoning and Motion Planning in a Unified Action Space

Vision-Language-Action (VLA) models have shown strong potential for robotic manipulation, but many existing methods still rely on direct mappings from language and visual observations to dense actions. This formulation can weaken the semantic reasoning capability inherited from pre-trained Vision-Language Models (VLMs), which are mainly optimized for visual-linguistic understanding rather than low-level control, and becomes fragile under spatial variations, including changes in object positions, scene layouts, robot embodiments, and camera viewpoints. To address these limitations, we propose H-VLA, a hierarchical VLA framework that decouples high-level key-action reasoning from low-level motion generation. H-VLA combines a Key-Action Model for predicting a key-action as the next manipulation subgoal, a Motion Planning Model for generating dense future actions conditioned on the predicted key-action, and a Unified Camera-Centric Action Space for consistent representation across datasets, embodiments, and viewpoints. We further adopt a two-stage training strategy that emphasizes key-action reasoning during pre-training and dense motion generation during fine-tuning. Experiments show that H-VLA achieves strong performance on SimplerEnv, reaching 91% on Google Robot visual matching, 84% on Google Robot variant aggregation, and 81% on WidowX visual matching. On Agilex real-robot tasks, H-VLA improves over the strongest baseline by 10, 47, and 16 percentage points under in-distribution, out-of-distribution position, and out-of-distribution scene/object settings, respectively.

Xiongfeng Peng, Lu Xu, Yandong Wang +10

Advanced Research Lab, Samsung R&D Institute China-Beijing (SRCB), China · Samsung AI Center, DS Division, South Korea

Vision-Language-Action Models Language-Conditioned Robot Manipulation Reasoning for VLA Models Robustness of VLA Models Robot Motion Planning

Ask PDF