CLaD: Planning with Grounded Foresight via Cross-Modal Latent Dynamics

Andrew Jeong; Jaemin Kim; Sebin Lee; Sung-Eui Yoon

cs.ROMar 31, 2026

CLaD: Planning with Grounded Foresight via Cross-Modal Latent Dynamics

Authors: Andrew Jeong, Jaemin Kim, Sebin Lee, Sung-Eui Yoon

Organizations: KAIST

Abstract

Robotic manipulation involves kinematic and semantic transitions that are inherently coupled via underlying actions. However, existing approaches plan within either semantic or latent space without explicitly aligning these cross-modal transitions. To address this, we propose CLaD, a framework that models how proprioceptive and semantic states jointly evolve under actions through asymmetric cross-attention that allows kinematic transitions to query semantic ones. CLaD predicts grounded latent foresights via self-supervised objectives with EMA target encoders and auxiliary reconstruction losses, preventing representation collapse while anchoring predictions to observable states. Predicted foresights are modulated with observations to condition a diffusion policy for action generation. On LIBERO-LONG benchmark, CLaD achieves 94.7% success rate, competitive with large VLAs with significantly fewer parameters.

Figures & tables

Figure 2: Detailed architecture of CLaD’s two-stage framework. Stage 1 (Left) - Semantic state $s_{t}$ and proprioceptive state $p_{t}$ are each encoded and fed into modality-specific cross-attention modules to extract transition representations $z_{s}$ and $z_{p}$ . Asymmetric cross-attention, where proprioceptive transitions query semantic ones, yields shared dynamics $z_{\text{dyn}}$ , from which lightweight MLPs predict future latent foresights $\hat{z}^{t+\tau}_{p}$ and $\hat{z}^{t+\tau}_{s}$ . These predictions are supervised via EMA target encoders using actual future states $s_{t+\tau}$ and $p_{t+\tau}$ , with auxiliary reconstruction decoders maintaining correspondence with observable quantities and preventing representation collapse. Stage 2 (Right) - Predicted latent foresights $\hat{z}^{t+\tau}=[\hat{z}^{t+\tau}_{p};\,\hat{z}^{t+\tau}_{s}]$ are adaptively modulated with current observations via FiLM layers, and the resulting representation conditions a diffusion policy to generate action sequences $a_{t:t+\tau}$ .

Task ID	Susie †	Seer †	LBP †	${\pi_{0}}^{\ddagger}$	${\pi_{0.5}}^{\ddagger}$	OpenVLA ‡	Ours †	Ours ‡
Task ID	(0.86B)	(0.32B)	(0.19B)	(3.3B)	(3.3B)	(7B)	(0.66B)
1	83.3	88.3	90.0	74.0	92.0	98.0	100.0	100.0
2	63.3	90.0	100.0	94.0	98.0	96.0	100.0	100.0
3	96.6	98.3	100.0	88.0	98.0	100.0	98.3	98.0
4	100.0	100.0	100.0	22.0	98.0	78.0	93.7	94.0
5	83.3	91.7	76.6	100.0	100.0	84.0	92.0	91.0

Table 1: Performance comparison on LIBERO-LONG benchmark . CLaD achieves competitive performance with significantly fewer parameters (0.66B) compared to baselines. LIBERO-LONG tasks include: (1) put soup and sauce in basket; (2) put box and butter in basket; (3) turn on stove and put pot; (4) put bowl in drawer and close it; (5) put mugs on left and right plates; (6) pick book and place it in back; (7) put mug on plate and put pudding to right; (8) put soup and box in basket; (9) put both pots on stove; (10) put mug in microwave and close it.

Method	OpenVLA	$\pi_{0.5}$	CLaD
Inference time (Hz)	6	10	25
Memory (GB)	15	19	4
Parameters (B)	7.0	3.3	0.66

Table 2: Comparison on computational requirements. CLaD achieves faster inference and lower resource consumption compared to large-scale VLA models, while maintaining competitive task performance on LIBERO-LONG.

Method	UVA	LBP	CLaD
Parameters (B)	0.5	0.19	0.66
Planning Time (s)	0.195	0.008	0.012
Avg. SR (%)	90.0	88.6	94.7

Table 3: Performance and efficiency of latent planning methods on LIBERO-LONG. Although CLaD requires more parameters than LBP (0.19B) and UVA (0.5B), it yields higher average success rate, respectively, suggesting a competitive trade-off between model capacity and task performance within the latent planning paradigm.

Figure 3: Modality contribution analysis on LIBERO-LONG tasks . We compare diffusion policy without foresight conditioning (Policy only), conditioning on proprioceptive foresight only ( $\text{CLaD}_{\textbf{p}}$ ), semantic foresight only ( $\text{CLaD}_{\textbf{s}}$ ), and full cross-modal foresight (CLaD). Results demonstrate that cross-modal foresight provides substantial gains over single-modality predictions, though semantic foresight alone achieves reasonable performance. Notably, proprioceptive foresight alone severely degrades performance, suggesting that kinematic predictions require semantic context for effective planning.

Learning objective	$\mathcal{L}_{\text{latent}}$	$\mathcal{L}_{\text{latent}}$ + $\mathcal{L}_{\text{recon}}$
Avg. SR	86.1 (-8.6)	94.7

Table 4: Ablation of the auxiliary reconstruction loss

\mathcal{L}_{\text{recon}}

. Removing

\mathcal{L}_{\text{recon}}

leads to 8.6% performance degradation, indicating that

\mathcal{L}_{\text{recon}}

is critical to ground latent dynamics

z_{\text{dyn}}

to observable states for learning robust latent foresights, not merely a regularizer.

Figure 4: UMAP of latent embedding $z_{\text{dyn}}$ with regard to $\mathcal{L}_{\text{recon}}$ . With $\mathcal{L}_{\text{recon}}$ (left), embeddings form distinct task-specific clusters. Without $\mathcal{L}_{\text{recon}}$ (right), clusters become diffuse and overlapping, indicating reduced semantic structure in the latent space.

Attention Configuration	Avg. SR(%)
Symmetric self-attention	86.7
Semantics queries proprioceptive	93.8
Proprioceptive queries semantic (ours)	94.7

Table 5: Ablation of cross-attention configurations. Using proprioceptive transitions as queries over semantic transitions (94.7%) outperforms both the reverse configuration (93.8%) and symmetric self-attention (86.7%), confirming that directing kinematic context to attend over semantic transitions is a beneficial inductive bias for extracting cross-modal dynamics.

Method	Spatial	Object	Goal	Long
$\pi_{0}^{\diamond}$	97.2	97.8	91.6	82.0
UniVLA	96.5	96.8	95.6	92.0
${\pi_{0.5}}^{\diamond}$	97.6	98.4	97.2	93.2
$\text{OpenVLA}^{\diamond}$	98.2	98.6	97.6	93.8
CLaD (Ours)	97.3	95.7	94.3	94.7

Table 6: Average success rates (%) over 50 rollouts on all suites in LIBERO benchmark. Color intensity is proportional to the performance level (thicker = higher).

Panel of Figure 5 — Figure 5: Pixel attribution for predicted latent foresight via Integrated Gradients. Heatmaps show pixel-level contributions toward the alignment between predicted foresight $\hat{\mathbf{z}}^{t+\tau}$ and target embedding. Brighter regions indicate higher attribution scores. While not yielding precise object boundaries, attributions consistently highlight task-relevant objects, suggesting that the model leverages semantic features for future state prediction.

Variants	Heavy Mask	Action-free	Curriculum	CLaD
Avg. SR (%)	88.2	90.8	85.1	94.7

Table 7: Ablation with action-free training variants.

Panel of Figure 6 — Figure 6: Demonstrations on task 1, 2, 3 of LIBERO-LONG.

Panel of Figure 7 — Figure 7: Demonstrations on task 4, 5, 6 of LIBERO-LONG.

Panel of Figure 8 — Figure 8: Demonstrations on task 7, 8, 9 of LIBERO-LONG.

Figure 9: Demonstrations on task 10 of LIBERO-LONG.

Robot Motion Planning

Robot motion planning finds safe, feasible paths and trajectories for robots moving through constrained, changing environments.

See all 271 papers

Oct 3, 2026

ForeAct3D: Policy-Grounded Future World Modeling for VLA Policies

Robots need to anticipate how their actions will change the world, since manipulation success hinges on the resulting contacts and object motions. However, existing Vision-Language-Action (VLA) policies that predict future observations from shared features leave the forecast decoupled from the actions the policy will actually execute, and impose no physical constraints on how the scene may evolve. We introduce ForeAct3D, a framework for policy-grounded future world modeling within VLA policies. Learnable geometric queries decode depth, semantic segmentation, and camera pose from the policy representation into current and future semantic 3D scene states, and the future queries are conditioned on the policy-generated action chunk to ground the forecast in the planned interaction. A physical-consistency closure relates the two states through background staticity and instance-level rigidity, and anchors the wrist-camera pose to end-effector kinematics. These objectives shape the shared representation used for action generation during training, and no future prediction is required at inference. Without robot pretraining, ForeAct3D achieves 98.3% average success on LIBERO and an average task length of 3.73 on CALVIN, outperforming its base policy on every suite. Ablations show that semantic 3D supervision, physical consistency, and action conditioning each improve manipulation performance, and that action conditioning substantially improves future object localization. Real-world experiments on spatial placement, object insertion, and sequential manipulation further raise average success from 6.7% to 37.8% over the base policy. The project page and code are available at https://github.com/anthonytao80-crypto/ForeAct3D.

Zhe Tao, Feiran Wang, Gaowen Liu +2

University of Illinois Urbana-Champaign · University of Illinois Chicago · Cisco

World Models for Robotics Vision-Language-Action Models Language-Conditioned Robot Manipulation 3D Scene Understanding

Ask PDF

Jul 13, 2026

Towards Predictive, Aligned, and Scalable Robot Learning

Learning, at its core, extends beyond memorization to the ability to reason and solve novel problems by navigating a space of possibilities. We introduce Lumo-2, a latent world-action model that generates actions by reasoning over world dynamics in latent space. The learned latent world dynamics capture physically grounded visual transitions, naturally encoding future possibilities and providing a unified substrate for cross-modal alignment. This formulation enables predictive reasoning akin to world modelling while remaining lightweight and focused on physical dynamics relevant to control. Central to our approach is the hypothesis that action generation quality is governed by the geometry of the latent space. We observe that standard reconstruction-based action tokenization objectives induce representations biased toward low-level signal fidelity, leading to misalignment between reconstruction quality and downstream control performance. To address this limitation, we propose a multi-stage modality pre-alignment strategy in which action representations are progressively aligned with latent world dynamics, vision, and language. This process enforces cross-modal consistency, promotes abstraction, and induces a structured latent space for predictive reasoning. We provide a systematic empirical study of latent world modelling and modality alignment, analyzing their roles in scaling laws and out-of-distribution generalization. Results show that Lumo-2 consistently outperforms strong vision-language-action (VLA) and world-action model (WAM) baselines, with gains on challenging real-world tasks requiring temporal reasoning, physical understanding, or high control complexity, including long-horizon and dexterous manipulation. These findings suggest that structured multimodal alignment and predictive reasoning are fundamental principles for advancing embodied intelligence.

Peijun Tang, Shangjin Xie, Baifu Huang +6

Astribot Team

Latent Action Learning Cross-Modal Alignment World Action Models Embodied World Models Robot Learning

Ask PDF

Oct 20, 2026

PLaCA: Learning Patch-Level Future Latent Change for Action Generation

Vision-Language-Action (VLA) models generate robot actions from visual observations and language instructions, but effective manipulation also requires anticipating how the environment will change. Existing world-action models explore future image prediction and latent dynamics modeling. However, generating complete future images can be computationally expensive, while highly compressed latent representations may obscure local changes. To address these challenges, we propose PLaCA, a world-to-action framework built around future latent change. PLaCA directly predicts spatially structured patch-level future latent change from the current multimodal input, preserving local information about expected environmental changes. We introduce patch-level change-aware supervision to emphasize regions undergoing larger changes and a future-to-action interface that explicitly incorporates predicted future latent change into action generation. Experiments on LIBERO-Long show that PLaCA achieves a 79.0% success rate, improving upon the Action-only baseline by 5.0 percentage points. On CALVIN, PLaCA improves the average sequence length from 2.784 to 3.564. These improvements are achieved while updating only 2.95% of the model parameters during the second training stage, demonstrating the effectiveness of future latent change modeling for downstream robotic manipulation.

Zhan Yi, Yuchen Wang, Chao Li +2

School of Robotics and Automation, Nanjing University, Suzhou, Jiangsu, China

Temporal Modeling for VLA Models World Models for VLA Models Robotic Manipulation

Ask PDF

Abstract

Robot Motion Planning

Explore similar work

ForeAct3D: Policy-Grounded Future World Modeling for VLA Policies

Towards Predictive, Aligned, and Scalable Robot Learning

PLaCA: Learning Patch-Level Future Latent Change for Action Generation