CLaD: Planning with Grounded Foresight via Cross-Modal Latent Dynamics

Andrew Jeong; Jaemin Kim; Sebin Lee; Sung-Eui Yoon

cs.ROMar 31, 2026

CLaD: Planning with Grounded Foresight via Cross-Modal Latent Dynamics

Authors: Andrew Jeong, Jaemin Kim, Sebin Lee, Sung-Eui Yoon

Organizations: KAIST

Abstract

Robotic manipulation involves kinematic and semantic transitions that are inherently coupled via underlying actions. However, existing approaches plan within either semantic or latent space without explicitly aligning these cross-modal transitions. To address this, we propose CLaD, a framework that models how proprioceptive and semantic states jointly evolve under actions through asymmetric cross-attention that allows kinematic transitions to query semantic ones. CLaD predicts grounded latent foresights via self-supervised objectives with EMA target encoders and auxiliary reconstruction losses, preventing representation collapse while anchoring predictions to observable states. Predicted foresights are modulated with observations to condition a diffusion policy for action generation. On LIBERO-LONG benchmark, CLaD achieves 94.7% success rate, competitive with large VLAs with significantly fewer parameters.

Figures & tables

Figure 2: Detailed architecture of CLaD’s two-stage framework. Stage 1 (Left) - Semantic state $s_{t}$ and proprioceptive state $p_{t}$ are each encoded and fed into modality-specific cross-attention modules to extract transition representations $z_{s}$ and $z_{p}$ . Asymmetric cross-attention, where proprioceptive transitions query semantic ones, yields shared dynamics $z_{\text{dyn}}$ , from which lightweight MLPs predict future latent foresights $\hat{z}^{t+\tau}_{p}$ and $\hat{z}^{t+\tau}_{s}$ . These predictions are supervised via EMA target encoders using actual future states $s_{t+\tau}$ and $p_{t+\tau}$ , with auxiliary reconstruction decoders maintaining correspondence with observable quantities and preventing representation collapse. Stage 2 (Right) - Predicted latent foresights $\hat{z}^{t+\tau}=[\hat{z}^{t+\tau}_{p};\,\hat{z}^{t+\tau}_{s}]$ are adaptively modulated with current observations via FiLM layers, and the resulting representation conditions a diffusion policy to generate action sequences $a_{t:t+\tau}$ .

Task ID	Susie †	Seer †	LBP †	${\pi_{0}}^{\ddagger}$	${\pi_{0.5}}^{\ddagger}$	OpenVLA ‡	Ours †	Ours ‡
Task ID	(0.86B)	(0.32B)	(0.19B)	(3.3B)	(3.3B)	(7B)	(0.66B)
1	83.3	88.3	90.0	74.0	92.0	98.0	100.0	100.0
2	63.3	90.0	100.0	94.0	98.0	96.0	100.0	100.0
3	96.6	98.3	100.0	88.0	98.0	100.0	98.3	98.0
4	100.0	100.0	100.0	22.0	98.0	78.0	93.7	94.0
5	83.3	91.7	76.6	100.0	100.0	84.0	92.0	91.0

Table 1: Performance comparison on LIBERO-LONG benchmark . CLaD achieves competitive performance with significantly fewer parameters (0.66B) compared to baselines. LIBERO-LONG tasks include: (1) put soup and sauce in basket; (2) put box and butter in basket; (3) turn on stove and put pot; (4) put bowl in drawer and close it; (5) put mugs on left and right plates; (6) pick book and place it in back; (7) put mug on plate and put pudding to right; (8) put soup and box in basket; (9) put both pots on stove; (10) put mug in microwave and close it.

Method	OpenVLA	$\pi_{0.5}$	CLaD
Inference time (Hz)	6	10	25
Memory (GB)	15	19	4
Parameters (B)	7.0	3.3	0.66

Table 2: Comparison on computational requirements. CLaD achieves faster inference and lower resource consumption compared to large-scale VLA models, while maintaining competitive task performance on LIBERO-LONG.

Method	UVA	LBP	CLaD
Parameters (B)	0.5	0.19	0.66
Planning Time (s)	0.195	0.008	0.012
Avg. SR (%)	90.0	88.6	94.7

Table 3: Performance and efficiency of latent planning methods on LIBERO-LONG. Although CLaD requires more parameters than LBP (0.19B) and UVA (0.5B), it yields higher average success rate, respectively, suggesting a competitive trade-off between model capacity and task performance within the latent planning paradigm.

Figure 3: Modality contribution analysis on LIBERO-LONG tasks . We compare diffusion policy without foresight conditioning (Policy only), conditioning on proprioceptive foresight only ( $\text{CLaD}_{\textbf{p}}$ ), semantic foresight only ( $\text{CLaD}_{\textbf{s}}$ ), and full cross-modal foresight (CLaD). Results demonstrate that cross-modal foresight provides substantial gains over single-modality predictions, though semantic foresight alone achieves reasonable performance. Notably, proprioceptive foresight alone severely degrades performance, suggesting that kinematic predictions require semantic context for effective planning.

Learning objective	$\mathcal{L}_{\text{latent}}$	$\mathcal{L}_{\text{latent}}$ + $\mathcal{L}_{\text{recon}}$
Avg. SR	86.1 (-8.6)	94.7

Table 4: Ablation of the auxiliary reconstruction loss

\mathcal{L}_{\text{recon}}

. Removing

\mathcal{L}_{\text{recon}}

leads to 8.6% performance degradation, indicating that

\mathcal{L}_{\text{recon}}

is critical to ground latent dynamics

z_{\text{dyn}}

to observable states for learning robust latent foresights, not merely a regularizer.

Figure 4: UMAP of latent embedding $z_{\text{dyn}}$ with regard to $\mathcal{L}_{\text{recon}}$ . With $\mathcal{L}_{\text{recon}}$ (left), embeddings form distinct task-specific clusters. Without $\mathcal{L}_{\text{recon}}$ (right), clusters become diffuse and overlapping, indicating reduced semantic structure in the latent space.

Attention Configuration	Avg. SR(%)
Symmetric self-attention	86.7
Semantics queries proprioceptive	93.8
Proprioceptive queries semantic (ours)	94.7

Table 5: Ablation of cross-attention configurations. Using proprioceptive transitions as queries over semantic transitions (94.7%) outperforms both the reverse configuration (93.8%) and symmetric self-attention (86.7%), confirming that directing kinematic context to attend over semantic transitions is a beneficial inductive bias for extracting cross-modal dynamics.

Method	Spatial	Object	Goal	Long
$\pi_{0}^{\diamond}$	97.2	97.8	91.6	82.0
UniVLA	96.5	96.8	95.6	92.0
${\pi_{0.5}}^{\diamond}$	97.6	98.4	97.2	93.2
$\text{OpenVLA}^{\diamond}$	98.2	98.6	97.6	93.8
CLaD (Ours)	97.3	95.7	94.3	94.7

Table 6: Average success rates (%) over 50 rollouts on all suites in LIBERO benchmark. Color intensity is proportional to the performance level (thicker = higher).

Panel of Figure 5 — Figure 5: Pixel attribution for predicted latent foresight via Integrated Gradients. Heatmaps show pixel-level contributions toward the alignment between predicted foresight $\hat{\mathbf{z}}^{t+\tau}$ and target embedding. Brighter regions indicate higher attribution scores. While not yielding precise object boundaries, attributions consistently highlight task-relevant objects, suggesting that the model leverages semantic features for future state prediction.

Variants	Heavy Mask	Action-free	Curriculum	CLaD
Avg. SR (%)	88.2	90.8	85.1	94.7

Table 7: Ablation with action-free training variants.

Panel of Figure 6 — Figure 6: Demonstrations on task 1, 2, 3 of LIBERO-LONG.

Panel of Figure 7 — Figure 7: Demonstrations on task 4, 5, 6 of LIBERO-LONG.

Panel of Figure 8 — Figure 8: Demonstrations on task 7, 8, 9 of LIBERO-LONG.

Figure 9: Demonstrations on task 10 of LIBERO-LONG.

Robot Motion Planning

Robot motion planning finds safe, feasible paths and trajectories for robots moving through constrained, changing environments.

See all 271 papers

ForeAct3D: Policy-Grounded Future World Modeling for VLA Policies
Oct 3, 2026Zhe Tao, Feiran Wang, Gaowen Liu +2World Models for Robotics Vision-Language-Action Models
University of Illinois Urbana-Champaign · University of Illinois Chicago · Cisco
Towards Predictive, Aligned, and Scalable Robot Learning
Jul 13, 2026Peijun Tang, Shangjin Xie, Baifu Huang +6Latent Action Learning Cross-Modal Alignment
Astribot Team
PLaCA: Learning Patch-Level Future Latent Change for Action Generation
Oct 20, 2026Zhan Yi, Yuchen Wang, Chao Li +2Temporal Modeling for VLA Models World Models for VLA Models
School of Robotics and Automation, Nanjing University, Suzhou, Jiangsu, China

Oct 3, 2026J/K move · Enter open · S save

ForeAct3D: Policy-Grounded Future World Modeling for VLA Policies

Zhe Tao, Feiran Wang, Gaowen Liu, Ramana Rao Kompella+1

University of Illinois Urbana-Champaign · University of Illinois Chicago · Cisco

World Models for Robotics Vision-Language-Action Models Language-Conditioned Robot Manipulation 3D Scene Understanding

Open paper AskPDF arXiv

Robots need to anticipate how their actions will change the world, since manipulation success hinges on the resulting contacts and object motions. However, existing Vision-Language-Action (VLA) policies that predict future observations from shared features leave the forecast decoupled from the actions the policy will actually execute, and impose no physical constraints on how the scene may evolve. We introduce ForeAct3D, a framework for policy-grounded future world modeling within VLA policies. Learnable geometric queries decode depth, semantic segmentation, and camera pose from the policy representation into current and future semantic 3D scene states, and the future queries are conditioned on the policy-generated action chunk to ground the forecast in the planned interaction. A physical-consistency closure relates the two states through background staticity and instance-level rigidity, and anchors the wrist-camera pose to end-effector kinematics. These objectives shape the shared representation used for action generation during training, and no future prediction is required at inference. Without robot pretraining, ForeAct3D achieves 98.3% average success on LIBERO and an average task length of 3.73 on CALVIN, outperforming its base policy on every suite. Ablations show that semantic 3D supervision, physical consistency, and action conditioning each improve manipulation performance, and that action conditioning substantially improves future object localization. Real-world experiments on spatial placement, object insertion, and sequential manipulation further raise average success from 6.7% to 37.8% over the base policy. The project page and code are available at https://github.com/anthonytao80-crypto/ForeAct3D.

Abstract

Robot Motion Planning

Explore similar work

ForeAct3D: Policy-Grounded Future World Modeling for VLA Policies

Towards Predictive, Aligned, and Scalable Robot Learning

PLaCA: Learning Patch-Level Future Latent Change for Action Generation