ForeAct3D: Policy-Grounded Future World Modeling for VLA Policies

Zhe Tao; Feiran Wang; Gaowen Liu; Ramana Rao Kompella; Yan Yan

cs.ROOct 3, 2026

ForeAct3D: Policy-Grounded Future World Modeling for VLA Policies

Authors: Zhe Tao, Feiran Wang, Gaowen Liu, Ramana Rao Kompella, Yan Yan

Organizations: University of Illinois Urbana-Champaign · University of Illinois Chicago · Cisco

Abstract

Robots need to anticipate how their actions will change the world, since manipulation success hinges on the resulting contacts and object motions. However, existing Vision-Language-Action (VLA) policies that predict future observations from shared features leave the forecast decoupled from the actions the policy will actually execute, and impose no physical constraints on how the scene may evolve. We introduce ForeAct3D, a framework for policy-grounded future world modeling within VLA policies. Learnable geometric queries decode depth, semantic segmentation, and camera pose from the policy representation into current and future semantic 3D scene states, and the future queries are conditioned on the policy-generated action chunk to ground the forecast in the planned interaction. A physical-consistency closure relates the two states through background staticity and instance-level rigidity, and anchors the wrist-camera pose to end-effector kinematics. These objectives shape the shared representation used for action generation during training, and no future prediction is required at inference. Without robot pretraining, ForeAct3D achieves 98.3% average success on LIBERO and an average task length of 3.73 on CALVIN, outperforming its base policy on every suite. Ablations show that semantic 3D supervision, physical consistency, and action conditioning each improve manipulation performance, and that action conditioning substantially improves future object localization. Real-world experiments on spatial placement, object insertion, and sequential manipulation further raise average success from 6.7% to 37.8% over the base policy. The project page and code are available at https://github.com/anthonytao80-crypto/ForeAct3D.

Figures & tables

Figure 1: Future world modeling for robotic manipulation. (a) Predictive models guide control via inverse dynamics or planning. (b) VLA policies learn future representations jointly with actions. (c) ForeAct3D predicts physically constrained semantic 3D futures conditioned on the policy action. Dashed arrows denote supervision linking the predicted scene states to policy learning.

Figure 2: Overview of the proposed framework. ForeAct3D integrates semantic 3D prediction, physical-consistency closure, and action-conditioned future forecasting into a unified VLA policy for predicting task-relevant scene evolution under the planned action chunk.

One model per suite
Method	Venue, Year	Spatial	Object	Goal	Long	Avg.
DreamVLA ( Zhang et al., 2026a )	NeurIPS’25	97.5	94.0	89.5	89.5	92.6
QDepth-VLA ( Li et al., 2025 )	arXiv’26	97.6	96.6	95.2	90.0	94.9
UniVLA ( Bu et al., 2025 )	RSS’25	96.5	96.8	95.6	92.0	95.2
ST4VLA ( Ye et al., 2026 )	ICLR’26	98.0	99.0	93.8	92.6	95.9
Spatial Forcing ( Li et al., 2026 )	ICLR’26	99.4	99.6	98.8	96.0	98.5

Table 1: Manipulation success rates (%) on LIBERO. Results cover four task suites and their average. Methods are grouped by training protocol and sorted within groups by average success.

With robot pretraining
Method	Venue, Year	Avg. Len.	Task 1	Task 2	Task 3	Task 4	Task 5
Retrieval-OpenVLA ( Zhang et al., 2026b )	CVPR’26	3.53	92.7	79.6	70.5	58.3	51.9
Dita ( Hou et al., 2025 )	ICCV’25	3.61	94.5	82.5	72.8	61.3	50.0
GHIL-Glue ( Hatch et al., 2025 )	ICRA’25	3.69	95.2	88.5	72.3	62.9	49.8
ProgressVLA ( Yan et al., 2026 )	arXiv’26	3.73	95.2	84.8	73.6	67.2	52.0
MoDE ( Reuss et al., 2025 )	ICLR’25	4.01	96.2	88.9	81.1	71.8	63.5

Table 2: Long-horizon performance on CALVIN ABC

\rightarrow

D benchmark. Methods are grouped by training protocol and sorted by average rollout length. ∗ marks results reproduced by us.

Method	Horizon	Scene CD (m) $\downarrow$	Inst-CD (m) $\downarrow$	Trans (cm) $\downarrow$	Recall $\uparrow$
CLOVER	5	0.0195	0.0139	1.741	75.82
DreamVLA	3	0.1350	0.1455	20.61	10.73
ForeAct3D	8	0.0206	0.01371	1.245	92.78

Table 3: Geometry, semantics, and future localization. (a) CALVIN geometry; instance metrics use GT masks and horizons differ by method. (b) ForeAct3D semantic mIoU and localization at

t+T

. GT / Pred denotes GT / predicted semantic masks; both retain GT instance IDs. Recall is in percent. Details and protocol qualifications are in Appendix A.5 .

	Yellow area		Pen in mug	Long task
Configuration	Success	Reach	Success	Full	First stage
StarVLA-OFT	0.0 (0/15)	6.7 (1/15)	20.0 (3/15)	0.0 (0/15)	13.3 (2/15)
+ Semantic 3D	13.3 (2/15)	33.3 (5/15)	46.7 (7/15)	6.7 (1/15)	33.3 (5/15)
+ Physical consistency	13.3 (2/15)	40.0 (6/15)	60.0 (9/15)	6.7 (1/15)	40.0 (6/15)
+ Action conditioning (Ours)	33.3 (5/15)	33.3 (5/15)	60.0 (9/15)	20.0 (3/15)	40.0 (6/15)

Table 4: Real-world cumulative ablations. Rates (%) are followed by counts in parentheses, with 15 trials per configuration and task. Reach means the object entered the yellow area at least once; success requires it to remain there at termination. First stage records long-task mug placement.

Panel of Figure 3 — Figure 3: Scene prediction and policy execution in simulation. Examples show opening a drawer in LIBERO-Goal and turning off a light in CALVIN. Rows present current observations, predicted semantic 3D states, and execution results. Matching colored borders link corresponding observation and execution frames across successive steps of each manipulation sequence.

Benchmark	Configuration	Score $\uparrow$
LIBERO	StarVLA-OFT	96.6
	+ Semantic 3D prediction	97.3
	+ Physical-consistency closure	97.9
	+ Action conditioning	98.3
CALVIN	Full	3.73
	w/o kinematics	3.68

Table 5: Policy and prediction ablations. (a) Cumulative policy ablations on LIBERO and closure ablations on CALVIN. CALVIN variants retain semantic 3D prediction and action conditioning. Scores report LIBERO success (%, averaged over four suites) and mean CALVIN task length. (b) Future geometry with and without action conditioning. Detailed metrics appear in Appendix A.6 .

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

		Depth			Scene		GT instance regions
Method	State	AbsRel	RMSE	$\delta_{1}$	CD	EMD	Inst-CD	Trans	Recall
CLOVER	Future	0.0083	0.1252	0.9955	0.0195	0.0601	0.0139	1.741	0.7582
DreamVLA	Future	0.0867	0.5583	0.9126	0.1350	0.3145	0.1455	20.61	0.1073
ForeAct3D	Current	0.0110	0.1551	0.9942	0.0206	0.0631	0.0129	1.165	0.9416
ForeAct3D	Future	0.0117	0.1633	0.9933	0.0206	0.0623	0.01371	1.245	0.9278

Appendix

Table 6: Complete depth-derived geometry on CALVIN. Instance metrics use GT masks. Rows separate current and future states; future horizons are 5 (CLOVER), 3 (DreamVLA), and 8 (ForeAct3D). Depth RMSE, CD, and EMD use meters; translation uses centimeters; recall is a fraction.

Benchmark	Suite	mIoU@ $t$	BG@ $t+T$	Robot@ $t+T$	Object@ $t+T$	mIoU@ $t+T$
LIBERO	Object	98.0	99.4	95.8	98.0	97.7
	Goal	87.2	94.8	94.8	70.8	86.8
	Spatial	78.1	91.5	95.7	45.7	77.6
	Long	93.7	97.2	94.5	87.8	93.2
	Average	89.3	95.7	95.2	75.6	88.8
CALVIN	ABC $\rightarrow$ D	88.0	89.8	90.3	77.3	85.8

Appendix

Table 7: Three-class semantic segmentation quality (%). Future class IoUs are reported for background, robot, and object. All objects share the same semantic class. LIBERO Average denotes the mean across the Object, Goal, Spatial, and Long task suites.

		Trans (cm) $\downarrow$		Recall@2cm $\uparrow$
Benchmark	Suite	GT mask	Pred mask	GT mask	Pred mask
LIBERO	Object	0.56	0.48	96.7	96.9
	Goal	1.58	1.20	91.2	92.8
	Spatial	0.71	0.67	95.7	96.3
	Long	1.38	1.13	92.4	93.8
	Average	1.06	0.87	94.0	94.9

Appendix

Table 8: Future localization with predicted depth and semantic masks. GT instance IDs are retained in both mask settings. Recall@2cm is in percent. These results assess localization with predicted semantic support, not autonomous instance detection without GT IDs.

Metric	ForeAct3D	w/o Action Conditioning
AbsRel $\downarrow$	0.0117	0.0165
RMSE (m) $\downarrow$	0.1633	0.2144
$\delta_{1}$ $\uparrow$	0.9933	0.9881
Scene CD (m) $\downarrow$	0.0206	0.0220
Scene EMD (m) $\downarrow$	0.0623	0.0655
Inst-CD (m) $\downarrow$	0.0137	0.0183

Appendix

Table 9: Effect of action conditioning on future depth-derived geometry on CALVIN. Both variants use the same evaluation protocol and prediction horizon of

T=8

Suite	Variant	AbsRel $\downarrow$	RMSE $\downarrow$	$\delta_{1}$ $\uparrow$	Inst-CD $\downarrow$	Trans $\downarrow$	Recall $\uparrow$
			(m)		(m)	(cm)	@2cm
Goal	Full	0.0225	0.214	0.976	0.0101	1.58	0.912
Goal	w/o action	0.0562	0.315	0.945	0.0126	2.65	0.879
Object	Full	0.0078	0.067	0.994	0.0068	0.56	0.967
Object	w/o action	0.0388	0.216	0.958	0.0115	2.82	0.908
Spatial	Full	0.0229	0.230	0.973	0.0076	0.71	0.957

Appendix

Table 10: Action-conditioning results by LIBERO suite. Values are measured at the future prediction horizon under the same GT-mask protocol as Table 8 . Full denotes ForeAct3D with action conditioning; w/o action denotes the variant without it. The main-text instance metrics are equal-weight averages of the four suites, computed before rounding.

Explore similar work

Aug 27, 2026cs.RO

PHR-VLA: Planning Horizon Reasoning for Vision-Language-Action Models

Vision-language-action models (VLAs) have shown strong promise for general-purpose robotic manipulation by mapping language instructions and vision observations directly to actions. However, most VLAs primarily condition action prediction on current observations and lack an explicit mechanism for reasoning over future task dynamics, which is particularly important for fine-grained, contact-rich manipulation. We present PHR-VLA, a framework that enables planning-horizon reasoning in VLAs through privileged latent representations of future dynamics. PHR-VLA introduces a lightweight auxiliary future head that, during training, aligns the VLA's internal representations with latent dynamics extracted from future observations. Evaluation results demonstrate that local, contact-centric, patch-level latent dynamics supervision from the wrist camera improves success rate on LIBERO from 84.1% to 88.4% and on real-world disassembly tasks from 63.3% to 82.5%. Patch-level supervision from a third-person camera also improves performance on Meta-World from 56.70% to 57.8%. These results demonstrate that privileged latent dynamics alignment provides an effective training signal for improving anticipatory reasoning in VLA policies. Project website: https://davoodsz.github.io/PHR-VLA.github.io/

Davood Soleymanzadeh, Kaidi Zhang, Zhiyuan Zhang +4

J. Mike Walker ’66 Department of Mechanical Engineering, Texas A&M University, College Station, TX 77843, USA · Department of Industrial Engineering, Purdue University, West Lafayette, IN 47907, USA · Zachry Department of Civil and Environmental Engineering, Texas A&M University, College Station, TX 77843 USA

Vision-Language-Action Models Reasoning for VLA Models Robotic Manipulation Language-Conditioned Robot Manipulation

Ask PDF

Sep 12, 2026cs.RO

GeomVLA: Unifying Scene, Motion, and Action in 3D

We present GeomVLA, a Vision-Language-Action (VLA) model that unifies perception, latent scene motion prediction, and action generation within a shared robot-centric 3D coordinate frame. Our approach lifts pretrained VLM features into spatially grounded 3D scene tokens using depth and camera calibration, while retaining the semantic representations learned during VLM pretraining. We further introduce a 3D Scene Trajectory Denoiser, a task-conditioned module that learns a latent representation of how scene points are expected to move in 3D. Rather than executing the predicted trajectory as an open-loop plan, GeomVLA extracts intermediate motion tokens from the trajectory denoiser and uses them to condition a 3D flow-based action denoiser through geometry-aware attention. GeomVLA achieves state-of-the-art performance on CALVIN, competitive performance on LIBERO and RoboTwin2.0, and outperforms strong baselines in real-world manipulation settings without robot-action pretraining. Extensive ablations show that future-motion reasoning alone is insufficient: the primary gains are associated with maintaining geometric consistency among scene representation, motion prediction, and robot actions throughout the perception-to-action pipeline.

Ziyin Xiong, Nikolaos Gkanatsios, Moritz Reuss +1

Carnegie Mellon University · NVIDIA

Robotic Manipulation 3D Scene Representation Vision-Language-Action Models VLMs for Robotics

Ask PDF

May 20, 2026cs.RO

GaussianDream: A Feed-Forward 3D Gaussian World Model for Robotic Manipulation

Vision-language-action (VLA) policies have advanced language-conditioned robotic manipulation by transferring semantic priors from pretrained vision-language models to action generation. However, standard action-imitation learning often lacks sufficient modeling of explicit 3D spatial information, dense geometric supervision, and future environment evolution, all critical for precise robotic interaction. To address this, we propose \textbf{GaussianDream}, a feed-forward 3D Gaussian world-model plug-in. Specifically, we introduce learnable GaussianDream Queries in the encoder, enabling the model to capture current-frame 3D spatial structure and short-horizon future evolution. During training, the latent GaussianDream prefix is processed by a static reconstruction head and a future prediction head to produce current 3D Gaussian scene states and future Gaussian evolution states. The current branch is supervised by RGB rendering and depth, while the future branch uses future RGB, depth, and pseudo 3D scene-flow signals. During inference, GaussianDream discards all auxiliary heads and retains only the learned prefix to condition action generation, without test-time Gaussian reconstruction or future prediction. Experimental results demonstrate that GaussianDream achieves state-of-the-art performance across multiple robotic manipulation benchmarks, reaching \textbf{98.4%} on LIBERO, \textbf{54.8%} on RoboCasa Human-50, and \textbf{50.0%} on real-robot tasks. Compared with existing 3D-enhanced VLA methods, GaussianDream achieves strong accuracy while providing higher inference efficiency than video-based world-model approaches.

Zijian Zhang, Yuqing Jiang, Qian Cheng +6

Tuojing Intelligence · University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences +5

World Models for Robotics Robotic Manipulation Vision-Language-Action Models

Ask PDF