HiFOR: Hierarchical Foresight with Action-Guided World Grounding for Vision-Language-Action Model

Rongxing Ding; Hongyu Qu; Kehong Wu; Xudong Wang; Fan Zhang; Xinguang Xiang; Xiangbo Shu

cs.CVOct 29, 2026

HiFOR: Hierarchical Foresight with Action-Guided World Grounding for Vision-Language-Action Model

Authors: Rongxing Ding, Hongyu Qu, Kehong Wu, Xudong Wang, Fan Zhang, Xinguang Xiang, Xiangbo Shu

Organizations: Nanjing University of Science and Technology · National University of Singapore · Shenyang Institute of Automation, Chinese Academy of Sciences · The Chinese University of Hong Kong

Abstract

Vision-Language-Action (VLA) models have demonstrated promising capabilities in robotic manipulation by integrating visual perception, language understanding, and action generation. Recent advances have further incorporated future prediction to enhance the understanding of environmental dynamics and task progression. However, existing methods often model short-term dynamics and long-term task objectives separately, while directly incorporating predicted futures into action policies without sufficiently considering their relevance to current action requirements. To address these limitations, we propose HiFOR, a hierarchical foresight framework with action-guided world retrieval. Specifically, Hierarchical Foresight Prediction jointly predicts short-term state evolution, semantic milestones, and final goal states through a Hierarchical Dependency Mask (HDM), establishing dependencies across temporal horizons. Building upon these predictions, Structure-Aware Latent World Modeling organizes future latent representations along the horizon, view, and spatial dimensions to facilitate fine-grained retrieval. Furthermore, Action-Guided Factorized World Retrieval adaptively retrieves relevant future information from the structured latent world based on evolving action representations during denoising, providing targeted guidance for action generation. Extensive experiments on two simulation benchmarks and six real-world robotic manipulation tasks demonstrate the effectiveness of our approach.

Figures & tables

Figure 1: Comparison of Foresight VLA Models. (a) Existing methods typically adopt single-horizon foresight and directly feed predictions into VLA models without selective utilization. (b) In contrast, HiFOR combines multi-horizon foresight with action-guided retrieval of relevant future information.

Figure 2: The overview of HIFOR. (a) We augment the VLA with hierarchical foresight prediction and action-guided world retrieval, enabling the Action Expert to leverage future information during action generation. (b) Hierarchical Foresight Prediction (§ 3.4 ) predicts short-term, milestone, and goal futures through hierarchical dependency modeling. (c) Structure-Aware Latent World Modeling (§ 3.5 ) organizes future latent representations along the horizon, view, and spatial dimensions. (d) Action-Guided Factorized World Retrieval (§ 3.6 ) selectively retrieves action-relevant future information through denoising-aware factorized attention to guide action generation.

Figure 3: Hierarchical Foresight Planning Data Construction. (a) A planning agent generates semantic milestones from task instructions. (b) Automatic milestone localization identifies their completion timestamps in expert demonstrations using event evidence and temporal consistency checks.

	RoboTwin 2.0 (Hard)
	Adjust	Blocks	Click	Dump Bin	Move	Open	Place Burger	Place Can	Place Cans	Stack Blocks
Method	Bottle	Ranking RGB	Bell	Bigbin	Playingcard	Laptop	Fries	Basket	Plasticbox	Two	Avg.
RDT [ 25 ]	75%	0%	9%	32%	11%	32%	27%	6%	5%	2%	14.60%
ACT [ 55 ]	23%	0%	3%	1%	0%	0%	0%	0%	0%	0%	2.90%
DP3 [ 51 ]	3%	0%	0%	53%	3%	7%	18%	2%	3%	0%	4.85%
EventVLA [ 46 ]	24%	6%	48%	64%	17%	23%	7%	7%	14%	13%	19.00%

Table 1: Performance comparison on RoboTwin 2.0 [ 10 ] . We report per-task success rates (SR) over 100 rollouts under the Hard setting.

\dagger

denotes reproduced results. Results for all 20 tasks are provided in the Appendix.

Method	Publication	Spatial	Object	Goal	Long	Avg.
Diffusion Policy [ 11 ]	IJRR’25	78.5	87.5	73.5	64.8	76.1
GR00T-N1 [ 1 ]	arXiv’25	94.4	97.6	93.0	90.6	93.9
OpenVLA [ 19 ]	arXiv’24	84.7	88.4	79.2	53.7	76.5
OpenVLA-OFT [ 17 ]	arXiv’25	97.6	98.4	97.9	94.5	97.1
MemoryVLA [ 34 ]	ICLR’26	98.4	98.4	96.4	93.4	96.7
UniVLA [ 5 ]	arXiv’25	95.4	98.8	95.8	85.2	94.2

Table 2: Success rates (%) on the LIBERO benchmark . Best results are shown in bold , and second-best results are underlined .

Figure 4: Real-world evaluation results .We evaluate HIFOR against leading models on the Generalization and Long-horizon task suites.

Method						LIBERO
Short	Milestone	Goal	AWR	Spatial	Object	Goal	Long	Avg.
				98.8	98.2	98.0	92.4	40.35
$\checkmark$				99.0	98.4	98.0	93.4	44.15
$\checkmark$	$\checkmark$			99.2	99.0	98.0	94.2	45.60
$\checkmark$	$\checkmark$	$\checkmark$		99.0	98.8	98.2	95.0	46.95
$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	99.4	99.2	98.6	95.8	49.40

Table 3: Ablation analysis of the core components in terms of Success Rate (%) on LIBERO [ 24 ] and RoboTwin 2.0 [ 10 ] .

Figure 5: Analysis of the Effects of Different Short-Horizon Prediction Lengths on LIBERO. [ 24 ] .

Figure 6: Analysis of different future prediction representations on LIBERO [ 24 ] .

Figure 7: The Visualization of Action-Guided Factorized World Retrieval Across Task Stages.

Vision-Language-Action Models

Vision-language-action models turn visual observations and natural-language instructions into actions for robots and other embodied agents.

See all 1,194 papers

PHR-VLA: Planning Horizon Reasoning for Vision-Language-Action Models
Aug 27, 2026Davood Soleymanzadeh, Kaidi Zhang, Zhiyuan Zhang +4Vision-Language-Action Models Reasoning for VLA Models
J. Mike Walker ’66 Department of Mechanical Engineering, Texas A&M University, College Station, TX 77843, USA · Department of Industrial Engineering, Purdue University, West Lafayette, IN 47907, USA · Zachry Department of Civil and Environmental Engineering, Texas A&M University, College Station, TX 77843 USA
PLaCA: Learning Patch-Level Future Latent Change for Action Generation
Oct 20, 2026Zhan Yi, Yuchen Wang, Chao Li +2Temporal Modeling for VLA Models World Models for VLA Models
School of Robotics and Automation, Nanjing University, Suzhou, Jiangsu, China
FutureVLA: Joint Visuomotor Prediction for Vision-Language-Action Model
Mar 11, 2026Xiaoxu Xu, Hao Li, Jinhui Ye +7Vision-Language-Action Models Visuomotor Policy Learning
Beihang University. · Shanghai Artificial Intelligence Laboratory. · University of Science and Technology of China. +1

Aug 27, 2026J/K move · Enter open · S save

PHR-VLA: Planning Horizon Reasoning for Vision-Language-Action Models

Davood Soleymanzadeh, Kaidi Zhang, Zhiyuan Zhang, Bihao Zhang+3

J. Mike Walker ’66 Department of Mechanical Engineering, Texas A&M University, College Station, TX 77843, USA · Department of Industrial Engineering, Purdue University, West Lafayette, IN 47907, USA · Zachry Department of Civil and Environmental Engineering, Texas A&M University, College Station, TX 77843 USA

Vision-Language-Action Models Reasoning for VLA Models Robotic Manipulation Language-Conditioned Robot Manipulation

Open paper AskPDF arXiv

Vision-language-action models (VLAs) have shown strong promise for general-purpose robotic manipulation by mapping language instructions and vision observations directly to actions. However, most VLAs primarily condition action prediction on current observations and lack an explicit mechanism for reasoning over future task dynamics, which is particularly important for fine-grained, contact-rich manipulation. We present PHR-VLA, a framework that enables planning-horizon reasoning in VLAs through privileged latent representations of future dynamics. PHR-VLA introduces a lightweight auxiliary future head that, during training, aligns the VLA's internal representations with latent dynamics extracted from future observations. Evaluation results demonstrate that local, contact-centric, patch-level latent dynamics supervision from the wrist camera improves success rate on LIBERO from 84.1% to 88.4% and on real-world disassembly tasks from 63.3% to 82.5%. Patch-level supervision from a third-person camera also improves performance on Meta-World from 56.70% to 57.8%. These results demonstrate that privileged latent dynamics alignment provides an effective training signal for improving anticipatory reasoning in VLA policies. Project website: https://davoodsz.github.io/PHR-VLA.github.io/

Abstract

Vision-Language-Action Models

Explore similar work

PHR-VLA: Planning Horizon Reasoning for Vision-Language-Action Models

PLaCA: Learning Patch-Level Future Latent Change for Action Generation

FutureVLA: Joint Visuomotor Prediction for Vision-Language-Action Model