H-VLA: Hierarchical Vision-Language-Action Model with Key-Action Reasoning and Motion Planning in a Unified Action Space

Xiongfeng Peng; Lu Xu; Yandong Wang; Jiaqian Yu; Zirui Zheng; Yamin Mao; Weiming Li; Inseop Chung

cs.ROSep 19, 2026

H-VLA: Hierarchical Vision-Language-Action Model with Key-Action Reasoning and Motion Planning in a Unified Action Space

Authors: Xiongfeng Peng, Lu Xu, Yandong Wang, Jiaqian Yu, Zirui Zheng, Yamin Mao, Weiming Li, Inseop Chung, +5 more

Organizations: Advanced Research Lab, Samsung R&D Institute China-Beijing (SRCB), China · Samsung AI Center, DS Division, South Korea

Abstract

Vision-Language-Action (VLA) models have shown strong potential for robotic manipulation, but many existing methods still rely on direct mappings from language and visual observations to dense actions. This formulation can weaken the semantic reasoning capability inherited from pre-trained Vision-Language Models (VLMs), which are mainly optimized for visual-linguistic understanding rather than low-level control, and becomes fragile under spatial variations, including changes in object positions, scene layouts, robot embodiments, and camera viewpoints. To address these limitations, we propose H-VLA, a hierarchical VLA framework that decouples high-level key-action reasoning from low-level motion generation. H-VLA combines a Key-Action Model for predicting a key-action as the next manipulation subgoal, a Motion Planning Model for generating dense future actions conditioned on the predicted key-action, and a Unified Camera-Centric Action Space for consistent representation across datasets, embodiments, and viewpoints. We further adopt a two-stage training strategy that emphasizes key-action reasoning during pre-training and dense motion generation during fine-tuning. Experiments show that H-VLA achieves strong performance on SimplerEnv, reaching 91% on Google Robot visual matching, 84% on Google Robot variant aggregation, and 81% on WidowX visual matching. On Agilex real-robot tasks, H-VLA improves over the strongest baseline by 10, 47, and 16 percentage points under in-distribution, out-of-distribution position, and out-of-distribution scene/object settings, respectively.

Figures & tables

Figure 1: H-VLA: A hierarchical vision-language-action framework for cross-embodiment robot manipulation. (a) Given third-person RGB observations, optional wrist-view observations, language instructions, and current end-effector states, H-VLA first predicts a structured manipulation key-action through a Key-Action Model and then generates future frame-by-frame actions through a Motion Planning Model. A unified camera-centric action space standardizes representations across heterogeneous embodiments and datasets. (b) H-VLA employs a two-stage training strategy, including Key-Action-Centric Pre-training and Motion-Planning-Oriented Fine-tuning. (c) H-VLA achieves strong performance across simulation and real-robot benchmarks.

Figure 2: Detailed architecture of H-VLA using the dual-arm task “tighten the cap of the bottle” as an example. Given third-person RGB observations, wrist-view observations, language instructions, and current end-effector (EEF) states, a pre-trained VLM extracts the cognition feature from a learnable action token and global visual features from the class (CLS) token. Wrist-view observations are encoded by the vision encoder shared with the VLM, and the availability of left and right wrist cameras is indicated by a wrist-camera mask. The Key-Action Model predicts a structured camera-frame end-effector key-action representing the next manipulation subgoal, while the Motion Planning Model generates dense future action chunks in the same camera-centric action space conditioned on the predicted key-action, cognition feature, global visual feature, wrist-view visual features, and current end-effector state.

	Google Robot (VM)					Google Robot (VA)					WidowX (VM)
Policy	Coke Can	Move Near	Open/ Close	Drawer+ Apple	Avg	Coke Can	Move Near	Open/ Close	Drawer+ Apple	Avg	Spoon Towel	Carrot Plate	Stack	Eggplant Basket	Avg
† RT-1-X [ 37 ] (PT)	59%	33%	56%	17%	41%	49%	33%	29%	10%	30%	4%	8%	0%	0%	3%
‡ RT-2-X [ 37 ] (PT)	79%	78%	25%	4%	46%	82%	79%	35%	21%	54%	–	–	–	–	–
† Octo [ 44 ] (PT)	18%	4%	24%	0%	11%	1%	4%	1%	0%	1%	4%	8%	0%	38%	13%
† OpenVLA [ 23 ] (PT)	14%	51%	48%	0%	28%	64%	64%	19%	1%	37%	4%	0%	0%	4%	2%
‡ EMMA-X [ 43 ] (PT)	2%	3%	18%	–	–	5%	7%	21%	–	–	–	–	–	–	–

Table 1: Task-wise SimplerEnv success rates. For Google Robot, we report Visual Matching (VM) and Variant Aggregation (VA); for WidowX, we report VM. The best result in each column is shown in bold. Methods with missing task results are marked as – and are not compared on the corresponding average. Task names are abbreviated. PT: pre-training only; FT: fine-tuning after robot-policy pre-training; T: downstream training without robot-data pre-training.

\pi_{0}^{*}

refers to open-pi-zero. † : evaluated by us; ‡ : reported in prior work.

	ID				OOD Position				OOD Scene/Object
Policy	PickBottles	HandOver	LiftPot	Avg	PickBottles	HandOver	LiftPot	Avg	PickBottles	HandOver	LiftPot	Avg
CogACT [ 27 ]	47%	53%	50%	50%	20%	33%	20%	24%	–	–	–	–
MolmoAct2 [ 13 ]	80%	60%	70%	70%	53%	13%	40%	36%	–	–	–	–
$\pi_{0}$ [ 3 ]	60%	100%	90%	83%	33%	30%	30%	31%	37%	23%	72%	44%
$\pi_{0.5}$ [ 20 ]	85%	80%	80%	82%	23%	37%	33%	31%	47%	37%	67%	50%
H-VLA (Ours)	100%	90%	90%	93%	77%	93%	80%	83%	73%	63%	61%	66%

Table 2: Real-robot success rates on the Agilex dual-arm benchmark under in-distribution (ID), out-of-distribution (OOD) position shifts, and out-of-distribution (OOD) scene/object shifts.

	Model Size				Pretraining	Inference
Policy	VLM	KAM	AM/MPM	Total	Robot Data	Precision	VLM	KAM	AM/MPM	Total	Chunk
$\pi_{0}$ [ 3 ]	3.0B	–	300M	3.3B	$>1000$ K	BF16 (mixed)	49 ms	–	22 ms	73 ms	50
$\pi_{0.5}$ [ 20 ]	3.0B	–	300M	3.3B	$>1000$ K	BF16 (mixed)	48 ms	–	24 ms	73 ms	50
MolmoAct2 [ 13 ]	4.9B	–	621M	5.5B	$>300$ K	BF16	90 ms	–	320 ms	420 ms	30
CogACT [ 27 ]	7.5B	–	89M	7.6B	$\geq 400$ K	BF16	57 ms	–	18 ms	76 ms	16
H-VLA (Ours)	7.5B	89M	89M	7.7B	183K	BF16	54 ms	17 ms	17 ms	103 ms	16

Table 3: Model size, robot pre-training data, and RTX 4090 inference latency. Robot data are reported in trajectories (K: thousands);

>

and

\geq

denote conservative estimates or lower bounds. KAM: Key-Action Model; AM/MPM: Action Model/Motion Planning Model. Total denotes end-to-end latency; for H-VLA, it also includes left/right wrist-RGB encoding that is not included in the VLM/KAM/MPM module times. Chunk is the training action-chunk length and the maximum per-query inference horizon.

	Google Robot (VM)					Google Robot (VA)					WidowX (VM)
Variant	Coke Can	Move Near	Open/ Close	Drawer+ Apple	Avg	Coke Can	Move Near	Open/ Close	Drawer+ Apple	Avg	Spoon Towel	Carrot Plate	Stack	Eggplant Basket	Avg
H-VLA (w/o Stage-2 FT)	75%	68%	74%	73%	73%	77%	36%	63%	21%	49%	50%	63%	13%	88%	53%
H-VLA	98%	90%	80%	98%	91%	98%	85%	79%	72%	84%	96%	88%	58%	83%	81%
w/o MPT+TST	98%	91%	72%	80%	85%	94%	84%	83%	63%	81%	71%	83%	42%	100%	74%
w/o MPT+TST+CC	91%	72%	69%	38%	68%	93%	70%	39%	33%	59%	71%	63%	42%	96%	68%
w/o MPT+TST+CC+KA	95%	59%	53%	18%	56%	93%	68%	45%	34%	60%	58%	54%	13%	92%	54%

Table 4: Task-wise ablation results on SimplerEnv [ 29 ] under the SimplerEnv evaluation setup. MPT denotes mixed single-/dual-arm pre-training, TST denotes two-stage training, CC denotes the unified camera-centric action space, and KA denotes the Key-Action Model.

Figure 3: Qualitative visualization of predicted key-actions on five representative single-arm and dual-arm simulation and real-robot tasks. H-VLA predicts task-relevant key-actions for diverse manipulation primitives, including picking, opening, placing, and closing.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Episodes	Tasks	Embodiment	Arm Type	3rd-Person	Wrist	Used Episodes
RoboCOIN Cobot Magic	20k	54	Agilex Robot	Dual-Arm	Yes	Yes	14k
DROID	76k	86	Franka Robot	Single-Arm	Yes	Yes	22k
BridgeDataV2	60k	13 Skills	WidowX Robot	Single-Arm	Yes	No	60k
FractalData	87k	700+	Google Robot	Single-Arm	Yes	No	87k
Agilex Real Data	150	3	Agilex Robot	Dual-Arm	Yes	Yes	150

Appendix

Table 5: Dataset statistics used in H-VLA. The 3rd-Person and Wrist columns indicate the visual streams used by H-VLA after dataset preprocessing.

Dataset	Camera View / Pose	Intrinsics / Extrinsics	Calibration Method
RoboCOIN Cobot Magic	Task-dependent	Orbbec factory intrinsics / Self-calibrated extrinsics	Task-specific PnP self-calibration using FK-derived gripper poses and image annotations
DROID	Task-dependent	Provided / Provided	Directly obtained from DROID calibration files
BridgeDataV2	Single pose	SimplerEnv / SimplerEnv	SimplerEnv reference calibration; several tasks further corrected using ECoT gripper markers
FractalData	Single pose	SimplerEnv / SimplerEnv	Directly obtained from SimplerEnv
Agilex Real Data	Fixed pose	Orbbec factory intrinsics / Self-calibrated extrinsics	PnP self-calibration using FK-derived gripper poses and image annotations

Appendix

Table 6: Camera configurations and calibration sources used by H-VLA after preprocessing.

Figure 4: Ground-truth key-action visualization on RoboCOIN Cobot Magic. Green dashed boxes indicate correctly identified key-actions, while red dashed boxes and red arrows indicate failure cases.

Figure 5: Ground-truth key-action visualization on DROID. Green dashed boxes indicate correctly identified key-actions, while red dashed boxes and red arrows indicate failure cases.

Figure 6: Ground-truth key-action visualization on BridgeDataV2. Green dashed boxes indicate correctly identified key-actions, while red dashed boxes and red arrows indicate failure cases.

Figure 7: Ground-truth key-action visualization on FractalData. Green dashed boxes indicate correctly identified key-actions, while red dashed boxes and red arrows indicate failure cases.

Figure 8: Ground-truth key-action visualization on Agilex Real Data. Green dashed boxes indicate correctly identified key-actions, while red dashed boxes and red arrows indicate failure cases.

Short Name	Full Task Name	VM	VA
Coke Can	Pick Coke Can	300	825
Move Near	Move Near	240	600
Open/Close	Open/Close Drawer	216	378
Drawer+Apple	Open Top Drawer and Place Apple	108	189
Spoon Towel	Put Spoon on Towel	24	–
Carrot Plate	Put Carrot on Plate	24	–

Appendix

Table 7: Task-name mapping and number of evaluation trials in the SimplerEnv benchmark sweep used for our evaluations. Google Robot tasks are evaluated under both VM and VA settings, while WidowX Robot tasks are evaluated under VM only.

Task	SimplerEnv Success Stop	X-VLA Success Stop	SimplerEnv Steps	X-VLA Steps	SimplerEnv Trials	X-VLA Trials
Coke Can	No	Yes	80	$80\times 2$	VM(300)+VA(825)	VM(300)+VA(825)
Move Near	No	Yes	80	$80\times 2$	VM(240)+VA(600)	VM(240)+VA(600)
Open/Close	No	Yes	113	$113\times 2$	VM(216)+VA(378)	VM(216)+VA(210)
Drawer+Apple	No	Yes	200	$200\times 2$	VM(108)+VA(189)	VM(108)+VA(189)

Appendix

Table 8: Comparison between the SimplerEnv evaluation setup and the X-VLA-modified configuration on Google Robot tasks. X-VLA enables success-triggered early termination, doubles the maximum episode horizon, and changes the Open/Close Drawer VA trial count from 378 to 210.

Task	SimplerEnv Success Stop	X-VLA Success Stop	SimplerEnv Steps	X-VLA Steps	Camera Transform
Spoon Towel	No	Yes	60	1200	Default
Carrot Plate	No	Yes	60	1200	Default
Stack	No	Yes	60	1200	Default
Eggplant Basket	No	Yes	60	1200	Modified by X-VLA

Appendix

Table 9: Comparison between the SimplerEnv evaluation setup and the X-VLA-modified configuration on WidowX Robot tasks. X-VLA enables success-triggered early termination, increases the maximum episode horizon from 60 to 1200 steps, and modifies the robot-base-to-camera transform for Eggplant Basket.

Figure 9: Key-action visualization on Google Robot. We select one short-horizon task and one long-horizon task to illustrate the predicted key-actions at different task phases.

Figure 10: Key-action visualization on WidowX Robot. The cube stacking task requires high precision and fine-grained spatial reasoning.

Component	Model / Configuration	Description
Robot platform	Agilex Cobot Magic-style platform	Dual-arm tabletop manipulation setup
Manipulation arms	Agilex PiPER arms	Two 6-DoF follower arms for policy execution
Teleoperation arms	Agilex PiPER arms	Two leader arms for demonstration collection
End-effectors	Parallel grippers	Used for grasping, handover, and lifting
Third-person camera	Orbbec-DABAI RGB-D camera	Global workspace observation
Wrist cameras	Two Orbbec-DABAI RGB-D cameras	Local left/right arm observations

Appendix

Table 10: Hardware configuration of the Agilex real-robot evaluation platform.

Figure 11: Agilex dual-arm robot platform used in the real-world experiments. The setup consists of two follower robot arms, two leader arms for teleoperation, one third-person RGB camera, two wrist-mounted RGB cameras, and a tabletop manipulation workspace.

Models	Setting	PickBottles	HandOver	LiftPot	Total
H-VLA / $\pi_{0}$ / $\pi_{0.5}$	ID	20	20	20	60
	OOD Position	30	30	30	90
	OOD Scene/Object	30	30	18	78
CogACT / MolmoAct2	ID	15	15	10	40
	OOD Position	15	15	10	40
	OOD Scene/Object	–	–	–	–

Appendix

Table 11: Number of real-robot evaluation trials per model and setting.

OOD Position

OOD Scene/Object

Policy

PickBottles

HandOver

LiftPot

Avg

PickBottles

HandOver

LiftPot

Avg

PickBottles

HandOver

LiftPot

Avg

CogACT [ 27 ]

7/15

8/15

5/10

50%

3/15

5/15

2/10

24%

–

MolmoAct2 [ 13 ]

12/15

9/15

7/10

70%

8/15

2/15

4/10

36%

–

\pi_{0}

[ 3 ]

12/20

20/20

18/20

83%

10/30

9/30

31%

11/30

7/30

13/18

44%

\pi_{0.5}

[ 20 ]

17/20

16/20

82%

7/30

11/30

10/30

31%

14/30

11/30

12/18

50%

H-VLA

20/20

18/20

93%

23/30

28/30

24/30

83%

22/30

19/30

11/18

66%

Appendix

Table 12: Success counts on the Agilex real-robot benchmark. Each entry reports successful trials over total trials. The average is the mean of the three task success rates, matching the main paper. Dashes indicate settings that were not evaluated. The best result in each column is shown in bold.

Figure 12: Key-action visualization on the Agilex robot. In the examples shown, the key-actions predicted for both the left and right arms are consistent with the intended manipulation stages.

Explore similar work

PHR-VLA: Planning Horizon Reasoning for Vision-Language-Action Models
Aug 27, 2026Davood Soleymanzadeh, Kaidi Zhang, Zhiyuan Zhang +4Vision-Language-Action Models Reasoning for VLA Models
J. Mike Walker ’66 Department of Mechanical Engineering, Texas A&M University, College Station, TX 77843, USA · Department of Industrial Engineering, Purdue University, West Lafayette, IN 47907, USA · Zachry Department of Civil and Environmental Engineering, Texas A&M University, College Station, TX 77843 USA
ThinkingVLA: Interleaved Vision and Language Reasoning for Robotic Manipulation
Jun 16, 2026Tianyi Lu, Hui Zhang, Zijie Diao +8Long-Horizon Robotic Manipulation Language-Conditioned Robot Manipulation
Fudan University · Shanghai Innovation Institue · China Unicom +1
In-Context VLA: Endowing Vision-Language-Action Models with Language via In-Context Post-Training and Agentic Tool Use
Aug 6, 2026Jiarui Yang, Wen Huang, Jiale Zhang +2ICL for Robotics Continual Learning for VLA Models
Nankai University · Tsinghua University · Swiss Federal Institute of Technology in Lausanne (EPFL)

Aug 27, 2026, cs.ROJ/K move · Enter open · S save

PHR-VLA: Planning Horizon Reasoning for Vision-Language-Action Models

Davood Soleymanzadeh, Kaidi Zhang, Zhiyuan Zhang, Bihao Zhang+3

J. Mike Walker ’66 Department of Mechanical Engineering, Texas A&M University, College Station, TX 77843, USA · Department of Industrial Engineering, Purdue University, West Lafayette, IN 47907, USA · Zachry Department of Civil and Environmental Engineering, Texas A&M University, College Station, TX 77843 USA

Vision-Language-Action Models Reasoning for VLA Models Robotic Manipulation Language-Conditioned Robot Manipulation

Open paper AskPDF arXiv

Vision-language-action models (VLAs) have shown strong promise for general-purpose robotic manipulation by mapping language instructions and vision observations directly to actions. However, most VLAs primarily condition action prediction on current observations and lack an explicit mechanism for reasoning over future task dynamics, which is particularly important for fine-grained, contact-rich manipulation. We present PHR-VLA, a framework that enables planning-horizon reasoning in VLAs through privileged latent representations of future dynamics. PHR-VLA introduces a lightweight auxiliary future head that, during training, aligns the VLA's internal representations with latent dynamics extracted from future observations. Evaluation results demonstrate that local, contact-centric, patch-level latent dynamics supervision from the wrist camera improves success rate on LIBERO from 84.1% to 88.4% and on real-world disassembly tasks from 63.3% to 82.5%. Patch-level supervision from a third-person camera also improves performance on Meta-World from 56.70% to 57.8%. These results demonstrate that privileged latent dynamics alignment provides an effective training signal for improving anticipatory reasoning in VLA policies. Project website: https://davoodsz.github.io/PHR-VLA.github.io/

Abstract

Explore similar work

PHR-VLA: Planning Horizon Reasoning for Vision-Language-Action Models

ThinkingVLA: Interleaved Vision and Language Reasoning for Robotic Manipulation

In-Context VLA: Endowing Vision-Language-Action Models with Language via In-Context Post-Training and Agentic Tool Use