Fast Plans, Faithful Actions: Closing the Planning-Execution Gap in Hierarchical Vision-Language-Action Models

Chuanliang Xie; Boyu Ma; Gen Li; Yizhou Liu; Houwang Chen; Xinyu Zhou; Jianfei Yang

cs.ROSep 25, 2026

Fast Plans, Faithful Actions: Closing the Planning-Execution Gap in Hierarchical Vision-Language-Action Models

Authors: Chuanliang Xie, Boyu Ma, Gen Li, Yizhou Liu, Houwang Chen, Xinyu Zhou, Jianfei Yang

Organizations: MARS Lab, Nanyang Technological University

Abstract

Hierarchical vision-language-action (VLA) systems consist of a high-level vision-language planner and a low-level action expert that generates continuous actions. This hierarchical design has practical value only if the planner can generate plans fast enough to meet real-time control requirements, and the resulting plans actually contribute to the generation of action. We study one such system, a waypoint hierarchy pipeline adapted from $π_{0.5}$ , and find that neither requirement is satisfied. This baseline relies on token-level autoregressive decoding (Token-AR) to generate a waypoint plan, requiring 57 very expensive vision-language model (VLM) forward passes. However, we find that erasing the waypoint endpoints has little effect on task success. Two findings reveal the misalignment of planner-executor: the planner generates outputs at an excessively fine granularity, and the executor underuses plans as a control condition. We address the latency issue with waypoint-aligned block-autoregressive decoding (Block-AR), and plan underuse issue with normalized goal modulation (NGM), a layer-wise goal path constrained by phase gating and anti-shortcut training so that the waypoint influences action generation maintaining other signals. Our method reduces the maximum number of VLM forward passes from 57 to 8 on LIBERO, including one prefix prefill, and achieves an $8.7\times$ reduction in planning latency on a Rokae dual-arm robot. With normalized goal modulation and anti-shortcut training, Block-AR's success rate on LIBERO-Long increases from 91.0% to 96.2%, while its average success rate across the four suites increases from 95.85% to 98.45%. On three bimanual tasks with this robot, success rates remain comparable across methods.

Figures & tables

Figure 1: Fast plans, faithful actions. (a) Inputs (Rokae AR5-5 dual arm, pepper–banana placement): external and wrist images, instruction $\ell$ , state $s$ . (b) A PaliGemma planner (Gemma-2B language model) predicts $(q,g,d)$ waypoints (target configuration, gripper, duration). The schematic shows four waypoints per arm (hollow rings; orange dashed rings: grasp/release), illustrative per-arm step counts (linked pills), and corresponding action beads. Deployed dual-arm blocks share one $d$ ; $d=0$ ends the plan. A 300M flow-matching expert, which also attends to the planner’s image and language prefix (dotted), maps each waypoint to $d$ actions; the robot executes, observes, and replans (dashed loop). (c) For one seven-waypoint LIBERO plan, Token-AR needs one prefix prefill plus one backbone pass per value token (dots), 57 passes in total; Block-AR needs the same prefill plus one pass per waypoint (pills), 8 in total. On the robot, plan latency falls from 1094 to 125 ms and the arms move 66% instead of 29% of the time (bars). (d) Endpoint-erasure test on LIBERO-Long: with images, instruction, state, and flow noise fixed, the commanded endpoint is set to the segment start. Erasure does not reduce success with suffix-token conditioning ( $U=0$ ), but reduces success by 7.4 points with normalized goal modulation (NGM), which injects the goal into every expert layer.

Method	Plan space	Executor	Serial cost grows with	Interface assessment
HAMSTER ( Li et al., 2025 )	2-D image path	separate policy	path points	failure attribution
3D HAMSTER ( Hwang et al., 2026 )	pixel + depth path	separate flow policy	path points	guidance ablation
GAE ( Liu et al., 2025 )	camera-frame 3-D points	frozen point-cloud expert	waypoints	goal-noise ablation
NoTVLA ( Huang et al., 2025 )	pixel + depth keyposes	spline + replanning	keyposes	deterministic execution
Coarse-to-Control ( Wu et al., 2026 )	coarse action blocks	same model, AR tokens	blocks + actions	qualitative diagnostics
Ours	robot configuration + duration	flow expert, shared prefix	value slots (Token-AR) / waypoints (Block-AR)	$S$ , $U$

Table 1: Planner-to-executor designs in hierarchical VLAs.

Figure 2: Method. (a) Waypoint representation: a LIBERO waypoint is the token group <wp>

\,q_{1}\cdots q_{6}\,g\,

<dur>

\,d

with

d\in\{1,\dots,32\}

(338 reserved token IDs; a robot waypoint has 19 tokens). For block

i

the expert denoises

D=32

actions from the measured state

s_{i}

and executes the first

d_{i}

(green); the next block starts from the measured state. (b) Block-AR: attention is bidirectional within a block (dark) and causal across blocks (light); inputs are shifted by one block (the first is a learned query, dashed) and targets are not, so one backbone pass yields a whole waypoint through three LM-head calls (configuration, gripper, duration). Decoding produces at most

M=7

blocks and stops early when a block predicts

d=0

. (c) NGM: besides the suffix tokens

s_{i},q_{i},g_{i},d_{i}

, the normalized goal

[\widehat{\Delta};e(g_{i})]

with

\widehat{\Delta}=(q_{i}-s_{i})/\sigma_{\Delta}

enters an MLP with a zero-initialized output projection. Its residual

r

, gated by

g(t)

, is added to the AdaRMS condition

\phi(t)+g(t)r

of every layer; the gate is open for

t\geq 0.5

and closed for

t\leq 0.2

. Training noises the endpoint seen by both routes (

\sigma_{c}=0.7

) and replaces both by null embeddings with probability 0.15; inference uses the clean endpoint and the gate only.

Figure 3: Waypoints extracted from demonstrations. (a) Pepper-swap episode 2: 634 steps (21 s) become 45 waypoints. The top row shows the start, four gripper transitions, and end. The curves show the 14 joint angles (left arm above, right arm below; centered with a shared scale); vertical cuts mark waypoints, and the blocks below encode segments $(q,g,d)$ with width proportional to duration $d$ . Brackets identify the three segments capped at $D=32$ steps; rings mark gripper transitions (g = grasp, r = release; left, right, right, left arm). The orange $d=0$ block marks the end of the plan. (b) Segment-duration distributions on a shared scale: robot (6,158 training segments; median 12, 5.8% at the cap) and LIBERO-Long (13,815 segments; median 7, longest 29).

Method	Spatial	Object	Goal	Long	Avg.	$\Delta$ vs $\pi_{0.5}$	$\Delta$ vs Token-AR
$\pi_{0.5}$ (released, as reported)	98.8	98.2	98.0	92.4	96.85	–	–
Ours, Token-AR + suffix cond.	98.0	99.0	97.6	92.2	96.70	$-0.15$	–
Ours, Block-AR + suffix cond.	98.6	97.6	96.2	91.0	95.85	$-1.00$	$-0.85$
Ours, Block-AR + NGM	99.2	99.4	99.0	96.2	98.45	$+1.60$	$+1.75$

Table 2: Success rate (%) on the four LIBERO suites.

Figure 4: Robot planning latency and motion duty cycle. (a) Recorded shelf-placement timelines for Token-AR and Block-AR on a common time axis. Waiting comprises planner, expert, and transport; motion denotes commanded execution at 30 Hz. Motion occupies 24% and 59% here, versus 29% and 66% over 35 telemetry episodes (Table 3 ). Both episodes were stopped before placement. (b) Mean request and commanded-motion durations (497 Token-AR / 936 Block-AR requests). Planner latency falls from 1094 to 125 ms (

8.7\times

); expert latency remains near 160 ms. Transport is the hairline segment. The request round trip drops from 1266 to 293 ms (

4.3\times

), below the roughly 0.5 s of commanded motion.

	Token-AR	Block-AR
Serial passes / plan, max (LIBERO / robot)	57 / 120	8 / 8
Plan latency, robot (ms)	1094	125
Request round trip, robot (ms)	1266	293
Motion duty cycle, robot	28.7%	65.6%

Table 3: Planning cost (A) and waypoint statistics (B).

Figure 5: Representative dual-arm sequences from Block-AR telemetry rollouts (external camera, seconds since start); success rates are reported separately in Table 5 . (a) Pepper–banana placement. (b) Pepper swap. (c) Shelf placement, stopped at 34 s with the red block carried to the shelf but not placed.

System	$B$	$S_{\text{endpoint}}$ (%)	$S_{\text{duration}}$ (%)	$S_{\text{image}}$	Retention	SR ${}_{\text{intact}}$ (%)	$U$ (pp)
Block-AR + suffix	0.118	0.6	19	0.330	100%	91.0	0.0
Token-AR + naive	0.074	348	106	0.090	27%	15	+14
Block-AR + NGM	0.105	42.6	26.4	0.27	82%	96.2	+7.4

Table 4: Waypoint dependence and visual responsiveness on LIBERO-Long.

Task	Token-AR	Block-AR	+NGM	Plan (ms)	Duty (%)
Pepper–banana	17/20	16/20	18/20	1144/129	31/62
Pepper swap	16/20	15/20	17/20	1055/123	30/68
Shelf placement	15/20	15/20	16/20	1085/122	23/65
All	48/60	46/60	51/60	1094/125	29/66

Table 5: Real-robot results.

Vision-Language-Action Models

Vision-language-action models turn visual observations and natural-language instructions into actions for robots and other embodied agents.

See all 1,190 papers

Coarse-to-Control: Action-Token Planning for Vision-Language-Action Models
Jun 5, 2026Jinhao Wu, Shiduo Zhang, Yicheng Liu +9Visuomotor Policy Learning AI Control
1Nanjing University · 2Shanghai Innovation Institute · 3Fudan University +1
PearlVLA: Progressive Embodied Action-Plan Refinement in Latent Space
Jun 16, 2026Bochen Yang, Lianlei ShanEfficient VLA Models Latent Visual Reasoning
Imperial College London · Tsinghua University
StairVLA: Stage-Aware Hierarchical Action Generation for Vision-Language-Action Models
Oct 6, 2026Shangyuan Yuan, Xinda Qi, Yujiang Pu +2Efficient VLA Model Inference Vision-Language-Action Models
Michigan State University · Ant Technology U.S., Inc.

Jun 5, 2026J/K move · Enter open · S save

Coarse-to-Control: Action-Token Planning for Vision-Language-Action Models

Jinhao Wu, Shiduo Zhang, Yicheng Liu, Xiaopeng Yu+8

1Nanjing University · 2Shanghai Innovation Institute · 3Fudan University+1

Visuomotor Policy Learning AI Control Robotic Manipulation Robot Task Planning Vision-Language-Action Models

Open paper AskPDF arXiv

Most vision-language-action (VLA) models map observations directly to actions without explicit intermediate planning, which limits performance on long-horizon tasks where early mistakes compound. We propose Coarse-to-Control, a plan-execute VLA that introduces planning natively in the action-token space. The key idea is to let the policy first predict a compact sequence of coarse action tokens that summarize the intended future trajectory, and then generate executable action tokens conditioned on this plan. Because both planning and execution share a unified discrete action vocabulary, the plan stays close to the control manifold and provides directly actionable guidance rather than an abstract hint that must be translated back to motor commands. Experiments on LIBERO, SimplerEnv-WidowX, and real-world manipulation tasks show that action-token planning consistently improves over direct action generation, with the largest gains on long-horizon multi-stage tasks.

Abstract

Vision-Language-Action Models

Explore similar work

Coarse-to-Control: Action-Token Planning for Vision-Language-Action Models

PearlVLA: Progressive Embodied Action-Plan Refinement in Latent Space

StairVLA: Stage-Aware Hierarchical Action Generation for Vision-Language-Action Models