Hierarchical vision-language-action (VLA) systems consist of a high-level vision-language planner and a low-level action expert that generates continuous actions. This hierarchical design has practical value only if the planner can generate plans fast enough to meet real-time control requirements, and the resulting plans actually contribute to the generation of action. We study one such system, a waypoint hierarchy pipeline adapted from π0.5, and find that neither requirement is satisfied. This baseline relies on token-level autoregressive decoding (Token-AR) to generate a waypoint plan, requiring 57 very expensive vision-language model (VLM) forward passes. However, we find that erasing the waypoint endpoints has little effect on task success. Two findings reveal the misalignment of planner-executor: the planner generates outputs at an excessively fine granularity, and the executor underuses plans as a control condition. We address the latency issue with waypoint-aligned block-autoregressive decoding (Block-AR), and plan underuse issue with normalized goal modulation (NGM), a layer-wise goal path constrained by phase gating and anti-shortcut training so that the waypoint influences action generation maintaining other signals. Our method reduces the maximum number of VLM forward passes from 57 to 8 on LIBERO, including one prefix prefill, and achieves an 8.7× reduction in planning latency on a Rokae dual-arm robot. With normalized goal modulation and anti-shortcut training, Block-AR's success rate on LIBERO-Long increases from 91.0% to 96.2%, while its average success rate across the four suites increases from 95.85% to 98.45%. On three bimanual tasks with this robot, success rates remain comparable across methods.
Figures & tables
Figure 1: Fast plans, faithful actions. (a) Inputs (Rokae AR5-5 dual arm, pepper–banana placement): external and wrist images, instruction ℓ , state s . (b) A PaliGemma planner (Gemma-2B language model) predicts (q,g,d) waypoints (target configuration, gripper, duration). The schematic shows four waypoints per arm (hollow rings; orange dashed rings: grasp/release), illustrative per-arm step counts (linked pills), and corresponding action beads. Deployed dual-arm blocks share one d ; d=0 ends the plan. A 300M flow-matching expert, which also attends to the planner’s image and language prefix (dotted), maps each waypoint to d actions; the robot executes, observes, and replans (dashed loop). (c) For one seven-waypoint LIBERO plan, Token-AR needs one prefix prefill plus one backbone pass per value token (dots), 57 passes in total; Block-AR needs the same prefill plus one pass per waypoint (pills), 8 in total. On the robot, plan latency falls from 1094 to 125 ms and the arms move 66% instead of 29% of the time (bars). (d) Endpoint-erasure test on LIBERO-Long: with images, instruction, state, and flow noise fixed, the commanded endpoint is set to the segment start. Erasure does not reduce success with suffix-token conditioning ( U=0 ), but reduces success by 7.4 points with normalized goal modulation (NGM), which injects the goal into every expert layer.
Method
Plan space
Executor
Serial cost grows with
Interface assessment
HAMSTER ( Li et al., 2025 )
2-D image path
separate policy
path points
failure attribution
3D HAMSTER ( Hwang et al., 2026 )
pixel + depth path
separate flow policy
path points
guidance ablation
GAE ( Liu et al., 2025 )
camera-frame 3-D points
frozen point-cloud expert
waypoints
goal-noise ablation
NoTVLA ( Huang et al., 2025 )
pixel + depth keyposes
spline + replanning
keyposes
deterministic execution
Coarse-to-Control ( Wu et al., 2026 )
coarse action blocks
same model, AR tokens
blocks + actions
qualitative diagnostics
Ours
robot configuration + duration
flow expert, shared prefix
value slots (Token-AR) / waypoints (Block-AR)
S , U
Table 1: Planner-to-executor designs in hierarchical VLAs.
Figure 2: Method. (a) Waypoint representation: a LIBERO waypoint is the token group <wp> q1⋯q6g <dur> d with d∈{1,…,32} (338 reserved token IDs; a robot waypoint has 19 tokens). For block i the expert denoises D=32 actions from the measured state si and executes the first di (green); the next block starts from the measured state. (b) Block-AR: attention is bidirectional within a block (dark) and causal across blocks (light); inputs are shifted by one block (the first is a learned query, dashed) and targets are not, so one backbone pass yields a whole waypoint through three LM-head calls (configuration, gripper, duration). Decoding produces at most M=7 blocks and stops early when a block predicts d=0 . (c) NGM: besides the suffix tokens si,qi,gi,di , the normalized goal [Δ;e(gi)] with Δ=(qi−si)/σΔ enters an MLP with a zero-initialized output projection. Its residual r , gated by g(t) , is added to the AdaRMS condition ϕ(t)+g(t)r of every layer; the gate is open for t≥0.5 and closed for t≤0.2 . Training noises the endpoint seen by both routes ( σc=0.7 ) and replaces both by null embeddings with probability 0.15; inference uses the clean endpoint and the gate only.
Figure 3: Waypoints extracted from demonstrations. (a) Pepper-swap episode 2: 634 steps (21 s) become 45 waypoints. The top row shows the start, four gripper transitions, and end. The curves show the 14 joint angles (left arm above, right arm below; centered with a shared scale); vertical cuts mark waypoints, and the blocks below encode segments (q,g,d) with width proportional to duration d . Brackets identify the three segments capped at D=32 steps; rings mark gripper transitions (g = grasp, r = release; left, right, right, left arm). The orange d=0 block marks the end of the plan. (b) Segment-duration distributions on a shared scale: robot (6,158 training segments; median 12, 5.8% at the cap) and LIBERO-Long (13,815 segments; median 7, longest 29).
Method
Spatial
Object
Goal
Long
Avg.
Δ vs π0.5
Δ vs Token-AR
π0.5 (released, as reported)
98.8
98.2
98.0
92.4
96.85
–
–
Ours, Token-AR + suffix cond.
98.0
99.0
97.6
92.2
96.70
−0.15
–
Ours, Block-AR + suffix cond.
98.6
97.6
96.2
91.0
95.85
−1.00
−0.85
Ours, Block-AR + NGM
99.2
99.4
99.0
96.2
98.45
+1.60
+1.75
Table 2: Success rate (%) on the four LIBERO suites.
Figure 4: Robot planning latency and motion duty cycle. (a) Recorded shelf-placement timelines for Token-AR and Block-AR on a common time axis. Waiting comprises planner, expert, and transport; motion denotes commanded execution at 30 Hz. Motion occupies 24% and 59% here, versus 29% and 66% over 35 telemetry episodes (Table 3 ). Both episodes were stopped before placement. (b) Mean request and commanded-motion durations (497 Token-AR / 936 Block-AR requests). Planner latency falls from 1094 to 125 ms ( 8.7× ); expert latency remains near 160 ms. Transport is the hairline segment. The request round trip drops from 1266 to 293 ms ( 4.3× ), below the roughly 0.5 s of commanded motion.
Token-AR
Block-AR
Serial passes / plan, max (LIBERO / robot)
57 / 120
8 / 8
Plan latency, robot (ms)
1094
125
Request round trip, robot (ms)
1266
293
Motion duty cycle, robot
28.7%
65.6%
Table 3: Planning cost (A) and waypoint statistics (B).
Figure 5: Representative dual-arm sequences from Block-AR telemetry rollouts (external camera, seconds since start); success rates are reported separately in Table 5 . (a) Pepper–banana placement. (b) Pepper swap. (c) Shelf placement, stopped at 34 s with the red block carried to the shelf but not placed.
System
B
Sendpoint (%)
Sduration (%)
Simage
Retention
SR intact (%)
U (pp)
Block-AR + suffix
0.118
0.6
19
0.330
100%
91.0
0.0
Token-AR + naive
0.074
348
106
0.090
27%
15
+14
Block-AR + NGM
0.105
42.6
26.4
0.27
82%
96.2
+7.4
Table 4: Waypoint dependence and visual responsiveness on LIBERO-Long.
Most vision-language-action (VLA) models map observations directly to actions without explicit intermediate planning, which limits performance on long-horizon tasks where early mistakes compound. We propose Coarse-to-Control, a plan-execute VLA that introduces planning natively in the action-token space. The key idea is to let the policy first predict a compact sequence of coarse action tokens that summarize the intended future trajectory, and then generate executable action tokens conditioned on this plan. Because both planning and execution share a unified discrete action vocabulary, the plan stays close to the control manifold and provides directly actionable guidance rather than an abstract hint that must be translated back to motor commands. Experiments on LIBERO, SimplerEnv-WidowX, and real-world manipulation tasks show that action-token planning consistently improves over direct action generation, with the largest gains on long-horizon multi-stage tasks.
Jinhao Wu, Shiduo Zhang, Yicheng Liu +9
1Nanjing University · 2Shanghai Innovation Institute · 3Fudan University +1
Current Vision-Language-Action (VLA) models face a trade-off between efficient action generation and explicit deliberation. Directly decoding actions from vision-language backbone representations enables low-latency control, whereas textual reasoning, pixel-level subgoals, or world-model evaluation of decoded actions can improve planning but incur substantial latency and computational cost. We propose PearlVLA, a VLA framework that progressively refines a VLM-derived latent plan using feedback from the predicted consequence of each intermediate plan. PearlVLA uses a frozen latent world model (LaWM) pretrained on action-free video. At each refinement round, the current plan produces a continuous latent action code, and the LaWM predicts the corresponding latent visual subgoal. A future-guided plan refiner uses this subgoal to update the plan, so each revision reshapes the next LaWM query. After K rounds, the refined plan is passed once to the host policy's action interface to produce an action chunk. We further introduce Causal Refinement-Grouped Process-Reward RL to optimize latent refinement by comparing rewards from the longer-horizon imagined futures of plan edits made at the same refinement state. Experiments on the LIBERO and RoboCasa benchmarks show that PearlVLA performs competitively against strong existing methods.
Vision-language-action (VLA) models increasingly rely on diffusion- or flow-matching-based action heads to generate continuous robot actions. These action heads typically process the denoising trajectory in a largely uniform manner. However, we observe that the conditioning focus naturally shifts across denoising stages: early stages combine language instructions and visual observations to establish a coarse action trajectory, whereas later stages place greater emphasis on current visual observations for action alignment. Based on this insight, we introduce StairVLA, a stage-aware hierarchical action generation framework that uses partially denoised actions as a natural interface between coarse long-horizon action generation and local refinement. A high-level VLA performs early denoising to produce a reusable long-horizon partially denoised action trajectory, while a lightweight refiner operates at a higher frequency to refine local action chunks using the latest observations. This design amortizes expensive high-level VLA computation while preserving frequent closed-loop correction. On LIBERO, our GR00T-style instantiation improves average success from 96.5% to 97.8% while reducing amortized inference latency from 115.0 ms to 44.2 ms per action chunk. More broadly, across two VLA backbones, simulation benchmarks, and real-robot tasks, StairVLA consistently reduces inference cost while maintaining strong task performance.
Shangyuan Yuan, Xinda Qi, Yujiang Pu +2
Michigan State University · Ant Technology U.S., Inc.