Organizations: Keio University · National Institute of Informatics Research and Development Center for Large Language Models · National Institute of Informatics
Coarse-to-fine autoregressive modeling has recently shown strong promise for visuomotor policy learning, combining the inference efficiency of autoregressive methods with the global trajectory coherence of diffusion-based policies. However, existing approaches rely on discrete action tokenizers that map continuous action sequences to codebook indices, a design inherited from image generation where learned compression is necessary for high-dimensional pixel data. We observe that robot actions are inherently low-dimensional continuous vectors, for which such tokenization introduces unnecessary quantization error and a multi-stage training pipeline. In this work, we propose Hierarchical Flow Policy (HiFlow), a tokenization-free coarse-to-fine autoregressive policy that operates directly on raw continuous actions. HiFlow constructs multi-scale continuous action targets from each action chunk via simple temporal pooling. Specifically, it averages contiguous action windows to produce coarse summaries that are refined at finer temporal resolutions. The entire model is trained end-to-end in a single stage, eliminating the need for a separate tokenizer. Experiments on MimicGen, RoboTwin 2.0, and real-world environments demonstrate that HiFlow consistently outperforms existing methods including diffusion-based and tokenization-based autoregressive policies.
Figures & tables
Fig. 1 : Tokenized vs tokenization-free scale-wise policy learning. Prior next-scale autoregressive policies discretize multi-scale action chunks using a VQ-VAE tokenizer and learn to predict code indices with cross-entropy, which introduces a separate tokenizer pretraining stage and incurs quantization error. HiFlow instead refines raw continuous actions from coarse to fine temporal scales. A scale-wise autoregressive transformer provides coarse-to-fine conditioning, and a shared conditional flow matching module generates continuous actions at each scale, eliminating the tokenizer while preserving fine-grained control.
Fig. 2 : Architecture overview of HiFlow . Given visual observations, proprioceptive states, and a task identifier, the scale-wise autoregressive Transformer (ScaleAR) produces conditioning features at progressively finer temporal scales via a scale-wise causal mask. A shared ActionFlowNet then generates continuous actions at each scale through conditional flow matching, progressively refining the trajectory from a single-token global summary (scale 1) to the full T -step action chunk. The entire pipeline operates in continuous action space without any discrete tokenization.
D
1024
T
8
Le
12
Lf
6
Batch size
128
Optimizer
AdamW
Learning rate
1×10−4
TABLE I : Experimental settings.
Fig. 3 : Task overview across the three evaluation benchmarks. Top: 8 single-arm manipulation tasks from MimicGen. Middle: 3 dual-arm tasks from RoboTwin 2.0. Bottom: 5 real-world tasks with a mobile manipulator, spanning object grasping, relocation, and target placement.
Policy
Coffee
Hammer
Mug
Nut
Square
Stack
Stack3
Thread
Avg.
TCD [ 15 ]
77
92
53
44
63
95
62
56
68
SDP [ 14 ]
82
100
62
54
82
96
80
70
78
CARP [ 6 ]
86
98
74
78
90
100
82
70
85
HiFlow (Ours)
100
100
76
80
82
98
76
90
88
TABLE II : Quantitative results on MimicGen in success rate (%). Best results in bold .
Policy
Click alarm
Move can
Place basket
ACT [ 2 ]
32
22
1
DP [ 4 ]
61
32
18
HiFlow (Ours)
69
42
39
TABLE III : Quantitative results on RoboTwin benchmark. Success rates (%). Best results in bold . “Click alarm”, “Move can”, and “Place basket” denote “click alarmclock”, “move can pot”, and “place can basket”, respectively.
Grasp
Relocation
Target Placement
Policy
Apple
Mustard
Coke
Ball → Dish
Orange → Plate
ACT [ 2 ]
50.0
33.3
8.3
16.7
8.3
DP [ 4 ]
66.7
58.3
66.7
25.0
16.7
CARP [ 6 ]
66.7
58.3
58.3
33.3
25.0
HiFlow (Ours)
75.0
83.3
75.0
58.3
41.7
TABLE IV : Quantitative results on real-world experiments. Success rates (%) over 12 trials per task. Best results in bold .
Fig. 4 : Qualitative results of HiFlow on representative tasks from each simulation benchmark. (a) Threading from MimicGen. (b) Place can basket from RoboTwin 2.0.
Fig. 5 : Qualitative results of HiFlow on the Orange → Plate task from the real-world experiments. Note that the third-person view are not included in the observation of the model.
Fig. 6 : Visualization of coarse-to-fine action generation. Each panel shows the predicted cumulative end-effector displacement trajectory in Cartesian task space (position components only) at a different scale level on the Mimicgen benchmark. ∘ , ⋆ , and ⋄ indicate token boundaries, trajectory start point and the end point, respectively.
Scale configuration
Avg. SR
{1,8}
84
{1,4,8}
86
{1,2,4,8,16}
85
{1,2,4,8}
88
TABLE V : Sensitivity analysis on temporal downsampling scales. Average success rate (%) on MimicGen. Best result in bold .
Visuomotor policy learning trains robots to turn camera observations, often with language and proprioception, into actions for manipulation and navigation.
South China University of Technology (scut.edu.cn) · The Chinese University of Hong Kong (cuhk.edu.hk) · Hong Kong Polytechnic University (polyu.edu.hk)