AR-VLA: True Autoregressive Action Expert for Vision-Language-Action Models
Authors: Yutong Hu, Jan-Nico Zaech, Nikolay Nikolov, Yuanqi Yao, Sombit Dey, Giuliano Albanese, Renaud Detry, Luc Van Gool, +1 more
Organizations: ∗INSAIT, Sofia University “St. Kliment Ohridski” · †KU Leuven, Dept. Mechanical Engineering, Research unit Robotics, Automation and Mechatronics · ‡KU Leuven, Dept. Electrical Engineering, Research unit Processing Speech and Images
We propose a standalone autoregressive (AR) Action Expert that generates actions as a continuous causal sequence while conditioning on refreshable vision-language prefixes. In contrast to existing Vision-Language-Action (VLA) models and diffusion policies that reset temporal context with each new observation and predict actions reactively, our Action Expert maintains its own history through a long-lived memory and is inherently context-aware. This structure addresses the frequency mismatch between fast control and slow reasoning, enabling efficient independent pretraining of kinematic syntax and modular integration with heavy perception backbones, naturally ensuring spatio-temporally consistent action generation across frames. To synchronize these asynchronous hybrid V-L-A modalities, we utilize a re-anchoring mechanism that mathematically accounts for perception staleness during both training and inference. Experiments on simulated and real-robot manipulation tasks demonstrate that the proposed method can effectively replace traditional chunk-based action heads for both specialist and generalist policies. AR-VLA exhibits superior history awareness and substantially smoother action trajectories while maintaining or exceeding the task success rates of state-of-the-art reactive VLAs. Overall, our work introduces a scalable, context-aware action generation schema that provides a robust structural foundation for training effective robotic policies. Code and Videos available at https://arvla.insait.ai
Figures & tables
Fig. 2: Performance Overview. (a) Quantitative Results: In both generalist (left) and specialist (right) benchmarks, AR-VLA achieves competitive or superior performance compared to state-of-the-art policies, including OpenVLA, Flow-Matching (FM), ACT, and Diffusion Policy (DP), details in Sec. IV-A . (b) Trajectory Quality: Qualitative visualization of joint trajectories over time reveals that AR-VLA produces significantly smoother and more kinematically consistent motion compared to reactive baselines that reset context at each step (analysis in Sec. IV-B ). (c) Long-Horizon Capability: AR-VLA successfully completes long-horizon tasks where baselines like DP and FM fail due to a lack of temporal context awareness. Detailed task defination and explanation in Sec. IV-C .
Fig. 3: The AR-VLA Framework . The system bridges an VLM backbone with a autoregressive Action Expert asynchronously. Atemporal features from the VLM are explicitly injected with temporal context via Dynamic Temporal Re-anchoring (DTR). Within the Hybrid KV Cache, re-anchored VL tokens (green) serve as a semantic prefix to the rolling kinematic history (orange). The Action Expert generates future action sequences by querying this shared cache using incrementally advancing step embeddings.
Fig. 4: Heterogeneous FIFO Update Rules for the Hybrid KV Cache. The framework manages memory through two distinct queueing strategies to ensure efficient context utilization. The VL Stream (green) operates as a short-lived, block-wise FIFO: In contrast, the Action Stream (orange) maintains a token-wise rolling FIFO, continuously appending the single latest action prediction while evicting the oldest kinematic state.
Fig. 5: Simulation benchmarks setups. We do simulation evaluation spanning generalist and specialist policies, with diverse embodiment, action space, and task.
Model
Spoon
Carrot
Block
Eggplant
Average
OpenVLA [ 19 ]
0
0
0
4.1
1.0
Octo-Base [ 29 ]
12.5
8.3
0
43.1
16.0
Octo-Small [ 29 ]
47.2
9.7
4.2
56.9
30.0
SpatialVLA [ 32 ]
16.7
25.0
29.2
100.0
42.7
CogACT [ 24 ]
58.3
37.5
20.8
91.7
52.1
Pi-0-Fast* [ 31 ]
62.5
29.2
20.8
83.3
49.0
TABLE I: BridgeV2 Pretraining to SIMPLER Simulation Success Rate (%)
Method
pushT
aloha-cube
aloha-insert
Max IoU
Success
Script
Human
Script
Human
DP [ 8 ]
0.957
65.20
33.33
10.00
22.67
1.66
ACT [ 47 ]
0.800
52.00
86.00
50.00
32.00
20.00
AR(Ours)
0.920
60.40
97.33
67.33
54.67
6.67
TABLE II: Specialist Policy Performance. We report average max IOU and success rate (%) for pushT, and success rate (%) for two ALOHA task training on scripted and human demonstrations. Per seed evaluation results with standard error is accessible from our project website.
Fig. 6: BridgeV2 pretraining to real-world WidowX Zero-Shot Performance Comparison. We set the camera pose so that all methods, except SpatialVLA [ 32 ] , reach a 100% success rate on an easy in-distribution task, then test them zero-shot on challenging tasks. Details of experiment protocol in Appendix.
Fig. 7: Smoothness Visualization. Joint states captured from success execution for the same task. X-axis in seconds, y-axis in rad
Model
Jerk ( ↓ )
Latency ( ↓ )
Avg
Max
VLM
Act. Expert
Total/Act
OpenVLA
10.13
42.14
321.72 / 1
321.72
Fast
8.15
80.24
744.78 / 4
186.20
FM
9.39
45.33
69.77 / 4
267.28 / 4
84.26
AR (Ours)
7.89
39.83
69.56 / 4
28.86 / 1
46.25
TABLE III: Inference Smoothness Metrics. Jerk in 102 rad/s 3 . VLM and Action Expert latencies in ms / chunk size. Total/Act represents the effective latency per single action.
Fig. 8: History-Awareness Evaluation. PushT2 requires visiting both goals, but which goal has been visited is unobservable midway. Stack3 requires stacking cups over a battery that becomes occluded. Both task require memory of unobservable past states. H donates the context window length of AR-VLA. Details about task defination, data collection, training and execution in Appendix.
Variant
Val. Error ( ↓ )
Sim. Success ( ↑ )
w/ Phase-1 (Full)
4.2
61.5
w/o Phase-1 (100% time)
4.5
37.5
w/o Phase-1 (200% time)
4.0
54.2
mask rate = 0.0
2.7
0.0
mask rate = 0.2
3.1
27.1
mask rate = 0.4
3.5
47.9
TABLE IV: Ablation Study of AR-VLA Design Choice. Validate Error calculate from the model output and ground truth when taking dataset trajectory as inout. Success rates are percentages in total 96 SIMPLER trials after training. Bold indicates the full AR-VLA configuration.
Fig. 9: Three different Action Experts sharing the same architecture and V-L backbone. The same networks are trained and used differently.
Fig. 10: AR Actor that share the exact same size and architecture of Action Chunking Transformer, the same decoders are trained and used differently.
Parameter
Generalist (AR-VLA)
Specialist (AR-Actor)
Vision Backbone
SigLIP-So400m
ResNet-18
Language/State Backbone
Gemma-2b (Fast)
Transformer Encoder (Trainable)
Action Expert Layers
18 Transformer Decoder
4 Transformer Decoder
Hidden Dimension ( D )
1024
512
Feed-forward Dim ( Dff )
4096
3200
Attention Heads
8
8
TABLE V: Architecture specifications for Generalist (AR-VLA) and Specialist (AR-Actor) models. The specialist configuration is based on the 4-4 Transformer-based Actor setup.
Hyperparameter
VLA (Phase 1)
VLA (Phase 2)
Specialist
Optimizer
AdamW
AdamW
AdamW
Learning Rate (LR)
1.0×10−4
5.0×10−5
1.0×10−5
LR Backbone
- (N/A)
- (Insulated)
1.0×10−5
Weight Decay
0.0
0.0
1.0×10−4
Grad. Clip Norm
1.0
1.0
10.0
Batch Size
1024
512
8
TABLE VI: Comparative training configurations for generalist (Phases 1 & 2) and specialist policies.
Parameter
BridgeV2 (Generalist)
ALOHA Sim (Specialist)
Training History Length ( H )
16 Steps
20 Steps
Causal Training Length ( H )
8 Steps
20 Steps
Test-Time History
20 Steps
30 Steps
History Masking Prob.
0.6
0.5
TABLE VII: Sequence and execution parameters across environments.
Fig. 11: Demonstration collection for history-aware tasks. (a) The PushT2 task can be complete in two different orders, we don’t constrain human demonstrators on their solution. (b) We collect demonstrations for Stack3 under different initial object layouts. On stage 2, the visual snapshot is identical (the battery in dark is covered, and the camera can only see two cups) but the next object to pick is different.
Fig. 12: Typical cases during PushT2 task execution.
Fig. 13: Typical cases during Stack3 task execution.
Fig. 14: AR-VLA Zero-shot task execution in SIMPLER simulator.
Fig. 15: AR-VLA Zero-shot task execution in real world.
School of Information, Renmin University of China · Key Laboratory of Data Engineering and Knowledge Engineering, Beijing, China · Beijing University of Posts and Telecommunications +1